dagger.Dagger

class dagger.Dagger(dag_dir: str, dag_name: str, overwrite_dag_dir: bool = False)

Bases: DagBuilderBase

Store DAG state, provide methods to wrap Python functions, turn them into submit scripts, and add layers to the DAG, define parent/child relationships, and submit the DAG to HTCondor.

property submit_functions: dict

Return the dictionary of functions that have been submitted to the DAG. This is useful for debugging and checking which functions are part of the DAG.

Returns:

Dictionary of submitted functions.

Return type:

dict

property layer_list: list

Return the list of layers in the DAG. This is useful for debugging and checking the structure of the DAG.

Returns:

List of layers in the DAG.

Return type:

list

property job_list: list

Return the list of jobs in the DAG. This is useful for debugging and checking the jobs that have been added to the DAG.

Returns:

List of jobs in the DAG.

Return type:

list

property job_names: list

Return the list of job names in the DAG. This is useful for debugging and checking the names of the jobs that have been added to the DAG.

Returns:

List of job names in the DAG.

Return type:

list

function_to_submit_obj(func: callable, **kwargs) → Submit

User-facing wrapper that automatically provides dag_dir.

add_function_to_layer(func: callable, py_script_name: str = '', submit_vars: dict = {}, layer_name: str = '', parent_layer_name: str = '', layer_vars: list[dict] = [], delimiter: str | None = None, **kwargs) → NodeLayer

Convert a Python function into a layer in the DAG. This is a convenience method that combines the function_to_submit_obj and dag_layer methods into one. It will automatically convert the function to a submit object and add it to the DAG as a layer.

Parameters:

• func (callable) – The function to convert.

• py_script_name (str) – The name of the Python script to create. If not provided, it defaults to the function name with a .py extension.

• submit_vars (dict) – Additional arguments to include in the submit script. This can include command line arguments, Condor requirements, container paths etc.

• layer_name (str) – Optional : Name of the layer to create. If not provided, a default name will be generated based on the current number of layers. The default format is “layer_{n}”, where n is the current number of layers.

• parent_layer_name (str) – Name of the parent layer to link to. If provided, this layer will be added as a child of the parent layer.

• layer_vars (list[dict]) – List of dictionaries representing variables to be used for each job in the layer. Each dictionary in the list represents a set of variables to be used for a single job in the layer.

• kwargs (dict) – Additional keyword arguments to pass to the layer creation. This can include any additional parameters supported by htcondor2.dags.NodeLayer.

• delimiter (str) – Optional delimiter to use for the function source code. This allows wrapping arbitrary code in the function source code. If provided, the function source code will be wrapped in the delimiter. Any arbitrary string can be used as a delimited, but it is recommended to use a string that is unlikely to appear in the function source code, such as """ or '''.

Raises:

TypeError – If the input is not a callable function.

Returns:

A NodeLayer object representing the new layer in the DAG.

Return type:

htcondor2.dags.NodeLayer

Example:

>>> from dagger import Dagger
>>> dag = Dagger(dag_dir="my_dag_dir", dag_name="my_dag")
>>> def my_function(x: int, y: str) -> None:
...     print(f"My function with x={x} and y={y}")
>>> layer = dag.function_to_layer(
...     func=my_function,
...     py_script_name="my_function.py",
...     submit_vars={"requirements": "Machine == 'my_machine'"},
...     layer_name="my_layer",
...     parent_layer_name="parent_layer",
...     layer_vars=[{"x": 1, "y": "test"}, {"x": 2, "y": "example"}]
... )

write_dag(**kwargs) → None

Write the current DAG to a file. This will create a .dag file that can be submitted to HTCondor. This is a simple wrapepr around the htcondor2.dags.write_dag method, for ease of use with this class.

Parameters:

kwargs (dict) – Keyword arguments to pass to the DAG writing function. This can include any parameters supported by htcondor2.dags.write_dag

Example:

>>> dag = Dagger()
>>> dag.write_dag(filename="my_dag.dag", submit_description="my_submit.sub")

Returns:

None