dagger.DagBuilderBase

class dagger.DagBuilderBase

Bases: object

Base class for building DAGs. This class contains the core DAG-building functionality without any user-facing concerns like directory management. It provides low-level methods for function parsing, submit object creation, and DAG layer management.

parse_function(func: callable, return_as_string: bool = True, return_name: bool = False, trim_whitespace: bool = True) → tuple[str | list[str], str]

Parse a Python function and turn it into a list or string representation. This function uses the Python inspect module to retrieve the source code. This can be dangerous on unverified code, so please make sure you know what the source code is doing before wrapping it.

The function can be return as a list of strings, or formatted into a valid string that can be directly placed into a Python script. Formatting as a list allows for further manipulation and modification of the code prior to writing it to a file, for example to add command line argument parsing, shebangs etc.

Parameters:

  • func (callable) – The function to parse.

  • return_as_string (bool) – If True, return the function as a formatted string. If False, return as a list of strings.

  • return_name (bool) – If True, return the function name as well. This is useful if you want to use the function name in the script.

  • trim_whitespace (bool) – If True, trim leading whitespace from the function source code. This will not trim _all_ the whitespace, but only the global indentation level.

Returns:

The function source code as a string or list of strings.

Return type:

str or list

Raises:

TypeError – If the input is not a callable function.

Example:

>>> def example_function(x: int, y: str) -> None:
...     print(f"Example function with x={x} and y={y}")
>>> dagger_instance = dagger()
>>> dagger_instance.parse_function(example_function)
function_to_submit_obj(func: callable, dag_dir: str, py_script_name: str = '', submit_vars: dict = {}, delimiter: str | None = None) → Submit

Convert a Python function into a HTCondor Submit object.

Parameters:

  • func (callable) – The function to convert.

  • py_script_name (str) – The name of the Python script to create. If not provided, it defaults to the function name with a .py extension. The Python script is created when this function is called. The name can contain the full path to the script. If only a name is provided, it will be generated in the current directory.

  • submit_script_name (str) – The name of the submit script to create. If not provided, it defaults to the function name with a .submit extension.

  • submit_vars (dict) – Additional arguments to include in the submit script. This can include command line arguments, Condor requirements, container paths etc.

  • delimiter (str) – Optional delimiter to use for the function source code. This allows wrapping arbitrary code in the function source code. If provided, the function source code will be wrapped in the delimiter. Any arbitrary string can be used as a delimited, but it is recommended to use a string that is unlikely to appear in the function source code, such as """ or '''.

Returns:

HTCondor Submit object

Return type:

htcondor2.dags.Submit

Raises:

TypeError – If the input is not a callable function.

dag_layer(submit_obj: Submit, submit_vars: list[dict], layer_name: str = '', parent_layer_name: str = '', **kwargs) → NodeLayer

Create a new layer in the DAG. If no layer name is provided, a default name will be generated based on the current number of layers, which is incremented each time a new layer is created.

Parameters:

  • submit_obj (htcondor2.Submit) – The HTCondor Submit object to associate with this layer.

  • submit_vars (list[dict]) – Additional arguments to include in the submit script for this layer. This can include command line arguments, Condor requirements, container paths etc. Each dictionary in the list represents a set of variables to be used for a single job in the layer.

  • layer_name (str) – Optional : Name of the layer to create. If not provided, a default name will be generated based on the current number of layers. The default format is “layer_{n}”, where n is the current number of layers.

  • parent_layer_name (str) – Name of the parent layer to link to. If provided, this layer will be added as a child of the parent layer.

  • kwargs (dict) – Additional keyword arguments to pass to the layer creation. This can include any additional parameters supported by htcondor2.dags.NodeLayer.

Raises:

  • ValueError – If parent_layer_name does not exist in the DAG.

  • TypeError – If submit_obj is not an instance of htcondor2.Submit.

Returns:

A NodeLayer object representing the new layer in the DAG.

Return type:

htcondor2.dags.NodeLayer

Example:

>>> dag = Dagger()
>>> dag.dag_layer(layer_name="my_layer")
>>> dag.dag_layer(parent_layer_name="my_layer")