hpcflow.app.Workflow#

class hpcflow.app.Workflow(path)#

Bases: Workflow

Class to represent a persistent hpcFlow workflow.

Methods

`add_loop`	Add a loop to a subset of workflow tasks.
`add_submission`
`add_task`
`add_task_after`	Add a new task after the specified task.
`add_task_before`	Add a new task before the specified task.
`batch_update`	A context manager that batches up structural changes to the workflow and commits them to disk all together when the context manager exits.
`check_parameters_exist`
`copy`	Copy the workflow to a new path and return the copied workflow.
`delete`
`elements`
`from_JSON_file`	Generate from a JSON file.
`from_JSON_string`	Generate from a JSON string.
`from_YAML_file`	Generate from a YAML file.
`from_YAML_string`	Generate from a YAML string.
`from_file`	Generate from either a YAML or JSON file, depending on the file extension.
`from_template`	Generate from a WorkflowTemplate object.
`from_template_data`	Generate from the data associated with a WorkflowTemplate object.
`get_EARs_from_IDs`	Return element action run objects from a list of five-tuples, representing the task insert ID, element index, iteration index, action index, and run index, respectively.
`get_all_parameter_data`
`get_element_iterations_from_IDs`	Return element iteration objects from a list of three-tuples, representing the task insert ID, element index, and iteration index, respectively.
`get_elements_from_IDs`	Return element objects from a list of two-tuples, representing the task insert ID, and element index, respectively.
`get_iteration_task_pathway`
`get_parameter_data`
`get_parameter_source`
`get_task_elements`
`get_task_elements_islice`
`get_task_unique_names`	Return the unique names of all workflow tasks.
`is_parameter_set`
`rename`
`resolve_jobscripts`
`save_parameter`
`save_parameters`	Save multiple parameters to a given EAR.
`set_EAR_end`	Set the end time on an EAR.
`set_EAR_start`	Set the start time on an EAR.
`set_EAR_submission_indices`	Set the submission index on an EAR.
`show_all_EAR_statuses`
`submit`
`write_commands`	Write run-time commands for a given EAR.

Attributes

`app`
`artifacts_path`
`creation_info`
`loops`
`name`	The workflow name may be different from the template name, as it includes the creation date-timestamp if generated.
`num_added_tasks`
`num_element_iterations`
`num_elements`
`num_loops`
`num_submissions`
`num_tasks`
`store_format`
`submissions`
`submissions_path`
`task_artifacts_path`
`tasks`
`template`
`template_components`
`ts_fmt`
`ts_name_fmt`

Parameters:: path (PathLike) –

add_loop(loop, parent_loop_indices=None)#

Add a loop to a subset of workflow tasks.

Parameters:

loop (Loop) –
parent_loop_indices (Dict) –

Return type:

None

add_submission(JS_parallelism=None)#

Parameters:: JS_parallelism (bool | None) –
Return type:: Submission

add_task(task, new_index=None)#

Parameters:

task (Task) –
new_index (int | None) –

Return type:

None

add_task_after(new_task, task_ref=None)#

Add a new task after the specified task.

Parameters:

task_ref (Task) – If not given, the new task will be added at the end of the workflow.
new_task (Task) –

Return type:

None

add_task_before(new_task, task_ref=None)#

Add a new task before the specified task.

Parameters:

task_ref (Task) – If not given, the new task will be added at the beginning of the workflow.
new_task (Task) –

Return type:

None

app = BaseApp(name='hpcFlow', version='0.2.0a50')#

property artifacts_path#

batch_update(is_workflow_creation=False)#

A context manager that batches up structural changes to the workflow and commits them to disk all together when the context manager exits.

Parameters:: is_workflow_creation (bool) –
Return type:: Iterator[None]

check_parameters_exist(indices)#

Parameters:: indices (int | List[int]) –
Return type:: bool | List[bool]

copy(path=None)#

Copy the workflow to a new path and return the copied workflow.

Return type:: Workflow

property creation_info#

delete()#

elements()#

Return type:: Iterator[Element]

classmethod from_JSON_file(JSON_path, path=None, name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from a JSON file.

Parameters:

JSON_path (PathLike) – The path to a workflow template in the JSON file format.
path (str | None) – The directory in which the workflow will be generated. The current directory if not specified.
name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified the WorkflowTemplate name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

classmethod from_JSON_string(JSON_str, path=None, name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from a JSON string.

Parameters:

JSON_str (PathLike) – The JSON string containing a workflow template parametrisation.
path (str | None) – The directory in which the workflow will be generated. The current directory if not specified.
name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified the WorkflowTemplate name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

classmethod from_YAML_file(YAML_path, path=None, name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from a YAML file.

Parameters:

YAML_path (PathLike) – The path to a workflow template in the YAML file format.
path (str | None) – The directory in which the workflow will be generated. The current directory if not specified.
name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified the WorkflowTemplate name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

classmethod from_YAML_string(YAML_str, path=None, name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from a YAML string.

Parameters:

YAML_str (PathLike) – The YAML string containing a workflow template parametrisation.
path (str | None) – The directory in which the workflow will be generated. The current directory if not specified.
name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified the WorkflowTemplate name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

classmethod from_file(template_path, template_format=None, path=None, name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from either a YAML or JSON file, depending on the file extension.

Parameters:

template_path (PathLike) – The path to a template file in YAML or JSON format, and with a “.yml”, “.yaml”, or “.json” extension.
template_format (str | None) – If specified, one of “json” or “yaml”. This forces parsing from a particular format regardless of the file extension.
path (str | None) – The directory in which the workflow will be generated. The current directory if not specified.
name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified the WorkflowTemplate name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

classmethod from_template(template, path=None, name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from a WorkflowTemplate object.

Parameters:

template (WorkflowTemplate) – The WorkflowTemplate object to make persistent.
path (PathLike | None) – The directory in which the workflow will be generated. The current directory if not specified.
name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified the WorkflowTemplate name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

classmethod from_template_data(template_name, tasks=None, loops=None, resources=None, path=None, workflow_name=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None)#

Generate from the data associated with a WorkflowTemplate object.

Parameters:

template_name (str) – Name of the new workflow template, from which the new workflow will be generated.
tasks (List[Task] | None) – List of Task objects to add to the new workflow.
loops (List[Loop] | None) – List of Loop objects to add to the new workflow.
resources (Dict[str, Dict] | None) – Mapping of action scopes to resource requirements, to be applied to all element sets in the workflow. resources specified in an element set take precedence of those defined here for the whole workflow.
path (PathLike | None) – The directory in which the workflow will be generated. The current directory if not specified.
workflow_name (str | None) – The name of the workflow. If specified, the workflow directory will be path joined with name. If not specified template_name will be used, in combination with a date-timestamp.
overwrite (bool | None) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str | None) – The persistent store to use for this workflow.
ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

Return type:

Workflow

get_EARs_from_IDs(indices)#

Return element action run objects from a list of five-tuples, representing the task insert ID, element index, iteration index, action index, and run index, respectively.

Parameters:: indices (List[EAR_ID]) –
Return type:: List[ElementActionRun]

get_all_parameter_data()#

Return type:: Dict[int, Any]

get_element_iterations_from_IDs(indices)#

Return element iteration objects from a list of three-tuples, representing the task insert ID, element index, and iteration index, respectively.

Parameters:: indices (List[IterationID]) –
Return type:: List[app.ElementIteration]

get_elements_from_IDs(indices)#

Return element objects from a list of two-tuples, representing the task insert ID, and element index, respectively.

Parameters:: indices (List[ElementID]) –
Return type:: List[app.Element]

get_iteration_task_pathway()#

get_parameter_data(index)#

Parameters:: index (int) –
Return type:: Tuple[bool, Any]

get_parameter_source(index)#

Parameters:: index (int) –
Return type:: Dict

get_task_elements(task, selection)#

Parameters:

task (Task) –
selection (slice) –

Return type:

List[Element]

get_task_elements_islice(task, selection)#

Parameters:

task (Task) –
selection (slice) –

Return type:

Iterator[Element]

get_task_unique_names(map_to_insert_ID=False)#

Return the unique names of all workflow tasks.

Parameters:: map_to_insert_ID (bool, optional) – If True, return a dict whose values are task insert IDs, otherwise return a list.
Return type:: List[str] | Dict[str, int]

is_parameter_set(index)#

Parameters:: index (int) –
Return type:: bool

property loops: WorkflowLoopList#

property name#: The workflow name may be different from the template name, as it includes the creation date-timestamp if generated.

property num_added_tasks: int#

property num_element_iterations: int#

property num_elements: int#

property num_loops: int#

property num_submissions: int#

property num_tasks: int#

rename(new_name)#

Parameters:: new_name (str) –

resolve_jobscripts()#

Return type:: List[Jobscript]

save_parameter(name, value, submission_idx, jobscript_idx, JS_element_idx, JS_action_idx)#

Parameters:

submission_idx (int) –
jobscript_idx (int) –
JS_element_idx (int) –
JS_action_idx (int) –

save_parameters(values, submission_idx, jobscript_idx, JS_element_idx, JS_action_idx)#

Save multiple parameters to a given EAR.

Parameters:

values (Dict) –
submission_idx (int) –
jobscript_idx (int) –
JS_element_idx (int) –
JS_action_idx (int) –

set_EAR_end(submission_idx, jobscript_idx, JS_element_idx, JS_action_idx)#

Set the end time on an EAR.

Parameters:

submission_idx (int) –
jobscript_idx (int) –
JS_element_idx (int) –
JS_action_idx (int) –

Return type:

None

set_EAR_start(submission_idx, jobscript_idx, JS_element_idx, JS_action_idx)#

Set the start time on an EAR.

Parameters:

submission_idx (int) –
jobscript_idx (int) –
JS_element_idx (int) –
JS_action_idx (int) –

Return type:

None

set_EAR_submission_indices(sub_idx, EAR_indices)#

Set the submission index on an EAR.

Parameters:

sub_idx (int) –
EAR_indices (Tuple[int, int, int, int]) –

Return type:

None

show_all_EAR_statuses()#

property store_format#

property submissions: List[Submission]#

property submissions_path#

submit(ignore_errors=False, JS_parallelism=None, print_stdout=False)#

Parameters:

ignore_errors (bool | None) –
JS_parallelism (bool | None) –
print_stdout (bool | None) –

Return type:

Dict[int, int]

property task_artifacts_path#

property tasks: WorkflowTaskList#

property template: WorkflowTemplate#

property template_components: Dict#

property ts_fmt#

property ts_name_fmt#

write_commands(submission_idx, jobscript_idx, JS_element_idx, JS_action_idx)#

Write run-time commands for a given EAR.

Parameters:

submission_idx (int) –
jobscript_idx (int) –
JS_element_idx (int) –
JS_action_idx (int) –

Return type:

None