hpcflow.sdk.persistence.zarr.ZarrZipPersistentStore#

class hpcflow.sdk.persistence.zarr.ZarrZipPersistentStore(app, workflow, path, fs)#

Bases: ZarrPersistentStore

A store designed mainly as an archive format that can be uploaded to data repositories such as Zenodo.

Note

Archive format persistent stores cannot be updated without being unzipped first.

Methods

`add_EAR`	Add a new EAR to an element iteration.
`add_element_iteration`	Add a new iteration to an element.
`add_element_iterations`	Add a new iteration to multiple elements.
`add_element_set`	Add an element set to a task.
`add_elements`	Add multiple new elements to a task.
`add_file`	Add a file that will be associated with a parameter.
`add_loop`	Add a new loop to the workflow.
`add_set_parameter`	Add a parameter that is set to a value.
`add_submission`	Add a new submission.
`add_task`	Add a new task to the workflow.
`add_template_components`	Add template components to the workflow.
`add_unset_parameter`	Add a parameter that is not set to any value.
`cache_ctx`	Context manager for using the persistent element/iteration/run cache.
`cached_load`	Context manager to cache the root attributes.
`check_parameters_exist`	For each parameter ID, return True if it exists, else False.
`clear_jobscript_at_submit_metadata_cache`	Clear the cache of at-submit-time jobscript metadata.
`copy`	Copy the workflow store.
`delete`	Delete the persistent workflow.
`delete_no_confirm`	Permanently delete the workflow data with no confirmation.
`get_EAR_skipped`	Whether the element action run with the given ID was skipped.
`get_EARs`	Get element action runs with the given IDs.
`get_binary_file`	Retrieve the contents of a binary file stored within the workflow.
`get_creation_info`	Get information about the creation of the workflow.
`get_dirs_array`	Retrieve the run directories array.
`get_element_iteration_dicts`	Get user-facing element-iteration data, including that of associated EARs.
`get_element_iterations`	Get element iterations with the given IDs.
`get_elements`	Get elements with the given IDs.
`get_jobscript_at_submit_metadata`	For the specified jobscript, retrieve the values of jobscript-submit-time attributes.
`get_jobscript_block_dependencies`	For the specified jobscript-block, retrieve the dependencies.
`get_jobscript_block_run_ID_array`	For the specified jobscript-block, retrieve the run ID array.
`get_jobscript_block_task_actions_array`	For the specified jobscript-block, retrieve the task-actions array.
`get_jobscript_block_task_elements_map`	For the specified jobscript-block, retrieve the task-elements mapping.
`get_loops`	Retrieve all loops, including pending.
`get_loops_by_IDs`	Retrieve loops by index (ID), including pending.
`get_name`	Get the name of the workflow.
`get_parameter_data_array_group`
`get_parameter_set_statuses`	Get whether the parameters with the given IDs are set.
`get_parameter_sources`	Get the sources of the parameters with the given IDs.
`get_parameter_sources_array`
`get_parameters`	Get parameters with the given IDs.
`get_submission_at_submit_metadata`	Retrieve the values of submission attributes that are stored at submit-time.
`get_submissions`	Retrieve all submissions, including pending.
`get_submissions_by_ID`	Get submissions with the given IDs.
`get_task`	Get a task.
`get_task_elements`	Get element data by an indices within a given task.
`get_tasks`	Retrieve all tasks, including pending.
`get_tasks_by_IDs`	Get tasks with the given IDs.
`get_template`	Get the workflow template.
`get_template_components`	Get all template components, including pending.
`get_text_file`	Retrieve the contents of a text file stored within the workflow.
`get_ts_fmt`	Get the format for timestamps.
`get_ts_name_fmt`	Get the format for timestamps to use in names.
`make_test_store_from_spec`	Generate an store for testing purposes.
`parameters_array_cache`	Context manager for the using the parameters-array cache, which here means caching the (existence of) Zarr array sub-groups within the parameter/arrays group.
`parameters_metadata_cache`	Context manager for using the parameters-metadata cache, which here means caching the parameter sources array object and its chunks.
`prepare_test_store_from_spec`	Generate a valid store from a specification in terms of nested elements/iterations/EARs.
`read_param_files`
`read_run_files`
`rechunk_parameter_base`	Rechunk the parameter data to be stored more efficiently.
`rechunk_runs`	Rechunk the run data to be stored more efficiently.
`reinstate_replaced_dir`	Reinstate the directory containing replaced workflow details.
`remove_path`	Try very hard to delete a directory or file.
`remove_replaced_dir`	Remove the directory containing replaced workflow details.
`rename_path`	Revert the replaced workflow path to its original name.
`save`	Commit pending changes to disk, if not in batch-update mode.
`set_EAR_end`	Mark an element action run as finished.
`set_EAR_skip`	Mark element action runs as skipped for the specified reasons.
`set_EAR_start`	Mark an element action run as started.
`set_EARs_initialised`	Mark an element action run as initialised.
`set_file`	Set details of a file, including whether it is associated with a parameter.
`set_jobscript_metadata`	Set the metadata for a job script.
`set_multi_run_ends`
`set_multi_run_starts`
`set_parameter_value`	Set the value of a parameter.
`set_parameter_values`	Set multiple non-file parameter values by parameter IDs.
`set_run_dirs`
`set_run_submission_data`	Set the run submission data, like the submission index for an element action run.
`unset_parameter_value`	Clear a parameter value without changing its source metadata.
`unzip`	Expand the persistent store.
`update_at_submit_metadata`	Update metadata that is set at submit-time.
`update_iter_data_indices`	Update data indices of one or more iterations.
`update_loop_num_iters`	Add iterations to a loop.
`update_loop_parents`	Set the parents of a loop.
`update_param_source`	Set the source of a parameter.
`update_run_data_indices`	Update data indices of one or more runs.
`using_resource`	Context manager for managing StoreResource objects associated with the store.
`write_empty_workflow`	Write an empty persistent workflow.
`write_param_files`
`write_run_files`
`zip`	Convert the persistent store to zipped form.

Attributes

`EAR_cache`	Cache for persistent EARs.
`element_cache`	Cache for persistent elements.
`element_iter_cache`	Cache for persistent element iterations.
`has_pending`	Whether there are any pending changes.
`is_submittable`	Does this store support workflow submission?
`logger`	The logger to use.
`num_EARs_cache`	Cache for total number of persistent EARs.
`num_iters_cache`	Cache for total number of persistent element iterations.
`num_params_cache`
`num_tasks_cache`	Cache for number of persistent tasks.
`param_sources_cache`	Cache for persistent parameter sources.
`parameter_cache`	Cache for persistent parameters.
`task_cache`	Cache for persistent tasks.
`ts_fmt`	The format for timestamps.
`use_cache`	Whether to use a cache.
`workflow`	The workflow this relates to.
`zarr_store`	The underlying store object.

Parameters:

path (str | Path)
fs (AbstractFileSystem)

property EAR_cache: dict[int, AnySEAR]#: Cache for persistent EARs.

add_EAR(elem_iter_ID, action_idx, commands_idx, data_idx, task_ID, element_idx, iteration_idx, metadata=None, save=True)#

Add a new EAR to an element iteration.

Parameters:

elem_iter_ID (int)
action_idx (int)
commands_idx (list[int])
data_idx (DataIndex)
task_ID (int)
element_idx (int)
iteration_idx (int)
metadata (Metadata | None)
save (bool)

Return type:

int

add_element_iteration(element_ID, data_idx, schema_parameters, task_ID, index, loop_idx=None, save=True)#

Add a new iteration to an element.

Parameters:

element_ID (int)
data_idx (DataIndex)
schema_parameters (list[str])
task_ID (int)
index (int)
loop_idx (Mapping[str, int] | None)
save (bool)

Return type:

int

add_element_iterations(element_IDs, data_idx_all, schema_parameters_all, task_ID, index, loop_idx_all=None, save=True)#

Add a new iteration to multiple elements.

Parameters:

element_IDs (Sequence[int])
data_idx_all (Sequence[DataIndex])
schema_parameters_all (Sequence[list[str]])
task_ID (int)
index (int)
loop_idx_all (Sequence[Mapping[str, int]] | None)
save (bool)

Return type:

list[int]

add_element_set(task_id, es_js, save=True)#

Add an element set to a task.

Parameters:

task_id (int)
es_js (Mapping)
save (bool)

add_elements(task_ID, es_idx, seq_idx, src_idx, save=True)#

Add multiple new elements to a task.

Parameters:

task_ID (int)
es_idx (int)
seq_idx (list[dict[str, int]])
src_idx (list[dict[str, int]])
save (bool)

add_file(store_contents, is_input, source, path, contents=None, filename=None, save=True)#

Add a file that will be associated with a parameter.

Parameters:

store_contents (bool)
is_input (bool)
source (ParamSource)
path (Path | str)
contents (str | None)
filename (str | None)
save (bool)

add_loop(loop_template, iterable_parameters, output_parameters, parents, num_added_iterations, iter_IDs, save=True)#

Add a new loop to the workflow.

Parameters:

loop_template (Mapping[str, Any])
iterable_parameters (Mapping[str, IterableParam])
output_parameters (Mapping[str, int])
parents (Sequence[str])
num_added_iterations (Mapping[tuple[int, ...], int])
iter_IDs (Iterable[int])
save (bool)

add_set_parameter(data, source, save=True)#

Add a parameter that is set to a value.

Parameters:

data (ParameterValue | list | tuple | set | dict | int | float | str | Any)
source (ParamSource)
save (bool)

Return type:

int

add_submission(sub_idx, sub_js, save=True)#

Add a new submission.

Parameters:

sub_idx (int)
sub_js (Mapping[str, JSONed])
save (bool)

add_task(idx, task_template, num_actions, save=True)#

Add a new task to the workflow.

Parameters:

idx (int)
task_template (Mapping)
num_actions (int)
save (bool)

add_template_components(temp_comps, save=True)#

Add template components to the workflow.

Parameters:

temp_comps (Mapping[str, dict])
save (bool)

Return type:

None

add_unset_parameter(source, save=True)#

Add a parameter that is not set to any value.

Parameters:

source (ParamSource)
save (bool)

Return type:

int

cache_ctx()#

Context manager for using the persistent element/iteration/run cache.

Return type:: Iterator[None]

cached_load()#

Context manager to cache the root attributes.

Return type:: Iterator[None]

check_parameters_exist(ids)#

For each parameter ID, return True if it exists, else False.

Parameters:: ids (Sequence[int])
Return type:: Iterator[bool]

clear_jobscript_at_submit_metadata_cache()#: Clear the cache of at-submit-time jobscript metadata.

copy(path=None)#

Copy the workflow store.

This does not work on remote filesystems.

Parameters:: path (PathLike)
Return type:: Path

delete()#

Delete the persistent workflow.

Return type:: None

delete_no_confirm()#

Permanently delete the workflow data with no confirmation.

Return type:: None

property element_cache: dict[int, AnySElement]#: Cache for persistent elements.

property element_iter_cache: dict[int, AnySElementIter]#: Cache for persistent element iterations.

get_EAR_skipped(EAR_ID)#

Whether the element action run with the given ID was skipped.

Parameters:: EAR_ID (int)
Return type:: int

get_EARs(ids)#

Get element action runs with the given IDs.

Parameters:: ids (Iterable[int])
Return type:: Sequence[AnySEAR]

get_binary_file(path)#

Retrieve the contents of a binary file stored within the workflow.

Parameters:: path (str | Path)
Return type:: bytes

get_creation_info()#: Get information about the creation of the workflow.

get_dirs_array()#

Retrieve the run directories array.

Return type:: NDArray

get_element_iteration_dicts(iter_IDs)#

Get user-facing element-iteration data, including that of associated EARs.

Parameters:: iter_IDs (Iterable[int])
Return type:: list[dict[str, Any]]

get_element_iterations(ids)#

Get element iterations with the given IDs.

Parameters:: ids (Iterable[int])
Return type:: Sequence[AnySElementIter]

get_elements(ids)#

Get elements with the given IDs.

Parameters:: ids (Iterable[int])
Return type:: Sequence[AnySElement]

get_jobscript_at_submit_metadata(sub_idx, js_idx, metadata_attr)#

For the specified jobscript, retrieve the values of jobscript-submit-time attributes.

Notes

If the cache does not exist, this method will retrieve and cache metadata for all jobscripts for which metadata has been set. If the cache does exist, but not for the requested jobscript, then this method will retrieve and cache metadata for all non-cached jobscripts for which metadata has been set. If metadata has not yet been set for the specified jobscript, and dict with all None values will be returned.

The cache can be cleared using the method clear_jobscript_at_submit_metadata_cache.

Parameters:

sub_idx (int)
js_idx (int)
metadata_attr (dict | None)

Return type:

dict[str, Any]

get_jobscript_block_dependencies(sub_idx, js_idx, blk_idx, js_dependencies)#

For the specified jobscript-block, retrieve the dependencies.

Parameters:

sub_idx (int)
js_idx (int)
blk_idx (int)
js_dependencies (dict[tuple[int, int], ResolvedJobscriptBlockDependencies] | None)

Return type:

dict[tuple[int, int], ResolvedJobscriptBlockDependencies]

get_jobscript_block_run_ID_array(sub_idx, js_idx, blk_idx, run_ID_arr)#

For the specified jobscript-block, retrieve the run ID array.

Parameters:

sub_idx (int)
js_idx (int)
blk_idx (int)
run_ID_arr (NDArray | None)

Return type:

NDArray

get_jobscript_block_task_actions_array(sub_idx, js_idx, blk_idx, task_actions_arr)#

For the specified jobscript-block, retrieve the task-actions array.

Parameters:

sub_idx (int)
js_idx (int)
blk_idx (int)
task_actions_arr (NDArray | list[tuple[int, int, int]] | None)

Return type:

NDArray

get_jobscript_block_task_elements_map(sub_idx, js_idx, blk_idx, task_elems_map)#

For the specified jobscript-block, retrieve the task-elements mapping.

Parameters:

sub_idx (int)
js_idx (int)
blk_idx (int)
task_elems_map (dict[int, list[int]] | None)

Return type:

dict[int, list[int]]

get_loops()#

Retrieve all loops, including pending.

Return type:: dict[int, LoopDescriptor]

get_loops_by_IDs(ids)#

Retrieve loops by index (ID), including pending.

Parameters:: ids (Iterable[int])
Return type:: dict[int, LoopDescriptor]

get_name()#: Get the name of the workflow.

get_parameter_data_array_group(parameter_idx)#

Parameters:: parameter_idx (int)
Return type:: Group | None

get_parameter_set_statuses(ids)#

Get whether the parameters with the given IDs are set.

Parameters:: ids (Iterable[int])
Return type:: list[bool]

get_parameter_sources(ids)#

Get the sources of the parameters with the given IDs.

Parameters:: ids (Iterable[int])
Return type:: list[ParamSource]

get_parameter_sources_array(mode='r')#

Return type:: Array

get_parameters(ids, **kwargs)#

Get parameters with the given IDs.

Parameters:: ids (Iterable[int]) – The IDs of the parameters to get.
Keyword Arguments:: dataset_copy (bool) – For Zarr stores only. If True, copy arrays as NumPy arrays.
Return type:: list[AnySParameter]

get_submission_at_submit_metadata(sub_idx, metadata_attr)#

Retrieve the values of submission attributes that are stored at submit-time.

Parameters:

sub_idx (int)
metadata_attr (dict | None)

Return type:

dict[str, Any]

get_submissions()#

Retrieve all submissions, including pending.

Return type:: dict[int, Mapping[str, JSONed]]

get_submissions_by_ID(ids)#

Get submissions with the given IDs.

Parameters:: ids (Iterable[int])
Return type:: dict[int, Mapping[str, JSONed]]

get_task(task_idx)#

Get a task.

Parameters:: task_idx (int)
Return type:: AnySTask

get_task_elements(task_id, idx_lst=None)#

Get element data by an indices within a given task.

Element iterations and EARs belonging to the elements are included.

Parameters:

task_id (int)
idx_lst (Iterable[int] | None)

Return type:

Iterator[Mapping[str, Any]]

get_tasks()#

Retrieve all tasks, including pending.

Return type:: list[AnySTask]

get_tasks_by_IDs(ids)#

Get tasks with the given IDs.

Parameters:: ids (Iterable[int])
Return type:: Sequence[AnySTask]

get_template()#

Get the workflow template.

Return type:: dict[str, JSONed]

get_template_components()#

Get all template components, including pending.

Return type:: dict[str, Any]

get_text_file(path)#

Retrieve the contents of a text file stored within the workflow.

Parameters:: path (str | Path) – The path to a text file stored within the workflow. This can either be an absolute path or a path that is relative to the workflow root.
Return type:: str

get_ts_fmt()#: Get the format for timestamps.

get_ts_name_fmt()#: Get the format for timestamps to use in names.

property has_pending: bool#: Whether there are any pending changes.

property is_submittable: bool#: Does this store support workflow submission?

property logger: Logger#: The logger to use.

classmethod make_test_store_from_spec(spec, dir=None, path='test_store', overwrite=False)#: Generate an store for testing purposes.

property num_EARs_cache: int | None#: Cache for total number of persistent EARs.

property num_iters_cache: int | None#: Cache for total number of persistent element iterations.

property num_params_cache: int | None#

property num_tasks_cache: int | None#: Cache for number of persistent tasks.

property param_sources_cache: dict[int, ParamSource]#: Cache for persistent parameter sources.

property parameter_cache: dict[int, AnySParameter]#: Cache for persistent parameters.

parameters_array_cache()#

Context manager for the using the parameters-array cache, which here means caching the (existence of) Zarr array sub-groups within the parameter/arrays group.

Return type:: Iterator[None]

parameters_metadata_cache()#

Context manager for using the parameters-metadata cache, which here means caching the parameter sources array object and its chunks.

Return type:: Iterator[None]

static prepare_test_store_from_spec(task_spec)#

Generate a valid store from a specification in terms of nested elements/iterations/EARs.

Parameters:: task_spec (Sequence[Mapping[str, Sequence[Mapping[str, Sequence[Mapping[str, Sequence]]]]]])
Return type:: tuple[list[dict], list[dict], list[dict], list[dict]]

read_param_files(run_file_lookup)#

Parameters:: run_file_lookup (Mapping[int, Mapping[int, Mapping[int, tuple[int, int]]]]) – Keys are submission indices. Values map file IDs to dictionaries mapping parameter source run IDs to indices within those files, and keys within
Return type:: dict[int, Any | None]

read_run_files(run_file_lookup)#

Parameters:: run_file_lookup (dict[int, dict[int, dict[int, int]]]) – Keys are submission indices. Values map file IDs to dictionaries mapping run IDs to indices within those files.
Return type:: dict[int, list[Any] | None]

rechunk_parameter_base(chunk_size=None, backup=True, status=True)#

Rechunk the parameter data to be stored more efficiently.

Parameters:

chunk_size (int | tuple[int, ...] | None)
backup (bool)
status (bool)

Return type:

Array

rechunk_runs(chunk_size=None, backup=True, status=True)#

Rechunk the run data to be stored more efficiently.

Parameters:

chunk_size (int | tuple[int, ...] | None)
backup (bool)
status (bool)

Return type:

Array

reinstate_replaced_dir()#

Reinstate the directory containing replaced workflow details.

Return type:: None

remove_path(path)#

Try very hard to delete a directory or file.

Dropbox (on Windows, at least) seems to try to re-sync files if the parent directory is deleted soon after creation, which is the case on a failed workflow creation (e.g. missing inputs), so in addition to catching PermissionErrors generated when Dropbox has a lock on files, we repeatedly try deleting the directory tree.

Parameters:: path (str | Path)
Return type:: None

remove_replaced_dir()#

Remove the directory containing replaced workflow details.

Return type:: None

rename_path(replaced, original)#

Revert the replaced workflow path to its original name.

This happens when new workflow creation fails and there is an existing workflow with the same name; the original workflow which was renamed, must be reverted.

Parameters:

replaced (str)
original (str | Path)

Return type:

None

save()#

Commit pending changes to disk, if not in batch-update mode.

Return type:: None

set_EAR_end(EAR_ID, exit_code, success, snapshot, save=True)#

Mark an element action run as finished.

Parameters:

EAR_ID (int)
exit_code (int)
success (bool)
snapshot (bool)
save (bool)

Return type:

datetime

set_EAR_skip(skip_reasons, save=True)#

Mark element action runs as skipped for the specified reasons.

Parameters:

skip_reasons (dict[int, int])
save (bool)

Return type:

None

set_EAR_start(EAR_ID, run_dir, port_number, save=True)#

Mark an element action run as started.

Parameters:

EAR_ID (int)
run_dir (Path | None)
port_number (int | None)
save (bool)

Return type:

datetime

set_EARs_initialised(iter_ID, save=True)#

Mark an element action run as initialised.

Parameters:

iter_ID (int)
save (bool)

Return type:

None

set_file(store_contents, is_input, param_id, path, contents=None, filename=None, clean_up=False, save=True)#

Set details of a file, including whether it is associated with a parameter.

Parameters:

store_contents (bool)
is_input (bool)
param_id (int | None)
path (Path | str)
contents (str | None)
filename (str | None)
clean_up (bool)
save (bool)

set_jobscript_metadata(sub_idx, js_idx, version_info=None, submit_time=None, submit_hostname=None, submit_machine=None, shell_idx=None, submit_cmdline=None, os_name=None, shell_name=None, scheduler_name=None, scheduler_job_ID=None, process_ID=None, save=True)#

Set the metadata for a job script.

Parameters:

sub_idx (int)
js_idx (int)
version_info (VersionInfo | None)
submit_time (str | None)
submit_hostname (str | None)
submit_machine (str | None)
shell_idx (int | None)
submit_cmdline (list[str] | None)
os_name (str | None)
shell_name (str | None)
scheduler_name (str | None)
scheduler_job_ID (str | None)
process_ID (int | None)
save (bool)

set_multi_run_ends(run_ids, run_dirs, exit_codes, successes, save=True)#

Parameters:

run_ids (list[int])
run_dirs (list[Path | None])
exit_codes (list[int])
successes (list[bool])
save (bool)

Return type:

datetime

set_multi_run_starts(run_ids, run_dirs, port_number, save=True)#

Parameters:

run_ids (list[int])
run_dirs (list[Path | None])
port_number (int)
save (bool)

Return type:

datetime

set_parameter_value(param_id, value, is_file=False, save=True)#

Set the value of a parameter.

Parameters:

param_id (int)
value (Any)
is_file (bool)
save (bool)

set_parameter_values(values, save=True)#

Set multiple non-file parameter values by parameter IDs.

Parameters:

values (dict[int, Any])
save (bool)

set_run_dirs(run_dir_indices, run_idx, save=True)#

Parameters:

run_dir_indices (ndarray)
run_idx (ndarray)
save (bool)

set_run_submission_data(EAR_ID, cmds_ID, sub_idx, run_file_ID, run_file_idx, save=True)#

Set the run submission data, like the submission index for an element action run.

Parameters:

EAR_ID (int)
cmds_ID (int | None)
sub_idx (int)
run_file_ID (int)
run_file_idx (int)
save (bool)

Return type:

None

property task_cache: dict[int, AnySTask]#: Cache for persistent tasks.

property ts_fmt: str#: The format for timestamps.

unset_parameter_value(param_id, save=True)#

Clear a parameter value without changing its source metadata.

Parameters:

param_id (int)
save (bool)

Return type:

None

unzip(path='.', log=None, status=True)#

Expand the persistent store.

Parameters:

path (str) – Path at which to create the new unzipped workflow. If this is an existing directory, the new workflow directory will be created within this directory. Otherwise, this path will represent the new workflow directory path.
log (str | None)
status (bool)

Return type:

str

update_at_submit_metadata(sub_idx, submission_parts, save=True)#

Update metadata that is set at submit-time.

Parameters:

sub_idx (int)
submission_parts (dict[str, list[int]])
save (bool)

update_iter_data_indices(data_indices)#

Update data indices of one or more iterations.

Parameters:: data_indices (dict[int, DataIndex])

update_loop_num_iters(index, num_added_iters, save=True)#

Add iterations to a loop.

Parameters:

index (int)
num_added_iters (Mapping[tuple[int, ...], int])
save (bool)

Return type:

None

update_loop_parents(index, num_added_iters, parents, save=True)#

Set the parents of a loop.

Parameters:

index (int)
num_added_iters (Mapping[tuple[int, ...], int])
parents (Sequence[str])
save (bool)

Return type:

None

update_param_source(param_sources, save=True)#

Set the source of a parameter.

Parameters:

param_sources (Mapping[int, ParamSource])
save (bool)

Return type:

None

update_run_data_indices(data_indices)#

Update data indices of one or more runs.

Parameters:: data_indices (dict[int, DataIndex])

property use_cache: bool#: Whether to use a cache.

using_resource(res_label, action)#

Context manager for managing StoreResource objects associated with the store.

Parameters:

res_label (Literal['metadata', 'submissions', 'parameters', 'attrs', 'runs'])
action (str)

Return type:

Iterator[Any]

property workflow: Workflow#: The workflow this relates to.

classmethod write_empty_workflow(app, *, template_js, template_components_js, wk_path, fs, name, replaced_wk, ts_fmt, ts_name_fmt, creation_info, compressor='blosc', compressor_kwargs=None)#

Write an empty persistent workflow.

Parameters:

app (BaseApp)
template_js (TemplateMeta)
template_components_js (dict[str, Any])
wk_path (str)
fs (AbstractFileSystem)
name (str)
replaced_wk (str | None)
ts_fmt (str)
ts_name_fmt (str)
creation_info (StoreCreationInfo)
compressor (str | None)
compressor_kwargs (dict[str, Any] | None)

Return type:

None

write_param_files(data)#

Parameters:

data (dict[int, dict[int, dict[tuple[int, int], Any]]]) –

Keys are submission indices.

Values map file IDs to dictionaries mapping indices within those files to dictionaries of parameter data.

Structure should be: like

dict[

int, # submission_idx dict[

int, # file_ID dict[

tuple(int, int), # local index and key within that index Any # data of a single parameter

],

],

]

write_run_files(data)#

Parameters:

data (dict[int, dict[int, dict[int, dict[int, Any]]]]) –

Keys are submission indices.

Values map file IDs to dictionaries mapping indices within those files to dictionaries of execution-metadata field indices and their new values.

Structure should be: like

dict[

int, # submission_idx dict[

int, # file_ID dict[

int, # local index dict[int, Any], # field index -> value

],

],

]

property zarr_store: Store#: The underlying store object.

zip(path='.', log=None, overwrite=False, include_execute=False, include_rechunk_backups=False, status=True)#

Convert the persistent store to zipped form.

Parameters:

path (str) – Path at which to create the new zipped workflow. If this is an existing directory, the zip file will be created within this directory. Otherwise, this path is assumed to be the full file path to the new zip file.
log (str | None)
overwrite (bool)
include_execute (bool)
include_rechunk_backups (bool)
status (bool)