hpcflow.sdk.persistence.pending.PendingChanges#

class hpcflow.sdk.persistence.pending.PendingChanges(app, store, resource_map)#

Bases: Generic[AnySTask, AnySElement, AnySElementIter, AnySEAR, AnySParameter]

Class to store pending changes and merge them into a persistent store.

Parameters:

app (App) – The main application context.
store (PersistentStore) – The persistent store that owns this object
resource_map (CommitResourceMap) – Map of resources, used when processing commits.

Methods

`commit_EAR_ends`	Commit pending element action run finish information to disk.
`commit_EAR_skips`	Commit pending element action skip flags to disk.
`commit_EAR_starts`	Commit pending element action run start information to disk.
`commit_EAR_submission_indices`	Commit pending element action run submission index updates to disk.
`commit_EARs`	Commit pending element action runs to disk.
`commit_EARs_initialised`	Commit pending element action run init state updates to disk.
`commit_all`	Commit all pending changes to disk.
`commit_elem_IDs`	Commit pending element ID updates to disk.
`commit_elem_iter_EAR_IDs`	Commit pending element action run ID updates to disk.
`commit_elem_iter_IDs`	Commit pending element iteration ID updates to disk.
`commit_elem_iters`	Commit pending element iterations to disk.
`commit_element_sets`	Commit pending element sets to disk.
`commit_elements`	Commit pending elements to disk.
`commit_files`	Add pending files to the files directory.
`commit_js_metadata`	Commit pending jobscript metadata changes to disk.
`commit_loop_indices`	Make pending update to element iteration loop indices persistent.
`commit_loop_num_iters`	Make pending update to the number of loop iterations.
`commit_loop_parents`	Make pending update to additional loop parents.
`commit_loops`	Commit pending loops to disk.
`commit_param_sources`	Make pending changes to parameter sources persistent.
`commit_parameters`	Make pending parameters persistent.
`commit_submission_parts`	Commit pending submission parts to disk.
`commit_submissions`	Commit pending submissions to disk.
`commit_tasks`	Commit pending tasks to disk.
`commit_template_components`	Commit pending template components to disk.
`reset`	Clear all pending data and prepare to accept new pending data.
`where_pending`	Get the list of items for which there is some outstanding pending items.

Attributes

`logger`	The logger.
`add_tasks`	Keys are new task IDs.
`add_loops`	Keys are loop IDs, values are loop descriptors.
`add_submissions`	Keys are submission IDs, values are submission descriptors.
`add_elements`	Keys are element IDs.
`add_elem_iters`	Keys are element iteration IDs.
`add_EARs`	Keys are element action run IDs.
`add_parameters`	Keys are parameter indices and values are tuples whose first element is data to add and whose second element is the source dict for the new data.
`add_files`	Workflow-related files (inputs, outputs) added to the persistent store.
`add_template_components`	Template components to add.
`add_element_sets`	Keys are element set IDs, values are descriptors.
`add_elem_IDs`	Keys are task IDs, and values are element IDs to add to that task.
`add_elem_iter_IDs`	Keys are element IDs, and values are iteration IDs to add to that element.
`add_elem_iter_EAR_IDs`	Keys are element iteration IDs, then EAR action index, and values are EAR IDs.
`add_submission_parts`	Submission parts to add.
`set_EARs_initialised`	IDs of EARs to mark as initialised.
`set_EAR_submission_indices`	Submission IDs to attach to EARs.
`set_EAR_skips`	IDs of EARs to mark as skipped.
`set_EAR_starts`	Keys are EAR IDs and values are tuples of start time, and start dir snapshot.
`set_EAR_ends`	Keys are EAR IDs and values are tuples of end time, end dir snapshot, exit code, and success boolean.
`set_js_metadata`	Keys are IDs of jobscripts.
`set_parameters`	Keys are IDs of parameters to add or modify.
`update_param_sources`	Keys are parameter indices and values are dict parameter sources to merge with existing source of that parameter.
`update_loop_indices`	Keys are indices of loops, values are descriptions of what to update.
`update_loop_num_iters`	Keys are indices of loops, values are number of iterations.
`update_loop_parents`	Keys are indices of loops, values are list of parent names.

add_EARs: dict[int, AnySEAR]#: Keys are element action run IDs.

add_elem_IDs: dict[int, list[int]]#: Keys are task IDs, and values are element IDs to add to that task.

add_elem_iter_EAR_IDs: dict[int, dict[int, list[int]]]#: Keys are element iteration IDs, then EAR action index, and values are EAR IDs. This is a list of EAR IDs to add to a given element iteration action.

add_elem_iter_IDs: dict[int, list[int]]#: Keys are element IDs, and values are iteration IDs to add to that element.

add_elem_iters: dict[int, AnySElementIter]#: Keys are element iteration IDs.

add_element_sets: dict[int, list[Mapping]]#: Keys are element set IDs, values are descriptors.

add_elements: dict[int, AnySElement]#: Keys are element IDs.

add_files: list[FileDescriptor]#: Workflow-related files (inputs, outputs) added to the persistent store.

add_loops: dict[int, LoopDescriptor]#: Keys are loop IDs, values are loop descriptors.

add_parameters: dict[int, AnySParameter]#: Keys are parameter indices and values are tuples whose first element is data to add and whose second element is the source dict for the new data.

add_submission_parts: dict[int, dict[str, list[int]]]#: Submission parts to add.

add_submissions: dict[int, JSONDocument]#: Keys are submission IDs, values are submission descriptors.

add_tasks: dict[int, AnySTask]#: Keys are new task IDs.

add_template_components: dict[str, dict[str, dict]]#: Template components to add.

commit_EAR_ends()#

Commit pending element action run finish information to disk.

Return type:: None

commit_EAR_skips()#

Commit pending element action skip flags to disk.

Return type:: None

commit_EAR_starts()#

Commit pending element action run start information to disk.

Return type:: None

commit_EAR_submission_indices()#

Commit pending element action run submission index updates to disk.

Return type:: None

commit_EARs()#

Commit pending element action runs to disk.

Return type:: None

commit_EARs_initialised()#

Commit pending element action run init state updates to disk.

Return type:: None

commit_all()#

Commit all pending changes to disk.

Return type:: None

commit_elem_IDs()#

Commit pending element ID updates to disk.

Return type:: None

commit_elem_iter_EAR_IDs()#

Commit pending element action run ID updates to disk.

Return type:: None

commit_elem_iter_IDs()#

Commit pending element iteration ID updates to disk.

Return type:: None

commit_elem_iters()#

Commit pending element iterations to disk.

Return type:: None

commit_element_sets()#

Commit pending element sets to disk.

Return type:: None

commit_elements()#

Commit pending elements to disk.

Return type:: None

commit_files()#

Add pending files to the files directory.

Return type:: None

commit_js_metadata()#

Commit pending jobscript metadata changes to disk.

Return type:: None

commit_loop_indices()#

Make pending update to element iteration loop indices persistent.

Return type:: None

commit_loop_num_iters()#

Make pending update to the number of loop iterations.

Return type:: None

commit_loop_parents()#

Make pending update to additional loop parents.

Return type:: None

commit_loops()#

Commit pending loops to disk.

Return type:: None

commit_param_sources()#

Make pending changes to parameter sources persistent.

Return type:: None

commit_parameters()#

Make pending parameters persistent.

Return type:: None

commit_submission_parts()#

Commit pending submission parts to disk.

Return type:: None

commit_submissions()#

Commit pending submissions to disk.

Return type:: None

commit_tasks()#

Commit pending tasks to disk.

Return type:: None

commit_template_components()#

Commit pending template components to disk.

Return type:: None

property logger: Logger#: The logger.

reset(is_init=False)#

Clear all pending data and prepare to accept new pending data.

Parameters:: is_init (bool) –
Return type:: None

set_EAR_ends: dict[int, tuple[datetime, dict[str, Any], int, bool]]#: Keys are EAR IDs and values are tuples of end time, end dir snapshot, exit code, and success boolean.

set_EAR_skips: list[int]#: IDs of EARs to mark as skipped.

set_EAR_starts: dict[int, tuple[datetime, dict[str, Any], str]]#: Keys are EAR IDs and values are tuples of start time, and start dir snapshot.

set_EAR_submission_indices: dict[int, int]#: Submission IDs to attach to EARs.

set_EARs_initialised: list[int]#: IDs of EARs to mark as initialised.

set_js_metadata: dict[int, dict[int, dict[str, Any]]]#: Keys are IDs of jobscripts.

set_parameters: dict[int, tuple[Any, bool]]#: Keys are IDs of parameters to add or modify.

update_loop_indices: dict[int, dict[str, int]]#: Keys are indices of loops, values are descriptions of what to update.

update_loop_num_iters: dict[int, list[list[list[int] | int]]]#: Keys are indices of loops, values are number of iterations.

update_loop_parents: dict[int, list[str]]#: Keys are indices of loops, values are list of parent names.

update_param_sources: dict[int, ParamSource]#: Keys are parameter indices and values are dict parameter sources to merge with existing source of that parameter.

where_pending()#

Get the list of items for which there is some outstanding pending items.

Return type:: list[str]