hpcflow.app.Jobscript

class hpcflow.app.Jobscript(task_insert_IDs, task_actions, task_elements, EAR_ID, resources, task_loop_idx, dependencies, submit_time=None, submit_hostname=None, submit_machine=None, submit_cmdline=None, scheduler_job_ID=None, process_ID=None, version_info=None, os_name=None, shell_name=None, scheduler_name=None, running=None, resource_hash=None, elements=None)

Bases: Jobscript

A group of actions that are submitted together to be executed by the underlying job management system as a single unit.

Parameters:

• task_insert_IDs (list[int]) – The task insertion IDs.

• task_actions (list[tuple]) – The actions of the tasks. task insert ID, action_idx, index into task_loop_idx for each JS_ACTION_IDX

• task_elements (dict[int, list[int]]) – The elements of the tasks. Maps JS_ELEMENT_IDX to list of TASK_ELEMENT_IDX for each TASK_INSERT_ID

• EAR_ID (NDArray) – Element action run information.

• resources (ElementResources) – Resources to use

• task_loop_idx (list[dict]) – Description of what loops are in play.

• dependencies (dict[int, dict]) – Description of dependencies.

• submit_time (datetime) – When the jobscript was submitted, if known.

• submit_hostname (str) – Where the jobscript was submitted, if known.

• submit_machine (str) – Description of what the jobscript was submitted to, if known.

• submit_cmdline (str) – The command line used to do the commit, if known.

• scheduler_job_ID (str) – The job ID from the scheduler, if known.

• process_ID (int) – The process ID of the subprocess, if known.

• version_info (dict[str, ...]) – Version info about the target system.

• os_name (str) – The name of the OS.

• shell_name (str) – The name of the shell.

• scheduler_name (str) – The scheduler used.

• running (bool) – Whether the jobscript is currently running.

• resource_hash (str | None) –

• elements (dict[int, list[int]] | None) –

Methods

cancel	Cancel this jobscript.
compose_jobscript	Prepare the jobscript file string.
from_json_like	Make an instance of this class from JSON (or YAML) data.
get_active_states	If this jobscript is active on this machine, return the state information from the scheduler.
get_commands_file_name	Get the name of a file containing commands for a particular jobscript action.
get_task_loop_idx_array	Get an array of task loop indices.
make_artifact_dirs	Create the directories that will hold artifacts associated with this jobscript.
submit	Submit the jobscript to the scheduler.
to_dict	Serialize this object as a dictionary.
to_json_like	Serialize this object as an object structure that can be trivially converted to JSON.
write_EAR_ID_file	Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.
write_element_run_dir_file	Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.
write_jobscript	Write the jobscript to its file.

Attributes

EAR_ID	The array of EAR IDs.
EAR_ID_file_name	The name of a file containing EAR IDs.
EAR_ID_file_path	The path to the file containing EAR IDs for this jobscript.
all_EAR_IDs	The IDs of all EARs in this jobscript.
all_EARs	Description of EAR information for this jobscript.
dependencies	The dependency descriptor.
direct_stderr_file_name	File for direct execution stderr.
direct_stderr_path	The path to the file containing the stderr from directly executed commands for this jobscript.
direct_stdout_file_name	File for direct execution stdout.
direct_stdout_path	The path to the file containing the stdout from directly executed commands for this jobscript.
direct_win_pid_file_name	File for holding the direct execution PID.
direct_win_pid_file_path	The path to the file containing PIDs for directly executed commands for this jobscript.
element_run_dir_file_name	The name of a file containing run directory names.
element_run_dir_file_path	The path to the file containing run directory names for this jobscript.
end_time	The last known end time of any EAR in this jobscript.
index	The index of this jobscript within its parent Submission.
is_array	Whether to generate an array job.
is_submitted	Whether this jobscript has been submitted.
jobscript_name	The name of the jobscript file.
jobscript_path	The path to the file containing the jobscript file.
num_actions	The number of actions in this jobscript.
num_elements	The number of elements in this jobscript.
os_name	The name of the OS to use.
process_ID	The process ID from direct execution, if known.
resources	The common resources that this jobscript requires.
scheduler	The scheduler that submissions go to from this jobscript.
scheduler_job_ID	The job ID from the scheduler, if known.
scheduler_js_ref	The reference to the submitted job for the jobscript.
scheduler_name	The name of the scheduler to use.
scheduler_ref	The generalised scheduler reference descriptor.
shell	The shell for composing submission scripts.
shell_name	The name of the shell to use.
start_time	The first known start time of any EAR in this jobscript.
submission	The parent submission.
submit_cmdline	The command line used to do the commit, if known.
submit_hostname	Where the jobscript was submitted, if known.
submit_machine	Description of what the jobscript was submitted to, if known.
submit_time	When the jobscript was submitted, if known.
task_actions	The IDs of actions of each task in this jobscript.
task_elements	The IDs of elements of each task in this jobscript.
task_insert_IDs	The insertion IDs of tasks in this jobscript.
task_loop_idx	The description of where various task loops are.
version_info	Version information about the execution environment (OS, etc).
workflow	The workflow this is all on behalf of.
workflow_app_alias	Alias for the workflow app in job scripts.

property EAR_ID: NDArray

The array of EAR IDs.

property EAR_ID_file_name: str

The name of a file containing EAR IDs.

property EAR_ID_file_path: Path

The path to the file containing EAR IDs for this jobscript.

property all_EAR_IDs: Iterable[int]

The IDs of all EARs in this jobscript.

property all_EARs: Sequence[ElementActionRun]

Description of EAR information for this jobscript.

cancel()

Cancel this jobscript.

Return type:

None

compose_jobscript(deps=None, os_name=None, shell_name=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)

Prepare the jobscript file string.

Parameters:

• deps (dict[int, tuple[str, bool]] | None) –

• os_name (str | None) –

• shell_name (str | None) –

• os_args (dict[str, Any] | None) –

• shell_args (dict[str, Any] | None) –

• scheduler_name (str | None) –

• scheduler_args (dict[str, Any] | None) –

Return type:

str

property dependencies: Mapping[int, ResolvedDependencies]

The dependency descriptor.

property direct_stderr_file_name: str

File for direct execution stderr.

property direct_stderr_path: Path

The path to the file containing the stderr from directly executed commands for this jobscript.

property direct_stdout_file_name: str

File for direct execution stdout.

property direct_stdout_path: Path

The path to the file containing the stdout from directly executed commands for this jobscript.

property direct_win_pid_file_name: str

File for holding the direct execution PID.

property direct_win_pid_file_path: Path

The path to the file containing PIDs for directly executed commands for this jobscript. Windows only.

property element_run_dir_file_name: str

The name of a file containing run directory names.

property element_run_dir_file_path: Path

The path to the file containing run directory names for this jobscript.

property end_time: None | datetime

The last known end time of any EAR in this jobscript.

classmethod from_json_like(json_like, shared_data=None)

Make an instance of this class from JSON (or YAML) data.

Parameters:

• json_like – The data to deserialise.

• shared_data – Shared context data.

Return type:

The deserialised object.

get_active_states(as_json=False)

If this jobscript is active on this machine, return the state information from the scheduler.

Parameters:

as_json (bool) –

Return type:

Mapping[int, JobscriptElementState] | dict[int, str]

get_commands_file_name(js_action_idx, shell=None)

Get the name of a file containing commands for a particular jobscript action.

Parameters:

• js_action_idx (int | str) –

• shell (Shell | None) –

Return type:

str

get_task_loop_idx_array()

Get an array of task loop indices.

Return type:

NDArray

property index: int

The index of this jobscript within its parent Submission.

property is_array: bool

Whether to generate an array job.

property is_submitted: bool

Whether this jobscript has been submitted.

property jobscript_name: str

The name of the jobscript file.

property jobscript_path: Path

The path to the file containing the jobscript file.

make_artifact_dirs()

Create the directories that will hold artifacts associated with this jobscript.

Return type:

list[list[Path]]

property num_actions: int

The number of actions in this jobscript.

property num_elements: int

The number of elements in this jobscript.

property os_name: str

The name of the OS to use.

property process_ID: int | None

The process ID from direct execution, if known.

property resources: ElementResources

The common resources that this jobscript requires.

property scheduler: Scheduler

The scheduler that submissions go to from this jobscript.

property scheduler_job_ID: str | None

The job ID from the scheduler, if known.

property scheduler_js_ref

The reference to the submitted job for the jobscript.

property scheduler_name: str | None

The name of the scheduler to use.

property scheduler_ref: SchedulerRef

The generalised scheduler reference descriptor.

property shell: Shell

The shell for composing submission scripts.

property shell_name: str | None

The name of the shell to use.

property start_time: None | datetime

The first known start time of any EAR in this jobscript.

property submission: Submission

The parent submission.

submit(scheduler_refs, print_stdout=False)

Submit the jobscript to the scheduler.

Parameters:

• scheduler_refs (dict[int, tuple[str, bool]]) –

• print_stdout (bool) –

Return type:

str

property submit_cmdline: list[str] | None

The command line used to do the commit, if known.

property submit_hostname: str | None

Where the jobscript was submitted, if known.

property submit_machine: str | None

Description of what the jobscript was submitted to, if known.

property submit_time: datetime | None

When the jobscript was submitted, if known.

property task_actions: Sequence[tuple[int, int, int]]

The IDs of actions of each task in this jobscript.

property task_elements: Mapping[int, Sequence[int]]

The IDs of elements of each task in this jobscript.

property task_insert_IDs: Sequence[int]

The insertion IDs of tasks in this jobscript.

property task_loop_idx: Sequence[Mapping[str, int]]

The description of where various task loops are.

to_dict()

Serialize this object as a dictionary.

Return type:

dict[str, Any]

to_json_like(dct=None, shared_data=None, exclude=(), path=None)

Serialize this object as an object structure that can be trivially converted to JSON. Note that YAML can also be produced from the result of this method; it just requires a different final serialization step.

Parameters:

• dct (dict[str, JSONable] | None) –

• shared_data (_JSONDeserState) –

• exclude (Container[str | None]) –

• path (list | None) –

Return type:

tuple[JSONDocument, _JSONDeserState]

property version_info: VersionInfo | None

Version information about the execution environment (OS, etc).

property workflow: Workflow

The workflow this is all on behalf of.

property workflow_app_alias: str

Alias for the workflow app in job scripts.

write_EAR_ID_file()

Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.

write_element_run_dir_file(run_dirs)

Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.

We assume a given task element’s actions all run in the same directory, but in general a jobscript “element” may cross task boundaries, so we need to provide the directory for each jobscript-element/jobscript-action combination.

Parameters:

run_dirs (list[list[Path]]) –

write_jobscript(os_name=None, shell_name=None, deps=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)

Write the jobscript to its file.

Parameters:

• os_name (str | None) –

• shell_name (str | None) –

• deps (dict[int, tuple[str, bool]] | None) –

• os_args (dict[str, Any] | None) –

• shell_args (dict[str, Any] | None) –

• scheduler_name (str | None) –

• scheduler_args (dict[str, Any] | None) –

Return type:

Path