hpcflow.app.Jobscript#

class hpcflow.app.Jobscript(task_insert_IDs, task_actions, task_elements, EAR_ID, resources, task_loop_idx, dependencies, submit_time=None, submit_hostname=None, submit_machine=None, submit_cmdline=None, scheduler_job_ID=None, process_ID=None, version_info=None, os_name=None, shell_name=None, scheduler_name=None, running=None)#

Bases: Jobscript

A group of actions that are submitted together to be executed by the underlying job management system as a single unit.

Parameters:
  • task_insert_IDs (list[int]) – The task insertion IDs.

  • task_actions (list[tuple]) – The actions of the tasks. task insert ID, action_idx, index into task_loop_idx for each JS_ACTION_IDX

  • task_elements (dict[int, list[int]]) – The elements of the tasks. Maps JS_ELEMENT_IDX to list of TASK_ELEMENT_IDX for each TASK_INSERT_ID

  • EAR_ID (NDArray) – Element action run information.

  • resources (ElementResources) – Resources to use

  • task_loop_idx (list[dict]) – Description of what loops are in play.

  • dependencies (dict[int, dict]) – Description of dependencies.

  • submit_time (datetime) – When the jobscript was submitted, if known.

  • submit_hostname (str) – Where the jobscript was submitted, if known.

  • submit_machine (str) – Description of what the jobscript was submitted to, if known.

  • submit_cmdline (str) – The command line used to do the commit, if known.

  • scheduler_job_ID (str) – The job ID from the scheduler, if known.

  • process_ID (int) – The process ID of the subprocess, if known.

  • version_info (tuple[str, ...]) – Version info about the target system.

  • os_name (str) – The name of the OS.

  • shell_name (str) – The name of the shell.

  • scheduler_name (str) – The scheduler used.

  • running (bool) – Whether the jobscript is currently running.

Methods

cancel

Cancel this jobscript.

compose_jobscript

Prepare the jobscript file string.

from_json_like

Make an instance of this class from JSON (or YAML) data.

get_active_states

If this jobscript is active on this machine, return the state information from the scheduler.

get_commands_file_name

Get the name of a file containing commands for a particular jobscript action.

get_task_loop_idx_array

Get an array of task loop indices.

make_artifact_dirs

Create the directories that will hold artifacts associated with this jobscript.

submit

Submit the jobscript to the scheduler.

to_dict

Serialize this object as a dictionary.

to_json_like

Serialize this object as an object structure that can be trivially converted to JSON.

write_EAR_ID_file

Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.

write_element_run_dir_file

Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.

write_jobscript

Write the jobscript to its file.

Attributes

EAR_ID

The array of EAR IDs.

EAR_ID_file_name

The name of a file containing EAR IDs.

EAR_ID_file_path

The path to the file containing EAR IDs for this jobscript.

all_EAR_IDs

The IDs of all EARs in this jobscript.

all_EARs

Description of EAR information for this jobscript.

app

dependencies

The dependency descriptor.

direct_stderr_file_name

File for direct execution stderr.

direct_stderr_path

The path to the file containing the stderr from directly executed commands for this jobscript.

direct_stdout_file_name

File for direct execution stdout.

direct_stdout_path

The path to the file containing the stdout from directly executed commands for this jobscript.

direct_win_pid_file_name

File for holding the direct execution PID.

direct_win_pid_file_path

The path to the file containing PIDs for directly executed commands for this jobscript.

element_run_dir_file_name

The name of a file containing run directory names.

element_run_dir_file_path

The path to the file containing run directory names for this jobscript.

end_time

The last known end time of any EAR in this jobscript.

index

The index of this jobscript within its parent Submission.

is_array

Whether to generate an array job.

is_submitted

Whether this jobscript has been submitted.

jobscript_name

The name of the jobscript file.

jobscript_path

The path to the file containing the jobscript file.

num_actions

The number of actions in this jobscript.

num_elements

The number of elements in this jobscript.

os_name

The name of the OS to use.

process_ID

The process ID from direct execution, if known.

resources

The common resources that this jobscript requires.

scheduler

The scheduler that submissions go to from this jobscript.

scheduler_job_ID

The job ID from the scheduler, if known.

scheduler_js_ref

The reference to the submitted job for the jobscript.

scheduler_name

The name of the scheduler to use.

scheduler_ref

The generalised scheduler reference descriptor.

shell

The shell for composing submission scripts.

shell_name

The name of the shell to use.

start_time

The first known start time of any EAR in this jobscript.

submission

The parent submission.

submit_cmdline

The command line used to do the commit, if known.

submit_hostname

Where the jobscript was submitted, if known.

submit_machine

Description of what the jobscript was submitted to, if known.

submit_time

When the jobscript was submitted, if known.

task_actions

The IDs of actions of each task in this jobscript.

task_elements

The IDs of elements of each task in this jobscript.

task_insert_IDs

The insertion IDs of tasks in this jobscript.

task_loop_idx

The description of where various task loops are.

version_info

Version information about the execution environment (OS, etc).

workflow

The workflow this is all on behalf of.

workflow_app_alias

Alias for the workflow app in job scripts.

property EAR_ID#

The array of EAR IDs.

property EAR_ID_file_name#

The name of a file containing EAR IDs.

property EAR_ID_file_path#

The path to the file containing EAR IDs for this jobscript.

property all_EAR_IDs: List[int]#

The IDs of all EARs in this jobscript.

property all_EARs: List#

Description of EAR information for this jobscript.

app = BaseApp(name='hpcFlow', version='0.2.0a180')#
cancel()#

Cancel this jobscript.

compose_jobscript(deps=None, os_name=None, shell_name=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)#

Prepare the jobscript file string.

Parameters:
  • deps (Dict | None) –

  • os_name (str) –

  • shell_name (str) –

  • os_args (Dict | None) –

  • shell_args (Dict | None) –

  • scheduler_name (str | None) –

  • scheduler_args (Dict | None) –

Return type:

str

property dependencies#

The dependency descriptor.

property direct_stderr_file_name#

File for direct execution stderr.

property direct_stderr_path#

The path to the file containing the stderr from directly executed commands for this jobscript.

property direct_stdout_file_name#

File for direct execution stdout.

property direct_stdout_path#

The path to the file containing the stdout from directly executed commands for this jobscript.

property direct_win_pid_file_name#

File for holding the direct execution PID.

property direct_win_pid_file_path#

The path to the file containing PIDs for directly executed commands for this jobscript. Windows only.

property element_run_dir_file_name#

The name of a file containing run directory names.

property element_run_dir_file_path#

The path to the file containing run directory names for this jobscript.

property end_time#

The last known end time of any EAR in this jobscript.

classmethod from_json_like(json_like, shared_data=None)#

Make an instance of this class from JSON (or YAML) data.

Parameters:
  • json_like – The data to deserialise.

  • shared_data – Shared context data.

Return type:

The deserialised object.

get_active_states(as_json=False)#

If this jobscript is active on this machine, return the state information from the scheduler.

Parameters:

as_json (bool) –

Return type:

Dict[int, JobscriptElementState]

get_commands_file_name(js_action_idx, shell=None)#

Get the name of a file containing commands for a particular jobscript action.

get_task_loop_idx_array()#

Get an array of task loop indices.

property index#

The index of this jobscript within its parent Submission.

property is_array#

Whether to generate an array job.

property is_submitted#

Whether this jobscript has been submitted.

property jobscript_name#

The name of the jobscript file.

property jobscript_path#

The path to the file containing the jobscript file.

make_artifact_dirs()#

Create the directories that will hold artifacts associated with this jobscript.

property num_actions#

The number of actions in this jobscript.

property num_elements#

The number of elements in this jobscript.

property os_name: str | None#

The name of the OS to use.

property process_ID#

The process ID from direct execution, if known.

property resources#

The common resources that this jobscript requires.

property scheduler#

The scheduler that submissions go to from this jobscript.

property scheduler_job_ID#

The job ID from the scheduler, if known.

property scheduler_js_ref#

The reference to the submitted job for the jobscript.

property scheduler_name: str | None#

The name of the scheduler to use.

property scheduler_ref#

The generalised scheduler reference descriptor.

property shell#

The shell for composing submission scripts.

property shell_name: str | None#

The name of the shell to use.

property start_time#

The first known start time of any EAR in this jobscript.

property submission#

The parent submission.

submit(scheduler_refs, print_stdout=False)#

Submit the jobscript to the scheduler.

Parameters:
  • scheduler_refs (Dict[int, (str, bool)]) –

  • print_stdout (Optional[bool]) –

Return type:

str

property submit_cmdline#

The command line used to do the commit, if known.

property submit_hostname#

Where the jobscript was submitted, if known.

property submit_machine#

Description of what the jobscript was submitted to, if known.

property submit_time#

When the jobscript was submitted, if known.

property task_actions#

The IDs of actions of each task in this jobscript.

property task_elements#

The IDs of elements of each task in this jobscript.

property task_insert_IDs#

The insertion IDs of tasks in this jobscript.

property task_loop_idx#

The description of where various task loops are.

to_dict()#

Serialize this object as a dictionary.

to_json_like(dct=None, shared_data=None, exclude=None, path=None)#

Serialize this object as an object structure that can be trivially converted to JSON. Note that YAML can also be produced from the result of this method; it just requires a different final serialization step.

property version_info#

Version information about the execution environment (OS, etc).

property workflow#

The workflow this is all on behalf of.

property workflow_app_alias#

Alias for the workflow app in job scripts.

write_EAR_ID_file()#

Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.

write_element_run_dir_file(run_dirs)#

Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.

We assume a given task element’s actions all run in the same directory, but in general a jobscript “element” may cross task boundaries, so we need to provide the directory for each jobscript-element/jobscript-action combination.

Parameters:

run_dirs (List[List[Path]]) –

write_jobscript(os_name=None, shell_name=None, deps=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)#

Write the jobscript to its file.

Parameters:
  • os_name (str) –

  • shell_name (str) –

  • deps (Dict | None) –

  • os_args (Dict | None) –

  • shell_args (Dict | None) –

  • scheduler_name (str | None) –

  • scheduler_args (Dict | None) –