hpcflow.app.Jobscript#
- class hpcflow.app.Jobscript(task_insert_IDs, task_actions, task_elements, EAR_ID, resources, task_loop_idx, dependencies, submit_time=None, submit_hostname=None, submit_machine=None, submit_cmdline=None, scheduler_job_ID=None, process_ID=None, version_info=None, os_name=None, shell_name=None, scheduler_name=None, running=None)#
Bases:
Jobscript
A group of actions that are submitted together to be executed by the underlying job management system as a single unit.
- Parameters:
task_actions (list[tuple]) – The actions of the tasks.
task insert ID, action_idx, index into task_loop_idx
for eachJS_ACTION_IDX
task_elements (dict[int, list[int]]) – The elements of the tasks. Maps
JS_ELEMENT_IDX
to list ofTASK_ELEMENT_IDX
for eachTASK_INSERT_ID
EAR_ID (NDArray) – Element action run information.
resources (ElementResources) – Resources to use
task_loop_idx (list[dict]) – Description of what loops are in play.
dependencies (dict[int, dict]) – Description of dependencies.
submit_time (datetime) – When the jobscript was submitted, if known.
submit_hostname (str) – Where the jobscript was submitted, if known.
submit_machine (str) – Description of what the jobscript was submitted to, if known.
submit_cmdline (str) – The command line used to do the commit, if known.
scheduler_job_ID (str) – The job ID from the scheduler, if known.
process_ID (int) – The process ID of the subprocess, if known.
version_info (tuple[str, ...]) – Version info about the target system.
os_name (str) – The name of the OS.
shell_name (str) – The name of the shell.
scheduler_name (str) – The scheduler used.
running (bool) – Whether the jobscript is currently running.
Methods
Cancel this jobscript.
Prepare the jobscript file string.
Make an instance of this class from JSON (or YAML) data.
If this jobscript is active on this machine, return the state information from the scheduler.
Get the name of a file containing commands for a particular jobscript action.
Get an array of task loop indices.
Create the directories that will hold artifacts associated with this jobscript.
Submit the jobscript to the scheduler.
Serialize this object as a dictionary.
Serialize this object as an object structure that can be trivially converted to JSON.
Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.
Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.
Write the jobscript to its file.
Attributes
The array of EAR IDs.
The name of a file containing EAR IDs.
The path to the file containing EAR IDs for this jobscript.
The IDs of all EARs in this jobscript.
Description of EAR information for this jobscript.
The dependency descriptor.
File for direct execution stderr.
The path to the file containing the stderr from directly executed commands for this jobscript.
File for direct execution stdout.
The path to the file containing the stdout from directly executed commands for this jobscript.
File for holding the direct execution PID.
The path to the file containing PIDs for directly executed commands for this jobscript.
The name of a file containing run directory names.
The path to the file containing run directory names for this jobscript.
The last known end time of any EAR in this jobscript.
The index of this jobscript within its parent
Submission
.Whether to generate an array job.
Whether this jobscript has been submitted.
The name of the jobscript file.
The path to the file containing the jobscript file.
The number of actions in this jobscript.
The number of elements in this jobscript.
The name of the OS to use.
The process ID from direct execution, if known.
The common resources that this jobscript requires.
The scheduler that submissions go to from this jobscript.
The job ID from the scheduler, if known.
The reference to the submitted job for the jobscript.
The name of the scheduler to use.
The generalised scheduler reference descriptor.
The shell for composing submission scripts.
The name of the shell to use.
The first known start time of any EAR in this jobscript.
The parent submission.
The command line used to do the commit, if known.
Where the jobscript was submitted, if known.
Description of what the jobscript was submitted to, if known.
When the jobscript was submitted, if known.
The IDs of actions of each task in this jobscript.
The IDs of elements of each task in this jobscript.
The insertion IDs of tasks in this jobscript.
The description of where various task loops are.
Version information about the execution environment (OS, etc).
The workflow this is all on behalf of.
Alias for the workflow app in job scripts.
- property EAR_ID#
The array of EAR IDs.
- property EAR_ID_file_name#
The name of a file containing EAR IDs.
- property EAR_ID_file_path#
The path to the file containing EAR IDs for this jobscript.
- app = BaseApp(name='hpcFlow', version='0.2.0a180')#
- cancel()#
Cancel this jobscript.
- compose_jobscript(deps=None, os_name=None, shell_name=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)#
Prepare the jobscript file string.
- property dependencies#
The dependency descriptor.
- property direct_stderr_file_name#
File for direct execution stderr.
- property direct_stderr_path#
The path to the file containing the stderr from directly executed commands for this jobscript.
- property direct_stdout_file_name#
File for direct execution stdout.
- property direct_stdout_path#
The path to the file containing the stdout from directly executed commands for this jobscript.
- property direct_win_pid_file_name#
File for holding the direct execution PID.
- property direct_win_pid_file_path#
The path to the file containing PIDs for directly executed commands for this jobscript. Windows only.
- property element_run_dir_file_name#
The name of a file containing run directory names.
- property element_run_dir_file_path#
The path to the file containing run directory names for this jobscript.
- property end_time#
The last known end time of any EAR in this jobscript.
- classmethod from_json_like(json_like, shared_data=None)#
Make an instance of this class from JSON (or YAML) data.
- Parameters:
json_like – The data to deserialise.
shared_data – Shared context data.
- Return type:
The deserialised object.
- get_active_states(as_json=False)#
If this jobscript is active on this machine, return the state information from the scheduler.
- Parameters:
as_json (bool) –
- Return type:
- get_commands_file_name(js_action_idx, shell=None)#
Get the name of a file containing commands for a particular jobscript action.
- get_task_loop_idx_array()#
Get an array of task loop indices.
- property index#
The index of this jobscript within its parent
Submission
.
- property is_array#
Whether to generate an array job.
- property is_submitted#
Whether this jobscript has been submitted.
- property jobscript_name#
The name of the jobscript file.
- property jobscript_path#
The path to the file containing the jobscript file.
- make_artifact_dirs()#
Create the directories that will hold artifacts associated with this jobscript.
- property num_actions#
The number of actions in this jobscript.
- property num_elements#
The number of elements in this jobscript.
- property process_ID#
The process ID from direct execution, if known.
- property resources#
The common resources that this jobscript requires.
- property scheduler#
The scheduler that submissions go to from this jobscript.
- property scheduler_job_ID#
The job ID from the scheduler, if known.
- property scheduler_js_ref#
The reference to the submitted job for the jobscript.
- property scheduler_ref#
The generalised scheduler reference descriptor.
- property shell#
The shell for composing submission scripts.
- property start_time#
The first known start time of any EAR in this jobscript.
- property submission#
The parent submission.
- submit(scheduler_refs, print_stdout=False)#
Submit the jobscript to the scheduler.
- property submit_cmdline#
The command line used to do the commit, if known.
- property submit_hostname#
Where the jobscript was submitted, if known.
- property submit_machine#
Description of what the jobscript was submitted to, if known.
- property submit_time#
When the jobscript was submitted, if known.
- property task_actions#
The IDs of actions of each task in this jobscript.
- property task_elements#
The IDs of elements of each task in this jobscript.
- property task_insert_IDs#
The insertion IDs of tasks in this jobscript.
- property task_loop_idx#
The description of where various task loops are.
- to_dict()#
Serialize this object as a dictionary.
- to_json_like(dct=None, shared_data=None, exclude=None, path=None)#
Serialize this object as an object structure that can be trivially converted to JSON. Note that YAML can also be produced from the result of this method; it just requires a different final serialization step.
- property version_info#
Version information about the execution environment (OS, etc).
- property workflow#
The workflow this is all on behalf of.
- property workflow_app_alias#
Alias for the workflow app in job scripts.
- write_EAR_ID_file()#
Write a text file with num_elements lines and num_actions delimited tokens per line, representing whether a given EAR must be executed.
- write_element_run_dir_file(run_dirs)#
Write a text file with num_elements lines and num_actions delimited tokens per line, representing the working directory for each EAR.
We assume a given task element’s actions all run in the same directory, but in general a jobscript “element” may cross task boundaries, so we need to provide the directory for each jobscript-element/jobscript-action combination.
- write_jobscript(os_name=None, shell_name=None, deps=None, os_args=None, shell_args=None, scheduler_name=None, scheduler_args=None)#
Write the jobscript to its file.