hpcflow.sdk.submission.shells.bash.Bash

class hpcflow.sdk.submission.shells.bash.Bash(executable=None, os_args=None)

Bases: Shell

Class to represent using bash on a POSIX OS to generate and submit a jobscript.

Methods

format_array	Format construction of a shell array.
format_array_get_item	Format retrieval of a shell array item at a specified index.
format_commands_file	Format the commands file.
format_env_var_get	Format retrieval of a shell environment variable.
format_save_parameter	Produce code to save a parameter's value into the workflow persistent store.
format_source_functions_file	Format sourcing (i.e.
format_stream_assignment	Produce code to assign the output of the command to a shell variable.
get_command_file_launch_command	Get the command for launching the commands file for a given run.
get_direct_submit_command	Get the command for submitting a non-scheduled jobscript.
get_version_info	Get bash version information.
get_wait_command	Get the command to wait for a workflow.
prepare_JS_path	Prepare the jobscript path for use.
prepare_element_run_dirs	Prepare the element run directory names for use.
process_JS_header_args	Process the application invocation key in the jobscript header arguments.
process_app_invoc_executable	Perform any post-processing of an application invocation command name.

Attributes

os_args	Arguments to pass to the shell.
DEFAULT_EXE	Default for executable name.
JS_ACT_MULTI	Template for the action-run processing loop in a jobscript.
JS_ACT_SINGLE	Template for the single-action-run execution in a jobscript.
JS_BLOCK_HEADER	Template for a jobscript-block header.
JS_BLOCK_LOOP	Template for the jobscript block loop in a jobscript.
JS_DIRECT_HEADER	Template for the jobscript header when directly executed.
JS_ELEMENT_MULTI_ARRAY	Template for the array handling code in a jobscript.
JS_ELEMENT_MULTI_LOOP	Template for the element processing loop in a jobscript.
JS_ELEMENT_SINGLE	Template for single-element execution.
JS_ENV_SETUP_INDENT	Indent for environment setup.
JS_EXT	File extension for jobscripts.
JS_FOOTER	Template for the jobscript footer.
JS_FUNCS	Template for the jobscript functions file.
JS_HEADER	Template for the common part of the jobscript header.
JS_INDENT	Basic indent.
JS_MAIN	Template for setting up environment variables and running one or more action-runs.
JS_RUN	Template for setting up run environment variables and executing the run.
JS_RUN_CMD	Template for the run execution command.
JS_RUN_CMD_COMBINED	Template for the execution command for multiple combined runs.
JS_RUN_LOG_PATH_DISABLE	Template for disabling writing of the app log.
JS_RUN_LOG_PATH_ENABLE	Template for enabling writing of the app log.
JS_SCHEDULER_HEADER	Template for the jobscript header when scheduled.
JS_SHEBANG	Template for the jobscript shebang line.
executable	The executable to use plus any mandatory arguments.
linux_release_file	The name of the file describing the Linux version.
shebang_executable	The executable to use in a shebang line.

Parameters:

• executable (str | None) –

• os_args (dict[str, str] | None) –

DEFAULT_EXE: ClassVar[str] = '/bin/bash'

Default for executable name.

JS_ACT_MULTI: ClassVar[str] = 'for ((block_act_idx=0;block_act_idx<{num_actions};block_act_idx++))\ndo      \n{run_block}\ndone  \n'

Template for the action-run processing loop in a jobscript.

JS_ACT_SINGLE: ClassVar[str] = 'block_act_idx=0        \n{run_block}\n'

Template for the single-action-run execution in a jobscript.

JS_BLOCK_HEADER: ClassVar[str] = 'block_idx=0\nexport {app_caps}_BLOCK_IDX=0\n'

Template for a jobscript-block header.

JS_BLOCK_LOOP: ClassVar[str] = 'num_elements={num_elements}\nnum_actions={num_actions}\nblock_start_elem_idx=0\nfor ((block_idx=0;block_idx<{num_blocks};block_idx++))\ndo\n    export {app_caps}_BLOCK_IDX=$block_idx\n{element_loop}\n    block_start_elem_idx=$(($block_start_elem_idx + ${{num_elements[$block_idx]}}))\ndone\n'

Template for the jobscript block loop in a jobscript.

JS_DIRECT_HEADER: ClassVar[str] = '{shebang}\n{header}\n{wait_command}\n'

Template for the jobscript header when directly executed.

JS_ELEMENT_MULTI_ARRAY: ClassVar[str] = 'JS_elem_idx=$(({scheduler_array_item_var} - 1))\n{main}\n'

Template for the array handling code in a jobscript.

JS_ELEMENT_MULTI_LOOP: ClassVar[str] = 'for ((JS_elem_idx={block_start_elem_idx};JS_elem_idx<$(({block_start_elem_idx} + {num_elements}));JS_elem_idx++))\ndo\n{main}\ndone\n'

Template for the element processing loop in a jobscript.

JS_ELEMENT_SINGLE: ClassVar[str] = 'JS_elem_idx={block_start_elem_idx}\n{main}\n'

Template for single-element execution.

JS_ENV_SETUP_INDENT: ClassVar[str] = '    '

Indent for environment setup.

JS_EXT: ClassVar[str] = '.sh'

File extension for jobscripts.

JS_FOOTER: ClassVar[str] = 'cd $WK_PATH\n'

Template for the jobscript footer.

JS_FUNCS: ClassVar[str] = '{workflow_app_alias} () {{\n(\n{env_setup}{app_invoc}\\\n        --with-config log_file_path "${app_caps}_LOG_PATH"\\\n        --config-dir "{config_dir}"\\\n        --config-key "{config_invoc_key}"\\\n        "$@"\n)\n}}\n'

Template for the jobscript functions file.

JS_HEADER: ClassVar[str] = 'WK_PATH=`pwd`\nWK_PATH_ARG="$WK_PATH"\nSUB_IDX={sub_idx}\nJS_IDX={js_idx}\nAPP_CAPS={app_caps}\n\nSUB_DIR="$WK_PATH/artifacts/submissions/${{SUB_IDX}}"\nJS_FUNCS_PATH="$SUB_DIR/{jobscript_functions_dir}/{jobscript_functions_name}"\n. "$JS_FUNCS_PATH"        \n\nEAR_ID_FILE="$WK_PATH/artifacts/submissions/${{SUB_IDX}}/{run_IDs_file_dir}/{run_IDs_file_name}"\nSUB_TMP_DIR="$SUB_DIR/{tmp_dir_name}"\nSUB_LOG_DIR="$SUB_DIR/{log_dir_name}"\nSUB_STD_DIR="$SUB_DIR/{app_std_dir_name}"\nSUB_SCRIPTS_DIR="$SUB_DIR/{scripts_dir_name}"\n\nexport {app_caps}_WK_PATH=$WK_PATH\nexport {app_caps}_WK_PATH_ARG=$WK_PATH_ARG\nexport {app_caps}_SUB_IDX={sub_idx}\nexport {app_caps}_SUB_SCRIPTS_DIR=$SUB_SCRIPTS_DIR\nexport {app_caps}_SUB_TMP_DIR=$SUB_TMP_DIR\nexport {app_caps}_SUB_LOG_DIR=$SUB_LOG_DIR\nexport {app_caps}_SUB_STD_DIR=$SUB_STD_DIR\nexport {app_caps}_LOG_PATH="$SUB_LOG_DIR/js_${{JS_IDX}}.log"\nexport {app_caps}_JS_FUNCS_PATH=$JS_FUNCS_PATH\nexport {app_caps}_JS_IDX={js_idx}\nexport {app_caps}_RUN_ID_FILE=$EAR_ID_FILE\n'

Template for the common part of the jobscript header.

JS_INDENT: ClassVar[str] = '  '

Basic indent.

JS_MAIN: ClassVar[str] = 'block_elem_idx=$(( $JS_elem_idx - {block_start_elem_idx} ))\nelem_EAR_IDs=`sed "$((${{JS_elem_idx}} + 1))q;d" "$EAR_ID_FILE"`\nexport {app_caps}_JS_ELEM_IDX=$JS_elem_idx\nexport {app_caps}_BLOCK_ELEM_IDX=$block_elem_idx\n\n{action}\n'

Template for setting up environment variables and running one or more action-runs.

JS_RUN: ClassVar[str] = 'EAR_ID="$(cut -d\'{EAR_files_delimiter}\' -f $(($block_act_idx + 1)) <<< $elem_EAR_IDs)"\nif [ "$EAR_ID" = "-1" ]; then\n    continue\nfi\n\nexport {app_caps}_RUN_ID=$EAR_ID\nexport {app_caps}_RUN_LOG_PATH={run_log_enable_disable}\nexport {app_caps}_LOG_PATH="${app_caps}_RUN_LOG_PATH"\nexport {app_caps}_RUN_STD_PATH="$SUB_STD_DIR/${app_caps}_RUN_ID.txt"\nexport {app_caps}_BLOCK_ACT_IDX=$block_act_idx\n\ncd "$SUB_TMP_DIR"\n\n{run_cmd}\n'

Template for setting up run environment variables and executing the run.

JS_RUN_CMD: ClassVar[str] = '{workflow_app_alias} internal workflow "$WK_PATH_ARG" execute-run $SUB_IDX $JS_IDX $block_idx $block_act_idx $EAR_ID\n'

Template for the run execution command.

JS_RUN_CMD_COMBINED: ClassVar[str] = '{workflow_app_alias} internal workflow "$WK_PATH_ARG" execute-combined-runs $SUB_IDX $JS_IDX\n'

Template for the execution command for multiple combined runs.

JS_RUN_LOG_PATH_DISABLE: ClassVar[str] = '" "'

Template for disabling writing of the app log.

JS_RUN_LOG_PATH_ENABLE: ClassVar[str] = '"$SUB_LOG_DIR/{run_log_file_name}"'

Template for enabling writing of the app log.

JS_SCHEDULER_HEADER: ClassVar[str] = '{shebang}\n\n{scheduler_options}\n{header}\n'

Template for the jobscript header when scheduled.

JS_SHEBANG: ClassVar[str] = '#!{shebang_executable} {shebang_args}'

Template for the jobscript shebang line.

property executable: list[str]

The executable to use plus any mandatory arguments.

format_array(lst)

Format construction of a shell array.

Parameters:

lst (list) –

Return type:

str

format_array_get_item(arr_name, index)

Format retrieval of a shell array item at a specified index.

Parameters:

• arr_name (str) –

• index (int | str) –

Return type:

str

format_commands_file(app_name, commands)

Format the commands file.

Parameters:

• app_name (str) –

• commands (str) –

Return type:

str

format_env_var_get(var)

Format retrieval of a shell environment variable.

Parameters:

var (str) –

Return type:

str

format_save_parameter(workflow_app_alias, param_name, shell_var_name, cmd_idx, stderr, app_name)

Produce code to save a parameter’s value into the workflow persistent store.

Parameters:

• workflow_app_alias (str) –

• param_name (str) –

• shell_var_name (str) –

• cmd_idx (int) –

• stderr (bool) –

• app_name (str) –

format_source_functions_file(app_name, commands)

Format sourcing (i.e. invocation) of the jobscript functions file.

Parameters:

• app_name (str) –

• commands (str) –

Return type:

str

format_stream_assignment(shell_var_name, command)

Produce code to assign the output of the command to a shell variable.

Parameters:

• shell_var_name (str) –

• command (str) –

Return type:

str

get_command_file_launch_command(cmd_file_path)

Get the command for launching the commands file for a given run.

Parameters:

cmd_file_path (str) –

Return type:

list[str]

get_direct_submit_command(js_path)

Get the command for submitting a non-scheduled jobscript.

Parameters:

js_path (str) –

Return type:

list[str]

get_version_info(exclude_os=False)

Get bash version information.

Parameters:

exclude_os (bool) – If True, exclude operating system information.

Return type:

VersionInfo

get_wait_command(workflow_app_alias, sub_idx, deps)

Get the command to wait for a workflow.

Parameters:

• workflow_app_alias (str) –

• sub_idx (int) –

• deps (Mapping[int, Any]) –

property linux_release_file: str

The name of the file describing the Linux version.

os_args

Arguments to pass to the shell.

prepare_JS_path(js_path)

Prepare the jobscript path for use.

Parameters:

js_path (Path) –

Return type:

str

prepare_element_run_dirs(run_dirs)

Prepare the element run directory names for use.

Parameters:

run_dirs (list[list[Path]]) –

Return type:

list[list[str]]

process_JS_header_args(header_args)

Process the application invocation key in the jobscript header arguments.

Parameters:

header_args (JobscriptHeaderArgs) –

Return type:

JobscriptHeaderArgs

static process_app_invoc_executable(app_invoc_exe)

Perform any post-processing of an application invocation command name.

Parameters:

app_invoc_exe (str) –

Return type:

str

property shebang_executable: list[str]

The executable to use in a shebang line.