hpcflow.sdk.app.App#

class hpcflow.sdk.app.App(*args, **kwargs)#

Bases: BaseApp

Class from which to instantiate downstream app objects (e.g. MatFlow).

Methods

`add_env`	Generate and save a new environment.
`cache_all_data_files`	Cache all data files, and return their paths.
`cache_all_programs`	Cache all built-in programs, and return their paths.
`cache_data_file`	Cache a data file and return its cached path.
`cache_file`	Cache and retrieve the path to a data file (demo data or built-in program), according to the config or app attributes data_dir or program_dir (which are fsspec-compatible URLs), and a file key that represents the relative path of the file within the respective data directory.
`cache_program`	Cache a built-in program and return its cached path.
`clear_data_cache_dir`	Delete the contents of the demonstration data files cache directory.
`clear_known_submissions_file`	Clear the known-submissions file of all submissions.
`clear_program_cache_dir`	Delete the contents of the built-in program cache directory.
`clear_user_cache_dir`	Delete the contents of the cache directory.
`clear_user_cache_hostname_dir`	Delete the contents of the hostname-scoped cache directory.
`clear_user_runtime_dir`	Delete the contents of the user runtime directory.
`copy_cacheable_file`	Copy a builtin demo data or program file to the specified location, or the current directory.
`copy_data_file`	Copy a demonstration data file to the specified location, or the current directory.
`copy_demo_workflow`	Copy a builtin demo workflow to the specified location.
`copy_program`	Copy a built-in program to the specified location, or the current directory.
`detect_env_manager`
`get_OS_supported_schedulers`	Retrieve a list of schedulers that are supported in principle by this operating system.
`get_config_path`	Return the full path to the config file, without loading the config.
`get_data_file_path`	Retrieve the local path to a cached data or builtin program file.
`get_data_files_manifest`	Get the demonstration data files manifest.
`get_data_manifest`	Get a dict whose keys are example data file or program names and whose values are the source files if the source file required unzipping or None otherwise.
`get_demo_data_file_path`	Get the full path to an example data file in the app cache directory.
`get_demo_workflow_template_file`	Context manager to get a (temporary) file path to an included demo workflow template.
`get_env_setup`	Generate shell commands to activate the current conda-like or Python venv environment.
`get_info`	Get miscellaneous runtime system information.
`get_parameter_task_schema_map`	Get a dict mapping parameter types to task schemas that input/output each parameter.
`get_program_path`	Get the full path to a built-in program in the app cache directory.
`get_programs_manifest`	Get the built-in programs manifest.
`get_scheduler`	Get an arbitrary scheduler object.
`get_shell_hook`	Get the shell hook string that, when executed, will initialise micromamba, mamba, or conda using the executable by which an environment is currently activated.
`list_data_files`	List available demonstration data files.
`list_demo_workflows`	Return a list of demo workflow templates included in the app.
`list_programs`	List available built-in program files.
`load_builtin_template_component_data`	Load the template component data built into the package.
`load_config`	Load the user's configuration.
`load_demo_workflow`	Load a WorkflowTemplate object from a builtin demo template file.
`load_template_components`	Load all template component data, warning by default if already loaded.
`perm_error_retry`	Return a decorator for retrying functions on permission and OS errors that might be associated with cloud-storage desktop sync.
`read_known_submissions_file`	Retrieve existing workflows that might be running.
`redirect_std_to_file`
`reload_config`	Reload the configuration.
`reload_template_components`	Reload all template component data, warning by default if not already loaded.
`remove_env`	Remove an environment identified by its name and specifiers, or remove all environments with a particular setup label and specifiers.
`reset_config`	Reset the config file to defaults, and reload the config.
`save_env`	Save an environment to the environment definitions file.
`show_demo_workflow`	Print the contents of a builtin demo workflow template file.
`template_components_from_json_like`	Get template components from a (simply parsed) JSON document.
`unload_config`	Discard any loaded configuration.
`update_known_subs_file`	Update submission records in the known-submission file.

Attributes

`API_logger`	The logger for API messages.
`Action`	The `Action` class.
`ActionEnvironment`	The `ActionEnvironment` class.
`ActionRule`	The `ActionRule` class.
`ActionScope`	The `ActionScope` class.
`ActionScopeType`	The `ActionScopeType` class.
`CLI_logger`	The logger for CLI messages.
`Command`	The `Command` class.
`CommandFilesList`	The `CommandFilesList` class.
`DirectPosix`	The `DirectPosix` class.
`DirectWindows`	The `DirectWindows` class.
`Element`	The `Element` class.
`ElementAction`	The `ElementAction` class.
`ElementActionRun`	The `ElementActionRun` class.
`ElementFilter`	The `ElementFilter` class.
`ElementGroup`	The `ElementGroup` class.
`ElementInputFiles`	The `ElementInputFiles` class.
`ElementInputs`	The `ElementInputs` class.
`ElementIteration`	The `ElementIteration` class.
`ElementOutputFiles`	The `ElementOutputFiles` class.
`ElementOutputs`	The `ElementOutputs` class.
`ElementParameter`	The `ElementParameter` class.
`ElementPropagation`	The `ElementPropagation` class.
`ElementResources`	The `ElementResources` class.
`ElementSet`	The `ElementSet` class.
`Environment`	The `Environment` class.
`EnvironmentsList`	The `EnvironmentsList` class.
`Executable`	The `Executable` class.
`ExecutableInstance`	The `ExecutableInstance` class.
`ExecutablesList`	The `ExecutablesList` class.
`FileNameExt`	The `FileNameExt` class.
`FileNameSpec`	The `FileNameSpec` class.
`FileNameStem`	The `FileNameStem` class.
`FileSpec`	The `FileSpec` class.
`GroupList`	The `GroupList` class.
`InputFileGenerator`	The `InputFileGenerator` class.
`InputSource`	The `InputSource` class.
`InputSourceType`	The `InputSourceType` class.
`InputValue`	The `InputValue` class.
`Jobscript`	The `Jobscript` class.
`Loop`	The `Loop` class.
`MultiPathSequence`	The `MultiPathSequence` class.
`NumCores`	The `NumCores` class.
`OutputFileParser`	The `OutputFileParser` class.
`Parameter`	The `Parameter` class.
`ParameterValue`	The `ParameterValue` class.
`Parameters`	The `Parameters` class.
`ParametersList`	The `ParametersList` class.
`QueuedScheduler`	The `QueuedScheduler` class.
`ResourceList`	The `ResourceList` class.
`ResourceSpec`	The `ResourceSpec` class.
`Rule`	The `Rule` class.
`RunDirAppFiles`	The `RunDirAppFiles` class.
`SGEPosix`	The `SGEPosix` class.
`SchemaInput`	The `SchemaInput` class.
`SchemaOutput`	The `SchemaOutput` class.
`SchemaParameter`	The `SchemaParameter` class.
`SlurmPosix`	The `SlurmPosix` class.
`Submission`	The `Submission` class.
`Task`	The `Task` class.
`TaskInputParameters`	The `TaskInputParameters` class.
`TaskList`	The `TaskList` class.
`TaskObjective`	The `TaskObjective` class.
`TaskOutputParameters`	The `TaskOutputParameters` class.
`TaskSchema`	The `TaskSchema` class.
`TaskSchemasList`	The `TaskSchemasList` class.
`TaskSourceType`	The `TaskSourceType` class.
`TaskTemplateList`	The `TaskTemplateList` class.
`ValueSequence`	The `ValueSequence` class.
`Workflow`	The `Workflow` class.
`WorkflowLoop`	The `WorkflowLoop` class.
`WorkflowLoopList`	The `WorkflowLoopList` class.
`WorkflowTask`	The `WorkflowTask` class.
`WorkflowTaskList`	The `WorkflowTaskList` class.
`WorkflowTemplate`	The `WorkflowTemplate` class.
`cancel`	Cancel the execution of a workflow submission.
`command_files`	The known template command files.
`config`	The configuration.
`config_logger`	The logger for configuration messages.
`data_cache_dir`	A directory for demonstration data caching.
`envs`	The known template execution environments.
`get_OS_info`	Get information about the operating system.
`get_known_submissions`	Retrieve information about active and recently inactive finished workflows.
`get_shell_info`	Get information about a given shell and the operating system.
`is_config_loaded`	Whether the configuration is loaded.
`is_template_components_loaded`	Whether any template component (e.g. parameters) has been loaded.
`jinja_templates`	The known Jinja template files.
`known_subs_file_path`	The path to the file describing known submissions.
`log`	The application log.
`logger`	The main underlying logger.
`make_and_submit_demo_workflow`	Generate and submit a new demo workflow from a file or string containing a workflow template parametrisation.
`make_and_submit_workflow`	Generate and submit a new workflow from a file or string containing a workflow template parametrisation.
`make_demo_workflow`	Generate a new workflow from a builtin demo workflow template.
`make_workflow`	Generate a new workflow from a file or string containing a workflow template parametrisation.
`parameters`	The known template parameters.
`persistence_logger`	The logger for persistence engine messages.
`program_cache_dir`	A directory for built-in program caching.
`programs`	The known programs.
`run_hpcflow_tests`	Run hpcflow test suite.
`run_tests`	Run the test suite.
`run_time_info`	Information about the runtime.
`runtime_info_logger`	The logger for runtime messages.
`scheduler_lookup`	The scheduler mapping.
`scripts`	The known template scripts.
`show`	Show information about running workflows.
`show_legend`	Output a legend for the jobscript-element and EAR states that are displayed by the show command.
`submission_logger`	The logger for job submission messages.
`submit_workflow`	Submit an existing workflow.
`task_schemas`	The known template task schemas.
`template_components`	The template component data.
`timeit`	Whether the timing analysis system is active.
`user_cache_dir`	The user's cache directory.
`user_cache_hostname_dir`	The hostname-scoped app cache directory.
`user_data_dir`	The user's data directory.
`user_data_hostname_dir`	The directory for holding user data.
`user_runtime_dir`	The user's temporary runtime directory.
`name`	The name of the application.
`package_name`	Name of package.
`version`	The version of the application.
`module`	The module name in which the app object is defined.
`description`	Description of the application.
`gh_org`	Name of Github organisation responsible for the application.
`gh_repo`	Github repository containing the application source.
`config_options`	Configuration options.
`pytest_args`	Arguments for pytest.
`scripts_dir`	Directory for scripts.
`jinja_templates_dir`	Directory for Jinja templates.
`workflows_dir`	Directory for workflows.
`data_manifest_dir`	Directory for JSON manifest files.
`data_dir`	Directory for demonstration data.
`program_dir`	Directory for built-in program data.
`docs_import_conv`	The convention for the app alias used in import statements in the documentation.
`docs_url`	URL to documentation.
`encoders`	Callable that returns additional parameter encoders.
`decoders`	Callable that returns additional parameter decoders.
`cli`	Command line interface subsystem.
`env_setup_CLI`	Environment setup CLI, to which downstream apps can add their own commands

Return type:: T

property API_logger: Logger#: The logger for API messages.

property CLI_logger: Logger#: The logger for CLI messages.

add_env(name, setup=None, executables=None, use_current=False, **kwargs)#

Generate and save a new environment.

Parameters:

name (str)
setup (str | Sequence[str] | None)
executables (list[_Executable] | None)
use_current (bool)

cache_all_data_files(exist_ok=True)#

Cache all data files, and return their paths.

Parameters:: exist_ok (bool)
Return type:: list[Path]

cache_all_programs(exist_ok=True)#

Cache all built-in programs, and return their paths.

Parameters:: exist_ok (bool)
Return type:: list[Path]

cache_data_file(file_key, exist_ok=True)#

Cache a data file and return its cached path.

Parameters:

file_key (str)
exist_ok (bool)

Return type:

Path

cache_file(data_type, file_key, manifest=None, exist_ok=True)#

Cache and retrieve the path to a data file (demo data or built-in program), according to the config or app attributes data_dir or program_dir (which are fsspec-compatible URLs), and a file key that represents the relative path of the file within the respective data directory.

For remote file systems, this will involve downloading the file to a temporary directory. For files that live within a zip file, as specified in the manifest, this method will first unzip the files before moving to the cache directory.

Parameters:

data_type (Literal['data', 'program'])
file_key (str)
manifest (dict[str, dict[str, str]] | None)
exist_ok (bool)

Return type:

Path

cache_program(file_key, exist_ok=True)#

Cache a built-in program and return its cached path.

Parameters:

file_key (str)
exist_ok (bool)

Return type:

Path

property cancel: _Cancel#

Cancel the execution of a workflow submission.

Parameters:

workflow_ref (int | str | Path) – Which workflow to cancel, by ID or path.
ref_is_path (str) – One of “id”, “path” or “assume-id” (the default)
status (bool) – Whether to show a live status during cancel.

clear_data_cache_dir()#

Delete the contents of the demonstration data files cache directory.

Return type:: None

clear_known_submissions_file()#

Clear the known-submissions file of all submissions. This shouldn’t be needed normally.

Return type:: None

clear_program_cache_dir()#

Delete the contents of the built-in program cache directory.

Return type:: None

clear_user_cache_dir()#

Delete the contents of the cache directory.

Return type:: None

clear_user_cache_hostname_dir()#

Delete the contents of the hostname-scoped cache directory.

Return type:: None

clear_user_runtime_dir()#

Delete the contents of the user runtime directory.

Return type:: None

cli#: Command line interface subsystem.

property command_files: _CommandFilesList#: The known template command files.

property config: Config#: The configuration.

property config_logger: Logger#: The logger for configuration messages.

config_options#: Configuration options.

copy_cacheable_file(data_type, file_key, dst=None)#

Copy a builtin demo data or program file to the specified location, or the current directory.

Parameters:

data_type (Literal['data', 'program']) – Type of the data: “data” or “program”.
file_key (str) – The file key to copy.
dst (PathLike | None) – File path to copy the file to, or, if an existing directory, the parent directory to copy the source file or directory to. If not specified, the current working directory will be used. Note that if the source is a directory, this must be specified as the parent directory to copy into.

Returns:

The new path created at or in the destination.

Return type:

new_path

Raises:

FileExistsError – If the source is a directory, and the destination directory already contains a directory of the same name, FileExistsError will be raised. However, if the source is a file, and the destination directory already contains a file of the same name, it will be over-written.

copy_data_file(file_key, dst=None)#

Copy a demonstration data file to the specified location, or the current directory.

Parameters:

file_key (str) – The file key to copy.
dst (PathLike | None) – File path to copy the file to, or, if an existing directory, the parent directory to copy the source file or directory to. If not specified, the current working directory will be used. Note that if the source is a directory, this must be specified as the parent directory to copy into.

Returns:

The new path created at or in the destination.

Return type:

new_path

Raises:

FileExistsError – If the source is a directory, and the destination directory already contains a directory of the same name, FileExistsError will be raised. However, if the source is a file, and the destination directory already contains a file of the same name, it will be over-written.

copy_demo_workflow(name, dst=None, doc=True)#

Copy a builtin demo workflow to the specified location.

Parameters:

name (str) – The name of the demo workflow to copy
dst (PathLike | None) – Directory or full file path to copy the demo workflow to. If not specified, the current working directory will be used.
doc (bool) – If False, the copied workflow template file will not include the doc attribute (if originally present).

Return type:

str

copy_program(file_key, dst=None)#

Copy a built-in program to the specified location, or the current directory.

Parameters:

file_key (str) – The file key to copy.
dst (PathLike | None) – File path to copy the file to, or, if an existing directory, the parent directory to copy the source file or directory to. If not specified, the current working directory will be used. Note that if the source is a directory, this must be specified as the parent directory to copy into.

Returns:

The new path created at or in the destination.

Return type:

new_path

Raises:

FileExistsError – If the source is a directory, and the destination directory already contains a directory of the same name, FileExistsError will be raised. However, if the source is a file, and the destination directory already contains a file of the same name, it will be over-written.

property data_cache_dir: Path#: A directory for demonstration data caching.

data_dir#: Directory for demonstration data.

data_manifest_dir#: Directory for JSON manifest files.

decoders#: Callable that returns additional parameter decoders.

description#: Description of the application.

static detect_env_manager()#

Return type:: EnvInfo

docs_import_conv#: The convention for the app alias used in import statements in the documentation.

docs_url#: URL to documentation.

encoders#: Callable that returns additional parameter encoders.

env_setup_CLI#: Environment setup CLI, to which downstream apps can add their own commands

property envs: _EnvironmentsList#: The known template execution environments.

property get_OS_info: Callable[[], Mapping[str, str]]#

Get information about the operating system.

Returns:: Key-value mapping containing system version information.
Return type:: dict[str, str]

get_OS_supported_schedulers()#

Retrieve a list of schedulers that are supported in principle by this operating system.

This does not necessarily mean all the returned schedulers are available on this system.

Return type:: Iterator[str]

get_config_path(config_dir=None)#

Return the full path to the config file, without loading the config.

Parameters:: config_dir (PathLike)
Return type:: Path

get_data_file_path(data_type, file_key)#

Retrieve the local path to a cached data or builtin program file.

Parameters:

data_type (Literal['data', 'program'])
file_key (str)

Return type:

Path

get_data_files_manifest()#

Get the demonstration data files manifest.

Return type:: dict[str, dict[str, str]]

get_data_manifest(data_type)#

Get a dict whose keys are example data file or program names and whose values are the source files if the source file required unzipping or None otherwise.

If the config items data_manifest_file/program_manifest_file is set, this is used as the manifest file path. Otherwise, the app attribute data_manifest_dir is used, and is expected to be a (fsspec-compatible) URL to a directory that contains data.json and programs.json manifest files.

Parameters:: data_type (Literal['data', 'program'])
Return type:: dict[str, dict[str, str]]

get_demo_data_file_path(file_name)#

Get the full path to an example data file in the app cache directory.

If the file does not already exist in the app cache directory, it will be added (and unzipped if required). The file may first be downloaded from a remote file system such as GitHub (see get_data_file_path for details).

Parameters:: file_name (str)
Return type:: Path

get_demo_workflow_template_file(name, doc=True, delete=True)#

Context manager to get a (temporary) file path to an included demo workflow template.

Parameters:

name (str) – Name of the builtin demo workflow template whose file path is to be retrieved.
doc (bool) – If False, the yielded path will be to a file without the doc attribute (if originally present).
delete (bool) – If True, remove the temporary file on exit.

Return type:

Iterator[Path]

get_env_setup(shell)#

Generate shell commands to activate the current conda-like or Python venv environment.

Parameters:: shell (str)
Return type:: list[str]

get_info()#

Get miscellaneous runtime system information.

Return type:: dict[str, Any]

property get_known_submissions: _GetKnownSubmissions#

Retrieve information about active and recently inactive finished workflows.

This method removes workflows from the known-submissions file that are found to be inactive on this machine (according to the scheduler/process ID).

Parameters:

max_recent (int) – Maximum number of inactive workflows to retrieve.
no_update (bool) – If True, do not update the known-submissions file to set submissions that are now inactive.
as_json (bool) – If True, only include JSON-compatible information. This will exclude the submission key, for instance.

Returns:

List of descriptions of known items.

Return type:

list[KnownSubmissionItem]

get_parameter_task_schema_map()#

Get a dict mapping parameter types to task schemas that input/output each parameter.

Return type:: dict[str, list[list[str]]]

get_program_path(file_name)#

Get the full path to a built-in program in the app cache directory.

If the file does not already exist in the app cache directory, it will be added (and unzipped if required). The file may first be downloaded from a remote file system such as GitHub (see get_data_file_path for details).

Parameters:: file_name (str)
Return type:: Path

get_programs_manifest()#

Get the built-in programs manifest.

Return type:: dict[str, dict[str, str]]

get_scheduler(scheduler_name, os_name, scheduler_args=None)#

Get an arbitrary scheduler object.

Parameters:

scheduler_name (str)
os_name (str)
scheduler_args (dict[str, Any] | None)

Return type:

Scheduler

get_shell_hook(shell, env_info)#

Get the shell hook string that, when executed, will initialise micromamba, mamba, or conda using the executable by which an environment is currently activated.

Parameters:

shell (str)
env_info (EnvInfo)

Return type:

str

property get_shell_info: Callable[[str, bool], VersionInfo]#

Get information about a given shell and the operating system.

Parameters:

shell_name (str) – One of the supported shell names.
exclude_os (bool) – If True, exclude operating system information.

Returns:

The shell version information descriptor.

Return type:

VersionInfo

gh_org#: Name of Github organisation responsible for the application.

gh_repo#: Github repository containing the application source.

property is_config_loaded: bool#: Whether the configuration is loaded.

property is_template_components_loaded: bool#: Whether any template component (e.g. parameters) has been loaded.

property jinja_templates: dict[str, Path]#: The known Jinja template files.

jinja_templates_dir#: Directory for Jinja templates.

property known_subs_file_path: Path#: The path to the file describing known submissions.

list_data_files()#

List available demonstration data files.

Return type:: tuple[str, …]

list_demo_workflows()#

Return a list of demo workflow templates included in the app.

Return type:: tuple[str, …]

list_programs()#

List available built-in program files.

Return type:: tuple[str, …]

classmethod load_builtin_template_component_data(package)#

Load the template component data built into the package. This is as opposed to the template components defined by users.

Parameters:: package (ModuleType | str)
Return type:: BasicTemplateComponents

load_config(config_dir=None, config_key=None, warn=True, **overrides)#

Load the user’s configuration.

Parameters:

config_dir (PathLike) – Directory containing the configuration, if not default.
config_key (str | None) – Key to the configuration within the config file.
warn (bool) – Whether to warn if a configuration is already loaded.

Return type:

None

load_demo_workflow(name, variables=None)#

Load a WorkflowTemplate object from a builtin demo template file.

Parameters:

name (str) – Name of the demo workflow to load.
variables (dict[str, str] | Literal[False] | None) – String variables to substitute in the demo workflow. Substitutions will be attempted if the file looks to contain variable references (like “<<var:name>>”). If set to False, no substitutions will occur, which may result in an invalid workflow template!

Return type:

_WorkflowTemplate

load_template_components(warn=True)#

Load all template component data, warning by default if already loaded.

Parameters:: warn (bool)
Return type:: None

property log: AppLog#: The application log.

property logger: Logger#: The main underlying logger.

property make_and_submit_demo_workflow: _MakeAndSubmitDemoWorkflow#

Generate and submit a new demo workflow from a file or string containing a workflow template parametrisation.

Parameters:

workflow_name (str) – Name of the demo workflow to make. Required.
template_format (str) – If specified, one of “json” or “yaml”. This forces parsing from a particular format.
path (str | Path) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, object]) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str]) – String variables to substitute in the demo workflow template file.
JS_parallelism (bool) – If True, allow multiple jobscripts to execute simultaneously. Raises if set to True but the store type does not support the jobscript_parallelism feature. If not set, jobscript parallelism will be used if the store type supports it.
wait (bool) – If True, this command will block until the workflow execution is complete.
add_to_known (bool) – If True, add the new submission to the known-submissions file, which is used by the show command to monitor current and recent submissions.
return_idx (bool) – If True, return a dict representing the jobscript indices submitted for each submission.
tasks (list[int]) – List of task indices to include in this submission. By default all tasks are included.
cancel (bool) – Immediately cancel the submission. Useful for testing and benchmarking.
status (bool) – If True, display a live status to track submission progress.

Returns:

Workflow – The created workflow.
dict[int, list[int]] – Mapping of submission handles. If requested by return_idx parameter.

property make_and_submit_workflow: _MakeAndSubmitWorkflow#

Generate and submit a new workflow from a file or string containing a workflow template parametrisation.

Parameters:

template_path_or_str (str) – Either a path to a template file in YAML or JSON format, or a YAML/JSON string.
is_string (str) – Determines whether template_path_or_str is a string or a file.
template_format (str) – If specified, one of “json” or “yaml”. This forces parsing from a particular format.
path (str | Path) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store to use for this workflow.
ts_fmt (str) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, object]) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str]) – String variables to substitute in template_file_or_str.
JS_parallelism (bool) – If True, allow multiple jobscripts to execute simultaneously. Raises if set to True but the store type does not support the jobscript_parallelism feature. If not set, jobscript parallelism will be used if the store type supports it.
wait (bool) – If True, this command will block until the workflow execution is complete.
add_to_known (bool) – If True, add the new submission to the known-submissions file, which is used by the show command to monitor current and recent submissions.
return_idx (bool) – If True, return a dict representing the jobscript indices submitted for each submission.
tasks (list[int]) – List of task indices to include in this submission. By default all tasks are included.
cancel (bool) – Immediately cancel the submission. Useful for testing and benchmarking.
status (bool) – If True, display a live status to track workflow creation and submission progress.

Returns:

Workflow – The created workflow.
dict[int, list[int]] – Mapping of submission handles. If requested by return_idx parameter.

property make_demo_workflow: _MakeDemoWorkflow#

Generate a new workflow from a builtin demo workflow template.

Parameters:

workflow_name (str) – Name of the demo workflow to make.
template_format (str) – If specified, one of “json” or “yaml”. This forces parsing from a particular format.
path (str | Path) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store type to use.
ts_fmt (str) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, object]) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str]) – String variables to substitute in the demo workflow template file.
status (bool) – If True, display a live status to track workflow creation progress.
add_submission – If True, add a submission to the workflow (but do not submit).

Returns:

Workflow – The created workflow, if add_submission is False.
Submission – The created submission object, if add_submission is True.

property make_workflow: _MakeWorkflow#

Generate a new workflow from a file or string containing a workflow template parametrisation.

Parameters:

template_path_or_str (str) – Either a path to a template file in YAML or JSON format, or a YAML/JSON string.
is_string (bool) – Determines if passing a file path or a string.
template_format (str) – If specified, one of “json” or “yaml”. This forces parsing from a particular format.
path (str | Path) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.
name (str) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).
name_add_timestamp (bool) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.
name_use_dir (bool) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.
overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.
store (str) – The persistent store type to use.
ts_fmt (str) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.
ts_name_fmt (str) – The datetime format to use when generating the workflow name, where it includes a timestamp.
store_kwargs (dict[str, object]) – Keyword arguments to pass to the store’s write_empty_workflow method.
variables (dict[str, str]) – String variables to substitute in template_file_or_str.
status (bool) – If True, display a live status to track workflow creation progress.
add_submission – If True, add a submission to the workflow (but do not submit).

Returns:

Workflow – The created workflow, if add_submission is False.
Submission – The created submission object, if add_submission is True.

module#: The module name in which the app object is defined.

name#: The name of the application.

package_name#: Name of package.

property parameters: _ParametersList#: The known template parameters.

perm_error_retry()#: Return a decorator for retrying functions on permission and OS errors that might be associated with cloud-storage desktop sync. engine operations.

property persistence_logger: Logger#: The logger for persistence engine messages.

property program_cache_dir: Path#: A directory for built-in program caching.

program_dir#: Directory for built-in program data.

property programs: dict[str, Path]#: The known programs.

pytest_args#: Arguments for pytest.

read_known_submissions_file()#

Retrieve existing workflows that might be running.

Return type:: list[KnownSubmission]

static redirect_std_to_file(*args, **kwargs)#

reload_config(config_dir=None, config_key=None, warn=True, **overrides)#

Reload the configuration. Use if a user has updated the configuration file outside the scope of this application.

Parameters:

config_dir (PathLike)
config_key (str | None)
warn (bool)

Return type:

None

reload_template_components(warn=True)#

Reload all template component data, warning by default if not already loaded.

Parameters:: warn (bool)
Return type:: None

remove_env(name=None, specifiers=None, label=None)#

Remove an environment identified by its name and specifiers, or remove all environments with a particular setup label and specifiers.

Parameters:

name (str | None)
specifiers (dict[str, Any] | None)
label (str | None)

reset_config(config_dir=None, config_key=None, warn=True, **overrides)#

Reset the config file to defaults, and reload the config.

Parameters:

config_dir (PathLike)
config_key (str | None)
warn (bool)

Return type:

None

property run_hpcflow_tests: _RunTests#: Run hpcflow test suite. This function is only available from derived apps.

property run_tests: _RunTests#: Run the test suite.

property run_time_info: RunTimeInfo#: Information about the runtime.

property runtime_info_logger: Logger#: The logger for runtime messages.

save_env(env, env_source_file=None, file_name='configured_envs.yaml')#

Save an environment to the environment definitions file.

Parameters:

env (Environment)
env_source_file (Path | None)
file_name (str)

property scheduler_lookup: dict[tuple[str, str], type[Scheduler]]#: The scheduler mapping.

property scripts: dict[str, Path]#: The known template scripts.

scripts_dir#: Directory for scripts.

property show: _Show#

Show information about running workflows.

Parameters:

max_recent (int) – Maximum number of inactive workflows to show.
full (bool) – If True, provide more information; output may spans multiple lines for each workflow submission.
no_update (bool) – If True, do not update the known-submissions file to remove workflows that are no longer running.

show_demo_workflow(name, syntax=True, doc=False)#

Print the contents of a builtin demo workflow template file.

Parameters:

name (str) – The name of the demo workflow file to print.
syntax (bool) – If True, use rich to syntax-highlight the output.
doc (bool) – If False, the printed workflow template file contents will not include the doc attribute (if originally present).

property show_legend: Callable[[], None]#: Output a legend for the jobscript-element and EAR states that are displayed by the show command.

property submission_logger: Logger#: The logger for job submission messages.

property submit_workflow: _SubmitWorkflow#

Submit an existing workflow.

Parameters:

workflow_path (str) – Path to an existing workflow
JS_parallelism (bool) – If True, allow multiple jobscripts to execute simultaneously. Raises if set to True but the store type does not support the jobscript_parallelism feature. If not set, jobscript parallelism will be used if the store type supports it.
tasks (list[int]) – List of task indices to include in this submission. By default all tasks are included.

Returns:

Mapping of submission handles. If requested by return_idx parameter.

Return type:

dict[int, list[int]]

property task_schemas: _TaskSchemasList#: The known template task schemas.

property template_components: TemplateComponents#: The template component data.

template_components_from_json_like(json_like)#

Get template components from a (simply parsed) JSON document.

Parameters:: json_like (dict[str, dict])
Return type:: TemplateComponents

property timeit: bool#: Whether the timing analysis system is active.

unload_config()#

Discard any loaded configuration.

Return type:: None

update_known_subs_file(inactive_IDs, start_times, end_times)#

Update submission records in the known-submission file.

Note

We aim for atomicity to help with the scenario where a new workflow submission is adding itself to the file at the same time as we have decided an existing workflow should no longer be part of this file. Ideally, such a scenario should not arise because both operations should only ever be interactively initiated by the single user (Workflow.submit and App.get_known_submissions). If this operation is atomic, then at least the known-submissions file should be left in a usable (but inaccurate) state.

Returns:

List of local IDs removed from the known-submissions file due to the maximum number of recent workflows to store being exceeded.

Return type:

list[int]

Parameters:

inactive_IDs (list[int])
start_times (dict[int, str])
end_times (dict[int, str])

property user_cache_dir: Path#: The user’s cache directory.

property user_cache_hostname_dir: Path#: The hostname-scoped app cache directory.

property user_data_dir: Path#: The user’s data directory.

property user_data_hostname_dir: Path#

The directory for holding user data.

We segregate by hostname to account for the case where multiple machines might use the same shared file system.

property user_runtime_dir: Path#: The user’s temporary runtime directory.

version#: The version of the application.

workflows_dir#: Directory for workflows.

hpcflow.sdk.app.App#

This Page