hpcflow.sdk.app.App#

class hpcflow.sdk.app.App(*args, **kwargs)#

Bases: BaseApp

Class from which to instantiate downstream app objects (e.g. MatFlow).

Methods

cache_all_demo_data_files

Get the name of all cached demo data file.

cache_demo_data_file

Get the name of a cached demo data file.

clear_demo_data_cache_dir

Delete the contents of the example data files cache directory.

clear_known_submissions_file

Clear the known-submissions file of all submissions.

clear_user_cache_dir

Delete the contents of the cache directory.

clear_user_cache_hostname_dir

Delete the contents of the hostname-scoped cache directory.

clear_user_runtime_dir

Delete the contents of the user runtime directory.

configure_env

Configure an execution environment.

copy_demo_data

Copy a builtin demo data file to the specified location.

copy_demo_workflow

Copy a builtin demo workflow to the specified location.

get_OS_supported_schedulers

Retrieve a list of schedulers that are supported in principle by this operating system.

get_config_path

Return the full path to the config file, without loading the config.

get_demo_data_file_path

Get the full path to an example data file in the app cache directory.

get_demo_data_files_manifest

Get a dict whose keys are example data file names and whose values are the source files if the source file required unzipping or None otherwise.

get_demo_workflow_template_file

Context manager to get a (temporary) file path to an included demo workflow template.

get_info

Get miscellaneous runtime system information.

get_parameter_task_schema_map

Get a dict mapping parameter types to task schemas that input/output each parameter.

get_scheduler

Get an arbitrary scheduler object.

list_demo_data_files

List available example data files.

list_demo_workflows

Return a list of demo workflow templates included in the app.

load_builtin_template_component_data

Load the template component data built into the package.

load_config

Load the user's configuration.

load_demo_workflow

Load a WorkflowTemplate object from a builtin demo template file.

load_template_components

Load all template component data, warning by default if already loaded.

perm_error_retry

Return a decorator for retrying functions on permission and OS errors that might be associated with cloud-storage desktop sync.

read_known_submissions_file

Retrieve existing workflows that might be running.

reload_config

Reload the configuration.

reload_template_components

Reload all template component data, warning by default if not already loaded.

reset_config

Reset the config file to defaults, and reload the config.

show_demo_workflow

Print the contents of a builtin demo workflow template file.

template_components_from_json_like

Get template components from a (simply parsed) JSOM document.

unload_config

Discard any loaded configuration.

update_known_subs_file

Update submission records in the known-submission file.

Attributes

API_logger

The logger for API messages.

CLI_logger

The logger for CLI messages.

command_files

The known template command files.

config

The configuration.

config_logger

The logger for configuration messages.

demo_data_cache_dir

A directory for example data caching.

envs

The known template execution environments.

is_config_loaded

Whether the configuration is loaded.

is_template_components_loaded

Whether any template component (e.g.

known_subs_file_path

The path to the file describing known submissions.

log

The application log.

logger

The main underlying logger.

parameters

The known template parameters.

persistence_logger

The logger for persistence engine messages.

run_time_info

Information about the runtime.

runtime_info_logger

The logger for runtime messages.

scheduler_lookup

The scheduler mapping.

scripts

The known template scripts.

submission_logger

The logger for job submission messages.

task_schemas

The known template task schemas.

template_components

The template component data.

timeit

Whether the timing analysis system is active.

user_cache_dir

The user's cache directory.

user_cache_hostname_dir

The hostname-scoped app cache directory.

user_data_dir

The user's data directory.

user_data_hostname_dir

The directory for holding user data.

user_runtime_dir

The user's temporary runtime directory.

name

The name of the application.

package_name

Name of package.

version

The version of the application.

module

The module name in which the app object is defined.

description

Description of the application.

gh_org

Name of Github organisation responsible for the application.

gh_repo

Github repository containing the application source.

config_options

Configuration options.

pytest_args

Arguments for pytest.

scripts_dir

Directory for scripts.

workflows_dir

Directory for workflows.

demo_data_dir

Directory for demonstration data.

demo_data_manifest_dir

Directory for demonstration data manifests.

docs_import_conv

The convention for the app alias used in import statements in the documentation.

docs_url

URL to documentation.

cli

Command line interface subsystem.

property API_logger: Logger#

The logger for API messages.

property CLI_logger: Logger#

The logger for CLI messages.

cache_all_demo_data_files()#

Get the name of all cached demo data file.

Return type:

List[Path]

cache_demo_data_file(file_name)#

Get the name of a cached demo data file.

Return type:

Path

clear_demo_data_cache_dir()#

Delete the contents of the example data files cache directory.

clear_known_submissions_file()#

Clear the known-submissions file of all submissions. This shouldn’t be needed normally.

clear_user_cache_dir()#

Delete the contents of the cache directory.

clear_user_cache_hostname_dir()#

Delete the contents of the hostname-scoped cache directory.

clear_user_runtime_dir()#

Delete the contents of the user runtime directory.

cli#

Command line interface subsystem.

property command_files: CommandFilesList#

The known template command files.

property config: Config#

The configuration.

property config_logger: Logger#

The logger for configuration messages.

config_options#

Configuration options.

configure_env(name, setup=None, executables=None, use_current_env=False, env_source_file=None)#

Configure an execution environment.

copy_demo_data(file_name, dst=None, doc=True)#

Copy a builtin demo data file to the specified location.

Parameters:
  • file_name (str) – The name of the demo data file to copy

  • dst (PathLike | None) – Directory or full file path to copy the demo data file to. If not specified, the current working directory will be used.

  • doc (bool) –

Return type:

str

copy_demo_workflow(name, dst=None, doc=True)#

Copy a builtin demo workflow to the specified location.

Parameters:
  • name (str) – The name of the demo workflow to copy

  • dst (PathLike | None) – Directory or full file path to copy the demo workflow to. If not specified, the current working directory will be used.

  • doc (bool) – If False, the copied workflow template file will not include the doc attribute (if originally present).

Return type:

str

property demo_data_cache_dir: Path#

A directory for example data caching.

demo_data_dir#

Directory for demonstration data.

demo_data_manifest_dir#

Directory for demonstration data manifests.

description#

Description of the application.

docs_import_conv#

The convention for the app alias used in import statements in the documentation.

docs_url#

URL to documentation.

property envs: EnvironmentsList#

The known template execution environments.

get_OS_supported_schedulers()#

Retrieve a list of schedulers that are supported in principle by this operating system.

This does not necessarily mean all the returned schedulers are available on this system.

get_config_path(config_dir=None)#

Return the full path to the config file, without loading the config.

get_demo_data_file_path(file_name)#

Get the full path to an example data file in the app cache directory.

If the file does not already exist in the app cache directory, it will be added (and unzipped if required). The file may first be downloaded from a remote file system such as GitHub (see _get_demo_data_file_source_path for details).

Return type:

Path

get_demo_data_files_manifest()#

Get a dict whose keys are example data file names and whose values are the source files if the source file required unzipping or None otherwise.

If the config item demo_data_manifest_file is set, this is used as the manifest file path. Otherwise, the app attribute demo_data_manifest_dir is used, and is expected to be the package/directory in the source code within which a file demo_data_manifest.json is expected.

Return type:

Dict[str, None | str]

get_demo_workflow_template_file(name, doc=True, delete=True)#

Context manager to get a (temporary) file path to an included demo workflow template.

Parameters:
  • name (str) – Name of the builtin demo workflow template whose file path is to be retrieved.

  • doc (bool) – If False, the yielded path will be to a file without the doc attribute (if originally present).

  • delete (bool) – If True, remove the temporary file on exit.

Return type:

Path

get_info()#

Get miscellaneous runtime system information.

Return type:

Dict[str, Any]

get_parameter_task_schema_map()#

Get a dict mapping parameter types to task schemas that input/output each parameter.

Return type:

Dict[str, List[List]]

get_scheduler(scheduler_name, os_name, scheduler_args=None)#

Get an arbitrary scheduler object.

gh_org#

Name of Github organisation responsible for the application.

gh_repo#

Github repository containing the application source.

property is_config_loaded: bool#

Whether the configuration is loaded.

property is_template_components_loaded: bool#

Whether any template component (e.g. parameters) has been loaded.

property known_subs_file_path#

The path to the file describing known submissions.

list_demo_data_files()#

List available example data files.

Return type:

Tuple[str]

list_demo_workflows()#

Return a list of demo workflow templates included in the app.

Return type:

Tuple[str]

classmethod load_builtin_template_component_data(package)#

Load the template component data built into the package. This is as opposed to the template components defined by users.

Return type:

Dict[str, List | Dict]

load_config(config_dir=None, config_key=None, warn=True, **overrides)#

Load the user’s configuration.

Return type:

None

load_demo_workflow(name)#

Load a WorkflowTemplate object from a builtin demo template file.

Parameters:

name (str) –

Return type:

WorkflowTemplate

load_template_components(warn=True)#

Load all template component data, warning by default if already loaded.

Return type:

None

property log: AppLog#

The application log.

property logger: Logger#

The main underlying logger.

module#

The module name in which the app object is defined.

name#

The name of the application.

package_name#

Name of package.

property parameters: ParametersList#

The known template parameters.

perm_error_retry()#

Return a decorator for retrying functions on permission and OS errors that might be associated with cloud-storage desktop sync. engine operations.

property persistence_logger: Logger#

The logger for persistence engine messages.

pytest_args#

Arguments for pytest.

read_known_submissions_file()#

Retrieve existing workflows that might be running.

Return type:

List[Dict]

reload_config(config_dir=None, config_key=None, warn=True, **overrides)#

Reload the configuration. Use if a user has updated the configuration file outside the scope of this application.

Return type:

None

reload_template_components(warn=True)#

Reload all template component data, warning by default if not already loaded.

Return type:

None

reset_config(config_dir=None, config_key=None, warn=True, **overrides)#

Reset the config file to defaults, and reload the config.

Return type:

None

property run_time_info: RunTimeInfo#

Information about the runtime.

property runtime_info_logger: Logger#

The logger for runtime messages.

property scheduler_lookup#

The scheduler mapping.

property scripts#

The known template scripts.

scripts_dir#

Directory for scripts.

show_demo_workflow(name, syntax=True, doc=False)#

Print the contents of a builtin demo workflow template file.

Parameters:
  • name (str) – The name of the demo workflow file to print.

  • syntax (bool) – If True, use rich to syntax-highlight the output.

  • doc (bool) – If False, the printed workflow template file contents will not include the doc attribute (if originally present).

property submission_logger: Logger#

The logger for job submission messages.

property task_schemas: TaskSchemasList#

The known template task schemas.

property template_components: Dict[str, ObjectList]#

The template component data.

template_components_from_json_like(json_like)#

Get template components from a (simply parsed) JSOM document.

property timeit: bool#

Whether the timing analysis system is active.

unload_config()#

Discard any loaded configuration.

update_known_subs_file(inactive_IDs, start_times, end_times)#

Update submission records in the known-submission file.

Note we aim for atomicity to help with the scenario where a new workflow submission is adding itself to the file at the same time as we have decided an existing workflow should no longer be part of this file. Ideally, such a scenario should not arise because both operations should only ever be interactively initiated by the single user (Workflow.submit and App.get_known_submissions). If this operation is atomic, then at least the known-submissions file should be left in a usable (but inaccurate) state.

Returns:

List of local IDs removed from the known-submissions file due to the maximum number of recent workflows to store being exceeded.

Return type:

removed_IDs

Parameters:
property user_cache_dir: Path#

The user’s cache directory.

property user_cache_hostname_dir: Path#

The hostname-scoped app cache directory.

property user_data_dir: Path#

The user’s data directory.

property user_data_hostname_dir: Path#

The directory for holding user data.

We segregate by hostname to account for the case where multiple machines might use the same shared file system.

property user_runtime_dir: Path#

The user’s temporary runtime directory.

version#

The version of the application.

workflows_dir#

Directory for workflows.