hpcflow.sdk.config.config.Config#

class hpcflow.sdk.config.config.Config(app, config_file, options, logger, config_key, uid=None, callbacks=None, variables=None, **overrides)#

Bases: object

Application configuration as defined in one or more config files.

This class supports indexing into the collection of properties via Python dot notation.

Notes

On modifying/setting existing values, modifications are not automatically copied to the configuration file; use save() to save to the file. Items in overrides are not saved into the file.

schedulers is used for specifying the available schedulers on this machine, and the default arguments that should be used when initialising the Scheduler object.

shells is used for specifying the default arguments that should be used when initialising the Shell object.

Parameters:

app (BaseApp) – The main hpcflow application instance.
config_file (ConfigFile) – The configuration file that contains this config.
options (ConfigOptions) – Configuration options to be applied.
logger (logging.Logger) – Where to log messages relating to configuration.
config_key (str | None) – The name of the configuration within the configuration file.
uid (int) – User ID.
callbacks (dict) – Overrides for the callback system.
variables (dict[str, str]) – Variables to substitute when processing the configuration.

user_name#

User to submit as. Mapped to a field in the configuration file.

Type:: str

user_orcid#

User’s ORCID. Mapped to a field in the configuration file.

Type:: str

user_affiliation#

User’s institutional affiliation. Mapped to a field in the configuration file.

Type:: str

linux_release_file#

Where to get the description of the Linux release version data. Mapped to a field in the configuration file.

Type:: str

log_file_level#

At what level to do logging to the file. Mapped to a field in the configuration file.

Type:: str

log_console_level#

At what level to do logging to the console. Usually coarser than to a file. Mapped to a field in the configuration file.

Type:: str

demo_data_manifest_file#

Where the manifest describing the demo data is. Mapped to a field in the configuration file.

Type:: str

Methods

`add_scheduler`	Add a scheduler.
`add_shell`	Add a shell.
`add_shell_WSL`	Add shell with WSL prefix.
`append`	Append a value to a list-like configuration item.
`get`	Get the value of a configuration item.
`get_all`	Get all configurable items.
`get_configurable`	Get a list of all configurable keys.
`import_from_file`	Import config items from a (remote or local) YAML file.
`init`	Configure from a known importable config.
`pop`	Remove a value from a specified index of a list-like configuration item.
`prepend`	Prepend a value to a list-like configuration item.
`register_config_get_callback`	Decorator to register a function as a configuration callback for a specified configuration item name, to be invoked on get of the item.
`register_config_set_callback`	Decorator to register a function as a configuration callback for a specified configuration item name, to be invoked on set of the item.
`reset`	Reset to the default configuration.
`save`	Save any modified/unset configuration items into the file.
`set`	Set the value of a configuration item.
`set_github_demo_data_dir`	Set the demo_data_dir item, to an fsspec Github URL.
`unset`	Unset the value of a configuration item.
`update`	Update a map-like configuration item.

Attributes

`command_file_sources`	Where to get command files.
`config_directory`	The directory containing the configuration file.
`config_file_contents`	The cached contents of the configuration file.
`config_file_name`	The name of the configuration file.
`config_file_path`	The full path to the configuration file.
`config_key`	The primary key to select the configuration within the configuration file.
`config_schemas`	The schemas that apply to the configuration file.
`default_scheduler`	The name of the default scheduler.
`default_shell`	The name of the default shell.
`demo_data_dir`	Location of demo data.
`environment_sources`	Where to get execution environment descriptors.
`host_user_id`	User ID as understood by the script.
`host_user_id_file_path`	Where user ID information is stored.
`invoking_user_id`	User ID that created the workflow.
`log_file_path`	Where to log to.
`machine`	Machine to submit to.
`parameter_sources`	Where to get parameter descriptors.
`schedulers`	Settings for supported scheduler(s).
`shells`	Settings for supported shell(s).
`task_schema_sources`	Where to get task schemas.

add_scheduler(scheduler, **defaults)#

Add a scheduler.

Parameters:: scheduler (str) –
Return type:: None

add_shell(shell, **defaults)#

Add a shell.

Parameters:: shell (str) –
Return type:: None

add_shell_WSL(**defaults)#

Add shell with WSL prefix.

Return type:: None

append(path, value, *, is_json=False)#

Append a value to a list-like configuration item.

Parameters:

path (str) – The name of or path to the configuration item.
value – The value to append.
is_json (bool) –

Return type:

None

property command_file_sources: Sequence[str]#: Where to get command files. Mapped to a field in the configuration file.

property config_directory: Path#: The directory containing the configuration file.

property config_file_contents: str#: The cached contents of the configuration file.

property config_file_name: str#: The name of the configuration file.

property config_file_path: Path#: The full path to the configuration file.

property config_key: str#: The primary key to select the configuration within the configuration file.

property config_schemas: Sequence[Schema]#: The schemas that apply to the configuration file.

property default_scheduler: str#: The name of the default scheduler. Mapped to a field in the configuration file.

property default_shell: str#: The name of the default shell. Mapped to a field in the configuration file.

property demo_data_dir: str | None#: Location of demo data. Mapped to a field in the configuration file.

property environment_sources: Sequence[Path]#: Where to get execution environment descriptors. Mapped to a field in the configuration file.

get(path: str, *, callback: bool = True, copy: bool = False, ret_root_and_parts: Literal[False] = False, default: Any | None = None) → Any#

get(path: str, *, callback: bool = True, copy: bool = False, ret_root_and_parts: Literal[True], default: Any | None = None) → tuple[Any, Any, list[str]]

Get the value of a configuration item.

Parameters:: path – The name of or path to the configuration item.

get_all(*, include_overrides: bool = True, as_str: Literal[True]) → Mapping[str, str]#
get_all(*, include_overrides: bool = True, as_str: Literal[False] = False) → Mapping[str, Any]: Get all configurable items.

get_configurable()#

Get a list of all configurable keys.

Return type:: Sequence[str]

property host_user_id: str#: User ID as understood by the script.

property host_user_id_file_path: Path#: Where user ID information is stored.

import_from_file(file_path, *, rename=True, make_new=False)#

Import config items from a (remote or local) YAML file. Existing config items of the same names will be overwritten.

Parameters:

file_path (Path | str) – Local or remote path to a config import YAML file which may have top-level keys “invocation” and “config”.
rename – If True, the current config will be renamed to the stem of the file specified in file_path. Ignored if make_new is True.
make_new – If True, add the config items as a new config, rather than modifying the current config. The name of the new config will be the stem of the file specified in file_path.

Return type:

None

init(known_name, path=None)#

Configure from a known importable config.

Parameters:

known_name (str) –
path (str | None) –

Return type:

None

property invoking_user_id: str#: User ID that created the workflow.

property log_file_path: str#: Where to log to. Mapped to a field in the configuration file.

property machine: str#: Machine to submit to. Mapped to a field in the configuration file.

property parameter_sources: Sequence[str]#: Where to get parameter descriptors. Mapped to a field in the configuration file.

pop(path, index)#

Remove a value from a specified index of a list-like configuration item.

Parameters:

path (str) – The name of or path to the configuration item.
index (int) – Where to remove the value from. 0 for the first item, -1 for the last.

Return type:

None

prepend(path, value, *, is_json=False)#

Prepend a value to a list-like configuration item.

Parameters:

path (str) – The name of or path to the configuration item.
value – The value to prepend.
is_json (bool) –

Return type:

None

register_config_get_callback(name)#

Decorator to register a function as a configuration callback for a specified configuration item name, to be invoked on get of the item.

Parameters:: name (str) –
Return type:: Callable[[GetterCallback], GetterCallback]

register_config_set_callback(name)#

Decorator to register a function as a configuration callback for a specified configuration item name, to be invoked on set of the item.

Parameters:: name (str) –
Return type:: Callable[[SetterCallback], SetterCallback]

reset()#

Reset to the default configuration.

Return type:: None

save()#

Save any modified/unset configuration items into the file.

Return type:: None

property schedulers: Mapping[str, SchedulerConfigDescriptor]#: Settings for supported scheduler(s). Mapped to a field in the configuration file.

set(path: str, value: Any, *, is_json: Literal[False] = False, quiet: bool = False) → None#

set(path: str, value: str, *, is_json: Literal[True], quiet: bool = False) → None

Set the value of a configuration item.

Parameters:

path – Which configuration item to set.
value – What to set it to.

set_github_demo_data_dir(sha)#

Set the demo_data_dir item, to an fsspec Github URL.

We use this (via the CLI) when testing the frozen app on Github, because, by default, the SHA is set to the current version tag, which might not include recent changes to the demo data.

Parameters:: sha (str) –
Return type:: None

property shells: Mapping[str, ShellConfigDescriptor]#: Settings for supported shell(s). Mapped to a field in the configuration file.

property task_schema_sources: Sequence[str]#: Where to get task schemas. Mapped to a field in the configuration file.

unset(name)#

Unset the value of a configuration item.

Parameters:: name (str) – The name of the configuration item.
Return type:: None

Notes

Only top level configuration items may be unset.

update(path, value, *, is_json=False)#

Update a map-like configuration item.

Parameters:

path (str) – A dot-delimited string of the nested path to update.
value (dict) – A dictionary to merge in.
is_json (bool) –

Return type:

None