hpcflow.app.make_and_submit_demo_workflow#

hpcflow.app.make_and_submit_demo_workflow(workflow_name, template_format=None, path=None, name=None, name_add_timestamp=None, name_use_dir=None, overwrite=False, store='zarr', ts_fmt=None, ts_name_fmt=None, store_kwargs=None, variables=None, updates=None, resources=None, config=None, JS_parallelism=None, min_jobscripts=True, wait=False, add_to_known=True, return_idx=False, tasks=None, cancel=False, status=True, quiet=False)#

Generate and submit a new hpcFlow workflow from a file or string containing a workflow template parametrisation.

Parameters:

  • workflow_name (str) – Name of the demo workflow to make.

  • template_format (Literal['json', 'yaml'] | None) – If specified, one of “json” or “yaml”. This forces parsing from a particular format.

  • path (PathLike | None) – The directory in which the workflow will be generated. If not specified, the config item default_workflow_path will be used; if that is not set, the current directory is used.

  • name (str | None) – The name to use for the workflow. If not provided, the name will be set to that of the template (optionally suffixed by a date-timestamp if name_add_timestamp is True).

  • name_add_timestamp (bool) – If True, suffix the name with a date-timestamp. A default value can be set with the config item workflow_name_add_timestamp; otherwise set to True.

  • name_use_dir (bool) – If True, and name_add_timestamp is also True, the workflow directory name will be just the date-timestamp, and will be contained within a parent directory corresponding to the workflow name. A default value can be set with the config item workflow_name_use_dir; otherwise set to False.

  • overwrite (bool) – If True and the workflow directory (path + name) already exists, the existing directory will be overwritten.

  • store (str) – The persistent store to use for this workflow.

  • ts_fmt (str | None) – The datetime format to use for storing datetimes. Datetimes are always stored in UTC (because Numpy does not store time zone info), so this should not include a time zone name.

  • ts_name_fmt (str | None) – The datetime format to use when generating the workflow name, where it includes a timestamp.

  • store_kwargs (dict[str, Any] | None) – Keyword arguments to pass to the store’s write_empty_workflow method.

  • variables (dict[str, str] | None) – String variables to substitute in the demo workflow template file.

  • updates (dict[tuple[str | int, ...], Any] | None) – Any updates to apply to the template data, in the form of a dict whose keys are paths (as tuples of strings or ints) to the relevant datum, and whose values are the new values to use at those paths.

  • resources (dict[str, Any] | None) – Any updates to template-level resource items for the “any” scope. This is merged into updates.

  • config (dict[str, Any] | None) – Any updates to template-level config items. This is merged into updates.

  • JS_parallelism (bool | Literal['direct', 'scheduled'] | None) – If True, allow multiple jobscripts to execute simultaneously. If ‘scheduled’/’direct’, only allow simultaneous execution of scheduled/direct jobscripts. Raises if set to True, ‘scheduled’, or ‘direct’, but the store type does not support the jobscript_parallelism feature. If not set, jobscript parallelism will be used if the store type supports it, for scheduled jobscripts only.

  • min_jobscripts (bool) – If True (the default), minimise the total number of jobscripts by performing as many merges as possible. This may merge otherwise independent jobscripts, such that they are run sequentially rather than in parallel.

  • wait (bool) – If True, this command will block until the workflow execution is complete.

  • add_to_known (bool) – If True, add the new submission to the known-submissions file, which is used by the show command to monitor current and recent submissions.

  • return_idx (bool) – If True, return a dict representing the jobscript indices submitted for each submission.

  • tasks (list[int] | None) – List of task indices to include in this submission. By default all tasks are included.

  • cancel (bool) – Immediately cancel the submission. Useful for testing and benchmarking.

  • status (bool) – If True, display a live status to track submission progress.

  • quiet (bool) – If True, do not print anything about submission.

Returns:

  • Workflow – The created workflow.

  • dict[int, list[int]] – Mapping of submission handles. If requested by return_idx parameter.

Return type:

tuple[_Workflow, Mapping[int, Sequence[int]]] | _Workflow