cwltool.process

Classes and methods relevant for all CWL Process types.

Attributes

INPUT_OBJ_VOCAB

MPIRequirementName

MapperEnt

Mapper entries.

INTERNAL_VERSION

ORDERED_VERSIONS

ORIGINAL_CWLVERSION

CWLObjectType

Typical raw dictionary found in lightly parsed CWL.

CWLOutputType

JobsGeneratorType

LoadListingType

OutputCallbackType

supportedProcessRequirements

cwl_files

salad_files

SCHEMA_CACHE

SCHEMA_FILE

SCHEMA_DIR

SCHEMA_ANY

custom_schemas

FILE_COUNT_WARNING

CWL_IANA

Exceptions

UnsupportedRequirement

Common base class for all non-exit exceptions.

WorkflowException

Common base class for all non-exit exceptions.

Classes

Builder

Helper class to construct a command line from a CWL CommandLineTool.

LoadingContext

Shared kwargs based initializer for RuntimeContext and LoadingContext.

RuntimeContext

Shared kwargs based initializer for RuntimeContext and LoadingContext.

PathMapper

Mapping of files from relative path provided in the file to a tuple.

SecretStore

Minimal implementation of a secret storage.

StdFsAccess

Local filesystem implementation.

HasReqsHints

Base class for get_requirement().

LogAsDebugFilter

Filter instances are used to perform arbitrary filtering of LogRecords.

Process

Abstract CWL Process.

Functions

getdefault(val, default)

Return the val using the default as backup in case the val is None.

adjustDirObjs(rec, op)

Apply an update function to each Directory object in the object rec.

aslist(thing)

Wrap any non-MutableSequence/list in a list.

cmp_like_py2(dict1, dict2)

Compare in the same manner as Python2.

ensure_writable(path[, include_root])

Ensure that 'path' is writable.

get_listing(fs_access, rec[, recursive])

Expand, recursively, any 'listing' fields in a Directory.

normalizeFilesDirs(job)

random_outdir()

Return the random directory name chosen to use for tool / workflow output.

visit_class(rec, cls, op)

Apply a function to with "class" in cls.

validate_js_expressions(tool, schema[, ...])

use_standard_schema(version)

use_custom_schema(version, name, text)

get_schema(version)

shortname(inputid)

stage_files(pathmapper[, stage_func, ignore_writable, ...])

Link or copy files to their targets. Create them as needed.

relocateOutputs(outputObj, destination_path, ...[, ...])

cleanIntermediate(output_dirs)

add_sizes(fsaccess, obj)

fill_in_defaults(inputs, job, fsaccess)

For each missing input in the input object, copy over the default.

avroize_type(field_type[, name_prefix])

Add missing information to a type so that CWL types are valid.

get_overrides(overrides, toolid)

Combine overrides for the target tool ID.

var_spool_cwl_detector(obj[, item, obj_key])

Detect any textual reference to /var/spool/cwl.

eval_resource(builder, resource_req)

uniquename(stem[, names])

nestdir(base, deps)

mergedirs(listing)

scandeps(base, doc, reffields, urlfields, loadref[, ...])

Search for external files references in a CWL document or input object.

compute_checksums(fs_access, fileobj)

Module Contents

cwltool.process.INPUT_OBJ_VOCAB: Dict[str, str]
class cwltool.process.Builder(job, files, bindings, schemaDefs, names, requirements, hints, resources, mutation_manager, formatgraph, make_fs_access, fs_access, job_script_provider, timeout, debug, js_console, force_docker_pull, loadListing, outdir, tmpdir, stagedir, cwlVersion, container_engine)

Bases: cwltool.utils.HasReqsHints

Inheritance diagram of cwltool.process.Builder

Helper class to construct a command line from a CWL CommandLineTool.

Parameters:
build_job_script(commands)
Parameters:

commands (List[str])

Return type:

Optional[str]

bind_input(schema, datum, discover_secondaryFiles, lead_pos=None, tail_pos=None)

Bind an input object to the command line.

Raises:
  • ValidationException – in the event of an invalid type union

  • WorkflowException – if a CWL Expression (“position”, “required”, “pattern”, “format”) evaluates to the wrong type or if a required secondary file is missing

Parameters:
  • schema (cwltool.utils.CWLObjectType)

  • datum (Union[cwltool.utils.CWLObjectType, List[cwltool.utils.CWLObjectType]])

  • discover_secondaryFiles (bool)

  • lead_pos (Optional[Union[int, List[int]]])

  • tail_pos (Optional[Union[str, List[int]]])

Return type:

List[MutableMapping[str, Union[str, List[int]]]]

tostr(value)

Represent an input parameter as a string.

Raises:

WorkflowException – if the item is a File or Directory and the “path” is missing.

Parameters:

value (Union[MutableMapping[str, str], Any])

Return type:

str

generate_arg(binding)
Parameters:

binding (cwltool.utils.CWLObjectType)

Return type:

List[str]

do_eval(ex, context=None, recursive=False, strip_whitespace=True)
Parameters:
  • ex (Optional[cwltool.utils.CWLOutputType])

  • context (Optional[Any])

  • recursive (bool)

  • strip_whitespace (bool)

Return type:

Optional[cwltool.utils.CWLOutputType]

class cwltool.process.LoadingContext(kwargs=None)

Bases: ContextBase

Inheritance diagram of cwltool.process.LoadingContext

Shared kwargs based initializer for RuntimeContext and LoadingContext.

Parameters:

kwargs (Optional[Dict[str, Any]])

copy()

Return a copy of this LoadingContext.

Return type:

LoadingContext

class cwltool.process.RuntimeContext(kwargs=None)

Bases: ContextBase

Inheritance diagram of cwltool.process.RuntimeContext

Shared kwargs based initializer for RuntimeContext and LoadingContext.

Parameters:

kwargs (Optional[Dict[str, Any]])

outdir: str | None = None
tmpdir: str = ''
tmpdir_prefix: str
tmp_outdir_prefix: str = ''
stagedir: str = ''
get_outdir()

Return outdir or create one with tmp_outdir_prefix.

Return type:

str

get_tmpdir()

Return tmpdir or create one with tmpdir_prefix.

Return type:

str

get_stagedir()

Return stagedir or create one with tmpdir_prefix.

Return type:

str

create_tmpdir()

Create a temporary directory that respects tmpdir_prefix.

Return type:

str

create_outdir()

Create a temporary directory that respects tmp_outdir_prefix.

Return type:

str

copy()

Return a copy of this RuntimeContext.

Return type:

RuntimeContext

cwltool.process.getdefault(val, default)

Return the val using the default as backup in case the val is None.

Parameters:
  • val (Any)

  • default (Any)

Return type:

Any

exception cwltool.process.UnsupportedRequirement

Bases: WorkflowException

Inheritance diagram of cwltool.process.UnsupportedRequirement

Common base class for all non-exit exceptions.

exception cwltool.process.WorkflowException

Bases: Exception

Inheritance diagram of cwltool.process.WorkflowException

Common base class for all non-exit exceptions.

cwltool.process.MPIRequirementName = 'http://commonwl.org/cwltool#MPIRequirement'
cwltool.process.MapperEnt

Mapper entries.

cwltool.process.resolved: str

The “real” path on the local file system (after resolving relative paths and traversing symlinks

cwltool.process.target: str

The path on the target file system (under stagedir)

cwltool.process.type: str

The object type. One of “File”, “Directory”, “CreateFile”, “WritableFile”, or “CreateWritableFile”.

cwltool.process.staged: bool

If the File has been staged yet

class cwltool.process.PathMapper(referenced_files, basedir, stagedir, separateDirs=True)

Mapping of files from relative path provided in the file to a tuple.

(absolute local path, absolute container path)

The tao of PathMapper:

The initializer takes a list of class: File and class: Directory objects, a base directory (for resolving relative references) and a staging directory (where the files are mapped to).

The purpose of the setup method is to determine where each File or Directory should be placed on the target file system (relative to stagedir).

If separatedirs=True, unrelated files will be isolated in their own directories under stagedir. If separatedirs=False, files and directories will all be placed in stagedir (with the possibility for name collisions…)

The path map maps the “location” of the input Files and Directory objects to a tuple (resolved, target, type). The “resolved” field is the “real” path on the local file system (after resolving relative paths and traversing symlinks). The “target” is the path on the target file system (under stagedir). The type is the object type (one of File, Directory, CreateFile, WritableFile, CreateWritableFile).

The latter three (CreateFile, WritableFile, CreateWritableFile) are used by InitialWorkDirRequirement to indicate files that are generated on the fly (CreateFile and CreateWritableFile, in this case “resolved” holds the file contents instead of the path because they file doesn’t exist) or copied into the output directory so they can be opened for update (“r+” or “a”) (WritableFile and CreateWritableFile).

Parameters:
  • referenced_files (List[cwltool.utils.CWLObjectType])

  • basedir (str)

  • stagedir (str)

  • separateDirs (bool)

visitlisting(listing, stagedir, basedir, copy=False, staged=False)
Parameters:
  • listing (List[cwltool.utils.CWLObjectType])

  • stagedir (str)

  • basedir (str)

  • copy (bool)

  • staged (bool)

Return type:

None

visit(obj, stagedir, basedir, copy=False, staged=False)
Parameters:
  • obj (cwltool.utils.CWLObjectType)

  • stagedir (str)

  • basedir (str)

  • copy (bool)

  • staged (bool)

Return type:

None

setup(referenced_files, basedir)
Parameters:
  • referenced_files (List[cwltool.utils.CWLObjectType])

  • basedir (str)

Return type:

None

mapper(src)
Parameters:

src (str)

Return type:

MapperEnt

files()

Return a dictionary keys view of locations.

Return type:

KeysView[str]

items()

Return a dictionary items view.

Return type:

ItemsView[str, MapperEnt]

items_exclude_children()

Return a dictionary items view minus any entries which are children of other entries.

Return type:

ItemsView[str, MapperEnt]

reversemap(target)

Find the (source, resolved_path) for the given target, if any.

Parameters:

target (str)

Return type:

Optional[Tuple[str, str]]

update(key, resolved, target, ctype, stage)

Update an existine entry.

Parameters:
Return type:

MapperEnt

__contains__(key)

Test for the presence of the given relative path in this mapper.

Parameters:

key (str)

Return type:

bool

__iter__()

Get iterator for the maps.

Return type:

Iterator[MapperEnt]

class cwltool.process.SecretStore

Minimal implementation of a secret storage.

add(value)

Add the given value to the store.

Returns a placeholder value to use until the real value is needed.

Parameters:

value (Optional[cwltool.utils.CWLOutputType])

Return type:

Optional[cwltool.utils.CWLOutputType]

store(secrets, job)

Sanitize the job object of any of the given secrets.

Parameters:
  • secrets (List[str])

  • job (cwltool.utils.CWLObjectType)

Return type:

None

has_secret(value)

Test if the provided document has any of our secrets.

Parameters:

value (cwltool.utils.CWLOutputType)

Return type:

bool

retrieve(value)

Replace placeholders with their corresponding secrets.

Parameters:

value (cwltool.utils.CWLOutputType)

Return type:

cwltool.utils.CWLOutputType

class cwltool.process.StdFsAccess(basedir)

Local filesystem implementation.

Parameters:

basedir (str)

glob(pattern)
Parameters:

pattern (str)

Return type:

List[str]

open(fn, mode)
Parameters:
Return type:

IO[Any]

exists(fn)
Parameters:

fn (str)

Return type:

bool

size(fn)
Parameters:

fn (str)

Return type:

int

isfile(fn)
Parameters:

fn (str)

Return type:

bool

isdir(fn)
Parameters:

fn (str)

Return type:

bool

listdir(fn)
Parameters:

fn (str)

Return type:

List[str]

join(path, *paths)
Parameters:
Return type:

str

realpath(path)
Parameters:

path (str)

Return type:

str

cwltool.process.INTERNAL_VERSION = 'v1.2'
cwltool.process.ORDERED_VERSIONS = ['v1.0', 'v1.1.0-dev1', 'v1.1', 'v1.2.0-dev1', 'v1.2.0-dev2', 'v1.2.0-dev3', 'v1.2.0-dev4',...
cwltool.process.ORIGINAL_CWLVERSION = 'http://commonwl.org/cwltool#original_cwlVersion'
cwltool.process.CWLObjectType

Typical raw dictionary found in lightly parsed CWL.

cwltool.process.CWLOutputType
class cwltool.process.HasReqsHints

Base class for get_requirement().

get_requirement(feature)

Retrieve the named feature from the requirements field, or the hints field.

Parameters:

feature (str)

Return type:

Tuple[Optional[CWLObjectType], Optional[bool]]

cwltool.process.JobsGeneratorType
cwltool.process.LoadListingType
cwltool.process.OutputCallbackType
cwltool.process.adjustDirObjs(rec, op)

Apply an update function to each Directory object in the object rec.

Parameters:
  • rec (Any)

  • op (Union[Callable[Ellipsis, Any], partial[Any]])

Return type:

None

cwltool.process.aslist(thing)

Wrap any non-MutableSequence/list in a list.

Parameters:

thing (Any)

Return type:

MutableSequence[Any]

cwltool.process.cmp_like_py2(dict1, dict2)

Compare in the same manner as Python2.

Comparison function to be used in sorting as python3 doesn’t allow sorting of different types like str() and int(). This function re-creates sorting nature in py2 of heterogeneous list of int and str

Parameters:
  • dict1 (Dict[str, Any])

  • dict2 (Dict[str, Any])

Return type:

int

cwltool.process.ensure_writable(path, include_root=False)

Ensure that ‘path’ is writable.

If ‘path’ is a directory, then all files and directories under ‘path’ are made writable, recursively. If ‘path’ is a file or if ‘include_root’ is True, then ‘path’ itself is made writable.

Parameters:
Return type:

None

cwltool.process.get_listing(fs_access, rec, recursive=True)

Expand, recursively, any ‘listing’ fields in a Directory.

Parameters:
Return type:

None

cwltool.process.normalizeFilesDirs(job)
Parameters:

job (Optional[Union[MutableSequence[MutableMapping[str, Any]], MutableMapping[str, Any], DirectoryType]])

Return type:

None

cwltool.process.random_outdir()

Return the random directory name chosen to use for tool / workflow output.

Return type:

str

cwltool.process.visit_class(rec, cls, op)

Apply a function to with “class” in cls.

Parameters:
  • rec (Any)

  • cls (Iterable[Any])

  • op (Callable[Ellipsis, Any])

Return type:

None

cwltool.process.validate_js_expressions(tool, schema, jshint_options=None, container_engine='docker', eval_timeout=60)
Parameters:
Return type:

None

class cwltool.process.LogAsDebugFilter(name, parent)

Bases: logging.Filter

Inheritance diagram of cwltool.process.LogAsDebugFilter

Filter instances are used to perform arbitrary filtering of LogRecords.

Loggers and Handlers can optionally use Filter instances to filter records as desired. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with “A.B” will allow events logged by loggers “A.B”, “A.B.C”, “A.B.C.D”, “A.B.D” etc. but not “A.BB”, “B.A.B” etc. If initialized with the empty string, all events are passed.

Parameters:
filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

Parameters:

record (logging.LogRecord)

Return type:

bool

cwltool.process.supportedProcessRequirements
cwltool.process.cwl_files = ('Workflow.yml', 'CommandLineTool.yml', 'CommonWorkflowLanguage.yml', 'Process.yml',...
cwltool.process.salad_files = ('metaschema.yml', 'metaschema_base.yml', 'salad.md', 'field_name.yml', 'import_include.md',...
cwltool.process.SCHEMA_CACHE: Dict[str, Tuple[schema_salad.ref_resolver.Loader, schema_salad.avro.schema.Names | schema_salad.avro.schema.SchemaParseException, cwltool.utils.CWLObjectType, schema_salad.ref_resolver.Loader]]
cwltool.process.SCHEMA_FILE: cwltool.utils.CWLObjectType | None = None
cwltool.process.SCHEMA_DIR: cwltool.utils.CWLObjectType | None = None
cwltool.process.SCHEMA_ANY: cwltool.utils.CWLObjectType | None = None
cwltool.process.custom_schemas: Dict[str, Tuple[str, str]]
cwltool.process.use_standard_schema(version)
Parameters:

version (str)

Return type:

None

cwltool.process.use_custom_schema(version, name, text)
Parameters:
Return type:

None

cwltool.process.get_schema(version)
Parameters:

version (str)

Return type:

Tuple[schema_salad.ref_resolver.Loader, Union[schema_salad.avro.schema.Names, schema_salad.avro.schema.SchemaParseException], cwltool.utils.CWLObjectType, schema_salad.ref_resolver.Loader]

cwltool.process.shortname(inputid)
Parameters:

inputid (str)

Return type:

str

cwltool.process.stage_files(pathmapper, stage_func=None, ignore_writable=False, symlink=True, secret_store=None, fix_conflicts=False)

Link or copy files to their targets. Create them as needed.

Raises:

WorkflowException – if there is a file staging conflict

Parameters:
Return type:

None

cwltool.process.relocateOutputs(outputObj, destination_path, source_directories, action, fs_access, compute_checksum=True, path_mapper=PathMapper)
Parameters:
Return type:

cwltool.utils.CWLObjectType

cwltool.process.cleanIntermediate(output_dirs)
Parameters:

output_dirs (Iterable[str])

Return type:

None

cwltool.process.add_sizes(fsaccess, obj)
Parameters:
Return type:

None

cwltool.process.fill_in_defaults(inputs, job, fsaccess)

For each missing input in the input object, copy over the default.

Raises:

WorkflowException – if a required input parameter is missing

Parameters:
Return type:

None

cwltool.process.avroize_type(field_type, name_prefix='')

Add missing information to a type so that CWL types are valid.

Parameters:
  • field_type (Union[cwltool.utils.CWLObjectType, MutableSequence[Any], cwltool.utils.CWLOutputType, None])

  • name_prefix (str)

Return type:

Union[cwltool.utils.CWLObjectType, MutableSequence[Any], cwltool.utils.CWLOutputType, None]

cwltool.process.get_overrides(overrides, toolid)

Combine overrides for the target tool ID.

Parameters:
  • overrides (MutableSequence[cwltool.utils.CWLObjectType])

  • toolid (str)

Return type:

cwltool.utils.CWLObjectType

cwltool.process.var_spool_cwl_detector(obj, item=None, obj_key=None)

Detect any textual reference to /var/spool/cwl.

Parameters:
  • obj (cwltool.utils.CWLOutputType)

  • item (Optional[Any])

  • obj_key (Optional[Any])

Return type:

bool

cwltool.process.eval_resource(builder, resource_req)
Parameters:
Return type:

Optional[Union[str, int, float]]

cwltool.process.FILE_COUNT_WARNING = 5000
class cwltool.process.Process(toolpath_object, loadingContext)

Bases: cwltool.utils.HasReqsHints

Inheritance diagram of cwltool.process.Process

Abstract CWL Process.

Parameters:
evalResources(builder, runtimeContext)
Parameters:
Return type:

Dict[str, Union[int, float]]

checkRequirements(rec, supported_process_requirements)

Check the presence of unsupported requirements.

Parameters:
  • rec (Union[MutableSequence[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType, cwltool.utils.CWLOutputType, None])

  • supported_process_requirements (Iterable[str])

Return type:

None

validate_hints(avsc_names, hints, strict)

Process the hints field.

Parameters:
Return type:

None

visit(op)
Parameters:

op (Callable[[ruamel.yaml.comments.CommentedMap], None])

Return type:

None

abstract job(job_order, output_callbacks, runtimeContext)
Parameters:
  • job_order (cwltool.utils.CWLObjectType)

  • output_callbacks (Optional[cwltool.utils.OutputCallbackType])

  • runtimeContext (cwltool.context.RuntimeContext)

Return type:

cwltool.utils.JobsGeneratorType

__str__()

Return the id of this CWL process.

Return type:

str

cwltool.process.uniquename(stem, names=None)
Parameters:
  • stem (str)

  • names (Optional[Set[str]])

Return type:

str

cwltool.process.nestdir(base, deps)
Parameters:
  • base (str)

  • deps (cwltool.utils.CWLObjectType)

Return type:

cwltool.utils.CWLObjectType

cwltool.process.mergedirs(listing)
Parameters:

listing (MutableSequence[cwltool.utils.CWLObjectType])

Return type:

MutableSequence[cwltool.utils.CWLObjectType]

cwltool.process.CWL_IANA = 'https://www.iana.org/assignments/media-types/application/cwl'
cwltool.process.scandeps(base, doc, reffields, urlfields, loadref, urljoin=urllib.parse.urljoin, nestdirs=True)

Search for external files references in a CWL document or input object.

Looks for objects with ‘class: File’ or ‘class: Directory’ and adds them to the list of dependencies.

Parameters:
  • base (str) – the base URL for relative references.

  • doc (Union[cwltool.utils.CWLObjectType, MutableSequence[cwltool.utils.CWLObjectType]]) – a CWL document or input object

  • urlfields (Set[str]) – added as a File dependency

  • reffields (Set[str]) – field name like a workflow step ‘run’; will be added as a dependency and also loaded (using the ‘loadref’ function) and recursively scanned for dependencies. Those dependencies will be added as secondary files to the primary file.

  • nestdirs (bool) – if true, create intermediate directory objects when a file is located in a subdirectory under the starting directory. This is so that if the dependencies are materialized, they will produce the same relative file system locations.

  • loadref (Callable[[str, str], Union[ruamel.yaml.comments.CommentedMap, ruamel.yaml.comments.CommentedSeq, str, None]])

  • urljoin (Callable[[str, str], str])

Returns:

A list of File or Directory dependencies

Return type:

MutableSequence[cwltool.utils.CWLObjectType]

cwltool.process.compute_checksums(fs_access, fileobj)
Parameters:
Return type:

None