cwltool.process

Classes and methods relevant for all CWL Process types.

Module Contents

Classes

LogAsDebugFilter	Filter instances are used to perform arbitrary filtering of LogRecords.
Process	Base class for get_requirement().

Functions

use_standard_schema(version)
use_custom_schema(version, name, text)
get_schema(version)
shortname(inputid)
checkRequirements(rec, supported_process_requirements)
stage_files(pathmapper, stage_func = None, ignore_writable = False, symlink = True, secret_store = None, fix_conflicts = False)	Link or copy files to their targets. Create them as needed.
relocateOutputs(outputObj, destination_path, source_directories, action, fs_access, compute_checksum = True, path_mapper = PathMapper)
cleanIntermediate(output_dirs)
add_sizes(fsaccess, obj)
fill_in_defaults(inputs, job, fsaccess)
avroize_type(field_type, name_prefix = '')	Add missing information to a type so that CWL types are valid.
get_overrides(overrides, toolid)
var_spool_cwl_detector(obj, item = None, obj_key = None)	Detect any textual reference to /var/spool/cwl.
eval_resource(builder, resource_req)
uniquename(stem, names = None)
nestdir(base, deps)
mergedirs(listing)
scandeps(base, doc, reffields, urlfields, loadref, urljoin = urllib.parse.urljoin, nestdirs = True)	Given a CWL document or input object, search for dependencies
compute_checksums(fs_access, fileobj)

Attributes

supportedProcessRequirements
cwl_files
salad_files
SCHEMA_CACHE
SCHEMA_FILE
SCHEMA_DIR
SCHEMA_ANY
custom_schemas
FILE_COUNT_WARNING
CWL_IANA

class cwltool.process.LogAsDebugFilter(name, parent)

Bases: logging.Filter

Filter instances are used to perform arbitrary filtering of LogRecords.

Loggers and Handlers can optionally use Filter instances to filter records as desired. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with “A.B” will allow events logged by loggers “A.B”, “A.B.C”, “A.B.C.D”, “A.B.D” etc. but not “A.BB”, “B.A.B” etc. If initialized with the empty string, all events are passed.

Parameters

• name (str) –

• parent (logging.Logger) –

filter(self, record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

Parameters

record (logging.LogRecord) –

Return type

bool

cwltool.process.supportedProcessRequirements

cwltool.process.cwl_files = ['Workflow.yml', 'CommandLineTool.yml', 'CommonWorkflowLanguage.yml', 'Process.yml',...

cwltool.process.salad_files = ['metaschema.yml', 'metaschema_base.yml', 'salad.md', 'field_name.yml', 'import_include.md',...

cwltool.process.SCHEMA_CACHE :Dict[str, Tuple[Loader, Union[Names, SchemaParseException], CWLObjectType, Loader]]

cwltool.process.SCHEMA_FILE :Optional[CWLObjectType]

cwltool.process.SCHEMA_DIR :Optional[CWLObjectType]

cwltool.process.SCHEMA_ANY :Optional[CWLObjectType]

cwltool.process.custom_schemas :Dict[str, Tuple[str, str]]

cwltool.process.use_standard_schema(version)

Parameters

version (str) –

Return type

None

cwltool.process.use_custom_schema(version, name, text)

Parameters

• version (str) –

• name (str) –

• text (str) –

Return type

None

cwltool.process.get_schema(version)

Parameters

version (str) –

Return type

Tuple[schema_salad.ref_resolver.Loader, Union[schema_salad.avro.schema.Names, schema_salad.avro.schema.SchemaParseException], cwltool.utils.CWLObjectType, schema_salad.ref_resolver.Loader]

cwltool.process.shortname(inputid)

Parameters

inputid (str) –

Return type

str

cwltool.process.checkRequirements(rec, supported_process_requirements)

Parameters

• rec (Union[MutableSequence[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType, cwltool.utils.CWLOutputType, None]) –

• supported_process_requirements (Iterable[str]) –

Return type

None

cwltool.process.stage_files(pathmapper, stage_func=None, ignore_writable=False, symlink=True, secret_store=None, fix_conflicts=False)

Link or copy files to their targets. Create them as needed.

Parameters

• pathmapper (cwltool.pathmapper.PathMapper) –

• stage_func (Optional[Callable[[str, str], None]]) –

• ignore_writable (bool) –

• symlink (bool) –

• secret_store (Optional[cwltool.secrets.SecretStore]) –

• fix_conflicts (bool) –

Return type

None

cwltool.process.relocateOutputs(outputObj, destination_path, source_directories, action, fs_access, compute_checksum=True, path_mapper=PathMapper)

Parameters

• outputObj (cwltool.utils.CWLObjectType) –

• destination_path (str) –

• source_directories (Set[str]) –

• action (str) –

• fs_access (cwltool.stdfsaccess.StdFsAccess) –

• compute_checksum (bool) –

• path_mapper (Type[cwltool.pathmapper.PathMapper]) –

Return type

cwltool.utils.CWLObjectType

cwltool.process.cleanIntermediate(output_dirs)

Parameters

output_dirs (Iterable[str]) –

Return type

None

cwltool.process.add_sizes(fsaccess, obj)

Parameters

• fsaccess (cwltool.stdfsaccess.StdFsAccess) –

• obj (cwltool.utils.CWLObjectType) –

Return type

None

cwltool.process.fill_in_defaults(inputs, job, fsaccess)

Parameters

• inputs (List[cwltool.utils.CWLObjectType]) –

• job (cwltool.utils.CWLObjectType) –

• fsaccess (cwltool.stdfsaccess.StdFsAccess) –

Return type

None

cwltool.process.avroize_type(field_type, name_prefix='')

Add missing information to a type so that CWL types are valid.

Parameters

• field_type (Union[cwltool.utils.CWLObjectType, MutableSequence[Any], cwltool.utils.CWLOutputType, None]) –

• name_prefix (str) –

Return type

Union[cwltool.utils.CWLObjectType, MutableSequence[Any], cwltool.utils.CWLOutputType, None]

cwltool.process.get_overrides(overrides, toolid)

Parameters

• overrides (MutableSequence[cwltool.utils.CWLObjectType]) –

• toolid (str) –

Return type

cwltool.utils.CWLObjectType

cwltool.process.var_spool_cwl_detector(obj, item=None, obj_key=None)

Detect any textual reference to /var/spool/cwl.

Parameters

• obj (cwltool.utils.CWLOutputType) –

• item (Optional[Any]) –

• obj_key (Optional[Any]) –

Return type

bool

cwltool.process.eval_resource(builder, resource_req)

Parameters

• builder (cwltool.builder.Builder) –

• resource_req (Union[str, int, float]) –

Return type

Optional[Union[str, int, float]]

cwltool.process.FILE_COUNT_WARNING = 5000

class cwltool.process.Process(toolpath_object, loadingContext)

Bases: cwltool.utils.HasReqsHints

Base class for get_requirement().

Parameters

• toolpath_object (ruamel.yaml.comments.CommentedMap) –

• loadingContext (cwltool.context.LoadingContext) –

evalResources(self, builder, runtimeContext)

Parameters

• builder (cwltool.builder.Builder) –

• runtimeContext (cwltool.context.RuntimeContext) –

Return type

Dict[str, Union[int, float]]

validate_hints(self, avsc_names, hints, strict)

Parameters

• avsc_names (schema_salad.avro.schema.Names) –

• hints (List[cwltool.utils.CWLObjectType]) –

• strict (bool) –

Return type

None

visit(self, op)

Parameters

op (Callable[[ruamel.yaml.comments.CommentedMap], None]) –

Return type

None

abstract job(self, job_order, output_callbacks, runtimeContext)

Parameters

• job_order (cwltool.utils.CWLObjectType) –

• output_callbacks (Optional[cwltool.utils.OutputCallbackType]) –

• runtimeContext (cwltool.context.RuntimeContext) –

Return type

cwltool.utils.JobsGeneratorType

__str__(self)

Return the id of this CWL process.

Return type

str

cwltool.process.uniquename(stem, names=None)

Parameters

• stem (str) –

• names (Optional[Set[str]]) –

Return type

str

cwltool.process.nestdir(base, deps)

Parameters

• base (str) –

• deps (cwltool.utils.CWLObjectType) –

Return type

cwltool.utils.CWLObjectType

cwltool.process.mergedirs(listing)

Parameters

listing (MutableSequence[cwltool.utils.CWLObjectType]) –

Return type

MutableSequence[cwltool.utils.CWLObjectType]

cwltool.process.CWL_IANA = https://www.iana.org/assignments/media-types/application/cwl

cwltool.process.scandeps(base, doc, reffields, urlfields, loadref, urljoin=urllib.parse.urljoin, nestdirs=True)

Given a CWL document or input object, search for dependencies (references to external files) of ‘doc’ and return them as a list of File or Directory objects.

The ‘base’ is the base URL for relative references.

Looks for objects with ‘class: File’ or ‘class: Directory’ and adds them to the list of dependencies.

Anything in ‘urlfields’ is also added as a File dependency.

Anything in ‘reffields’ (such as workflow step ‘run’) will be added as a dependency and also loaded (using the ‘loadref’ function) and recursively scanned for dependencies. Those dependencies will be added as secondary files to the primary file.

If “nestdirs” is true, create intermediate directory objects when a file is located in a subdirectory under the starting directory. This is so that if the dependencies are materialized, they will produce the same relative file system locations.

Parameters

• base (str) –

• doc (Union[cwltool.utils.CWLObjectType, MutableSequence[cwltool.utils.CWLObjectType]]) –

• reffields (Set[str]) –

• urlfields (Set[str]) –

• loadref (Callable[[str, str], Union[ruamel.yaml.comments.CommentedMap, ruamel.yaml.comments.CommentedSeq, str, None]]) –

• urljoin (Callable[[str, str], str]) –

• nestdirs (bool) –

Return type

MutableSequence[cwltool.utils.CWLObjectType]

cwltool.process.compute_checksums(fs_access, fileobj)

Parameters

• fs_access (cwltool.stdfsaccess.StdFsAccess) –

• fileobj (cwltool.utils.CWLObjectType) –

Return type

None