cwltool.process

Classes and methods relevant for all CWL Process types.

Module Contents

Classes

LogAsDebugFilter

Filter instances are used to perform arbitrary filtering of LogRecords.

Process

Base class for get_requirement().

Functions

use_standard_schema(version)

use_custom_schema(version, name, text)

get_schema(version)

shortname(inputid)

checkRequirements(rec, supported_process_requirements)

stage_files(pathmapper, stage_func = None, ignore_writable = False, symlink = True, secret_store = None, fix_conflicts = False)

Link or copy files to their targets. Create them as needed.

relocateOutputs(outputObj, destination_path, source_directories, action, fs_access, compute_checksum = True, path_mapper = PathMapper)

cleanIntermediate(output_dirs)

add_sizes(fsaccess, obj)

fill_in_defaults(inputs, job, fsaccess)

avroize_type(field_type, name_prefix = '')

Add missing information to a type so that CWL types are valid.

get_overrides(overrides, toolid)

var_spool_cwl_detector(obj, item = None, obj_key = None)

Detect any textual reference to /var/spool/cwl.

eval_resource(builder, resource_req)

uniquename(stem, names = None)

nestdir(base, deps)

mergedirs(listing)

scandeps(base, doc, reffields, urlfields, loadref, urljoin = urllib.parse.urljoin, nestdirs = True)

Given a CWL document or input object, search for dependencies

compute_checksums(fs_access, fileobj)

Attributes

supportedProcessRequirements

cwl_files

salad_files

SCHEMA_CACHE

SCHEMA_FILE

SCHEMA_DIR

SCHEMA_ANY

custom_schemas

FILE_COUNT_WARNING

CWL_IANA

class cwltool.process.LogAsDebugFilter(name, parent)

Bases: logging.Filter

Inheritance diagram of cwltool.process.LogAsDebugFilter

Filter instances are used to perform arbitrary filtering of LogRecords.

Loggers and Handlers can optionally use Filter instances to filter records as desired. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with “A.B” will allow events logged by loggers “A.B”, “A.B.C”, “A.B.C.D”, “A.B.D” etc. but not “A.BB”, “B.A.B” etc. If initialized with the empty string, all events are passed.

Parameters
  • name (str) –

  • parent (logging.Logger) –

filter(self, record)

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

Parameters

record (logging.LogRecord) –

Return type

bool

cwltool.process.supportedProcessRequirements
cwltool.process.cwl_files = ['Workflow.yml', 'CommandLineTool.yml', 'CommonWorkflowLanguage.yml', 'Process.yml',...
cwltool.process.salad_files = ['metaschema.yml', 'metaschema_base.yml', 'salad.md', 'field_name.yml', 'import_include.md',...
cwltool.process.SCHEMA_CACHE :Dict[str, Tuple[Loader, Union[Names, SchemaParseException], CWLObjectType, Loader]]
cwltool.process.SCHEMA_FILE :Optional[CWLObjectType]
cwltool.process.SCHEMA_DIR :Optional[CWLObjectType]
cwltool.process.SCHEMA_ANY :Optional[CWLObjectType]
cwltool.process.custom_schemas :Dict[str, Tuple[str, str]]
cwltool.process.use_standard_schema(version)
Parameters

version (str) –

Return type

None

cwltool.process.use_custom_schema(version, name, text)
Parameters
  • version (str) –

  • name (str) –

  • text (str) –

Return type

None

cwltool.process.get_schema(version)
Parameters

version (str) –

Return type

Tuple[schema_salad.ref_resolver.Loader, Union[schema_salad.avro.schema.Names, schema_salad.avro.schema.SchemaParseException], cwltool.utils.CWLObjectType, schema_salad.ref_resolver.Loader]

cwltool.process.shortname(inputid)
Parameters

inputid (str) –

Return type

str

cwltool.process.checkRequirements(rec, supported_process_requirements)
Parameters
  • rec (Union[MutableSequence[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType, cwltool.utils.CWLOutputType, None]) –

  • supported_process_requirements (Iterable[str]) –

Return type

None

cwltool.process.stage_files(pathmapper, stage_func=None, ignore_writable=False, symlink=True, secret_store=None, fix_conflicts=False)

Link or copy files to their targets. Create them as needed.

Parameters
Return type

None

cwltool.process.relocateOutputs(outputObj, destination_path, source_directories, action, fs_access, compute_checksum=True, path_mapper=PathMapper)
Parameters
Return type

cwltool.utils.CWLObjectType

cwltool.process.cleanIntermediate(output_dirs)
Parameters

output_dirs (Iterable[str]) –

Return type

None

cwltool.process.add_sizes(fsaccess, obj)
Parameters
Return type

None

cwltool.process.fill_in_defaults(inputs, job, fsaccess)
Parameters
Return type

None

cwltool.process.avroize_type(field_type, name_prefix='')

Add missing information to a type so that CWL types are valid.

Parameters
  • field_type (Union[cwltool.utils.CWLObjectType, MutableSequence[Any], cwltool.utils.CWLOutputType, None]) –

  • name_prefix (str) –

Return type

Union[cwltool.utils.CWLObjectType, MutableSequence[Any], cwltool.utils.CWLOutputType, None]

cwltool.process.get_overrides(overrides, toolid)
Parameters
  • overrides (MutableSequence[cwltool.utils.CWLObjectType]) –

  • toolid (str) –

Return type

cwltool.utils.CWLObjectType

cwltool.process.var_spool_cwl_detector(obj, item=None, obj_key=None)

Detect any textual reference to /var/spool/cwl.

Parameters
  • obj (cwltool.utils.CWLOutputType) –

  • item (Optional[Any]) –

  • obj_key (Optional[Any]) –

Return type

bool

cwltool.process.eval_resource(builder, resource_req)
Parameters
Return type

Optional[Union[str, int, float]]

cwltool.process.FILE_COUNT_WARNING = 5000
class cwltool.process.Process(toolpath_object, loadingContext)

Bases: cwltool.utils.HasReqsHints

Inheritance diagram of cwltool.process.Process

Base class for get_requirement().

Parameters
evalResources(self, builder, runtimeContext)
Parameters
Return type

Dict[str, Union[int, float]]

validate_hints(self, avsc_names, hints, strict)
Parameters
  • avsc_names (schema_salad.avro.schema.Names) –

  • hints (List[cwltool.utils.CWLObjectType]) –

  • strict (bool) –

Return type

None

visit(self, op)
Parameters

op (Callable[[ruamel.yaml.comments.CommentedMap], None]) –

Return type

None

abstract job(self, job_order, output_callbacks, runtimeContext)
Parameters
  • job_order (cwltool.utils.CWLObjectType) –

  • output_callbacks (Optional[cwltool.utils.OutputCallbackType]) –

  • runtimeContext (cwltool.context.RuntimeContext) –

Return type

cwltool.utils.JobsGeneratorType

__str__(self)

Return the id of this CWL process.

Return type

str

cwltool.process.uniquename(stem, names=None)
Parameters
  • stem (str) –

  • names (Optional[Set[str]]) –

Return type

str

cwltool.process.nestdir(base, deps)
Parameters
  • base (str) –

  • deps (cwltool.utils.CWLObjectType) –

Return type

cwltool.utils.CWLObjectType

cwltool.process.mergedirs(listing)
Parameters

listing (MutableSequence[cwltool.utils.CWLObjectType]) –

Return type

MutableSequence[cwltool.utils.CWLObjectType]

cwltool.process.CWL_IANA = https://www.iana.org/assignments/media-types/application/cwl
cwltool.process.scandeps(base, doc, reffields, urlfields, loadref, urljoin=urllib.parse.urljoin, nestdirs=True)

Given a CWL document or input object, search for dependencies (references to external files) of ‘doc’ and return them as a list of File or Directory objects.

The ‘base’ is the base URL for relative references.

Looks for objects with ‘class: File’ or ‘class: Directory’ and adds them to the list of dependencies.

Anything in ‘urlfields’ is also added as a File dependency.

Anything in ‘reffields’ (such as workflow step ‘run’) will be added as a dependency and also loaded (using the ‘loadref’ function) and recursively scanned for dependencies. Those dependencies will be added as secondary files to the primary file.

If “nestdirs” is true, create intermediate directory objects when a file is located in a subdirectory under the starting directory. This is so that if the dependencies are materialized, they will produce the same relative file system locations.

Parameters
  • base (str) –

  • doc (Union[cwltool.utils.CWLObjectType, MutableSequence[cwltool.utils.CWLObjectType]]) –

  • reffields (Set[str]) –

  • urlfields (Set[str]) –

  • loadref (Callable[[str, str], Union[ruamel.yaml.comments.CommentedMap, ruamel.yaml.comments.CommentedSeq, str, None]]) –

  • urljoin (Callable[[str, str], str]) –

  • nestdirs (bool) –

Return type

MutableSequence[cwltool.utils.CWLObjectType]

cwltool.process.compute_checksums(fs_access, fileobj)
Parameters
Return type

None