cwltool.process =============== .. py:module:: cwltool.process .. autoapi-nested-parse:: Classes and methods relevant for all CWL Process types. Attributes ---------- .. autoapisummary:: cwltool.process.supportedProcessRequirements cwltool.process.cwl_files cwltool.process.salad_files cwltool.process.SCHEMA_CACHE cwltool.process.SCHEMA_FILE cwltool.process.SCHEMA_DIR cwltool.process.SCHEMA_ANY cwltool.process.custom_schemas cwltool.process.FILE_COUNT_WARNING cwltool.process.CWL_IANA Classes ------- .. autoapisummary:: cwltool.process.LogAsDebugFilter cwltool.process.Process Functions --------- .. autoapisummary:: cwltool.process.use_standard_schema cwltool.process.use_custom_schema cwltool.process.get_schema cwltool.process.shortname cwltool.process.stage_files cwltool.process.relocateOutputs cwltool.process.cleanIntermediate cwltool.process.add_sizes cwltool.process.fill_in_defaults cwltool.process.avroize_type cwltool.process.get_overrides cwltool.process.var_spool_cwl_detector cwltool.process.eval_resource cwltool.process.uniquename cwltool.process.nestdir cwltool.process.mergedirs cwltool.process.scandeps cwltool.process.compute_checksums Module Contents --------------- .. py:class:: LogAsDebugFilter(name, parent) Bases: :py:obj:`logging.Filter` .. autoapi-inheritance-diagram:: cwltool.process.LogAsDebugFilter :parts: 1 Filter instances are used to perform arbitrary filtering of LogRecords. Loggers and Handlers can optionally use Filter instances to filter records as desired. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with "A.B" will allow events logged by loggers "A.B", "A.B.C", "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with the empty string, all events are passed. .. py:attribute:: parent .. py:method:: filter(record) Determine if the specified record is to be logged. Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place. .. py:data:: supportedProcessRequirements .. py:data:: cwl_files :value: ('Base.yml', 'Workflow.yml', 'CommandLineTool.yml', 'CommonWorkflowLanguage.yml', 'Process.yml',... .. py:data:: salad_files :value: ('metaschema.yml', 'metaschema_base.yml', 'salad.md', 'field_name.yml', 'import_include.md',... .. py:data:: SCHEMA_CACHE :type: dict[str, tuple[schema_salad.ref_resolver.Loader, schema_salad.avro.schema.Names | schema_salad.avro.schema.SchemaParseException, cwl_utils.types.CWLObjectType, schema_salad.ref_resolver.Loader]] .. py:data:: SCHEMA_FILE :type: cwl_utils.types.CWLObjectType | None :value: None .. py:data:: SCHEMA_DIR :type: cwl_utils.types.CWLObjectType | None :value: None .. py:data:: SCHEMA_ANY :type: cwl_utils.types.CWLObjectType | None :value: None .. py:data:: custom_schemas :type: dict[str, tuple[str, str]] .. py:function:: use_standard_schema(version) .. py:function:: use_custom_schema(version, name, text) .. py:function:: get_schema(version) .. py:function:: shortname(inputid) .. py:function:: stage_files(pathmapper, stage_func = None, ignore_writable = False, symlink = True, secret_store = None, fix_conflicts = False) Link or copy files to their targets. Create them as needed. :raises WorkflowException: if there is a file staging conflict .. py:function:: relocateOutputs(outputObj, destination_path, source_directories, action, fs_access, compute_checksum = True, path_mapper = PathMapper) .. py:function:: cleanIntermediate(output_dirs) .. py:function:: add_sizes(fsaccess, obj) .. py:function:: fill_in_defaults(inputs, job, fsaccess) For each missing input in the input object, copy over the default. :raises WorkflowException: if a required input parameter is missing .. py:function:: avroize_type(field_type, name_prefix = '') Add missing information to a type so that CWL types are valid. .. py:function:: get_overrides(overrides, toolid) Combine overrides for the target tool ID. .. py:function:: var_spool_cwl_detector(obj, item = None, obj_key = None) Detect any textual reference to /var/spool/cwl. .. py:function:: eval_resource(builder, resource_req) Evaluate any CWL expressions inside a ResourceRequirement. .. py:data:: FILE_COUNT_WARNING :value: 5000 .. py:class:: Process(toolpath_object, loadingContext) Bases: :py:obj:`cwltool.utils.HasReqsHints` .. autoapi-inheritance-diagram:: cwltool.process.Process :parts: 1 Abstract CWL Process. .. py:attribute:: metadata :type: cwl_utils.types.CWLObjectType .. py:attribute:: provenance_object :type: Optional[cwltool.cwlprov.provenance_profile.ProvenanceProfile] :value: None .. py:attribute:: parent_wf :type: Optional[cwltool.cwlprov.provenance_profile.ProvenanceProfile] :value: None .. py:attribute:: names .. py:attribute:: tool .. py:attribute:: requirements .. py:attribute:: hints .. py:attribute:: original_requirements .. py:attribute:: original_hints .. py:attribute:: doc_loader .. py:attribute:: doc_schema .. py:attribute:: formatgraph :type: rdflib.Graph | None :value: None .. py:attribute:: schemaDefs :type: collections.abc.MutableMapping[str, cwl_utils.types.CWLObjectType] .. py:attribute:: inputs_record_schema :type: cwl_utils.types.CWLObjectType .. py:attribute:: outputs_record_schema :type: cwl_utils.types.CWLObjectType .. py:attribute:: container_engine :value: 'docker' .. py:method:: evalResources(builder, runtimeContext) .. py:method:: checkRequirements(rec, supported_process_requirements) Check the presence of unsupported requirements. .. py:method:: validate_hints(avsc_names, hints, strict) Process the hints field. .. py:method:: visit(op) .. py:method:: job(job_order, output_callbacks, runtimeContext) :abstractmethod: .. py:method:: __str__() Return the id of this CWL process. .. py:function:: uniquename(stem, names = None) Construct a thread-unique name using the given stem as a prefix. .. py:function:: nestdir(base, deps) Add intermediate directory objects to preserve the relative layout. .. py:function:: mergedirs(listing) .. py:data:: CWL_IANA :value: 'https://www.iana.org/assignments/media-types/application/cwl' .. py:function:: scandeps(base, doc, reffields, urlfields, loadref, urljoin = urllib.parse.urljoin, nestdirs = True) Search for external files references in a CWL document or input object. Looks for objects with 'class: File' or 'class: Directory' and adds them to the list of dependencies. :param base: the base URL for relative references. :param doc: a CWL document or input object :param urlfields: added as a File dependency :param reffields: field name like a workflow step 'run'; will be added as a dependency and also loaded (using the 'loadref' function) and recursively scanned for dependencies. Those dependencies will be added as secondary files to the primary file. :param nestdirs: if true, create intermediate directory objects when a file is located in a subdirectory under the starting directory. This is so that if the dependencies are materialized, they will produce the same relative file system locations. :returns: A list of File or Directory dependencies .. py:function:: compute_checksums(fs_access, fileobj)