cwltool.singularity

Support for executing Docker format containers using Singularity {2,3}.x or Apptainer 1.x.

Attributes

MapperEnt

Mapper entries.

CWLObjectType

Typical raw dictionary found in lightly parsed CWL.

Exceptions

WorkflowException

Common base class for all non-exit exceptions.

Classes

Builder

Helper class to construct a command line from a CWL CommandLineTool.

RuntimeContext

Shared kwargs based initializer for RuntimeContext and LoadingContext.

DockerCommandLineJob

Runs a CommandLineJob in a software container using the Docker engine.

ContainerCommandLineJob

Commandline job using containers.

PathMapper

Mapping of files from relative path provided in the file to a tuple.

SingularityCommandLineJob

Commandline job using containers.

Functions

singularity_supports_userns()

Confirm if the version of Singularity install supports the --userns flag.

create_tmp_dir(tmpdir_prefix)

Create a temporary directory that respects the given tmpdir_prefix.

ensure_non_writable(path)

Attempt to change permissions to ensure that a path is not writable.

ensure_writable(path[, include_root])

Ensure that 'path' is writable.

get_version()

Parse the output of 'singularity --version' to determine the flavor and version.

is_apptainer_1_or_newer()

Check if apptainer singularity distribution is version 1.0 or higher.

is_version_2_6()

Check if this singularity version is exactly version 2.6.

is_version_3_or_newer()

Check if this version is singularity version 3 or newer or equivalent.

is_version_3_1_or_newer()

Check if this version is singularity version 3.1 or newer or equivalent.

is_version_3_4_or_newer()

Detect if Singularity v3.4+ is available.

is_version_3_9_or_newer()

Detect if Singularity v3.9+ is available.

Module Contents

class cwltool.singularity.Builder(job, files, bindings, schemaDefs, names, requirements, hints, resources, mutation_manager, formatgraph, make_fs_access, fs_access, job_script_provider, timeout, debug, js_console, force_docker_pull, loadListing, outdir, tmpdir, stagedir, cwlVersion, container_engine)

Bases: cwltool.utils.HasReqsHints

Inheritance diagram of cwltool.singularity.Builder

Helper class to construct a command line from a CWL CommandLineTool.

Parameters:
build_job_script(commands)
Parameters:

commands (List[str])

Return type:

Optional[str]

bind_input(schema, datum, discover_secondaryFiles, lead_pos=None, tail_pos=None)

Bind an input object to the command line.

Raises:
  • ValidationException – in the event of an invalid type union

  • WorkflowException – if a CWL Expression (“position”, “required”, “pattern”, “format”) evaluates to the wrong type or if a required secondary file is missing

Parameters:
  • schema (cwltool.utils.CWLObjectType)

  • datum (Union[cwltool.utils.CWLObjectType, List[cwltool.utils.CWLObjectType]])

  • discover_secondaryFiles (bool)

  • lead_pos (Optional[Union[int, List[int]]])

  • tail_pos (Optional[Union[str, List[int]]])

Return type:

List[MutableMapping[str, Union[str, List[int]]]]

tostr(value)

Represent an input parameter as a string.

Raises:

WorkflowException – if the item is a File or Directory and the “path” is missing.

Parameters:

value (Union[MutableMapping[str, str], Any])

Return type:

str

generate_arg(binding)
Parameters:

binding (cwltool.utils.CWLObjectType)

Return type:

List[str]

do_eval(ex, context=None, recursive=False, strip_whitespace=True)
Parameters:
  • ex (Optional[cwltool.utils.CWLOutputType])

  • context (Optional[Any])

  • recursive (bool)

  • strip_whitespace (bool)

Return type:

Optional[cwltool.utils.CWLOutputType]

class cwltool.singularity.RuntimeContext(kwargs=None)

Bases: ContextBase

Inheritance diagram of cwltool.singularity.RuntimeContext

Shared kwargs based initializer for RuntimeContext and LoadingContext.

Parameters:

kwargs (Optional[Dict[str, Any]])

outdir: str | None = None
tmpdir: str = ''
tmpdir_prefix: str
tmp_outdir_prefix: str = ''
stagedir: str = ''
get_outdir()

Return outdir or create one with tmp_outdir_prefix.

Return type:

str

get_tmpdir()

Return tmpdir or create one with tmpdir_prefix.

Return type:

str

get_stagedir()

Return stagedir or create one with tmpdir_prefix.

Return type:

str

create_tmpdir()

Create a temporary directory that respects tmpdir_prefix.

Return type:

str

create_outdir()

Create a temporary directory that respects tmp_outdir_prefix.

Return type:

str

copy()

Return a copy of this RuntimeContext.

Return type:

RuntimeContext

class cwltool.singularity.DockerCommandLineJob(builder, joborder, make_path_mapper, requirements, hints, name)

Bases: cwltool.job.ContainerCommandLineJob

Inheritance diagram of cwltool.singularity.DockerCommandLineJob

Runs a CommandLineJob in a software container using the Docker engine.

Parameters:
get_image(docker_requirement, pull_image, force_pull, tmp_outdir_prefix)

Retrieve the relevant Docker container image.

Returns:

True upon success

Parameters:
  • docker_requirement (Dict[str, str])

  • pull_image (bool)

  • force_pull (bool)

  • tmp_outdir_prefix (str)

Return type:

bool

get_from_requirements(r, pull_image, force_pull, tmp_outdir_prefix)
Parameters:
  • r (cwltool.utils.CWLObjectType)

  • pull_image (bool)

  • force_pull (bool)

  • tmp_outdir_prefix (str)

Return type:

Optional[str]

static append_volume(runtime, source, target, writable=False, skip_mkdirs=False)

Add binding arguments to the runtime list.

Parameters:
Return type:

None

add_file_or_directory_volume(runtime, volume, host_outdir_tgt)

Append volume a file/dir mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

Return type:

None

add_writable_file_volume(runtime, volume, host_outdir_tgt, tmpdir_prefix)

Append a writable file mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

  • tmpdir_prefix (str)

Return type:

None

add_writable_directory_volume(runtime, volume, host_outdir_tgt, tmpdir_prefix)

Append a writable directory mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

  • tmpdir_prefix (str)

Return type:

None

create_runtime(env, runtimeContext)

Return the list of commands to run the selected container engine.

Parameters:
Return type:

Tuple[List[str], Optional[str]]

exception cwltool.singularity.WorkflowException

Bases: Exception

Inheritance diagram of cwltool.singularity.WorkflowException

Common base class for all non-exit exceptions.

class cwltool.singularity.ContainerCommandLineJob(builder, joborder, make_path_mapper, requirements, hints, name)

Bases: JobBase

Inheritance diagram of cwltool.singularity.ContainerCommandLineJob

Commandline job using containers.

Parameters:
CONTAINER_TMPDIR: str = '/tmp'
abstract get_from_requirements(r, pull_image, force_pull, tmp_outdir_prefix)
Parameters:
  • r (cwltool.utils.CWLObjectType)

  • pull_image (bool)

  • force_pull (bool)

  • tmp_outdir_prefix (str)

Return type:

Optional[str]

abstract create_runtime(env, runtime_context)

Return the list of commands to run the selected container engine.

Parameters:
Return type:

Tuple[List[str], Optional[str]]

static append_volume(runtime, source, target, writable=False)
Abstractmethod:

Parameters:
Return type:

None

Add binding arguments to the runtime list.

abstract add_file_or_directory_volume(runtime, volume, host_outdir_tgt)

Append volume a file/dir mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

Return type:

None

abstract add_writable_file_volume(runtime, volume, host_outdir_tgt, tmpdir_prefix)

Append a writable file mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

  • tmpdir_prefix (str)

Return type:

None

abstract add_writable_directory_volume(runtime, volume, host_outdir_tgt, tmpdir_prefix)

Append a writable directory mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

  • tmpdir_prefix (str)

Return type:

None

create_file_and_add_volume(runtime, volume, host_outdir_tgt, secret_store, tmpdir_prefix)

Create the file and add a mapping.

Parameters:
Return type:

str

add_volumes(pathmapper, runtime, tmpdir_prefix, secret_store=None, any_path_okay=False)

Append volume mappings to the runtime option list.

Parameters:
Return type:

None

run(runtimeContext, tmpdir_lock=None)
Parameters:
Return type:

None

docker_monitor(cidfile, tmpdir_prefix, cleanup_cidfile, docker_exe, process)

Record memory usage of the running Docker container.

Parameters:
Return type:

None

cwltool.singularity.MapperEnt

Mapper entries.

cwltool.singularity.resolved: str

The “real” path on the local file system (after resolving relative paths and traversing symlinks

cwltool.singularity.target: str

The path on the target file system (under stagedir)

cwltool.singularity.type: str

The object type. One of “File”, “Directory”, “CreateFile”, “WritableFile”, or “CreateWritableFile”.

cwltool.singularity.staged: bool

If the File has been staged yet

class cwltool.singularity.PathMapper(referenced_files, basedir, stagedir, separateDirs=True)

Mapping of files from relative path provided in the file to a tuple.

(absolute local path, absolute container path)

The tao of PathMapper:

The initializer takes a list of class: File and class: Directory objects, a base directory (for resolving relative references) and a staging directory (where the files are mapped to).

The purpose of the setup method is to determine where each File or Directory should be placed on the target file system (relative to stagedir).

If separatedirs=True, unrelated files will be isolated in their own directories under stagedir. If separatedirs=False, files and directories will all be placed in stagedir (with the possibility for name collisions…)

The path map maps the “location” of the input Files and Directory objects to a tuple (resolved, target, type). The “resolved” field is the “real” path on the local file system (after resolving relative paths and traversing symlinks). The “target” is the path on the target file system (under stagedir). The type is the object type (one of File, Directory, CreateFile, WritableFile, CreateWritableFile).

The latter three (CreateFile, WritableFile, CreateWritableFile) are used by InitialWorkDirRequirement to indicate files that are generated on the fly (CreateFile and CreateWritableFile, in this case “resolved” holds the file contents instead of the path because they file doesn’t exist) or copied into the output directory so they can be opened for update (“r+” or “a”) (WritableFile and CreateWritableFile).

Parameters:
  • referenced_files (List[cwltool.utils.CWLObjectType])

  • basedir (str)

  • stagedir (str)

  • separateDirs (bool)

visitlisting(listing, stagedir, basedir, copy=False, staged=False)
Parameters:
  • listing (List[cwltool.utils.CWLObjectType])

  • stagedir (str)

  • basedir (str)

  • copy (bool)

  • staged (bool)

Return type:

None

visit(obj, stagedir, basedir, copy=False, staged=False)
Parameters:
  • obj (cwltool.utils.CWLObjectType)

  • stagedir (str)

  • basedir (str)

  • copy (bool)

  • staged (bool)

Return type:

None

setup(referenced_files, basedir)
Parameters:
  • referenced_files (List[cwltool.utils.CWLObjectType])

  • basedir (str)

Return type:

None

mapper(src)
Parameters:

src (str)

Return type:

MapperEnt

files()

Return a dictionary keys view of locations.

Return type:

KeysView[str]

items()

Return a dictionary items view.

Return type:

ItemsView[str, MapperEnt]

items_exclude_children()

Return a dictionary items view minus any entries which are children of other entries.

Return type:

ItemsView[str, MapperEnt]

reversemap(target)

Find the (source, resolved_path) for the given target, if any.

Parameters:

target (str)

Return type:

Optional[Tuple[str, str]]

update(key, resolved, target, ctype, stage)

Update an existine entry.

Parameters:
Return type:

MapperEnt

__contains__(key)

Test for the presence of the given relative path in this mapper.

Parameters:

key (str)

Return type:

bool

__iter__()

Get iterator for the maps.

Return type:

Iterator[MapperEnt]

cwltool.singularity.singularity_supports_userns()

Confirm if the version of Singularity install supports the –userns flag.

Return type:

bool

cwltool.singularity.CWLObjectType

Typical raw dictionary found in lightly parsed CWL.

cwltool.singularity.create_tmp_dir(tmpdir_prefix)

Create a temporary directory that respects the given tmpdir_prefix.

Parameters:

tmpdir_prefix (str)

Return type:

str

cwltool.singularity.ensure_non_writable(path)

Attempt to change permissions to ensure that a path is not writable.

Parameters:

path (str)

Return type:

None

cwltool.singularity.ensure_writable(path, include_root=False)

Ensure that ‘path’ is writable.

If ‘path’ is a directory, then all files and directories under ‘path’ are made writable, recursively. If ‘path’ is a file or if ‘include_root’ is True, then ‘path’ itself is made writable.

Parameters:
Return type:

None

cwltool.singularity.get_version()

Parse the output of ‘singularity –version’ to determine the flavor and version.

Both pieces of information will be cached.

Returns:

A tuple containing: - A tuple with major and minor version numbers as integer. - A string with the name of the singularity flavor.

Return type:

Tuple[List[int], str]

cwltool.singularity.is_apptainer_1_or_newer()

Check if apptainer singularity distribution is version 1.0 or higher.

Apptainer v1.0.0 is compatible with SingularityCE 3.9.5. See: https://github.com/apptainer/apptainer/releases

Return type:

bool

cwltool.singularity.is_version_2_6()

Check if this singularity version is exactly version 2.6.

Also returns False if the flavor is not singularity or singularity-ce.

Return type:

bool

cwltool.singularity.is_version_3_or_newer()

Check if this version is singularity version 3 or newer or equivalent.

Return type:

bool

cwltool.singularity.is_version_3_1_or_newer()

Check if this version is singularity version 3.1 or newer or equivalent.

Return type:

bool

cwltool.singularity.is_version_3_4_or_newer()

Detect if Singularity v3.4+ is available.

Return type:

bool

cwltool.singularity.is_version_3_9_or_newer()

Detect if Singularity v3.9+ is available.

Return type:

bool

class cwltool.singularity.SingularityCommandLineJob(builder, joborder, make_path_mapper, requirements, hints, name)

Bases: cwltool.job.ContainerCommandLineJob

Inheritance diagram of cwltool.singularity.SingularityCommandLineJob

Commandline job using containers.

Parameters:
static get_image(dockerRequirement, pull_image, tmp_outdir_prefix, force_pull=False)

Acquire the software container image in the specified dockerRequirement.

Uses Singularity and returns the success as a bool. Updates the provided dockerRequirement with the specific dockerImageId to the full path of the local image, if found. Likewise the dockerRequirement[‘dockerPull’] is updated to a docker:// URI if needed.

Parameters:
  • dockerRequirement (Dict[str, str])

  • pull_image (bool)

  • tmp_outdir_prefix (str)

  • force_pull (bool)

Return type:

bool

get_from_requirements(r, pull_image, force_pull, tmp_outdir_prefix)

Return the filename of the Singularity image.

(e.g. hello-world-latest.{img,sif}).

Parameters:
  • r (cwltool.utils.CWLObjectType)

  • pull_image (bool)

  • force_pull (bool)

  • tmp_outdir_prefix (str)

Return type:

Optional[str]

static append_volume(runtime, source, target, writable=False)

Add binding arguments to the runtime list.

Parameters:
Return type:

None

add_file_or_directory_volume(runtime, volume, host_outdir_tgt)

Append volume a file/dir mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

Return type:

None

add_writable_file_volume(runtime, volume, host_outdir_tgt, tmpdir_prefix)

Append a writable file mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

  • tmpdir_prefix (str)

Return type:

None

add_writable_directory_volume(runtime, volume, host_outdir_tgt, tmpdir_prefix)

Append a writable directory mapping to the runtime option list.

Parameters:
  • runtime (List[str])

  • volume (cwltool.pathmapper.MapperEnt)

  • host_outdir_tgt (Optional[str])

  • tmpdir_prefix (str)

Return type:

None

create_runtime(env, runtime_context)

Return the Singularity runtime list of commands and options.

Parameters:
Return type:

Tuple[List[str], Optional[str]]