spacr.remote_execution

Workflow inputs and outputs

Distributed Jobs

Submit a configured run to a remote execution target and inspect its status and logs. Retrieve the actual module artifacts before downstream analysis.

Open: the application’s Help/tools menus.

Inputs and outputs below include conditional alternatives. The guidance and handoff notes say which route applies.

Inputs

• Run queue — Saved module/plate/settings job definitions and dependency order.

Outputs

• Run history and artifacts — Project run records, settings, output paths, artifact provenance, status and logs.

API reference.

Module tutorial.

Persistent remote and distributed execution for spaCR pipelines.

The local GUI and spacr.cli already agree on one headless contract:

spacr-run MODULE --settings SETTINGS.json

This module transports that contract to a workstation over SSH, submits it to Slurm, or hands it to a user-configured cloud/HPC command. Submitted jobs are recorded locally and can be polled or cancelled after spaCR itself has closed.

No command uses shell=True. SSH necessarily invokes a remote login shell; all user-controlled values interpolated into its small fixed scripts are quoted with shlex.quote(), and host names are validated separately. Custom command profiles are parsed into argument vectors and substitute each placeholder inside one argument, so shell operators have no special meaning.

Exceptions

RemoteExecutionError

A profile, submission, polling, or cancellation error.

Classes

CommandResult	Result returned by the injectable command runner.
ExecutionProfile	Connection and scheduler settings for one execution target.
JobStore	Atomic persistent store for remote job metadata.
ProfileStore	Atomic persistent store for execution profiles.
RemoteJob	Persistent local record of one submitted job.
RemoteJobManager	Submit, monitor, cancel and inspect persistent remote jobs.

Functions

map_settings_paths(→ Any)	Recursively map absolute paths below one local root to a remote root.
state_directory(→ pathlib.Path)	Return the persistent directory for profiles, jobs, settings and logs.

Module Contents

exception spacr.remote_execution.RemoteExecutionError

Bases: RuntimeError

A profile, submission, polling, or cancellation error.

These errors are safe to show directly in the GUI: passwords and environment variables are never included in command rendering.

Initialize self. See help(type(self)) for accurate signature.

class spacr.remote_execution.CommandResult

Result returned by the injectable command runner.

Parameters:

• returncode – process exit status, where zero denotes success.

• stdout – captured standard-output text.

• stderr – captured standard-error text.

class spacr.remote_execution.ExecutionProfile

Connection and scheduler settings for one execution target.

local_root and remote_root describe the same shared or mirrored dataset. Every absolute string nested in the settings is rewritten when it lies below local_root. Image datasets are deliberately not copied: accidental recursive transfer of a multi-terabyte plate is worse than a clear pre-flight error.

For command profiles, command strings are tokenized with shlex.split() and support {job_id}, {module}, {settings} and {external_id} placeholders. They are argument templates, not shell scripts.

Parameters:

• name – human-readable profile name used to select this target.

• backend – execution mechanism, one of BACKENDS.

• host – OpenSSH target for SSH and remote Slurm; blank permits a Slurm scheduler running on the local host.

• workdir – absolute execution directory where spacr-jobs artifacts are created.

• local_root – local dataset prefix paired with remote_root for nested absolute-path rewriting.

• remote_root – equivalent dataset prefix visible to the target host.

• runner – headless executable invoked as runner module --settings path by SSH and Slurm backends.

• scheduler_options – additional shell-free arguments passed to sbatch.

• submit_command – custom-backend argument template containing {settings} and printing a job identifier.

• status_command – custom status template containing {external_id} and returning a state spaCR can normalize.

• cancel_command – custom cancellation template containing {external_id}.

• log_command – optional custom template for fetching job output.

• job_id_pattern – optional regular expression extracting a named id, first capture, or full match from submission output.

• poll_seconds – recommended client polling interval, from 2 through 3600 seconds.

classmethod from_dict(value: Mapping[str, Any]) → ExecutionProfile

Construct and validate a profile from JSON-compatible data.

Parameters:

value – stored profile fields keyed by their dataclass names.

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

validate() → ExecutionProfile

Validate the profile and return self for fluent callers.

class spacr.remote_execution.JobStore(path: os.PathLike | None = None)

Atomic persistent store for remote job metadata.

Parameters:

path – optional jobs JSON path; by default state_directory()/jobs.json. The advisory lock uses a sibling path with .lock appended.

Open the remote-job store.

Parameters:

path – where the jobs live; None uses the state directory. A sibling lock file is derived from it, so two processes writing jobs cannot interleave.

get(job_id: str) → RemoteJob

Return a job by full ID or unambiguous prefix.

Parameters:

job_id – complete job identifier or an unambiguous prefix.

list() → List[RemoteJob]

Return newest jobs first.

save(job: RemoteJob) → None

Insert or replace one job atomically.

Parameters:

job – remote-job record to persist.

class spacr.remote_execution.ProfileStore(path: os.PathLike | None = None)

Atomic persistent store for execution profiles.

Parameters:

path – optional profiles JSON path; by default state_directory()/profiles.json. The advisory lock uses a sibling path with .lock appended.

Open the execution-profile store.

Parameters:

path – where the profiles live; None uses the state directory. A sibling lock file is derived from it, so two processes writing profiles cannot interleave.

delete(name: str) → bool

Delete one profile; return whether it existed.

Parameters:

name – profile name to delete case-insensitively.

get(name: str) → ExecutionProfile

Return a named profile or raise a user-facing error.

Parameters:

name – profile name to match case-insensitively.

list() → List[ExecutionProfile]

Return profiles sorted by case-insensitive name.

save(profile: ExecutionProfile) → None

Insert or replace one profile atomically.

Parameters:

profile – validated execution profile to persist.

class spacr.remote_execution.RemoteJob

Persistent local record of one submitted job.

Parameters:

• job_id – locally generated stable identifier for the job.

• module – resolved spaCR command-line module key to execute.

• profile_name – name of the execution profile used for submission.

• backend – backend recorded at submission time.

• status – compact lifecycle state, initially "submitting".

• external_id – validated process, scheduler, or provider identifier.

• created_utc – ISO-8601 construction timestamp.

• updated_utc – timestamp replaced whenever JobStore persists the record.

• settings_path – retained local path of the mapped settings JSON.

• settings_sha256 – SHA-256 digest of the exact serialized settings.

• remote_settings_path – uploaded settings path for SSH and Slurm.

• remote_job_dir – target-side directory containing job artifacts.

• log_reference – backend log location or custom-backend marker.

• log_tail – most recently fetched output or availability message.

• exit_code – normalized process exit status, or None until known.

• error – latest submission, polling, or cancellation failure; cleared after a successful refresh or cancellation.

• profile – submission-time profile snapshot, retained so the job remains operable after profile edits or deletion.

classmethod from_dict(value: Mapping[str, Any]) → RemoteJob

Construct a job from a stored mapping, tolerating future fields.

Parameters:

value – stored job fields keyed by their dataclass names.

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

class spacr.remote_execution.RemoteJobManager(profile_store: ProfileStore | None = None, job_store: JobStore | None = None, runner: CommandRunner = _run_command)

Submit, monitor, cancel and inspect persistent remote jobs.

All methods are synchronous and may perform network I/O. GUI callers must invoke them on a worker thread; the shipped Distributed Jobs screen does.

Parameters:

• profile_store – injectable persistent profile store; omitted creates the default ProfileStore.

• job_store – injectable persistent job store; omitted creates the default JobStore.

• runner – shell-free callable accepting an argument vector, timeout, and optional input and returning CommandResult.

Create the manager over its two stores and a command runner.

Parameters:

• profile_store – where execution profiles are read from; None opens the default store.

• job_store – where submitted jobs are recorded; None opens the default store.

• runner – how commands are executed. Injectable so a test can drive the manager without running anything.

cancel(job_id: str) → RemoteJob

Request cancellation and persist the result.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

logs(job_id: str, lines: int = 200) → str

Retrieve and persist the tail of one remote job’s log.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

refresh(job_id: str, *, include_logs: bool = True) → RemoteJob

Poll one non-terminal job and optionally retain its latest log tail.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

refresh_all(*, include_logs: bool = False) → List[RemoteJob]

Poll every active job and return the complete newest-first list.

submit(module: str, settings: Mapping[str, Any], profile_name: str) → RemoteJob

Submit resolved settings through a named execution profile.

Parameters:

• module – headless spaCR module name accepted by spacr-run.

• settings – resolved module settings to serialize for the job.

• profile_name – name of the execution profile to use.

spacr.remote_execution.map_settings_paths(value: Any, local_root: str, remote_root: str) → Any

Recursively map absolute paths below one local root to a remote root.

Non-path strings and paths outside the configured root are unchanged. Mapping keys are intentionally preserved: setting names are not paths.

Parameters:

• value – nested settings value whose absolute path strings are mapped.

• local_root – local dataset root that mapped paths must lie below.

• remote_root – remote dataset root that replaces local_root.

spacr.remote_execution.state_directory() → pathlib.Path

Return the persistent directory for profiles, jobs, settings and logs.

SPACR_REMOTE_STATE_DIR is intentionally supported for tests, portable deployments, and managed lab installations. Otherwise XDG state storage is used on Linux and a conventional per-user directory elsewhere.