spacr.data_manager¶

Workflow inputs and outputs¶

Data Manager¶

Inspect project files and disk usage. Removing derived artifacts invalidates downstream uses of those files; original images must remain recoverable.

Open: the application’s Help/tools menus.

Inputs and outputs below include conditional alternatives. The guidance and handoff notes say which route applies.

Inputs

Run history and artifacts — Project run records, settings, output paths, artifact provenance, status and logs.

Outputs

Run history and artifacts — Project run records, settings, output paths, artifact provenance, status and logs.

API reference.

Module tutorial.

What a project costs in disk, and how to get it back without losing data.

An imaging screen is mostly intermediates. stack/ is the raw files re-shaped, merged/ is stack/ with the masks concatenated on, data/ is one PNG per object, and a plate that arrives as 40 GB of TIFFs leaves 400 GB behind. A multi-terabyte project has no in-app answer to “where did the space go?”, and no way to clear derived data that does not also risk the one thing on the disk nobody can make again — the images off the microscope.

This module answers both, and it answers the second one conservatively.

Disk usage¶

scan_project() walks the project once and reports bytes per artifact kind — raw images, channel stacks, merged arrays, masks, crops, databases, model weights — measured from the filesystem, not from what the registry happens to have recorded. Every file is reconciled against spacr.artifacts.Registry:

a file under a registered artifact’s path is attributed to that artifact’s kind;
a file under none of them is unregistered — its bytes are reported (and labelled by the project layout, as a guess, so the number is readable) but its provenance is unknown;
a registered artifact whose path is no longer on disk is reported as missing, because a registry that claims files that are gone is a registry nobody should prune from.

Pruning¶

The safety property is the whole feature. An artifact is prunable only when it is regenerable, and “regenerable” is not a guess:

the registry knows it — anything unregistered is never a candidate;
its kind is not an original (ORIGINAL_KINDS): nothing in spacr.ports declares that it produces raw images or sequencing reads, so nothing can make them again;
a declared producing module exists for its kind, and the module the registry recorded is one of them;
that module’s inputs are complete — every recorded input artifact is still registered and still on disk, and spacr.ports.check_ready() says the module could run in this project right now;
it was written by a run that finished (status == "complete");
what is on disk still fingerprints to what was registered, so a folder somebody has since dropped a file into is not prunable;
it lives inside the project, reached without following a symlink;
every artifact sharing its path passes all of the above — three kinds live in measurements.db, and deleting the file for one of them destroys the other two;
no other registered artifact sits inside it or around it, because the bytes belong to the innermost one and a plan that under-reports what it deletes is the failure this module exists to prevent.

Anything that fails any of those is kept, and PrunePlan.kept says which rule kept it. Unknown provenance always means keep.

Rule 3 is doing more work than it looks. ports.producers_of is empty for raw-images and sequencing-reads, which is why originals are safe — and it is empty today for channel-stacks too, so stack/ is reported as unregistered bytes and never offered, even though it is one of the largest intermediates. That is the correct answer while nothing declares it produces them: the fix is a port declaration, not an exception here.

Nothing is deleted without a plan. plan_prune() returns exactly what would go and how much it frees; prune() refuses to run unless it is handed PrunePlan.token, a digest over that exact path set and byte total, so a confirmation cannot authorise a deletion other than the one that was shown.

Deleting is count first, delete second, verify. The tree is re-fingerprinted against the plan before anything is removed; the registry write is a count and a write on one predicate, checked for equality and rolled back on any difference. See _verified_write() for why that is the only property worth asserting here.

Archiving¶

plan_archive() and archive() move a project, or a subset of it, somewhere else and leave a record: a manifest at the destination, a ledger at the origin, and rows in the destination’s registry carrying the provenance the artifacts arrived with. The registry still knows where everything went.

Public API¶

scan_project, ProjectUsage, KindUsage, format_usage: Where the space went.
plan_prune, PrunePlan, PruneCandidate, PruneSkip, format_prune_plan, prune, PruneResult: What can safely go, and the deletion that is gated on it.
plan_archive, ArchivePlan, archive, ArchiveResult: Moving it, with a record.
is_prunable: The predicate, on its own, for anything that wants to ask.

Exceptions¶

`ArchiveError`	An archive could not be carried out, or could not be verified.
`ConfirmationRequired`	A destructive call arrived without the plan's confirmation token.
`DataManagerError`	Anything this module refuses to do.
`PruneAborted`	A prune stopped before removing anything, and nothing was removed.
`PruneIncomplete`	A prune deleted some of the plan and could not finish it.

Classes¶

`ArchiveItem`	One top-level entry an archive would move.
`ArchivePlan`	What an archive would move, and where to.
`ArchiveResult`	What an archive actually did.
`ArtifactUsage`	One registered artifact, measured on disk.
`KindUsage`	What one artifact kind costs in this project.
`ProjectUsage`	Where a project's disk went, reconciled against the registry.
`PruneCandidate`	One thing a prune would delete.
`PrunePlan`	Exactly what a prune would delete, and what it would leave.
`PruneResult`	What a prune actually did.
`PruneSkip`	One thing that was considered and kept, and why.

Functions¶

`archive`(→ ArchiveResult)	Move a project somewhere else and leave a record of where it went.
`format_prune_plan`(→ str)	Render a `PrunePlan` as the text shown before a deletion.
`format_usage`(→ str)	Render a `ProjectUsage` as a block of text.
`human_bytes`(→ str)	Render a byte count the way a disk report should read.
`is_prunable`() → str)	Return `""` when `artifact` may be pruned, else the reason to keep.
`plan_archive`(→ ArchivePlan)	Work out what moving a project — or part of one — would move.
`plan_prune`(→ PrunePlan)	Work out what could be deleted, and prove it before deleting anything.
`prune`(→ PruneResult)	Carry out a plan. Irreversible, and gated on the plan being unchanged.
`scan_project`(→ ProjectUsage)	Measure a project and reconcile it against the artifact registry.