spacr.cli_download¶
spacr-download — fetch spaCR’s published example data, with no GUI.
Every example dataset spaCR publishes has, until now, been reachable only by pressing a button inside the application: “Load test data”, “Load example data”, the screen-data picker. That is fine on a laptop and useless on a cluster, where the data has to be on disk BEFORE a batch job starts and there is no display to press a button on. This is that download as a command.
WHAT IT WILL AND WILL NOT DO WITHOUT BEING ASKED. With no arguments it fetches
the example sets – Import, Mask, Measure, Annotate/Classify, Replication,
Recruitment and the synthetic Invasion set – which come to about 1.8 GB. The
OPS and Align & Stitch samples are fetched only when named (spacr-download
ops stitch) or with all. It does NOT fetch the published TSG101 screen,
which is 33 GB.
A command that spent 33 GB of somebody’s quota because they typed its name
with no arguments would be a bug however well documented, so the screen is
opt-in, is asked for in pieces, and is confirmed before it starts. The pieces
are spacr.screen_data’s, unchanged: four measurement databases of about
0.5 GB and four crop folders of about 8 GB, any subset of which can be named.
NOTHING IS DOWNLOADED THAT CANNOT FIRST BE PRICED. --list prints every
piece with its size and whether it is already on disk, and the total the
current selection would cost, without opening a socket. --dry-run is the
same listing for the same reason: a user deciding what to spend an hour of
network on should be able to see the bill first.
Importing this module must stay light – no Qt, no torch, no matplotlib – so
spacr-download --help answers instantly on a login node with a cold NFS
cache. That is why the download primitives live in
spacr.example_archives rather than in spacr.qt.hf_download, which
imports PySide6 at module scope. tests/test_cli_download.py pins it.
Usage:
spacr-download # every example set (~1.8 GB)
spacr-download --list # what exists, what is here
spacr-download measure annotate # two of the seven
spacr-download --screen measurements # the four databases (~2.1 GB)
spacr-download --screen crops --plate 1 # one plate of crops (~8.9 GB)
spacr-download all --yes # everything, screen included
Exit codes (a job that exits 0 having downloaded nothing is the classic footgun, so these are exact):
0 everything asked for is on disk, or a confirmation was declined
1 a download failed; the pieces that succeeded are still on disk
2 bad arguments, not enough disk space, or a large download that could not
be confirmed because nothing was there to confirm it
Exceptions¶
A name, kind or plate number the user got wrong. |
Classes¶
One archive this command can fetch, priced and located. |
Functions¶
|
Return the |
|
One |
|
|
|
|
|
Fetch every piece, and keep going after one fails. |
|
The one plate folder the three plate example sets unpack into. |
|
Download one archive, unpack it, and throw the archive away. |
|
|
|
The plan minus what is already on disk. |
|
Every piece, its size, whether it is here, and what the total would be. |
|
Turn what the user typed into the exact list of things to fetch. |
|
Complaint about free disk space, or |
|
One folder per screen plate, and it has to be. |
|
How to ask for the screen, for a run that did not. |
Module Contents¶
- exception spacr.cli_download.SelectionError[source]¶
Bases:
ExceptionA name, kind or plate number the user got wrong.
Always exit code 2: nothing was attempted, so it is an argument problem rather than a failed download.
Initialize self. See help(type(self)) for accurate signature.
- class spacr.cli_download.Piece[source]¶
One archive this command can fetch, priced and located.
Examples and screen pieces are described by two different dataclasses –
ExampleSetandScreenAsset– because they answer to two different publishers. This is the one shape the listing, the size arithmetic and the download loop all work in, so none of them has to know which kind of thing it is holding.- Parameters:
key – how a selection names it, unique across both publishers.
name – what the listing’s first column shows.
detail – the rest of the row – a summary for an example set, the archive name for a screen piece.
repo – the dataset repository it is published in.
archive – the
.tarto fetch from it.folder – where it unpacks.
bytes – the archive’s size, so a selection can be priced before it is paid.
present – whether it is already unpacked where it would go. Computed when the plan is built rather than looked up later, so the listing and the download agree about what will happen.
expands_npz – whether the unpacked arrays have to be rewritten as
.npyafterwards.
- spacr.cli_download.build_parser() argparse.ArgumentParser[source]¶
Return the
spacr-downloadargument parser.Building it imports nothing beyond the standard library and two dependency-light spaCR modules, so
--helpis instant.
- spacr.cli_download.build_plan(examples: Sequence[spacr.example_archives.ExampleSet], assets: Sequence[spacr.screen_data.ScreenAsset], dest) List[Piece][source]¶
One
Pieceper thing to fetch, with its size and its folder.- Parameters:
examples – the example sets to include.
assets – the screen pieces to include.
dest – the root everything unpacks under.
- spacr.cli_download.cmd_download(args: argparse.Namespace, *, out=None, err=None) int[source]¶
spacr-downloadproper. Seemain()for the exit codes.- Parameters:
args – the parsed command line.
out – stream for ordinary output;
sys.stdoutwhen None.err – stream for errors;
sys.stderrwhen None.
- Returns:
the process exit code.
- spacr.cli_download.default_destination() pathlib.Path[source]¶
~/.cache/spacr/example_data– where the GUI already looks.Chosen so that a plate fetched by this command is the plate the application’s own “Load example data” buttons point at:
spacr.example_archives.example_plate_folder()is<this>/plate1. Someone who ranspacr-downloadbefore opening the GUI should find the data already there, not download it twice.
- spacr.cli_download.download(pieces: Sequence[Piece], *, out=None, err=None, quiet: bool = False) Tuple[List[Piece], List[Tuple[Piece, BaseException]]][source]¶
Fetch every piece, and keep going after one fails.
ONE FAILURE DOES NOT ABANDON THE REST. A crop plate is an hour of network; losing the three that would have succeeded because the second one’s connection dropped means starting the hour again. Each failure is reported as it happens and repeated in the summary, and the caller turns a non-empty result into exit code 1.
- Parameters:
pieces – what to fetch, in order.
out – stream for progress;
sys.stdoutwhen None.err – stream for failures;
sys.stderrwhen None.quiet – print neither the per-piece line nor the percentage.
- Returns:
(finished, failed). Both are needed and neither can be derived from the other: an interrupt stops the loop, so the pieces that were never attempted are in neither list, and a summary that subtracted the failures from the selection would report them as downloaded.
- spacr.cli_download.example_folder(dest) pathlib.Path[source]¶
The one plate folder the three plate example sets unpack into.
ONE FOLDER because the sets compose: the Measure example’s
merged/, the Annotate example’sdata/andmeasurements/, and the Mask demo’s raw images are three stages of the same plate, and spaCR expects to be pointed at a plate.The assay sets are not stages of that plate and each unpacks into a folder of its own; see
spacr.example_archives.example_set_folder().- Parameters:
dest – the root everything unpacks under.
- spacr.cli_download.fetch_piece(piece: Piece, *, out=None, progress: bool = True) int[source]¶
Download one archive, unpack it, and throw the archive away.
The archive is removed as soon as it is unpacked: it is a second copy of everything just written, and for a crop plate that is another 8 GB sitting on the disk for no reason.
- Parameters:
piece – what to fetch.
out – where progress is drawn;
sys.stdoutwhen None.progress – draw a percentage while it arrives.
- Returns:
how many members were unpacked.
- spacr.cli_download.main(argv: Sequence[str] | None = None) int[source]¶
spacr-downloadentry point.- Parameters:
argv – argument list;
sys.argv[1:]when None.- Returns:
0 done, 1 a download failed, 2 bad arguments or no room.
- spacr.cli_download.pieces_to_fetch(plan: Sequence[Piece], *, force: bool = False) List[Piece][source]¶
The plan minus what is already on disk.
--forcekeeps everything, which is how a truncated or edited copy gets repaired: the archive is re-fetched and unpacked over what is there.- Parameters:
plan – the pieces a selection resolved to.
force – fetch even what is already on disk.
- spacr.cli_download.render_listing(plan: Sequence[Piece], chosen: Sequence[Piece], dest, *, force: bool = False) str[source]¶
Every piece, its size, whether it is here, and what the total would be.
planis the whole inventory andchosenis the selection, so the listing answers both “what is there?” and “what would this command do?” at once. A*marks the selected rows.- Parameters:
plan – every piece there is.
chosen – the pieces this run would fetch.
dest – the root everything unpacks under.
force – whether pieces already on disk count towards the total.
- spacr.cli_download.resolve_selection(what: Sequence[str] = (), *, screen: str | None = None, plates: Sequence[int] = ()) Tuple[List[spacr.example_archives.ExampleSet], List[spacr.screen_data.ScreenAsset]][source]¶
Turn what the user typed into the exact list of things to fetch.
Kept apart from argparse so the rules can be read – and tested – as rules rather than as a parser’s side effects.
- Parameters:
what – names and groups: an example key,
examples,screenorall. Empty means the default.screen –
measurements,crops,allorNone. Naming a kind is itself a request for the screen, so--screen cropsneeds no positional argument to go with it.plates – which screen plates; empty means all of them.
- Returns:
(example sets, screen assets)in listing order.- Raises:
SelectionError – on a name, kind or plate that does not exist. A typo must not quietly select nothing and then report success.
- spacr.cli_download.room_for(pieces: Sequence[Piece], dest) str | None[source]¶
Complaint about free disk space, or
Nonewhen there is enough.The peak is not the total: each archive is unpacked and then deleted, so what has to fit at once is everything that will be kept plus the largest single archive still on disk beside its own unpacked copy.
Returns
Nonewhen the free space cannot be read at all. A filesystem that will not answer is not evidence of a full one, and refusing a download over a failedstatvfswould break the command on exactly the network filesystems a cluster user has.- Parameters:
pieces – what would be downloaded.
dest – where it would go; the nearest existing parent is measured, because the destination itself may not have been made yet.
- spacr.cli_download.screen_folder(dest, plate: int) pathlib.Path[source]¶
One folder per screen plate, and it has to be.
Every plate’s measurements archive unpacks to
measurements/measurements.dband every plate’s crop archive unpacks todata/– the same two paths, four times over. Unpacked into one folder the fourth plate would silently overwrite the third, andspacr.screen_data.ScreenAsset.is_present()would report a plate as downloaded because a DIFFERENT plate’s file is sitting where its own would go. So the plate number is in the path, and a selection of all four is four plate folders that can each be opened as itself.- Parameters:
dest – the root everything unpacks under.
plate – which screen plate.