spacr.frame_handoff¶

Hand a frame to the next step without a file parse in between.

A step that has built a frame IN MEMORY and hands it to a step in the same process has no need of a serialisation round trip. Writing one out and parsing it back costs twice: once to write and once to read, and both costs scale with the frame rather than with the work being done.

The merged measurement frame is the case this was built for. A four-plate screen merges to about 2.75 GB; writing that as CSV takes around 160 seconds and parsing it back takes longer still, all of it AFTER the frame already existed in the writing process’s memory, and all of it before the fit that needs it can start.

Two halves, and both are needed:

hold/held: the frame itself is offered under the path it was written to, so a reader that would have parsed that path gets the object instead. The reference is WEAK – when the producer drops the frame the offer disappears and the file on disk is read as before, so nothing here can keep a multi-gigabyte frame alive after its owner is done with it.
stage: the durable copy is written in a COLUMNAR format when one is available. The artefact is kept because a user can open it and because every fit of a queue then reads the same numbers; only the format changes, and Parquet both writes and reads several times faster than CSV at a fraction of the size.

A reader that knows nothing about this module keeps working: the path is a real path to a real file, and the handoff is an optimisation on top of it.

Functions¶

`describe`(→ str)	One line naming what was handed over for `path`, for a log.
`held`(→ Optional[pandas.DataFrame])	The frame offered for `path`, or `None` when there is none.
`hold`(→ str)	Offer `frame` as the contents of `path`.
`key_for`(→ str)	The absolute path two callers must agree on to meet here.
`release`(→ int)	Withdraw one offer, or every offer when `path` is `None`.
`stage`(→ str)	Write the durable copy and offer the frame in memory under its path.

Module Contents¶

spacr.frame_handoff.describe(path: Any) → str[source]¶

One line naming what was handed over for path, for a log.

Parameters:: path – filesystem path whose in-memory offer is to be described.
Returns:: one-line shape and handoff description, or "" when no live offer matches.

Empty when nothing was offered, so a caller can print it unconditionally and say nothing when there is nothing to say.

spacr.frame_handoff.held(path: Any) → pandas.DataFrame | None[source]¶

The frame offered for path, or None when there is none.

Parameters:: path – filesystem path whose in-memory offer is requested.
Returns:: offered DataFrame, or None when no live offer matches.

None is not a failure: it means the reader should read the file, which is what it would have done anyway.

spacr.frame_handoff.hold(path: Any, frame: pandas.DataFrame) → str[source]¶

Offer frame as the contents of path.

Parameters:

path – where the frame was (or will be) written.
frame – the frame the caller already has.

Returns:

the key the offer is filed under.

Raises:

ValueError – if frame is None.

The caller keeps ownership. Nothing here extends the frame’s life, so a producer that finishes and drops it withdraws the offer by doing so.

spacr.frame_handoff.key_for(path: Any) → str[source]¶

The absolute path two callers must agree on to meet here.

Parameters:: path – a path, or anything os.fspath() accepts.
Returns:: the expanded, absolute path used as the offer’s key.

spacr.frame_handoff.release(path: Any = None) → int[source]¶

Withdraw one offer, or every offer when path is None.

Parameters:: path – one offered path to withdraw, or None to withdraw all.
Returns:: how many offers were withdrawn.

Withdrawing is optional – the references are weak – but a producer that knows it has finished can say so, which makes the fallback to the file deterministic instead of dependent on when the frame is collected.

spacr.frame_handoff.stage(frame: pandas.DataFrame, folder: Any, stem: str, *, columnar: bool = True, report=print) → str[source]¶

Write the durable copy and offer the frame in memory under its path.

Parameters:

frame – the frame to hand on.
folder – directory for the artefact; created when absent.
stem – the file name without a suffix.
columnar – write Parquet when an engine is installed. False forces CSV, which is what a caller wants when the file is meant to be opened in a spreadsheet rather than read back by spaCR.
report – called with one line saying what was written and how long it took; None to say nothing.

Returns:

the path written.

Raises:

ValueError – if frame is None.
OSError – if the destination directory or durable artefact cannot be written. Other table-writer errors also propagate after the offer registry and any pre-existing durable artefact are left as they were.

The frame is offered BEFORE the write returns, so a reader that starts while the artefact is still being written gets the object rather than a half-written file.