spacr.sequencing_qc

Workflow inputs and outputs

Barcode QC

Inspect mapping depth, coverage, collisions and positional effects before guide-count filtering.

Open: Map Barcodes → Barcode QC.

Inputs and outputs below include conditional alternatives. The guidance and handoff notes say which route applies.

Inputs

• Guide counts per well — Map Barcodes run folder: unique_combinations.csv and annotated_reads.h5. Well identity requires the corresponding barcode references. Relevant columns, depending on the route: count.

Outputs

• Quality-control results — Stored project checks and QC reports; a missing check is not a passing result.

• Figures and table exports — The output location chosen by the tool; exports describe the selected data and filters.

Before this module

• Map Barcodes: Check mapping and coverage before guide filtering.

API reference.

Module tutorial.

QC for a barcode-mapping run, and a target-driven abundance-threshold sweep.

This module is the analysis that happens after spacr.sequencing.generate_barecode_mapping() has written its unique_combinations.csv / qc.csv. It answers the two questions a pooled-screen experimenter actually asks of a mapping run:

1. Did the run work? — reads per well and which wells are starved, how many reads never mapped, whether the barcode references are distinguishable at all, whether a plate row or column is systematically under-read, and how evenly the library is covered.

2. Where do I cut? — a gRNA is kept in a well when its share of that well’s reads reaches an abundance threshold. Too low and a well collects bleed-through guides it never contained, so no phenotype can be attributed to any one of them; too high and wells lose the guides they did contain, and with them the statistical power of the screen.

The second question used to be answered with a hand-picked number (2% of a well’s reads, read off a histogram once and then copied forward). Here the user states the biological quantity instead — how many gRNAs per well the design intends — and derive_threshold() solves for the abundance cutoff that delivers it. threshold_sweep() then walks a range around that cutoff so the trade-off is visible rather than asserted, and recommend_threshold() writes the answer out in words, with the derived number stated explicitly. The user picks from the curve; nothing here silently picks for them.

Why a module of its own rather than more of spacr.sequencing: that module is the read path — FASTQ in, count table out — and it is imported into every multiprocessing worker of a mapping run. The QC here is a separate, later job over the table that path produced; it pulls in plotting and statistics that the read workers must not pay for, and it is the piece a user re-runs a dozen times while choosing a threshold, long after the reads are mapped.

Example

from spacr.sequencing_qc import barcode_qc
result = barcode_qc({
    'count_data': '/data/screen/sample1_paired/unique_combinations.csv',
    'qc_data':    '/data/screen/sample1_paired/qc.csv',
    'target_grnas_per_well': 4,
})
print(result['recommendation'])

See also

spacr.sequencing.generate_barecode_mapping() — produces the inputs. spacr.ml.process_reads() — applies the chosen threshold as fraction_threshold when the screen is regressed.

Classes

ThresholdChoice	The abundance threshold a stated gRNAs-per-well target implies.
WellFractions	Per-well gRNA abundance fractions, prepared for repeated thresholding.

Functions

barcode_collisions(→ pandas.DataFrame)	Find barcode pairs a sequencing error could turn into each other.
barcode_qc(→ Dict[str, Any])	QC a barcode-mapping run and derive its abundance threshold from a target.
barcode_qc_defaults(→ Dict[str, Any])	Return the default settings for barcode_qc().
collision_summary(→ pandas.DataFrame)	Per-reference collision rate, and the share of reads it touches.
derive_threshold(→ ThresholdChoice)	Solve for the abundance threshold that delivers a stated gRNAs-per-well target.
library_depth(→ Dict[str, Any])	Summarise how evenly the gRNA library is covered by the run.
load_count_table(→ pandas.DataFrame)	Read one or more per-well gRNA count tables into one normalised frame.
plot_barcode_qc(counts, *, per_well, starved, ...[, ...])	Draw the four QC panels of a barcode-mapping run.
plot_threshold_sweep(sweep, choice[, dst])	Plot the sweep, with the derived threshold marked and labelled.
position_effects(→ pandas.DataFrame)	Flag plate rows and columns whose read depth departs from their plate.
reads_per_well(→ pandas.DataFrame)	Return one row per well: its read total and how many gRNAs it saw.
recommend_threshold(→ str)	Write the threshold recommendation out in words.
starvation_cutoff(→ float)	Return the read count below which a well counts as starved.
starved_wells(→ pandas.DataFrame)	Return the wells whose read depth is below the starvation cutoff.
sweep_grid(→ numpy.ndarray)	Log-spaced thresholds spanning span-fold either side of threshold.
threshold_sweep(→ pandas.DataFrame)	Evaluate the whole trade-off at each threshold.
unmapped_read_fractions(→ Dict[str, Any])	Read the mapping run's qc.csv and report what failed to map.

Module Contents

class spacr.sequencing_qc.ThresholdChoice

The abundance threshold a stated gRNAs-per-well target implies.

Parameters:

• threshold – the derived cutoff — a gRNA’s minimum share of its well’s reads.

• achieved – the gRNAs-per-well statistic actually obtained at it.

• target – what the user asked for.

• statistic – 'median' or 'mean'.

• n_wells – size of the well population it was derived over.

• attainable – False when even keeping every observed gRNA falls short of the target — the library, not the threshold, is the limit, and threshold is then the most permissive cutoff there is.

• n_candidates – how many distinct observed fractions were searched. The derivation is exact over this set: the statistic can only change at a fraction that is actually in the data.

• interval_low – exclusive lower end of the plateau of thresholds that all yield achieved. threshold is its geometric middle.

• interval_high – inclusive upper end of the same plateau — one step further and the statistic drops below achieved.

as_dict() → Dict[str, Any]

Return the choice as a plain dict, for CSV/JSON output.

class spacr.sequencing_qc.WellFractions(counts: pandas.DataFrame, wells: Iterable[str] | None = None)

Per-well gRNA abundance fractions, prepared for repeated thresholding.

Sorting each well’s fractions once turns “how many gRNAs survive threshold t” into a binary search, which is what makes an exact derivation over every observed fraction affordable — the sweep and the bisection between them evaluate the same population dozens of times.

Parameters:

• counts – normalised table from load_count_table().

• wells – optional restriction of the well population, as prc keys. Excluding starved wells here is what keeps them from dragging the derived threshold: a well with nine reads reports one gRNA at any cutoff and pulls the median down.

Raises:

ValueError – when the restriction leaves no wells.

Prepare a nonempty, optionally restricted well population.

counts_at(thresholds) → numpy.ndarray

gRNAs surviving each threshold, as [n_wells, n_thresholds].

Parameters:

thresholds – scalar or array of abundance cutoffs. A gRNA is kept when its fraction is greater than or equal to the cutoff, matching spacr.ml.process_reads().

Returns:

integer array of per-well surviving gRNA counts.

reads_retained_at(thresholds) → numpy.ndarray

Share of all mapped reads surviving each threshold.

Parameters:

thresholds – one or more minimum guide-fraction thresholds.

statistic_at(thresholds, statistic: str = 'median') → numpy.ndarray

Median or mean gRNAs per well, over the fixed well population.

The population is fixed — a well that loses its last gRNA stays in the denominator as a zero — and that is what makes the result a non-increasing function of the threshold: every well’s count only ever falls, so every order statistic of them only ever falls. Dropping emptied wells instead would let the median jump up when a sparse well is removed, and a target could then be met at two thresholds far apart with no way to say which was meant.

Parameters:

• thresholds – scalar or array of cutoffs.

• statistic – 'median' or 'mean'.

Returns:

array of the statistic, aligned with thresholds.

Raises:

ValueError – on an unknown statistic.

spacr.sequencing_qc.barcode_collisions(references: Mapping[str, Any], max_distance: int = 1) → pandas.DataFrame

Find barcode pairs a sequencing error could turn into each other.

Two barcodes of the same length within max_distance substitutions are a collision: one miscalled base moves a read from one well (or one guide) to another, and nothing downstream can tell that it happened. Exact duplicates are reported too, at distance 0 — those are fatal rather than risky, and spacr.sequencing.map_sequences_to_names() refuses to run on them.

Only substitutions are considered, and only within a reference set. Indels would change the barcode’s length and so shift every field after it in the read, which the regex rejects outright rather than mis-assigning; and a row barcode cannot be confused with a gRNA barcode because they are read out of different positions.

Parameters:

• references – {label: source}, where each source is a name,sequence CSV path, a FASTA path, or a {name: sequence} mapping. The label (“row”, “column”, “grna”) names the set in the output.

• max_distance – maximum number of substitutions. 1 is the default because a single miscalled base is the common event; 0 reports only exact duplicates.

Returns:

DataFrame [reference, name_a, name_b, distance, sequence_a, sequence_b], one row per colliding pair, sorted by reference then distance.

Raises:

ValueError – on a negative max_distance.

Example

from spacr.sequencing_qc import barcode_collisions
pairs = barcode_collisions({'row': '/data/barcodes/row.csv'})

spacr.sequencing_qc.barcode_qc(settings: Dict[str, Any] | None = None) → Dict[str, Any]

QC a barcode-mapping run and derive its abundance threshold from a target.

Runs every panel this module provides over one run’s outputs, derives the threshold that delivers target_grnas_per_well, sweeps around it, writes the figures and tables, and returns everything it computed along with a recommendation in words.

Parameters:

settings – dict; see barcode_qc_defaults() for every key and _TOOLTIPS for what each one does. The two that matter are count_data (the run’s unique_combinations.csv) and target_grnas_per_well.

Returns:

dict with choice (ThresholdChoice), threshold (the derived number), sweep, recommendation, per_well, starved, positions, depth, collisions, collision_summary, unmapped and dst.

Raises:

ValueError – from load_count_table() or derive_threshold() on unusable inputs.

Example

from spacr.sequencing_qc import barcode_qc
out = barcode_qc({'count_data': 'unique_combinations.csv',
                  'target_grnas_per_well': 4})
print(out['threshold'], out['recommendation'], sep='\n')

spacr.sequencing_qc.barcode_qc_defaults(settings=None) → Dict[str, Any]

Return the default settings for barcode_qc().

Parameters:

settings – optional dict to fill in place; a new one is made when omitted.

Returns:

the settings dict with defaults applied.

spacr.sequencing_qc.collision_summary(references: Mapping[str, Any], collisions: pandas.DataFrame, counts: pandas.DataFrame | None = None) → pandas.DataFrame

Per-reference collision rate, and the share of reads it touches.

Parameters:

• references – the same mapping passed to barcode_collisions().

• collisions – its output.

• counts – optional normalised count table. When given, the gRNA reference also reports reads_at_risk — the share of mapped reads carrying a barcode that has a near neighbour, which is what turns a list of risky pairs into an amount of data at risk.

Returns:

DataFrame [reference, n_barcodes, n_colliding_pairs, n_barcodes_at_risk, collision_rate, reads_at_risk].

spacr.sequencing_qc.derive_threshold(counts: pandas.DataFrame, target_grnas_per_well: float, statistic: str = 'median', wells: Iterable[str] | None = None) → ThresholdChoice

Solve for the abundance threshold that delivers a stated gRNAs-per-well target.

This replaces choosing a cutoff by eye. The experimenter states the biological quantity — how many gRNAs a well is meant to carry, which is a design decision about power versus attributability — and the cutoff is whatever number delivers it in this run’s data. Two runs at different depth get different numbers for the same target, which is the point.

The search is exact rather than gridded: the gRNAs-per-well statistic is a step function that can only change at a fraction actually present in the table, so every distinct observed fraction is a candidate and the monotonicity of the statistic (see WellFractions.statistic_at()) lets a bisection find the answer in log2(n) evaluations. Where no cutoff hits the target exactly — the statistic is a median of integers and jumps — the candidate landing closest is returned, preferring the one that still meets the target over the one that falls short.

The number returned sits in the middle of its plateau, not on its edge. A whole range of thresholds gives the same gRNAs-per-well answer — on a clean run that range is the empty space between the guides a well really carried and the bleed-through tail below them, which is exactly what a histogram is being read for when a cutoff is picked by eye. Returning either edge of it would put the cutoff where a re-sequenced run or a rounded count flips guides across it. The geometric middle is the same answer and holds up; both edges are reported as interval_low / interval_high so the width of the plateau — how much slack the choice has — is visible too.

Parameters:

• counts – normalised table from load_count_table().

• target_grnas_per_well – the target. Must be positive.

• statistic – 'median' (default) or 'mean'.

• wells – optional prc restriction of the well population — pass the non-starved wells to keep unusable wells out of the fit.

Returns:

a ThresholdChoice.

Raises:

ValueError – on a non-positive target or an unknown statistic.

Example

from spacr.sequencing_qc import load_count_table, derive_threshold
counts = load_count_table('unique_combinations.csv')
choice = derive_threshold(counts, target_grnas_per_well=4)
print(choice.threshold, choice.achieved)

spacr.sequencing_qc.library_depth(counts: pandas.DataFrame, expected_grnas: Iterable[str] | None = None) → Dict[str, Any]

Summarise how evenly the gRNA library is covered by the run.

Parameters:

• counts – normalised table from load_count_table().

• expected_grnas – the library as designed — names from the gRNA reference. Supplying it is what turns “we saw 4,900 guides” into “2% of the library was never seen”, which is the number that says whether the screen has the coverage it was powered for.

Returns:

dict with n_grnas_observed, n_grnas_expected, dropout_fraction, dropped_grnas (sorted names), gini, skew_ratio (90th/10th percentile of per-gRNA read totals — the standard pooled-library evenness number), top_decile_share and reads_per_grna (a Series, descending).

spacr.sequencing_qc.load_count_table(count_data, plate: str | None = None) → pandas.DataFrame

Read one or more per-well gRNA count tables into one normalised frame.

Accepts what a barcode-mapping run writes (unique_combinations.csv: rowID, columnID, grna_name, count) as well as already-loaded DataFrames, and a list mixing both. Each source that carries no plateID is given one — plate when supplied, otherwise plate1, plate2, … in the order the sources are listed — so several plates can be QC’d together without their wells colliding.

Parameters:

• count_data – path, DataFrame, or list of either.

• plate – plate name for the first source that does not carry one. Any further nameless source is called plate<N> for its 1-based position in count_data, so the name traces back to the file it came from — and two plates can never share a name, which would merge their wells into one.

Returns:

DataFrame with COUNT_COLUMNS plus prc (the plate_row_column well key), well_reads (the well’s read total) and fraction (this gRNA’s share of it).

Raises:

ValueError – when a source is missing a required column, or when no source holds any usable row.

Example

from spacr.sequencing_qc import load_count_table
counts = load_count_table(
    ['/data/p1/unique_combinations.csv',
     '/data/p2/unique_combinations.csv'])

spacr.sequencing_qc.plot_barcode_qc(counts: pandas.DataFrame, *, per_well: pandas.DataFrame, starved: pandas.DataFrame, positions: pandas.DataFrame, depth: Mapping[str, Any], unmapped: Mapping[str, Any] | None = None, dst: str | None = None)

Draw the four QC panels of a barcode-mapping run.

reads per well (with the starvation cut marked), position effects (every plate row and column against its plate median), library coverage (the Lorenz curve of per-gRNA read totals, with its Gini) and read fate (what fraction of reads mapped).

Parameters:

• counts – normalised table from load_count_table().

• per_well – output of reads_per_well().

• starved – output of starved_wells().

• positions – output of position_effects().

• depth – output of library_depth().

• unmapped – optional output of unmapped_read_fractions().

• dst – folder to write barcode_qc.pdf into; None returns the figure without saving.

Returns:

the matplotlib Figure.

spacr.sequencing_qc.plot_threshold_sweep(sweep: pandas.DataFrame, choice: ThresholdChoice, dst: str | None = None)

Plot the sweep, with the derived threshold marked and labelled.

Two stacked panels share one log-scaled threshold axis: gRNAs per well above, well retention and collision rate below. Two panels rather than a twin y-axis because the quantities have nothing in common — a count and two percentages — and overlaying them puts the flat 100% retention line on top of the frame, where it cannot be read. The derived threshold is a labelled vertical line carrying its own numeric value in both panels: the user must be able to see what their target translated to without reading it off the axis.

The gRNAs-per-well axis is symlog around 1. A relaxed threshold puts tens of guides in a well while the interesting region is a handful, and on a linear axis the answer is a flat line at the bottom of the plot.

Parameters:

• sweep – output of threshold_sweep().

• choice – output of derive_threshold().

• dst – folder to write threshold_sweep.pdf into; None returns the figure without saving.

Returns:

the matplotlib Figure.

spacr.sequencing_qc.position_effects(counts: pandas.DataFrame, ratio: float = DEFAULT_POSITION_RATIO) → pandas.DataFrame

Flag plate rows and columns whose read depth departs from their plate.

A pooled screen is pipetted, and pipetting has geometry: an edge row that dried, a column the multichannel missed. Both show up as a whole row or column of wells sitting at a different depth from the rest of the plate, and both bias every per-well fraction computed inside them.

Parameters:

• counts – normalised table from load_count_table().

• ratio – fold-change from the plate median at which a row or column is flagged. Must be greater than 1.

Returns:

DataFrame [plateID, axis, label, n_wells, median_reads, plate_median, ratio_to_plate, flagged], one row per plate row and per plate column, sorted worst-first.

Raises:

ValueError – when ratio is not above 1 — at 1 every row is flagged and the report says nothing.

spacr.sequencing_qc.reads_per_well(counts: pandas.DataFrame) → pandas.DataFrame

Return one row per well: its read total and how many gRNAs it saw.

Parameters:

counts – normalised table from load_count_table().

Returns:

DataFrame [prc, plateID, rowID, columnID, reads, n_grnas] sorted by reads ascending, so the starved end of the plate reads off the top.

spacr.sequencing_qc.recommend_threshold(sweep: pandas.DataFrame, choice: ThresholdChoice) → str

Write the threshold recommendation out in words.

A curve tells a reader where the knee is only if they already know what they are looking for. This states the derived number, what it buys, what relaxing and tightening it cost, and where the collision rate turns — in sentences, so the choice can be quoted in a methods section.

Parameters:

• sweep – output of threshold_sweep().

• choice – output of derive_threshold().

Returns:

a multi-line string.

spacr.sequencing_qc.starvation_cutoff(per_well: pandas.DataFrame, min_reads: int = 0, starved_read_fraction: float = DEFAULT_STARVED_READ_FRACTION) → float

Return the read count below which a well counts as starved.

min_reads above zero is used verbatim — an absolute floor the experimenter knows from the library prep. Otherwise the cut is starved_read_fraction of the median well’s depth, which is the only rule that transfers between runs of different total depth.

Parameters:

• per_well – output of reads_per_well().

• min_reads – absolute floor; 0 (the default) means derive one.

• starved_read_fraction – share of the median well’s reads used when deriving.

Returns:

the cutoff as a float. A well is starved when its read total is strictly below it.

Raises:

ValueError – on a negative min_reads or a starved_read_fraction outside (0, 1] — both would mark either no well or every well and say nothing.

spacr.sequencing_qc.starved_wells(counts: pandas.DataFrame, min_reads: int = 0, starved_read_fraction: float = DEFAULT_STARVED_READ_FRACTION) → pandas.DataFrame

Return the wells whose read depth is below the starvation cutoff.

Parameters:

• counts – normalised table from load_count_table(), or an already-computed reads_per_well() frame.

• min_reads – absolute floor; 0 derives one.

• starved_read_fraction – share of the median used when deriving.

Returns:

the starved subset of reads_per_well(), with the cutoff recorded in .attrs['cutoff'].

spacr.sequencing_qc.sweep_grid(threshold: float, span: float = DEFAULT_SWEEP_SPAN, points: int = DEFAULT_SWEEP_POINTS, *, low: float | None = None, high: float | None = None) → numpy.ndarray

Log-spaced thresholds spanning span-fold either side of threshold.

Parameters:

• threshold – the centre — the derived cutoff.

• span – multiplicative half-width. 4.0 sweeps a quarter to four times the derived value.

• points – how many points, before the centre is inserted.

• low – absolute lower end, overriding threshold / span when it is lower. barcode_qc() uses it to make sure the sweep reaches down into the bleed-through tail, where the collision rate turns — a curve that stops above the junk cannot show the user the cost of relaxing into it.

• high – absolute upper end, overriding threshold * span.

Returns:

sorted unique array of cutoffs in (0, 1], always containing threshold itself so the derived point is on the curve and not merely near it.

Raises:

ValueError – on a non-positive threshold, a span at or below 1, fewer than 3 points, or a low that is not below high.

spacr.sequencing_qc.threshold_sweep(counts: pandas.DataFrame, thresholds, target_grnas_per_well: float, statistic: str = 'median', wells: Iterable[str] | None = None) → pandas.DataFrame

Evaluate the whole trade-off at each threshold.

Parameters:

• counts – normalised table from load_count_table().

• thresholds – array of cutoffs — usually sweep_grid() around a derive_threshold() result.

• target_grnas_per_well – the attribution budget. A well holding more gRNAs than this is counted as a collision: its phenotype is a mixture of more guides than the design set out to disentangle, so it cannot be attributed to any one of them.

• statistic – 'median' or 'mean', for the headline gRNAs-per-well column.

• wells – optional prc restriction of the well population.

Returns:

DataFrame, one row per threshold, with

• grnas_per_well — the requested statistic over all wells; non-increasing in the threshold.

• grnas_per_well_retained — the same statistic over wells that still hold at least one gRNA. Easier to read, and not monotone: it rises when a one-gRNA well drops out.

• wells_retained / well_retention — non-increasing.

• collision_rate — share of all wells over the budget; non-increasing, because each well’s gRNA count only falls.

• collision_rate_retained — the same numerator over retained wells only. Not monotone, for the same reason as above.

• n_calls — surviving (well, gRNA) pairs; non-increasing.

• reads_retained — share of mapped reads kept; non-increasing.

spacr.sequencing_qc.unmapped_read_fractions(qc_data, counts: pandas.DataFrame | None = None) → Dict[str, Any]

Read the mapping run’s qc.csv and report what failed to map.

qc.csv accumulates, per barcode field, the number of reads whose sequence matched no entry in that field’s reference CSV, alongside total_reads.

What the denominator is. total_reads counts reads that matched the barcode regex — a read that never found the anchor sequence is not in the file at all. So these fractions are “of the reads that reached barcode lookup”, not “of the FASTQ”. That is the number that diagnoses a wrong or reverse-complemented barcode reference, which is what this panel is for; a run where the regex itself misses is already loud in the mapping log.

Parameters:

• qc_data – path, DataFrame, or list of either.

• counts – optional normalised count table for the same run. When given, unmapped_fraction is exact — the count table holds only reads whose three barcodes all resolved, so the shortfall against total_reads is the true joint unmapped share. Without it only the per-field fractions and bounds are reported.

Returns:

dict with total_reads, per_field (field -> unmapped fraction), mapped_reads/unmapped_fraction (only with counts), and unmapped_fraction_lower/_upper — the bounds implied by the per-field numbers alone.

Raises:

ValueError – when no total_reads column is present, or it sums to zero.

Nested helpers

derive_threshold.choice_at(index: int, achieved: float, attainable: bool) → ThresholdChoice

Build the result for candidate index, centred in its plateau.

index is the strictest candidate yielding achieved. The plateau reaches down to just above the last candidate that yielded MORE than achieved, found by a second bisection on the same monotone statistic.

spacr/sequencing_qc.py:862

derive_threshold.stat(value: float) → float

Evaluate the captured well-fraction statistic at one threshold.

Parameters:

value – abundance-fraction threshold to evaluate.

Returns:

the selected mean or median gRNAs-per-well value as a float; the retained-well details returned alongside it are discarded.

spacr/sequencing_qc.py:853

plot_barcode_qc._natural(value)

Split a position label into text and numeric sort parts.

Parameters:

value – row or column position label to stringify.

Returns:

all non-digits followed by the concatenated digits as an integer, or zero when none are present, so labels such as c1, c2, and c10 follow physical plate order.

spacr/sequencing_qc.py:1275