spacr.guide_permutation

Plate-aware marginal guide association tests for pooled screens.

The screen unit is the well. Each guide is tested separately after both its well-level read fraction and the well-level phenotype have been residualized against nuisance covariates. Empirical two-sided P values are obtained with Freedman–Lane residual permutations restricted within the experimental block (normally plateID), followed by a user-selected multiple-testing correction within each outcome and minimum-support family.

This module deliberately reports marginal associations. It does not claim to estimate a simultaneous conditional coefficient for every guide when the number or correlation structure of guides makes that model unidentified.

Functions

adjusted_value_label(→ str)	Axis/legend label for the adjusted value produced by method.
analyse_long_gene_table(, well_column, guide_column, ...)	The gene pass over spaCR's saved regression_data.csv.
analyse_long_guide_table(, well_column, guide_column, ...)	Convenience wrapper for spaCR's saved regression_data.csv.
gene_fraction_matrix(→ pandas.DataFrame)	Sum each gene's guide columns into ONE column per gene.
gene_freedman_lane_test(→ pandas.DataFrame)	Test each GENE's guides as a SET, against the same permutation null.
guide_freedman_lane_test(→ pandas.DataFrame)	Test marginal guide associations across one or more support families.
guide_support_sensitivity(, random_state, ...)	Run the same support-threshold analysis for one or more outcomes.
plot_guide_permutation_volcano(results, *, outcome, ...)	Draw and publish a guide-permutation volcano plot.
prepare_long_gene_data(data, outcome_columns, *[, ...])	Well-by-GENE fractions, the well table, and how many guides each gene has.
prepare_long_guide_data(data, outcome_columns, *[, ...])	Convert spaCR's long regression table into aligned well-level tables.
save_guide_permutation_results(→ Mapping[str, ...)	Save the long result and one source-data CSV per support threshold.

Module Contents

spacr.guide_permutation.adjusted_value_label(method) → str

Axis/legend label for the adjusted value produced by method.

Parameters:

method – multiple-testing correction name or accepted alias.

An FDR method yields a q value, a family-wise method an adjusted P, and none leaves the raw P value. Labelling every correction “BH q” – as this module did while it offered only four methods – mislabels the axis of every plot drawn with any other correction.

spacr.guide_permutation.analyse_long_gene_table(data: pandas.DataFrame, outcome_columns: str | Sequence[str], *, min_wells: int | Sequence[int] = (1, 2, 3, 4), well_column: str = 'prc', guide_column: str = 'grna', gene_column: str = 'gene', fraction_column: str = 'fraction', block_column: str = 'plateID', nuisance_columns: Sequence[str] | None = None, random_state: int = 0, **kwargs) → pandas.DataFrame

The gene pass over spaCR’s saved regression_data.csv.

Parameters:

• data – long well-guide-gene regression table to analyse.

• outcome_columns – phenotype column name or names to test.

The counterpart of analyse_long_guide_table(). Its BH correction is computed over GENES ONLY: two families, never one. Pooling them would be wrong twice over – the same wells produce both, and a gene’s regressor is literally the sum of its guides’ regressors, so they are not independent; and doubling the family size costs power for no protection.

spacr.guide_permutation.analyse_long_guide_table(data: pandas.DataFrame, outcome_columns: str | Sequence[str], *, min_wells: int | Sequence[int] = (1, 2, 3, 4), well_column: str = 'prc', guide_column: str = 'grna', fraction_column: str = 'fraction', block_column: str = 'plateID', nuisance_columns: Sequence[str] | None = None, **kwargs) → pandas.DataFrame

Convenience wrapper for spaCR’s saved regression_data.csv.

Parameters:

• data – long well-guide regression table to analyse.

• outcome_columns – phenotype column name or names to test.

spacr.guide_permutation.gene_fraction_matrix(fractions: pandas.DataFrame, gene_of_guide: Mapping[str, str]) → pandas.DataFrame

Sum each gene’s guide columns into ONE column per gene.

This is exactly the quantity the parametric gene fit regresses on: spacr.ml.check_and_clean_data() defines gene_fraction as the sum of the gene’s gRNA fractions within a well, and this builds the same number from the well-by-guide matrix. The two paths therefore test the same regressor, one with a t statistic and one with a permutation null.

Parameters:

• fractions – well-by-guide fractions from prepare_long_guide_data().

• gene_of_guide – guide id -> gene id.

Returns:

a well-by-gene frame, columns sorted, one column per gene that has at least one guide in fractions.

Raises:

ValueError – when a guide column has no gene.

spacr.guide_permutation.gene_freedman_lane_test(gene_fractions: pandas.DataFrame, outcomes: pandas.DataFrame, outcome_column: str, *, gene_metadata: pandas.DataFrame | None = None, **kwargs) → pandas.DataFrame

Test each GENE’s guides as a SET, against the same permutation null.

The design asks the nonparametric path for a gene pass beside its guide pass. The gene’s regressor is the SUM of its guides’ fractions – one column, one test, one degree of freedom – residualized against the same block/nuisance design and permuted with the same Freedman–Lane scheme. Passing the same random_state as the guide pass makes the permutations literally identical, because the permuted object is the residual of the same outcome against the same nuisance design.

IT IS NOT A COMBINATION OF PER-GUIDE P VALUES, and that is deliberate. Fisher’s and Stouffer’s methods both assume the p-values being combined are independent, and two guides measured in the same wells are not: they share the well’s phenotype, the well’s plate effect and the well’s cells. Combining them would report a confidence the design cannot support.

WHAT A ONE-DEGREE-OF-FREEDOM SET TEST CANNOT SEE, and the reason guides_in_gene travels with every row: a gene whose guides push the phenotype in OPPOSITE directions cancels in the sum, exactly as it does in the parametric y ~ gene_fraction:gene fit that this mirrors. Such a gene is not evidence of no effect; it is evidence the guides disagree, and the guide-level table is where that shows. A gene resting on one guide (guides_in_gene == 1) is the same test as that guide’s own row.

Parameters:

• gene_fractions – well-by-gene matrix from prepare_long_gene_data().

• outcomes – well-indexed phenotype and nuisance table aligned to gene_fractions.

• outcome_column – numeric phenotype column to test.

• gene_metadata – the per-gene guide counts, joined onto the result.

• kwargs – forwarded to guide_freedman_lane_test().

Returns:

one row per gene per minimum-wells family, BH-corrected WITHIN the gene family and never pooled with the guide family.

spacr.guide_permutation.guide_freedman_lane_test(fractions: pandas.DataFrame, outcomes: pandas.DataFrame, outcome_column: str, *, min_wells: int | Sequence[int] = 4, block_column: str = 'plateID', nuisance_columns: Sequence[str] | None = None, n_permutations: int = 200000, random_state: int | numpy.random.Generator = 0, multiple_testing: str = 'fdr_bh', alpha: float = 0.05, presence_threshold: float = 0.0, batch_size: int = 500, statistic: str = 'pearson', guide_metadata: pandas.DataFrame | None = None) → pandas.DataFrame

Test marginal guide associations across one or more support families.

THE METHOD, IN FULL

Let n be the number of wells and G the number of guides. For well i, y_i is the phenotype and x_ig is guide g’s read fraction. Z is the n x q nuisance design: indicator columns for the block (normally plateID) and for each column in nuisance_columns (normally rowID and columnID). A rank-deficient Z is REFUSED rather than pseudo-inverted, because a nuisance basis that cannot be inverted is a design question, not a numerical one.

1. Projection. Z = QR (reduced QR), and Q is the orthonormal basis. Residualising is M = I - QQ', the projection onto the orthogonal complement of Z:

x~_g = M x_g        y^ = QQ'y        y~ = y - y^ = My

QR rather than (Z'Z)^-1 because the indicator design a plate layout produces is close to collinear and the normal equations lose precision on it.

2. The statistic. For each guide:

t_g = (x~_g . y~) / (||x~_g|| ||y~||)

the cosine between the two residualised vectors – that is, the PARTIAL CORRELATION of x_g and y given Z. It is scale-free in both arguments, so guides are directly comparable without standardisation. A zero-norm residual on either side is refused by name rather than returned as a zero correlation.

With statistic='rank' the phenotype is ranked (average ranks for ties) BEFORE the projection, so M removes the block from the RANKS and t_g is a Spearman-type partial correlation.

3. Freedman–Lane. For permutation b, with P_b shuffling rows only WITHIN a block:

y_b  = y^ + P_b y~        y~_b = M y_b
t_g_b = (x~_g . y~_b) / (||x~_g|| ||y~_b||)

Three things this does that a naive shuffle does not. The nuisance fit is added back before re-projecting, so the reference distribution is “no guide effect GIVEN Z” rather than “no structure at all”. The denominator is recomputed per permutation, because M P_b y~ does not preserve norm and a fixed one would let the null drift with the shuffle. And the shuffle is confined to a block, because permuting across plates would let a plate effect leak into the null and inflate significance.

4. The P value. Two-sided, by absolute value:

p_g = (1 + #{b : |t_g_b| >= |t_g| - eps}) / (B + 1)

The +1 in both places is not a fudge: including the observed configuration in its own reference set is what makes the test EXACT, giving P(p <= alpha) <= alpha under the null for any B. It also sets a floor of 1 / (B + 1) – with the default 200,000 permutations, 5e-06. A guide at the floor is one no permutation reached, and guides there cannot be ranked against each other; raising B is the only remedy and costs linearly. The eps keeps a permutation that reproduces the observed statistic exactly from failing its own >=.

Permutations run in batches of batch_size so an n x batch_size block is materialised instead of n x B. Exceedances accumulate across batches; the result is identical and the memory is bounded.

What it assumes. Exchangeability of y~ within a block under the null. Structure inside a plate that Z does not capture – an edge effect rowID/columnID misses, a batch within a plate – breaks it, and the P values are then anti-conservative.

What it does not give. A conditional coefficient. Each guide is tested alone, so two guides in one well both take credit and the test cannot apportion the effect between them. That is the price of having no rank requirement: guides may outnumber wells here, and may not in a model that fits them all at once.

Permutation P values are computed once for every guide satisfying the smallest requested support threshold. For each displayed threshold, the eligible subset is then corrected as its own multiple-testing family. This keeps the observed statistic and empirical P value for a guide identical across thresholds; only the adjusted value changes with family size.

param fractions:

well-by-guide matrix of finite, non-negative guide fractions. Its index defines the well order and its columns name the tested guides.

param outcomes:

table indexed by well containing the phenotype, block, and any nuisance columns. It is reordered to fractions when the same well identifiers arrive in a different order.

param outcome_column:

numeric phenotype column in outcomes to test.

param statistic:

'pearson' (default) or 'rank'.

THE INFERENCE WAS ALWAYS DISTRIBUTION-FREE; THE STATISTIC WAS NOT. The permutation builds its own null, so the P value needs no assumption – but the quantity being permuted is a partial CORRELATION, and a correlation is a least-squares object: one extreme well moves it, exactly as it moves a regression coefficient.

'rank' replaces the phenotype with its ranks before residualising, which makes the statistic a Spearman-type partial correlation: monotone rather than linear, and bounded in its response to an outlier because a rank cannot be extreme.

WHY IT MATTERS HERE, measured on a real screen: the residuals came back with excess kurtosis +8.89 and skew +1.64, with 455 of 1,781 observations above the leverage guide and a maximum Cook’s distance of 1.01. That is a design where a handful of wells move a least-squares statistic, and ranking is the cheapest defence.

IT COSTS NOTHING IN SPEED, which is why it is offered and a robust regression is not. Both are one matrix product, so 200,000 permutations stay feasible; a quantile or Huber fit would need one fit per guide per permutation – here 789 x 200,000 – which is not a slower option, it is a different program.

spacr.guide_permutation.guide_support_sensitivity(fractions: pandas.DataFrame, outcomes: pandas.DataFrame, outcome_columns: str | Sequence[str], *, min_wells: int | Sequence[int] = (1, 2, 3, 4), random_state: int = 0, **kwargs) → pandas.DataFrame

Run the same support-threshold analysis for one or more outcomes.

Parameters:

• fractions – zero-filled well-by-guide fraction matrix.

• outcomes – well-indexed phenotype and nuisance table aligned with fractions.

• outcome_columns – phenotype column name or names to test.

spacr.guide_permutation.plot_guide_permutation_volcano(results: pandas.DataFrame, *, outcome: str, minimum_wells: int, save_path: str | pathlib.Path, label_guides: Mapping[str, str] | None = None, title: str | None = None, effect_threshold: float | None = None, effect_threshold_label: str | None = None, fmt: str | None = None, dpi: int | None = None)

Draw and publish a guide-permutation volcano plot.

Parameters:

• results (pandas.DataFrame) – Permutation results containing outcome, minimum_wells_threshold, standardized_marginal_effect, adjusted_p_value, significant, multiple_testing_method, and alpha. A guide column is also required when label_guides is provided.

• outcome (str) – Outcome to select from results.

• minimum_wells (int) – Guide-support threshold to select from results.

• save_path (str or pathlib.Path) – Destination path. The final suffix may change to match the selected output format.

• label_guides (mapping of str to str, optional) – Guide IDs mapped to annotation labels.

• title (str, optional) – Figure title. By default, describe the outcome and support threshold.

• effect_threshold (float, optional) – Positive half-width of the effect-size cut. Draw vertical lines at both signs; non-finite, non-positive, and None values draw no cut.

• effect_threshold_label (str, optional) – Legend text for the effect-size cut. By default, display its value.

• fmt (str, optional) – Explicit figure format. By default, use a recognized save_path suffix or the configured preference.

• dpi (int, optional) – Explicit output resolution. By default, use the configured preference.

Returns:

pathlib.Path – Path actually written by spacr.figure_sink.publish().

Raises:

ValueError – If no rows match outcome and minimum_wells.

spacr.guide_permutation.prepare_long_gene_data(data: pandas.DataFrame, outcome_columns: str | Sequence[str], *, well_column: str = 'prc', guide_column: str = 'grna', gene_column: str = 'gene', fraction_column: str = 'fraction', block_column: str = 'plateID', nuisance_columns: Sequence[str] | None = None)

Well-by-GENE fractions, the well table, and how many guides each gene has.

Parameters:

• data – long regression table containing well, guide, gene, fraction, and phenotype columns.

• outcome_columns – phenotype column name or names to retain per well.

Returns:

(gene_fractions, outcomes, gene_metadata), the gene-level counterpart of prepare_long_guide_data().

Raises:

ValueError – when gene_column is absent.

spacr.guide_permutation.prepare_long_guide_data(data: pandas.DataFrame, outcome_columns: str | Sequence[str], *, well_column: str = 'prc', guide_column: str = 'grna', fraction_column: str = 'fraction', block_column: str = 'plateID', nuisance_columns: Sequence[str] | None = None)

Convert spaCR’s long regression table into aligned well-level tables.

Parameters:

• data – long regression table with one fraction per well-guide pair.

• outcome_columns – phenotype column name or names to retain per well.

The long table must have one fraction per well/guide pair. Phenotype, block and nuisance values may repeat across the guide rows of a well, but they must be identical within that well. Duplicate well/guide rows are summed, matching the existing spaCR design-matrix construction.

Returns (fractions, outcomes, guide_metadata) where fractions is a zero-filled well-by-guide matrix, outcomes has one row per well, and guide_metadata reports the number of wells with a positive fraction.

spacr.guide_permutation.save_guide_permutation_results(results: pandas.DataFrame, destination: str | pathlib.Path, *, prefix: str = 'guide_permutation') → Mapping[str, pathlib.Path]

Save the long result and one source-data CSV per support threshold.

Parameters:

• results – long guide-permutation result table to write.

• destination – output directory for the long and threshold-specific CSV files.