spacr.regression_qc

Quality-control figures for the regression step of a pooled CRISPR screen.

Why this exists

spacr.ml.regression() fits a model to one row per well and emits a volcano plot. A volcano plot shows which coefficients are large and small-p; it shows nothing at all about whether the fit those numbers came out of is trustworthy. Every failure mode that has actually cost this project a screen is invisible on a volcano:

• a handful of 20-cell wells with leverage 0.4 each, dragging a gene’s coefficient wherever they like;

• a plate whose column 1 and column 24 are systematically dim, so the “hits” are the genes that happened to be plated on the edge;

• a design matrix in which gene and grna are nearly aliased, so the standard errors are 30x too small and every p-value is spuriously tiny;

• a Poisson fit on data whose variance is six times its mean, which makes every interval too narrow by a factor of 2.5;

• a logistic fit that is badly calibrated, so the predicted fractions are systematically wrong even though the ranking is fine.

Each of those is obvious in one glance at the right panel, and each of them produces a believable volcano plot. That asymmetry — silent, plausible garbage — is the documented failure mode of this pipeline, so the diagnostics are not optional decoration; they are the thing that says “do not believe this run”.

What this module does NOT do

It does not duplicate the volcano plot (spacr.plot.volcano_plot() and spacr.toxo.custom_volcano_plot() already draw it, and a second implementation would drift). The report carries a panel that names where the volcano was written instead. It also never changes a fit: nothing here drops a well, refits, or reweights. It reports; the user decides.

Degrading by model type

spacr.ml.regression_model can return a statsmodels OLS, WLS, RLM, QuantReg, a GLM in any of six families, a MixedLM, a BetaModel, a sklearn Lasso/Ridge/ElasticNet or a LinearSVC. Those objects agree on almost nothing: a sklearn Lasso has no p-values, no covariance matrix and no fittedvalues; a MixedLM has no hat matrix; a Gaussian model has no calibration curve worth drawing. Every panel therefore either draws or raises PanelUnavailable carrying the reason, and the reason is printed on the combined report page and returned in the manifest. A panel is never silently omitted, and never faked with a substitute statistic that does not mean what the axis label says.

The trap here is model.scale. Every statsmodels results object has one, and it means a different thing on each of them — see resolve_residual_standardisation():

• OLS / MixedLM: the error variance, in the metric of y - fitted;

• WLS: the error variance in the metric of sqrt(w) * (y - fitted), which for cell-count weights is hundreds of times larger than the unweighted residual variance;

• RLM (rlm/huber): a robust estimate of the standard DEVIATION, so the variance is its square;

• QuantReg: the constant 1.0, a placeholder — quantile regression has no error-variance parameter at all;

• BetaModel: the constant 1.0, which is correct only against the Pearson residual, never against y - fitted;

• sklearn estimators: absent.

Treating all of those as “the error variance of y - fitted” is how a QC panel reports the wrong wells as outliers, in a direction that depends on the units of the response. Standardisation therefore goes through one registry keyed on the fitted model’s class, and where no correct scale exists the six panels that need one are SKIPPED with that stated on the report.

Reading the output

regression_qc_report(...) writes into <dst>/regression_qc/:

• one file per panel (residuals_vs_fitted and friends),

• regression_qc_report — every panel on one page, skipped panels shown as a grey box stating why,

• regression_qc_report.txt — the same thing as text, with the numbers, so it can be grepped or pasted into a lab notebook,

and returns a manifest dict listing every panel, its status, its path and the statistics it computed. The statistics are the point: the manifest is what a caller (or a test) reads to find out that Cook’s distance flagged well plate1_r3_c11, not just that a file exists.

THE FIGURE EXTENSIONS ARE LEFT OFF ABOVE ON PURPOSE. They follow the user’s figure-format preference, and writing .pdf down as if it were fixed is exactly how the code came to record residuals_vs_fitted.pdf in the manifest for a file that is residuals_vs_fitted.png on disk.

Figures are built through matplotlib.figure.Figure directly rather than through pyplot. That is deliberate: a Figure that pyplot never sees cannot be leaked into pyplot’s global registry, cannot be picked up by a later plt.savefig() in another module, and needs no plt.close() discipline to stay out of the way. This repo has been bitten by figure leaks more than once; this module cannot leak one.

Exceptions

PanelUnavailable

A panel cannot be computed from this model, for a stated reason.

Classes

PanelVerdict	What one diagnostic concluded, and what the number behind it was.
QCPanelResult	One panel's outcome.
RegressionQCContext	Everything the panels need, normalised across model types.
ResidualStandardisation	How a fit's residuals are put on a comparable scale, or why they are not.

Functions

build_context(model, X, y, *[, weights, metadata, ...])	Normalise a fitted model into the view the QC panels read.
calibration_curve(y_true, y_pred[, n_bins, weights, ...])	Bin predictions and return the observed frequency in each bin.
condition_number(X)	Return the condition number of a design matrix, scaled and unscaled.
condition_verdict(scaled_condition_number)	Return the plain-English reading of a scaled condition number.
context_from_model(model, *[, coef_df, ...])	Build a context from a fitted model that still carries its own design.
cooks_distance(std_resid, leverage, n_params)	Return Cook's distance per observation.
dffits(std_resid, leverage, n_obs, n_params)	Return DFFITS per observation, the change in that observation's own fit.
diagnose_p_value_histogram(p_values[, n_bins])	Classify the shape of a screen's p-value distribution.
draw_panel(name, ctx, ax)	Draw one panel onto an axes and return the statistics it computed.
draw_verdict(→ None)	Stamp a verdict onto the panel it belongs to.
format_qc_report(manifest)	Render a manifest as the plain-text report that is written next to the PDFs.
leverage_from_design(X[, weights])	Return the hat-matrix diagonal computed from a design matrix.
overdispersion_statistic(y, mu, df_resid[, variance, ...])	Return the Pearson dispersion of a count fit and its verdict.
panel_names([group])	Return the panel names, optionally restricted to one report section.
regression_qc_report(model, X, y, dst, *[, weights, ...])	Write the full regression QC suite and return a manifest of what was written.
residual_normality(resid, *[, min_n])	Skew, excess kurtosis and a normality P value for one residual vector.
resolve_residual_standardisation(model, resid, n_obs, ...)	Return how this fit's residuals can be standardised, or why they cannot.
score_panel(→ PanelVerdict)	The verdict for one panel's statistics. Never raises.
variance_inflation_factors(X[, tol])	Return the VIF of every non-constant column of a design matrix.
worst_verdict(verdicts)	The verdict a suite should be summarised by, or None when there is none.

Module Contents

exception spacr.regression_qc.PanelUnavailable

Bases: Exception

A panel cannot be computed from this model, for a stated reason.

Raised by panel drawing functions and caught by regression_qc_report(), which records the reason on the report rather than dropping the panel. The message is user-facing prose: it must say what was missing and, where there is one, what to do instead.

Parameters:

reason – Why the panel cannot be drawn.

Initialize self. See help(type(self)) for accurate signature.

class spacr.regression_qc.PanelVerdict

Bases: NamedTuple

What one diagnostic concluded, and what the number behind it was.

Parameters:

• level – one of VERDICT_LEVELS.

• headline – the phrase drawn on the panel. A CLAIM, not a label – “variance is stable across the fit”, never “homoscedasticity”.

• detail – one sentence saying what the score means and which rule produced it, for the text report and for a caption.

• score – the number the verdict was read off, or None where the verdict came from a categorical result.

• statistic – what that number IS, so the report can name it.

worse_than(other) → bool

Whether this verdict is the one a summary should report.

Parameters:

other – verdict whose severity is compared with this one.

property word: str

The badge text.

class spacr.regression_qc.QCPanelResult

One panel’s outcome.

Parameters:

• name – Stable machine name (also the file stem).

• title – Human-readable panel title.

• group – Report section: 'fit', 'influence', 'design', 'response' or 'screen'.

• status – 'written' (drawn, nothing missing), 'partial' (drawn, but with a stated limitation — e.g. a coefficient plot with no confidence intervals), 'skipped' (not computable, see reason) or 'failed' (raised unexpectedly, see reason).

• path – Absolute path of the per-panel figure, or None.

• reason – Why the panel was skipped, or what limits it.

• stats – Numbers the panel computed, for callers and tests.

• verdict – What the panel CONCLUDED – a PanelVerdict, or None for a panel that never drew. The design: a diagnostic that reports a number and no judgement leaves the judging to a reader who is not going to do it, so the verdict travels with the panel into the report, into the manifest and onto the picture itself.

class spacr.regression_qc.RegressionQCContext

Everything the panels need, normalised across model types.

Built by build_context(); panels only ever read it. Holding the normalisation in one place is what keeps twenty panels from each having their own opinion about where the residuals of a MixedLM live.

Parameters:

• model – The fitted model object (statsmodels results or sklearn estimator).

• X – Design matrix as a DataFrame, one row per well.

• y – Response, 1-D, aligned to X.

• fitted – Fitted values on the response scale.

• resid – y - fitted (response-scale residuals).

• std_resid – Internally studentised residuals, or all-NaN when this model class has no error scale (see resolve_residual_standardisation()). Panels must ask standardisation.available rather than test for NaN.

• leverage – Diagonal of the hat matrix, one entry per well.

• leverage_source – How leverage was obtained, so a panel can say so on the axis.

• scale – Error VARIANCE used to standardise the residuals, in the metric of standardisation.base — which is not always y - fitted. NaN when no correct scale exists for this model class.

• standardisation – The ResidualStandardisation that produced std_resid: what was standardised, where the variance came from, or the reason there is none.

• prediction_note – Set when the fitted values are not a conditional mean of y — a hinge/SVM classifier predicts class labels — so the response-scale panels can state that on the report instead of quoting an R² that means nothing.

• decision_score – The continuous score wells are RANKED by, which is not always fitted: a hinge/SVM’s fitted is the hard 0/1 label, and an ROC computed on hard labels has exactly two operating points and understates the model. build_context sets this to decision_function(X) for a classifier and to fitted for everything else, oriented so that LARGER means MORE LIKELY POSITIVE for both.

• labels – Per-well labels (prc where available), used to name outliers on the influence panels.

• weights – Per-well weights passed to the fit (cell counts, for the GLM-binomial path), or None.

• metadata – Per-well metadata (plateID/rowID/columnID/ cell_count/prc), or None.

• coef_df – The coefficient table built by spacr.ml.process_model_coefficients(), or None.

• regression_type – The spaCR regression type string, if known.

• family – Name of the GLM family, or 'Gaussian (least squares)'.

• link – Name of the link function, when there is one.

• volcano_path – Where the volcano plot for this run was written.

• notes – Free-text notes accumulated while building the context.

property fit_unit: str

Singular name for one row to which influence is attributed.

property is_binary_response: bool

True when every response value is exactly 0 or 1.

spaCR routes logit/probit through GLM-Binomial with a continuous fraction response weighted by cell count, so a binomial family does not imply binary labels — and ROC/PR are undefined without them. This is the distinction that decides it.

property is_binomial: bool

True when the response is a probability/fraction under a binomial family.

property is_classifier: bool

True when the fit is a discriminative classifier (hinge).

Separate from is_binomial, which asks whether a binomial likelihood was fitted. A hinge/SVM has no likelihood at all, so it is not binomial — but it is the one model spaCR offers whose whole output is a discrimination, which is what ROC and precision-recall measure. Excluding it from those two panels, which is what asking only is_binomial did, left the classifier as the single model type with no discrimination QC.

property is_count: bool

True when the response is modelled as a count (Poisson / negative binomial).

property n: int

Number of fitted design rows (one per well in ordinary designs).

property n_unique_wells: int

Independent well identifiers represented by the fitted rows.

Historical long-format screen OLS has one design row per well-guide pair, so n and the number of wells are not interchangeable. The metadata is the only trustworthy place to make that distinction.

property p: int

Number of columns in the design matrix, intercept included.

property ranking_score: numpy.ndarray

The score ROC and precision-recall rank wells by.

decision_score when the context carries one, otherwise fitted. Larger is more likely positive in both cases; see decision_score.

property sample_description: str

Human-readable fitted-row count without calling duplicates wells.

property standardisation_available: bool

True when std_resid means what its name says for this model.

class spacr.regression_qc.ResidualStandardisation

How a fit’s residuals are put on a comparable scale, or why they are not.

std_resid = base / sqrt(variance * (1 - h)). Both halves of that matter and both are model-class-dependent: base is a Pearson residual for a GLM and for beta regression, sqrt(w) * (y - fitted) for WLS and y - fitted for OLS; variance is model.scale for OLS, its square for RLM, and does not exist at all for quantile regression.

Parameters:

• available – True when a correct error scale exists for this fit. When it is False, base and variance are None/NaN and reason says why — the panels that need a standardised residual skip with that reason rather than standardise by a number that happens to be there.

• metric – What base is, in words, for the axis and the report.

• source – Where variance came from, in words.

• base – The residual that is standardised, length n.

• variance – The error variance in the metric of base.

• reason – Why no standardisation exists, when available is False.

spacr.regression_qc.build_context(model, X, y, *, weights=None, metadata=None, coef_df=None, regression_type=None, volcano_path=None)

Normalise a fitted model into the view the QC panels read.

The residual standardisation is the one piece of real statistics here, and it is resolved per model class by resolve_residual_standardisation():

std_resid = base / sqrt(variance * (1 - h))

where base and variance come from that registry — the Pearson residual and the GLM dispersion for a GLM or a beta fit, sqrt(w) * (y - fitted) and the weighted error variance for WLS, y - fitted and scale ** 2 for a robust fit, and nothing at all for quantile regression or a classifier. When the registry reports that no correct scale exists, std_resid is all-NaN, standardisation.reason says why, and the six panels built on a standardised residual skip with that reason.

Parameters:

• model – Fitted statsmodels results object or sklearn estimator.

• X – Design matrix used for the fit (DataFrame preferred).

• y – Response used for the fit.

• weights – Per-observation weights passed to the fit, if any.

• metadata – Per-well metadata frame; aligned by index, else by length.

• coef_df – Coefficient table from spacr.ml.process_model_coefficients().

• regression_type – The spaCR regression type string.

• volcano_path – Where the run’s volcano plot was written.

Returns:

RegressionQCContext.

Raises:

ValueError – if X and y disagree in length, or if metadata cannot be aligned to the fitted rows.

spacr.regression_qc.calibration_curve(y_true, y_pred, n_bins=10, weights=None, strategy='quantile')

Bin predictions and return the observed frequency in each bin.

Works for a binary response and for the continuous per-well fraction that spaCR’s logit/probit path actually fits: in both cases the question is “of the wells where the model said 0.3, what fraction were positive?”. With weights (cell counts) the observed value is the weighted mean, so a 2,000-cell well is not given the same say as a 20-cell one.

Parameters:

• y_true – Observed response in [0, 1].

• y_pred – Predicted probability in [0, 1].

• n_bins – Number of bins. Default 10.

• weights – Optional per-observation weights.

• strategy – 'quantile' (equal counts per bin, the default, which avoids sparsely populated bins) or 'uniform' (equal width).

Returns:

dict with pred_mean, obs_mean, counts, weight, ece (weighted mean absolute gap), max_gap and brier.

Raises:

ValueError – if the inputs disagree in length or n_bins < 2.

Example

>>> import numpy as np
>>> p = np.linspace(0.02, 0.98, 500)
>>> rng = np.random.default_rng(0)
>>> y = (rng.uniform(size=500) < p).astype(float)
>>> out = calibration_curve(y, p, n_bins=5)
>>> bool(out['ece'] < 0.1)          # a calibrated model hugs y = x
True

spacr.regression_qc.condition_number(X)

Return the condition number of a design matrix, scaled and unscaled.

Two numbers, because they answer different questions. The unscaled condition number is what statsmodels prints in a summary, and it is dominated by the units of the columns: a predictor measured in cells rather than thousands of cells changes it by 1000 with no change in the science. The scaled one (each column normalised to unit length first, the Belsley-Kuh-Welsch definition) is unit-free and is the one whose thresholds — 30, 100, 1000 — mean anything.

Parameters:

X – Design matrix, array-like (n, p).

Returns:

(scaled, unscaled, singular_values) where singular_values are those of the column-scaled matrix, largest first.

Example

>>> import numpy as np
>>> ortho = np.eye(3)
>>> round(condition_number(ortho)[0], 6)
1.0

spacr.regression_qc.condition_verdict(scaled_condition_number)

Return the plain-English reading of a scaled condition number.

Parameters:

scaled_condition_number – Output of condition_number().

Returns:

A short sentence naming the severity band.

spacr.regression_qc.context_from_model(model, *, coef_df=None, regression_type=None, metadata=None, weights=None, volcano_path=None)

Build a context from a fitted model that still carries its own design.

build_context() needs X and y because the pipeline has them: spacr.ml.regression() is the only scope where the fitted design exists, which is why the report is written from there. A caller who receives only the FITTED MODEL — the Qt results panel gets one on perform_regression’s return payload, and nothing else — has no design matrix to hand over, and had no way to compute a residual at all.

A statsmodels results object does carry it: results.model.exog is the matrix that was fitted and results.model.exog_names names its columns, so the design does not have to be reconstructed or guessed. This recovers it and defers everything statistical to build_context().

Parameters:

model – a fitted statsmodels results object.

Returns:

RegressionQCContext.

Raises:

PanelUnavailable – when the model does not carry its own design, so the caller can put the REASON on screen. sklearn’s Lasso, Ridge and ElasticNet — spaCR’s penalised backends — keep neither the design nor the response, and a diagnostics tab that went blank without saying so would look like a broken tab rather than a model that cannot answer.

spacr.regression_qc.cooks_distance(std_resid, leverage, n_params)

Return Cook’s distance per observation.

Uses the identity D_i = r_i^2 / p * h_i / (1 - h_i) where r_i is the internally studentised residual. Written this way it needs no second pass over the data and works for any model for which a studentised residual and a leverage exist — including the GLM families, where the textbook e_i^2 form would be on the wrong scale.

Parameters:

• std_resid – Internally studentised residuals, length n.

• leverage – Hat-matrix diagonal, length n.

• n_params – Number of estimated parameters p (intercept included).

Returns:

1-D array of length n; inf where h_i == 1 (an observation the model fits exactly, which is maximally influential by construction).

Example

>>> import numpy as np
>>> d = cooks_distance(np.array([0.1, 4.0]), np.array([0.2, 0.2]), 2)
>>> bool(d[1] > 100 * d[0])
True

spacr.regression_qc.dffits(std_resid, leverage, n_obs, n_params)

Return DFFITS per observation, the change in that observation’s own fit.

DFFITS_i = t_i * sqrt(h_i / (1 - h_i)) with t_i the externally studentised residual, obtained from the internally studentised one by t_i = r_i * sqrt((n - p - 1) / (n - p - r_i^2)). The conventional threshold is 2 * sqrt(p / n).

Parameters:

• std_resid – Internally studentised residuals.

• leverage – Hat-matrix diagonal.

• n_obs – Number of observations.

• n_params – Number of estimated parameters.

Returns:

(dffits, threshold) — the per-observation values (nan where the external studentisation is undefined) and the threshold.

spacr.regression_qc.diagnose_p_value_histogram(p_values, n_bins=20)

Classify the shape of a screen’s p-value distribution.

A screen in which most genes do nothing should give p-values that are uniform on [0, 1] with a spike in the first bin (the real hits). Two other shapes are diagnostic of a broken fit and are the reason this panel exists:

• a spike near 1 means the test is conservative — usually a variance component soaking up the signal, or duplicated rows inflating n;

• a U shape (both ends enriched) means the null is mis-specified — typically unmodelled plate structure, which pushes half the genes one way and half the other.

Parameters:

• p_values – Iterable of p-values; non-finite entries are dropped. Every remaining value must lie in the closed interval [0, 1].

• n_bins – Histogram resolution. Default 20 (bins of width 0.05).

Returns:

dict with verdict (one of 'uniform-with-spike', 'uniform', 'excess-large', 'u-shaped', 'anti-uniform', 'too-few'), message, counts, expected, first_bin_ratio, last_bin_ratio and frac_below_0.05.

Raises:

ValueError – if a finite value lies outside [0, 1].

Example

>>> import numpy as np
>>> rng = np.random.default_rng(1)
>>> diagnose_p_value_histogram(rng.uniform(size=2000))['verdict']
'uniform'
>>> spiky = np.concatenate([rng.uniform(0.9, 1.0, 800),
...                         rng.uniform(size=200)])
>>> diagnose_p_value_histogram(spiky)['verdict']
'excess-large'

spacr.regression_qc.draw_panel(name, ctx, ax)

Draw one panel onto an axes and return the statistics it computed.

This is the unit the tests drive: it takes an axes the caller owns, so the contents (how many points the scatter has, what the annotation says) can be asserted directly rather than inferred from a PDF’s existence.

Parameters:

• name – A name from PANEL_ORDER.

• ctx – RegressionQCContext from build_context().

• ax – A matplotlib Axes to draw into.

Returns:

dict of statistics; the exact keys are per panel.

Raises:

• KeyError – if name is not a known panel.

• PanelUnavailable – if the panel cannot be computed for this model, with the reason as the message.

spacr.regression_qc.draw_verdict(ax, verdict) → None

Stamp a verdict onto the panel it belongs to.

Parameters:

• ax – Matplotlib axes containing the diagnostic panel.

• verdict – panel verdict to stamp; None or an unknown verdict leaves the panel unchanged.

BOTTOM LEFT, in a box, because every panel in this module already spends its top corners on the statistics block and its own annotations – and a verdict written over a data point is a verdict that gets moved instead of read. The box is what separates it from the panel’s own annotation: this is the suite talking about the panel rather than the panel talking about the data.

spacr.regression_qc.format_qc_report(manifest)

Render a manifest as the plain-text report that is written next to the PDFs.

Parameters:

manifest – The dict returned by regression_qc_report().

Returns:

A multi-line string, one block per report section.

The text form exists because the PDF cannot be grepped and because the numbers — dispersion, condition number, the p-value verdict — are the part a reviewer quotes.

spacr.regression_qc.leverage_from_design(X, weights=None)

Return the hat-matrix diagonal computed from a design matrix.

h_i = x_i' (X' W X)^+ x_i * w_i. The pseudo-inverse is used rather than the inverse because screen design matrices are routinely rank deficient (a gRNA present in exactly one well produces a column that is a multiple of another); a LinAlgError there would take the whole QC report down for a property of the data that the report exists to show.

Parameters:

• X – Design matrix, (n, p), array-like.

• weights – Optional per-observation weights (IRLS weights, or the var_weights spaCR passes for the cell-count-weighted binomial fit).

Returns:

1-D array of length n, each entry in [0, 1].

Example

>>> import numpy as np
>>> X = np.column_stack([np.ones(4), [0., 0., 0., 10.]])
>>> h = leverage_from_design(X)
>>> bool(h[3] > h[0])          # the far-out point has high leverage
True
>>> bool(abs(h.sum() - 2) < 1e-9)   # trace(H) == p for a full-rank X
True

spacr.regression_qc.overdispersion_statistic(y, mu, df_resid, variance=None, weights=None)

Return the Pearson dispersion of a count fit and its verdict.

phi = sum((y - mu)^2 / V(mu)) / df_resid. Under a correctly specified Poisson model phi == 1. phi well above 1 means the standard errors are too small by sqrt(phi) — at phi = 6, a 2.4-fold inflation, which turns noise into a screen full of hits. That is why this number is on the report as a number and not as a shape to eyeball.

Parameters:

• y – Observed counts.

• mu – Fitted means.

• df_resid – Residual degrees of freedom (n - p).

• variance – Callable V(mu); defaults to the Poisson V(mu) = mu.

• weights – Optional per-observation weights.

Returns:

dict with dispersion, pearson_chi2, df_resid and verdict.

Example

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> mu = np.full(500, 5.0)
>>> y = rng.poisson(5.0, size=500).astype(float)
>>> out = overdispersion_statistic(y, mu, 499)
>>> bool(0.7 < out['dispersion'] < 1.4)
True

spacr.regression_qc.panel_names(group=None)

Return the panel names, optionally restricted to one report section.

Parameters:

group – 'fit', 'influence', 'design', 'response', 'screen' or None for all.

Returns:

Tuple of panel names in report order.

Raises:

ValueError – on an unknown group.

spacr.regression_qc.regression_qc_report(model, X, y, dst, *, weights=None, metadata=None, coef_df=None, regression_type=None, volcano_path=None, panels=None, fmt=None, combined=True, strict=False, verbose=True, renderer=None)

Write the full regression QC suite and return a manifest of what was written.

Every panel is attempted. A panel that cannot be computed for this model — no p-values from a Lasso, no plate column in the metadata, no calibration for a Gaussian fit — is recorded as skipped with the reason, shown as a grey tile on the combined page and listed in the text report. It is never dropped in silence, because “the panel is not there” and “the panel is fine” must not look the same.

Parameters:

• model – Fitted model, as returned by spacr.ml.regression_model().

• X – The design matrix that was fitted (DataFrame preferred, so the panels can name the predictors).

• y – The response that was fitted.

• dst – Results folder for the run. The report goes into <dst>/regression_qc/.

• weights – Per-observation weights passed to the fit (spaCR passes cell counts for the GLM-binomial path).

• metadata – Per-well frame carrying plateID / rowID / columnID / prc / cell_count. Aligned to the fitted rows by index, or by length; anything else raises rather than risk labelling the wrong well.

• coef_df – The coefficient table from spacr.ml.process_model_coefficients(), used for the p-value histogram so it shows the screen’s p-values.

• regression_type – The spaCR regression type string, used to pick the family-specific panels when the model object does not say.

• volcano_path – Path of the volcano plot for this run, named on the report instead of drawing a second volcano.

• panels – Optional subset of PANEL_ORDER.

• fmt – force a figure format for the individual panels. None, the default, lets the user’s figure-format preference decide – which is what it used to say 'pdf' for, so a user who had chosen PNG got PDFs anyway and the manifest named files that were not there. An explicit format still wins, for a caller that genuinely needs one.

• combined – Also write the single-page multi-panel report.

• strict – Re-raise a panel’s unexpected exception instead of recording it as failed. Tests use this; the pipeline should not, because a broken diagnostic must not take down a fit that already succeeded.

• verbose – Print the destination and any failure.

• renderer – force 'pyqtgraph' or 'matplotlib'. None asks spacr.figures.scene.scene_renderer(), which prefers the screen’s renderer and falls back where Qt is not available – and the choice is made ONCE here rather than per panel, so a suite cannot come out half in one library and half in the other over an environment that changed while it ran.

Returns:

dict manifest with directory, combined, report, panels (list of QCPanelResult), written, skipped, failed, renderer and the model description.

Raises:

ValueError – if dst is falsy — a QC report nobody can find is worse than no QC report.

Example

from spacr.regression_qc import regression_qc_report
manifest = regression_qc_report(
    model, X, y, dst=res_folder,
    metadata=merged_df.loc[X.index,
                           ['plateID', 'rowID', 'columnID',
                            'prc', 'cell_count']],
    coef_df=coef_df, regression_type='ols')
print(len(manifest['written']), 'panels written')

spacr.regression_qc.residual_normality(resid, *, min_n=NORMALITY_MIN_N)

Skew, excess kurtosis and a normality P value for one residual vector.

Below min_n residuals, the K-squared test is not run. In that case normality_p is NaN and test contains NORMALITY_TOO_FEW; callers should display both fields.

Parameters:

• resid – residuals; non-finite entries are dropped first.

• min_n – fewest residuals the test will run on. Default NORMALITY_MIN_N.

Returns:

{'skew', 'excess_kurtosis', 'normality_statistic', 'normality_p', 'test', 'n'}. excess_kurtosis is Fisher’s, so 0 is normal.

>>> import numpy as np
>>> out = residual_normality(np.arange(4.0))
>>> out["test"]
'normality test needs n >= 8'

spacr.regression_qc.resolve_residual_standardisation(model, resid, n_obs, n_params)

Return how this fit’s residuals can be standardised, or why they cannot.

This is the one place that knows what model.scale means, and it knows it per model class rather than per attribute: an attribute that exists is not an attribute that means what the caller hoped. See _SCALE_RESOLVERS for the table and the module docstring for why it is not one formula.

Parameters:

• model – Fitted statsmodels results object or sklearn estimator.

• resid – Response-scale residuals y - fitted, length n_obs.

• n_obs – Number of observations.

• n_params – Number of columns in the design matrix.

Returns:

ResidualStandardisation. Check .available before reading .base / .variance.

Example

>>> import numpy as np, statsmodels.api as sm
>>> rng = np.random.default_rng(0)
>>> X = np.column_stack([np.ones(60), rng.normal(size=60)])
>>> y = X @ [1.0, 2.0] + rng.normal(size=60)
>>> fit = sm.RLM(y, X).fit()
>>> std = resolve_residual_standardisation(fit, y - fit.fittedvalues, 60, 2)
>>> bool(np.isclose(std.variance, fit.scale ** 2))   # a variance, not an SD
True
>>> resolve_residual_standardisation(
...     sm.QuantReg(y, X).fit(q=0.5), np.zeros(60), 60, 2).available
False

spacr.regression_qc.score_panel(name, stats) → PanelVerdict

The verdict for one panel’s statistics. Never raises.

Parameters:

• name – a name from PANEL_ORDER.

• stats – the dict draw_panel() returned.

Returns:

a PanelVerdict. An unknown panel, absent statistics or a scorer that trips over an unexpected value all come back as 'unknown' rather than raising – a diagnostic that crashes while judging a diagnostic would take down a fit that already succeeded, for the sake of a sentence.

spacr.regression_qc.variance_inflation_factors(X, tol=1e-10)

Return the VIF of every non-constant column of a design matrix.

Computed from the inverse of the predictor correlation matrix — the identity VIF_j = (R^-1)_jj — rather than by running p auxiliary regressions. On a screen design with 1,200 gRNA columns the auxiliary- regression route is O(p^4) and is impractical; the correlation route is one O(p^3) decomposition. The two agree exactly when the design contains an intercept (see the test that pins this against statsmodels.stats.outliers_influence.variance_inflation_factor).

Constant columns (the intercept, and any predictor that is constant on the rows that survived cleaning) have no VIF — a variance of zero cannot be inflated — and are reported as NaN rather than dropped, so the caller can see that they were there.

Exactly collinear columns get inf: they are identified by the near-null eigenvectors of the correlation matrix, so which columns are aliased is reported rather than the whole matrix being declared unusable.

Parameters:

• X – Design matrix; DataFrame or array-like.

• tol – Relative eigenvalue below which a direction counts as null.

Returns:

pandas.Series of VIFs indexed by column name.

Example

>>> import numpy as np, pandas as pd
>>> rng = np.random.default_rng(0)
>>> a = rng.normal(size=200)
>>> df = pd.DataFrame({'a': a, 'b': a + 0.01 * rng.normal(size=200),
...                    'c': rng.normal(size=200)})
>>> vif = variance_inflation_factors(df)
>>> bool(vif['a'] > 100), bool(vif['c'] < 2)
(True, True)

spacr.regression_qc.worst_verdict(verdicts)

The verdict a suite should be summarised by, or None when there is none.

Parameters:

verdicts – iterable of panel verdicts or None entries to compare.

THE WORST ONE, not the commonest and not an average. Nineteen panels passing and one saying the design is rank deficient is a run whose coefficients are one of infinitely many solutions, and a summary that reports “95% passed” is a summary that hides exactly the panel the suite was run for.

Nested helpers

_write_qc_numbers._plain(value)

Whatever JSON can hold, and a string for everything else.

spacr/regression_qc.py:4024

condition_number._ratio(sv)

Convert descending singular values to a stable condition number.

Parameters:

sv – non-empty descending singular-value array from the design.

Returns:

largest divided by smallest as a float, or infinity when the smallest is at or below NumPy’s numerical-rank tolerance computed from its dtype and the captured design shape.

spacr/regression_qc.py:600