spacr.qt.widgets.outlier_model

Robust outlier detection over a measurement table — per object and per well.

Not a z-score

The obvious thing to write here is |x - mean| / sd > 3, and it is the one thing this module refuses to do. spaCR object measurements are skewed and heavy-tailed by construction: cell_area is a positive quantity with a long right tail, pathogen_area is zero-inflated, an intensity ratio has a denominator that can approach zero. The mean and the SD of such a column are themselves dragged by the very objects they are supposed to find — one debris particle measured at 40× the median area moves the mean, inflates the SD, and so raises the threshold that was meant to catch it. The failure is silent and it gets worse the worse the contamination is: this is the masking effect, and it is the reason robust statistics exist.

Every method here therefore estimates the centre and the spread from statistics that a minority of arbitrary values cannot move: the median, the quartiles, and — in several dimensions — the Minimum Covariance Determinant.

Three methods, one idea

1. MAD — METHOD_MAD

Flag when |x - median(x)| / (1.4826 * MAD) > k, default k = 3.5 (DEFAULT_MAD_K), where MAD = median(|x - median(x)|).

The 1.4826. For a Gaussian, MAD = Phi^-1(0.75) * sigma = 0.6745 * sigma, so sigma = MAD / 0.6745 = 1.4826 * MAD. Multiplying by MAD_TO_SIGMA makes the MAD a consistent estimator of sigma under normality, which is what lets the threshold be read on the familiar scale — “3.5 SD” — while never being computed from an SD that the outliers could inflate. The constant is the whole reason k is comparable to a z-score cut-off at all; without it, k would be in units of “MADs” and 3.5 would mean something different for every column.

MAD == 0 is handled, not divided by. When more than half the objects share one value — a zero-inflated pathogen_area, an integer count, a saturated intensity — the MAD is exactly zero. Naively, every value that differs at all then scores infinity and the entire tail is flagged, which is the opposite of robust. So robust_scale() falls back to the mean absolute deviation from the median, scaled by MEAN_AD_TO_SIGMA = sqrt(pi/2) = 1.2533 (for a Gaussian E|X - mu| = sigma * sqrt(2/pi)), and says so in OutlierResult.notes and in OutlierResult.caveats().

The alternative considered and not taken is the Rousseeuw–Croux Sn (1.1926 * med_i med_j |x_i - x_j|), which is the statistically better answer: it has a 50% breakdown point, needs no symmetry assumption, and does not collapse on tied data. It is not used because the naive form is O(n^2) — unusable on the 10^5–10^6 object rows spaCR routinely produces — and the O(n log n) algorithm is a page of intricate code to maintain for a path that is only ever reached when the MAD has already collapsed. The mean absolute deviation has a 0% breakdown point and that is stated plainly rather than glossed: it is a stopgap that keeps a degenerate column from flagging its whole tail, not a robust estimator. If the fallback fires, the honest reading is that the column is not continuous enough for this test.

A column with no variation at all (mean absolute deviation zero too) is scored 0 everywhere and flagged nowhere. One value cannot be an outlier from itself.

2. IQR / Tukey — METHOD_IQR

Flag outside [Q1 - c*IQR, Q3 + c*IQR], default c = 1.5 (DEFAULT_IQR_C).

The asymmetry problem is real and is not fixed by choosing a bigger c. Tukey’s fence is symmetric in the quartiles, and a right-skewed distribution therefore has far more mass beyond the upper fence than beyond the lower one by construction, whatever the data. For a lognormal with sigma = 1 the standard fence flags about 8% of the sample on the right and essentially nothing on the left — and it does so for a perfectly clean sample. Read uncorrected, “8% of my cells are outliers” is a statement about the shape of the distribution, not about the cells.

The offered remedy is a log10 transform, TRANSFORM_LOG10, which turns a lognormal into a Gaussian and makes the fence mean what it appears to mean. It is never implicit: a transform silently applied would change what outlier_score is in units of without the user asking. And it refuses rather than fudges on non-positive values — log10 of a zero pathogen_area is -inf, and an implicit +1 or a dropped row would be an invented measurement or a self-selected population. The refusal names the feature and the count.

OutlierResult.caveats() says the asymmetry out loud whenever the IQR rule ran untransformed and the flags landed lopsidedly.

3. Robust Mahalanobis on an MCD covariance — METHOD_MAHALANOBIS

Fit sklearn.covariance.MinCovDet — the Minimum Covariance Determinant — to the selected features, take each object’s squared Mahalanobis distance to the robust centre in the robust metric, and flag it against a chi-square quantile.

Why MCD and not an isolation forest. Both are “multivariate outlier detectors” and they are not interchangeable:

• The threshold is a stated false-positive rate, not a tuned fraction. Under multivariate normality the squared robust Mahalanobis distance is distributed chi-square with p degrees of freedom, so the cut point is chi2.ppf(1 - alpha, p) for a chosen alpha — default DEFAULT_ALPHA = 0.001, meaning one clean object in a thousand is expected to be flagged. Over 10^5 objects that is about 100 false flags, and over the 200,000 objects of a typical two-plate screen it is about 200. That number is knowable in advance and is printed in OutlierResult.caveats(). An isolation forest’s score has no such null distribution and therefore no calibrated cut point at all.

• Isolation forest requires contamination — the share of the data that is bad — declared in advance. That share is precisely the unknown the analysis is trying to estimate. Setting contamination=0.01 guarantees 1% of the objects come back flagged whether the plate is pristine or ruined, which makes the output uninterpretable in exactly the case it matters.

• MCD is affine-equivariant. Rescale area from px^2 to um^2, or rotate to any linear combination of the features, and the fitted centre and covariance transform with it: every distance, and so every flag, is unchanged. This is the same invariance argument spacr.qt.widgets.pca_model makes for standardising before a PCA, and it is why no scaling option is needed here — the metric is the covariance. A tree ensemble splits on raw coordinates and is not equivariant; its answer depends on the units.

• MCD is the multivariate generalisation of the median and the MAD. The MCD centre is a multivariate median (the mean of the h-subset of minimum covariance determinant) and the MCD scatter is its spread. So all three methods offered here are the same idea at one, one, and p dimensions, and a user who understands the MAD already understands this.

• MCD is deterministic given a seed. The subset search is randomised, so random_state is set from OutlierSpec.seed and the same table gives the same flags every time — a figure in a report matches the screen it came from. An isolation forest is a randomised ensemble whose per-object score wobbles between fits.

The cost, stated. MCD needs n > p to have a covariance at all, and it is unstable until roughly n >= 2p (MCD_MIN_OBJECTS_PER_FEATURE * p): the estimate is fitted on a subset of about n/2 rows, so n = 2p already means the subset is only barely larger than the dimension. Its breakdown point is set by support_fraction: the sklearn default of (n + p + 1) / 2n gives the maximum ~50%, and raising it trades robustness for efficiency. Too few objects for the number of features is refused with a message that names the way out: run a PCA first (spacr.qt.widgets.pca_model.pca()) and flag on the first few components, which are fewer, uncorrelated, and carry the same information.

Multiple testing is not optional here. Flagging N objects at level alpha is N hypothesis tests; OutlierResult.caveats() states both the expected count of false flags at the chosen alpha and the Bonferroni-corrected alpha (alpha / N) that would hold the family-wise rate at the nominal level, so the user can pick which of the two questions they are asking.

Per object AND per well

The whole-plate failure that actually happens is a bad well: one well pipetted wrong, one well out of focus, one well where the cells were seeded at twice the density. Per-object flags are close to useless for finding it, and the reason is arithmetic rather than opinion.

Take 60 objects per well, a feature whose within-well spread is sigma, and a well whose whole population is shifted by 0.9 sigma. The object rule at k = 3.5 asks each object to exceed 3.5 sigma; a 0.9 sigma shift moves that requirement to 2.6 sigma, which about one object in 200 clears — so the bad well contributes a fraction of one extra flagged object and disappears into the ordinary tail. The well rule compares well medians, whose sampling spread is only 1.2533 * sigma / sqrt(60) = 0.16 sigma; the same shift is then 5.6 robust SDs and the well is unmistakable. The two tests have wildly different power against the same defect, which is why both are run and both are reported.

So the well pass is emphatically not “the well contains many flagged objects”. That statistic exists — it is reported as flagged_share — and it answers a different question: it finds a well containing a few catastrophic objects (a segmentation blow-up, a piece of dust measured as a cell) while being blind to a uniform shift. The well-level robust score finds the uniform shift while being blind to the isolated catastrophe. Both are on the well frame because neither subsumes the other.

The well pass reduces each well to a robust summary — the median of each feature over the well’s objects, plus n — and then runs the same rule (METHOD_MAD, METHOD_IQR or METHOD_MAHALANOBIS) across wells. The median rather than the mean for the same reason as everywhere else here: a single blown-up object must not become a bad well.

A well is not scored on three objects. Below DEFAULT_MIN_WELL_OBJECTS objects the median is too noisy for a comparison across wells to mean anything, so the well appears in the well frame with well_scored = False and a reason saying how many objects it had. It is never silently dropped and never quietly scored — a low-n well excluded without a word is how an empty well becomes an unremarkable row.

Well identity comes from the columns spaCR already writes, in WELL_KEY_SETS order: the canonical (plateID, rowID, columnID) of spacr.schema.WELL_KEY_COLUMNS, then the composed prc key, then (plateID, well), then the legacy spellings spacr.schema renames (plate / row_name / column_name), then a bare well for a single-plate table. Any list of columns may be given explicitly instead. When none can be found the error names the columns the table actually has, and points at per_well=False for a table that has no wells at all.

Flagging is never deletion

detect_outliers() adds columns; it never drops a row and never changes one. OutlierResult.object_frame() returns the input frame with outlier / outlier_score / outlier_reason / outlier_method appended, and len() of it always equals len() of the input. A name that would collide with an existing column is suffixed rather than overwritten, and the substitution is reported in OutlierResult.column_names and in the notes — a QC pass that silently overwrote a user’s own outlier column would be a data-loss bug.

OutlierResult.filtered() exists and drops the flagged rows. It is an explicit call whose docstring says, in those words, that the caller is choosing to delete objects from their analysis.

No Qt in here

Pure numpy, pandas, scipy and scikit-learn, like spacr.qt.widgets.pca_model and spacr.qt.widgets.graph_spec: usable from a notebook, testable without a display, and free for the screen — and for any later QC report — to reuse without inheriting a widget.

Exceptions

OutlierError

An outlier test that cannot mean anything, with the reason in the message.

Classes

OutlierResult	Flags, scores and reasons — per object and per well — plus the caveats.
OutlierSpec	What to test, how, and where the line is.

Functions

candidate_features(→ Tuple[str, ...])	The columns worth offering as outlier features, sorted.
detect_outliers(→ OutlierResult)	Flag outlying objects and outlying wells in frame.
median_absolute_deviation(→ float)	median(|x - median(x)|) over the finite values. Unscaled.
robust_scale(→ Tuple[float, float, str])	(median, sigma estimate, note) for one feature.
tukey_fences(→ Tuple[float, float, float, float, str])	(q1, q3, lower fence, upper fence, note) for one feature.
well_key_columns(→ Tuple[str, ...])	The columns that identify a well in frame, auto-detected.

Module Contents

exception spacr.qt.widgets.outlier_model.OutlierError

Bases: ValueError

An outlier test that cannot mean anything, with the reason in the message.

Raised rather than returned as an empty result. Every one of these is a sentence the user can act on — “cell_area has 412 non-positive values and log10 of them is not a number”, “3 features and 4 complete objects is fewer than the 6 the MCD needs” — and a caller that swallowed it would show an empty table with nothing to explain it. Screens catch it and print the message.

Initialize self. See help(type(self)) for accurate signature.

class spacr.qt.widgets.outlier_model.OutlierResult

Flags, scores and reasons — per object and per well — plus the caveats.

Parameters:

• method – the METHODS member that produced this.

• features – the columns actually tested, in matrix order.

• scores – one per input row, positionally aligned with the frame detect_outliers() was given. nan for a row that could not be scored. Positional because a measurement frame carries a duplicated or reset index often enough that positions are the only safe currency — the same reason PCAResult uses them.

• flags – one per input row. True is “flagged”, never “deleted”.

• reasons – one per input row; "" for an unflagged, scored row.

• threshold – the number scores was compared against.

• n_rows_in – number of rows in the frame that was analysed; object_frame() and filtered() require a frame of this length.

• n_scored – how many of those rows received a finite score.

• centres – per feature, the robust centre used (the median, or the MCD centre’s coordinate).

• scales – per feature, the robust sigma used. Empty for METHOD_MAHALANOBIS, whose scale is a matrix.

• fences – per feature, (low, high) for METHOD_IQR.

• wells – one row per well, always including the wells that were not scored. See well_frame().

• well_keys – the columns wells is keyed on. Empty when the well pass did not run.

• column_names – logical name -> the column object_frame() will actually add, which differs from OBJECT_COLUMNS only when the input already had that name.

__len__() → int

Objects the test was given — not the objects it flagged.

Deliberately the input count: a length that shrank with the flags would make len(result) mean something different for a clean plate and a ruined one.

caveats() → Tuple[str, ...]

Everything a reader needs before believing the flags.

filtered(source: pandas.DataFrame) → pandas.DataFrame

source without the flagged rows.

Calling this is choosing to delete objects from your analysis. Nothing else in this module removes a row; detect_outliers() only ever adds columns, so that a flag can be reviewed, argued with and reversed. This method is the explicit escape hatch for a caller who has looked at the flags and decided.

Objects that could not be scored are kept — they were not flagged, and dropping them here would delete rows for missingness under the name of outlier removal.

Parameters:

source – the frame detect_outliers() was given, or one with the same number of rows in the same order; any other length raises OutlierError.

flagged_wells() → Tuple[Any, ...]

The wells the across-well rule flagged, as key tuples.

A one-column key yields plain scalars rather than one-tuples, because prc is already the whole name of the well and wrapping it would make every caller unwrap it.

headline() → str

One sentence with the counts, including the bad news.

method_label() → str

The method and its threshold in one token, for the added column.

object_frame(source: pandas.DataFrame) → pandas.DataFrame

source with the flag columns added. Nothing is dropped.

len() of the returned frame equals len(source), always, and no input column is touched. A name that would collide with one of source’s own columns is suffixed — see column_names for what was actually written.

Parameters:

source – the frame detect_outliers() was given, or one with the same number of rows in the same order; any other length raises OutlierError.

Raises:

OutlierError – when source is not the frame that was analysed. The flags are positional, so silently aligning them to a frame of a different length would attribute one object’s badness to another.

report() → str

The whole story, as the panel shows it and a report file writes it.

unscored_wells() → Tuple[Any, ...]

The wells reported but not scored — too few objects, in the same key form as flagged_wells().

well_frame() → pandas.DataFrame

One row per well: its key, its n, its medians, and both signals.

Both, on purpose. flagged_share is the share of the well’s objects the per-object rule flagged — it finds a well with a few catastrophic objects in it. well_outlier_score is the same robust rule applied across wells to their medians — it finds a well whose whole population is shifted, which the object rule is nearly blind to. See the module docstring for the arithmetic of why they differ.

Wells with fewer than min_well_objects objects are present with well_scored = False and a reason. They are never dropped.

property flagged_share: float

Flagged objects over scored objects, in [0, 1].

property has_wells: bool

Whether the across-well pass produced anything.

property n_features: int

How many columns were tested.

property n_flagged: int

Objects flagged.

property n_not_scored: int

Objects with no usable value, which are never flagged.

property n_wells: int

Wells found, scored or not. 0 when the well pass did not run.

class spacr.qt.widgets.outlier_model.OutlierSpec

What to test, how, and where the line is.

Frozen and JSON round-tripping, like PCASpec and GraphSpec, so a QC pass is something a settings file or a methods section can carry verbatim.

Parameters:

• features – the columns to test. Empty means candidate_features() of whatever frame it runs against — a default, not a promise; the result records what it actually used.

• method – one of METHODS.

• k – modified-z threshold for METHOD_MAD.

• c – fence multiplier for METHOD_IQR.

• alpha – per-object false-positive rate for METHOD_MAHALANOBIS. The chi-square cut point is chi2.ppf(1 - alpha, p).

• transform – TRANSFORM_NONE or TRANSFORM_LOG10. Applied before anything else and never chosen for the user.

• well_keys – columns identifying a well. Empty means well_key_columns() auto-detects them.

• min_well_objects – a well with fewer objects than this is reported “not scored” rather than scored or dropped.

• per_well – run the across-well pass. False for measurements that did not come off a plate — and the only thing that makes a table with no well columns analysable at all.

• support_fraction – MCD’s support_fraction. None is sklearn’s default (n + p + 1) / 2n, the maximum-breakdown choice.

• seed – MCD’s random_state, so the same table gives the same flags every time.

Raises:

OutlierError – on an unknown method or transform, a non-positive threshold, an alpha outside (0, 1) or a support fraction outside (0, 1] — at the point the spec is built, not half way through a 200,000-row fit.

__post_init__() → None

De-duplicate the features and well keys, and validate the detector.

Raises:

OutlierError – if the method or transform is not one this module offers; if k or c is not positive – they are how many robust SDs and how many IQRs place the fence; if alpha is not a per-object false-positive rate in (0, 1); if min_well_objects is below 1, since a well with no objects has no median; or if support_fraction is given and is not in (0, 1] – None picks sklearn’s maximum-breakdown default.

describe() → str

One line, for a caption.

classmethod from_dict(payload: Mapping[str, Any]) → OutlierSpec

Rebuild from to_dict(); unknown keys ignored, missing keys defaulted, so a QC pass written by another build still opens.

Parameters:

payload – mapping written by to_dict(); only the spec’s own field names are read, and the constructor still validates them.

classmethod from_json(text: str) → OutlierSpec

Inverse of to_json().

Parameters:

text – JSON text written by to_json().

threshold(n_features: int = 1) → float

The number OutlierResult.scores is compared against.

k for METHOD_MAD, c for METHOD_IQR, and the chi-square quantile chi2.ppf(1 - alpha, p) for METHOD_MAHALANOBIS — which is the one that depends on how many features are in play, because the null distribution does.

to_dict() → Dict[str, Any]

A plain JSON-able dict. Every field, always — a stable schema beats a compact one for something a report reads back.

to_json() → str

to_dict() as sorted JSON.

with_features(features: Sequence[str]) → OutlierSpec

A copy testing features.

Parameters:

features – column names to test; stored as a de-duplicated tuple.

with_method(method: str) → OutlierSpec

A copy using method.

Parameters:

method – one of METHODS ("mad", "iqr" or "mahalanobis"); anything else raises OutlierError.

with_transform(transform: str) → OutlierSpec

A copy applying transform first.

Parameters:

transform – one of TRANSFORMS ("none" or "log10"); anything else raises OutlierError.

with_well_keys(keys: Sequence[str]) → OutlierSpec

A copy grouping wells by keys.

Parameters:

keys – columns that together identify a well; stored as a de-duplicated tuple.

spacr.qt.widgets.outlier_model.candidate_features(frame: pandas.DataFrame) → Tuple[str, ...]

The columns worth offering as outlier features, sorted.

Continuous by spacr.qt.widgets.graph_spec.column_kinds(), exactly as spacr.qt.widgets.pca_model.candidate_features() picks its features — the one column classifier in this codebase, re-read rather than re-derived, so PCA and this screen cannot disagree about what cell_count is.

That rule already excludes the object keys (they identify rather than describe, and the modified z-score of a prcfo is nonsense) and the small-cardinality numeric codes — a class label, a plate number — which are labels rather than measured quantities.

A user who wants a particular column anyway names it in OutlierSpec.features; nothing here refuses it.

Parameters:

frame – measurement table whose columns are classified.

spacr.qt.widgets.outlier_model.detect_outliers(frame: pandas.DataFrame, spec: OutlierSpec | None = None) → OutlierResult

Flag outlying objects and outlying wells in frame.

The policy is in the module docstring; the short version is that nothing is estimated from a mean or an SD, that a zero MAD falls back to a stated alternative rather than dividing by zero, that a log transform is never applied unless asked for, that the multivariate test is an MCD Mahalanobis distance cut at a chi-square quantile so its threshold is a stated false-positive rate, and that no row is ever dropped or altered — the result adds columns.

The well pass reduces each well to the median of each feature and runs the same rule across wells, because a well shifted as a whole flags almost none of its individual objects and is nevertheless unmistakable as a point among wells.

Parameters:

• frame – an object-level measurement table.

• spec – what to test and how. None is OutlierSpec’s defaults: the MAD rule at k = 3.5 over every continuous column, with the well pass on.

Returns:

an OutlierResult. Write the flags back with OutlierResult.object_frame().

Raises:

OutlierError – whenever the answer would be meaningless — with the reason and the way out in the message.

spacr.qt.widgets.outlier_model.median_absolute_deviation(values: Sequence[float]) → float

median(|x - median(x)|) over the finite values. Unscaled.

Returned raw rather than pre-multiplied by MAD_TO_SIGMA so that a caller can see the zero when it is zero — which is the case the whole fallback in robust_scale() exists for.

Parameters:

values – the feature’s values; they are converted to float and every non-finite entry is ignored.

Returns:

the MAD, or nan when nothing is finite.

spacr.qt.widgets.outlier_model.robust_scale(values: Sequence[float]) → Tuple[float, float, str]

(median, sigma estimate, note) for one feature.

The sigma estimate is MAD_TO_SIGMA times the MAD whenever the MAD is non-zero. When it is zero — more than half the objects share one value — dividing by it would score every other value infinity and flag the entire tail, so the estimate falls back to MEAN_AD_TO_SIGMA times the mean absolute deviation from the median and note says "mad-zero". See the module docstring for why that fallback and not Rousseeuw–Croux Sn, and for the honest reading of it.

Parameters:

values – the feature’s values; they are converted to float and every non-finite entry is ignored.

Returns:

(centre, scale, note). note is "" when the MAD did the work, "mad-zero" when the fallback fired, "constant" when the feature has no variation at all (scale 0, nothing can be an outlier), and "empty" when there is nothing finite to measure.

spacr.qt.widgets.outlier_model.tukey_fences(values: Sequence[float], c: float = DEFAULT_IQR_C) → Tuple[float, float, float, float, str]

(q1, q3, lower fence, upper fence, note) for one feature.

Quartiles by numpy.percentile’s default linear interpolation, fences at Q1 - c*IQR and Q3 + c*IQR.

When the IQR is zero — half the objects tied at one value — the fences collapse onto that value and everything else is “outside” them. Rather than flag the whole tail, the quartiles are reconstructed from the robust sigma of robust_scale() at their Gaussian positions (median +/- 0.6745 sigma, IQR = 1.349 sigma), which reproduces the ordinary fence exactly for Gaussian data and is reported as "iqr-zero".

Parameters:

values – the feature’s values; they are converted to float and every non-finite entry is ignored.

Returns:

(q1, q3, low, high, note); note is "", "iqr-zero", "constant" or "empty".

spacr.qt.widgets.outlier_model.well_key_columns(frame: pandas.DataFrame) → Tuple[str, ...]

The columns that identify a well in frame, auto-detected.

Tries WELL_KEY_SETS in order and returns the first set whose columns are all present.

Parameters:

frame – measurement table whose column names are searched.

Raises:

OutlierError – when none of them is, with the table’s own columns in the message — the user can then either name the right ones explicitly or turn the well pass off.