spacr.qt.screens.agreement

Annotator Agreement — how much do two annotation passes actually agree?

The Annotate app writes one INTEGER column per pass into png_list. Two people labelling the same crops therefore leave two columns side by side, and the only question that matters before either set is used as ground truth is: do they agree, and where don’t they?

Layout:

┌───────────────────────────────────────────────────────────────────┐
│ /data/plate1/measurements/measurements.db  [DB…] [Run folder…]    │
│ Read-only (mode=ro) — nothing here can modify your annotations.   │
├──────────────┬────────────────────────────────────────────────────┤
│ Annotators   │ a       b      n   abstain   raw    κ   band       │
│ ☑ alice      │ alice   bob  410      92   88.0% +0.61 substantial │
│ ☑ bob        ├────────────────────────────────────────────────────┤
│ ☐ carol      │ Confusion  [alice vs bob ▾]     1     2            │
│              │                              1  310    22          │
│ [Compute]    │                              2   27    51          │
├──────────────┴────────────────────────────────────────────────────┤
│ Overall Cohen's κ +0.61 (substantial) · raw 88.0% · 49 to review   │
├───────────────────────────────────┬───────────────────────────────┤
│ Disagreements  png_path  alice bob│      [crop preview]           │
│  …/A01_1_cell_3.png        1    2 │                               │
└───────────────────────────────────┴───────────────────────────────┘

Design notes:

• Read-only, structurally. Every connection goes through spacr.agreement. It opens the file with file:…?mode=ro and PRAGMA query_only = ON, the same approach the Database Browser takes. Adjudicating a disagreement is a job for the Annotate app. This screen only ever looks.

• Off the GUI thread. The report reads the whole png_list annotation block, so it runs through spacr.qt.bridge.make_thread() like every other spaCR job. Tests pass threaded=False.

• No modal dialogs on any error path. “You only picked one column”, “that file isn’t a database”, “this table has no annotation columns” — all of it lands in an inline status label. A QMessageBox would hang a headless run.

• κ is never quoted alone. Raw percent agreement, the number of compared rows, the number of abstentions and the reason κ is undefined (when it is) are all on screen next to it. See spacr.agreement for why that matters on a screen where 98 % of cells are negative.

Classes

AgreementScreen

Inter-annotator agreement over the annotation columns of a database.

Functions

format_kappa(→ str)

Render a κ for display — "undefined" rather than a fake number.

Module Contents

class spacr.qt.screens.agreement.AgreementScreen(parent=None, threaded: bool = True)

Bases: PySide6.QtWidgets.QWidget

Inter-annotator agreement over the annotation columns of a database.

Parameters:

• threaded – compute the report on a worker thread (the default). Tests pass False for deterministic, synchronous behaviour.

• parent – parent widget; ownership only.

Variables:

last_error – text of the most recent failure, "" when the last operation succeeded. Errors are only ever reported here and in the inline status label — never in a modal dialog.

Build the screen and arm its drop zone.

Parameters:

• parent – parent widget, or None.

• threaded – run the agreement computation on a worker thread. Set False in tests so compute finishes before it returns.

active_jobs() → int

How many compute threads are still winding down.

available_columns() → List[str]

Annotation columns offered for comparison.

closeEvent(event)

Let every in-flight compute thread finish before the widget dies.

Parameters:

event – the close event, passed on to the base class once each running job thread has been asked to quit and waited on (up to five seconds each).

compute() → bool

Build the agreement report for the ticked columns.

Everything that can go wrong here is a normal state — no database, one column ticked, a column that turns out to be empty — so every failure is inline text and a False return.

Returns:

for the synchronous path, whether the report was built; for the threaded path, True once the job has started.

confusion_rows() → List[List[str]]

The confusion grid as plain strings, row label first.

crop_message() → str

Text shown instead of a crop when there is no image.

current_crop_path() → str

Text of the crop-preview caption’s first line (test helper).

database_path() → str

Path of the open database, or ''.

disagreement_paths() → List[str]

The png_path of every row in the review list.

disagreement_rows() → List[List[str]]

The review list as plain strings — key first, then each label.

is_busy() → bool

Whether anything is still running.

What the window asks before closing.

Returns:

True while work is outstanding.

kappa_rows() → List[List[str]]

The κ table as plain strings — one list per pair.

report() → spacr.agreement.AgreementReport | None

The most recent AgreementReport, or None.

select_columns(names) → bool

Tick exactly names; report inline for anything unknown.

Parameters:

names – annotation column names to check; every other listed column is unchecked. Names not in the list are reported in the status line.

Returns:

True when every requested column was found.

select_disagreement(row: int) → bool

Show the crop for review row row.

A missing PNG is a fact about the dataset (crops get moved, or the database was copied without them), not an error worth a dialog — the preview says so and the row stays selected.

Parameters:

row – zero-based row of the disagreement table; out of range clears the preview.

Returns:

True when an image was actually rendered.

selected_columns() → List[str]

Ticked annotation columns, in list order.

set_database(path: str) → bool

Open path read-only and list its annotation columns.

Accepts the database file or a run src folder. Any problem (missing file, not a database, no png_list) is reported in the status label and returns False — this never raises.

Parameters:

path – measurements.db file, run src folder or measurements folder; resolved with spacr.qt.screens.db_browser.resolve_db_path().

Returns:

True when a database with at least one annotation column was opened.

status_text() → str

Current inline status message (test/introspection helper).

summary_text() → str

The overall-κ summary line, or '' before a report exists.

spacr.qt.screens.agreement.format_kappa(value: Any) → str

Render a κ for display — "undefined" rather than a fake number.

Parameters:

value – a κ, possibly nan.

Returns:

signed 3-decimal string, or "undefined".

Nested helpers

AgreementScreen._run_job._job(payload: Dict[str, Any]) → None

Call the wrapped function, stashing its result in the payload.

The payload is how a value crosses back from the worker: a return would be swallowed by the runner.

spacr/qt/screens/agreement.py:842

AgreementScreen.compute._job() → Dict[str, Any]

Build the agreement report and its disagreements. Off the GUI thread.

spacr/qt/screens/agreement.py:528