spacr.barcode_search¶

Decide which barcode tables a sequencing run really contains.

WHAT IT IS FOR¶

Map Barcodes has to be told three things it cannot guess: which mate carries each barcode, whether the reference table is stored in the same orientation as the reads, and where in the read the barcode sits. Getting any of them wrong produces a run that finishes normally and maps nothing, because every stage after the extraction window is working on the wrong bases. This module reads a bounded sample of the reads, measures each reference table against them in both orientations, and turns the measurements into settings the mapping run can use. It holds no graphical code and opens no dialogs, so the same engine serves the live search in the interface, a notebook, and the tests.

WHY A RAW HIT RATE IS NOT EVIDENCE¶

A short barcode turns up in a long read by pure coincidence. Thirty-two distinct eight base barcodes scanned across a hundred and fifty base read have about a seven percent chance of a spurious match, so a table that is genuinely absent still reports roughly seven percent of reads as hits. A detector that announced a finding at that rate would send someone hunting for a bug that does not exist, or would quietly configure a mapping that yields nothing. Every rate this module reports therefore arrives with the rate expected by chance for that table against those reads, and with the ratio between them. Nothing is called present unless it stands well clear of its own coincidence rate.

WHY A LARGE ENRICHMENT IS STILL NOT ENOUGH¶

Enrichment over chance answers whether the sequences are in the reads. It does not answer whether they are the plate barcodes. Sequencing adapters are fixed sequences repeated in every read, and a handful of barcodes in any table will resemble part of an adapter closely enough to match it. That produces a large, perfectly real enrichment driven by two barcodes sitting in the adapter rather than by a plate. The measurement that separates the two is position. A plate barcode occupies one place in the construct and so lands at one offset in almost every read, which is exactly why an extraction window can be proposed for it at all. A match scattered across the read cannot yield a window and is not treated as a finding, however enriched it is.

WHAT IT PRODUCES¶

A report carrying one finding per combination of reference table, orientation and input file. Each finding holds the observed rate, the chance rate, the enrichment, the offset distribution, the narrowest offset window that accounts for most of the hits, how many distinct barcodes of the table were seen, and a verdict with a sentence saying why that verdict was reached. The report can be rendered as a table for a log, and it can be folded into the settings that spacr.sequencing.generate_barecode_mapping() reads.

WHAT TO DO NEXT¶

Read the verdicts before the proposed settings.

A table reported as absent in both directions of both mates usually means the wrong reference file, not a failed experiment.

The offset column is the quickest way to tell a plate barcode from an adapter that happens to look like one.

Classes¶

`BarcodeHit`	One barcode occurrence inside one read.
`BarcodeSearchReport`	Every finding from one search, plus how much was read to get them.
`BarcodeTable`	One reference table of named barcode sequences.
`OrientationFinding`	What one reference table did against one file in one orientation.
`ProposedMapping`	Settings the search believes a mapping run should use, and why.
`SearchThresholds`	The numbers a barcode search judges by, and what each one judges.

Functions¶

`annotate_read`(read, tables[, orientations])	Locate every barcode inside one read so a display can colour them.
`expected_chance_rate`(table, read_lengths)	Return the share of reads expected to contain a barcode by coincidence.
`infer_barcode_role`(path)	Guess whether a reference file holds row, column or guide barcodes.
`iter_annotated_reads`(fastq_file, tables[, ...])	Yield reads together with the barcodes found inside them.
`iter_barcode_search`(fastq_files, tables[, max_reads, ...])	Search for barcodes a chunk at a time, reporting after every chunk.
`iter_fastq_reads`(path[, limit])	Yield read sequences from a sequencing file without loading it.
`load_barcode_table`(path[, name, role])	Read a reference table of barcodes from a comma separated file.
`load_barcode_tables`(paths)	Read several reference tables at once.
`propose_map_barcodes_settings`(report[, base_settings, ...])	Turn a finished search into settings for a barcode mapping run.
`reverse_complement`(sequence)	Return the reverse complement of a nucleotide sequence.
`sample_fastq_reads`(path[, limit])	Read a bounded sample of sequences from a sequencing file.
`search_barcodes`(fastq_files, tables[, max_reads, ...])	Search a bounded sample of reads and return the finished report.

Module Contents¶

class spacr.barcode_search.BarcodeHit[source]¶

One barcode occurrence inside one read.

The stretch it covers is given so that a display can highlight exactly those bases, and the colour index is a stable number for the barcode itself rather than for its table, so that two different barcodes of the same table are drawn in two different colours.

Parameters:

start – index of the barcode’s first base within the read.
end – index one past its last base, so read[start:end] is the barcode.
table – label of the table the barcode came from.
role – the role that table fills, or None.
barcode – the matched sequence itself.
orientation – whether it matched AS_GIVEN or REVERSE_COMPLEMENT.
colour_index – a stable number for the barcode, not for its table, so two barcodes of one table are drawn in two colours.

class spacr.barcode_search.BarcodeSearchReport[source]¶

Every finding from one search, plus how much was read to get them.

A report from a partly finished search has the same shape as a finished one and is meant to be shown, so a live display can render each refinement without special casing the first one.

Parameters:

findings – one OrientationFinding per table, file and orientation examined.
reads_by_file – how many reads were examined in each file.
read_length_by_file – mean read length per file, which is what sizes an offset histogram.
complete – whether the search finished. False on the partial reports a live display renders while it is still refining.

best_for_role(role)[source]¶

Return the most convincing present finding for a barcode role.

A role may be filled by more than one table and by either mate, and the caller wants the one that would actually be used. Only findings that cleared every check are considered, so this returns nothing when the role was not established.

Parameters:: role – the barcode role wanted.
Returns:: the finding with the highest observed rate, or None.

for_table(table)[source]¶

Return the findings belonging to one reference table.

Parameters:: table – the label of the table.
Returns:: a tuple of findings, strongest enrichment first.

format_table()[source]¶

Render the findings as fixed width text for a log or a console.

The expected rate sits beside the observed one on purpose. A reader who sees only the observed column will believe a coincidence, and the whole point of the module is that the two numbers are never separated.

Returns:: the rendered table, as a string.

roles()[source]¶

Return every barcode role the searched tables were given.

Returns:: a tuple of role names, in a stable order, without the anchor.

property reads[source]¶

Return how many reads were examined across all files.

Returns:: the total read count.

class spacr.barcode_search.BarcodeTable[source]¶

One reference table of named barcode sequences.

The sequences are held as a mapping from sequence to name so that a match can be named without a second search, and the per-length counts are kept because the chance rate depends on how many barcodes of each length there are rather than on the size of the table alone.

Parameters:

name – the label this table is known by for the rest of the search, and the label every finding it produces refers back to.
sequences – mapping from barcode sequence to the name of that barcode, so a hit can be named without searching the table again.
role – which part of the screen’s identity the table decodes – plate row, plate column, guide – or None for a table searched without one.
path – where the table was read from, kept so a proposed mapping can name the file rather than the label.

__post_init__()[source]¶

Give the table a private store for the indexes built from it.

Building the search index over a table of more than a thousand guide sequences takes long enough that doing it once per read would make a live display unusable, so each index is built once and kept here.

barcode_order()[source]¶

Return a stable number for each stored sequence of the table.

A display paints one colour per barcode rather than per table, and the number has to be the same every time the same table is loaded, so it comes from sorting the sequences rather than from the order of the file.

Returns:: a mapping from stored sequence to its position when sorted.

matcher(orientation)[source]¶

Return the reusable search index for one orientation of the table.

Parameters:: orientation – which orientation of the table to search in.
Returns:: the index, built on first use and kept afterwards.

oriented(orientation)[source]¶

Return the same table with its sequences in a chosen orientation.

Parameters:: orientation – either the label for the stored orientation or the label for the reverse complemented one.
Returns:: a table whose sequences read in the requested orientation.
Raises:: ValueError – when the orientation label is not one of the two, or when the table is not DNA and so cannot be complemented at all.

A NON-NUCLEOTIDE TABLE IS REFUSED RATHER THAN MANGLED. Peptide tags and amino-acid barcodes pass through reverse_complement unchanged – it maps only the four bases and N – so a flipped table would be the same letters reversed, searched for, and never found. Saying no is the honest answer; silently searching for a sequence that cannot match reports “this reference is absent from the reads”, which is a conclusion about the data rather than about the request.

property length_counts[source]¶

Return how many distinct barcodes the table holds at each length.

Returns:: a mapping from sequence length to the number of barcodes.

property size[source]¶

Return how many distinct sequences the table holds.

Returns:: the count of distinct barcode sequences.

class spacr.barcode_search.OrientationFinding[source]¶

What one reference table did against one file in one orientation.

Everything needed to judge the finding travels with it, so a caller never has to recompute the coincidence rate in order to decide whether the observed rate means anything.

Parameters:

table – label of the reference table this finding is about.
role – the role that table fills, or None when it was searched without one.
file_label – the read file the table was searched against.
orientation – whether the table matched AS_GIVEN or REVERSE_COMPLEMENT.
table_path – where the table was read from, or None when it was supplied inline.
reads – how many reads were examined to reach this finding.
hits – how many of those reads contained a barcode from the table.
observed_rate – hits divided by reads.
expected_rate – the rate the same table would reach by chance, given how many barcodes it holds at each length. The observed rate means nothing on its own; this is what makes it readable.
enrichment – observed_rate divided by expected_rate.
offset_start – earliest start offset counted as the barcode’s usual position.
offset_span – how many consecutive offsets that position covers.
modal_offset – the single commonest start offset, or None when nothing was found.
barcode_lengths – the distinct lengths present in the table. The extraction window has to reach past the longest, not the modal one.
distinct_barcodes_seen – how many different barcodes of the table were actually observed. A table that keeps matching the same one sequence is matching something other than its barcodes.
table_size – how many distinct sequences the table holds.
top_barcode_share – fraction of hits taken by the commonest single barcode. Near one, alongside a low distinct_barcodes_seen, is the signature of a spurious match rather than a real one.
verdict – PRESENT, ABSENT or INDETERMINATE.
reason – why that verdict was reached, worded to be shown to a user.
offset_counts – hit count at each start offset, for the histogram.

offset_histogram(read_length=None)[source]¶

Return the hit count at each start offset as an array.

Position in the array is the offset itself. A caller can draw the array as it is, and does not have to work out which offsets were used.

Parameters:: read_length – how long the array should be; long enough to hold the largest observed offset when omitted.
Returns:: an array of hit counts indexed by start offset.

property window_end[source]¶

Return the first offset past the barcode in its usual position.

The window has to reach the end of the last barcode, not its start.

When the barcodes in a set are not all the same length, the longest one decides where the window ends.

Returns:: the offset just past the end of the barcode.

class spacr.barcode_search.ProposedMapping[source]¶

Settings the search believes a mapping run should use, and why.

Every key in the proposal is one the mapping run already reads. The run can take the proposal as it stands.

What the search learned that has no matching key is carried alongside.

A caller that quietly dropped the orientation would bring back the very failure this module exists to prevent.

A screen that decodes a barcode beyond the plate column, the guide and the plate row has no settings key waiting for it. The table chosen for every role is therefore listed among the reference tables, whether or not a key could be filled in for it.

Parameters:

settings – only keys the mapping run already reads, so the mapping can be handed this dictionary directly.
reference_file – the file whose reads the window offsets are expressed against, since offsets mean nothing without their frame.
source_files – mapping from role to the read file that established it.
orientations – mapping from table label to the orientation the run should search that table in.
reference_tables – the table chosen for each role, listed whether or not a settings key exists to carry it – which is the point, because a role with no key is exactly the one that would otherwise be lost.
reverse_complement_needed – tables the run has to reverse complement to recognise, because they were found in the second mate as stored.
unresolved_roles – roles no table established. Named rather than guessed, so a caller can say so instead of starting a run that will map nothing.
notes – what the search learned that has no home among the settings keys.

class spacr.barcode_search.SearchThresholds[source]¶

The numbers a barcode search judges by, and what each one judges.

THESE WERE MODULE CONSTANTS. They were chosen against one screen’s data, which is exactly the kind of number that is right until someone runs a different assay – a shorter read, a smaller reference, a library where 10% of reads carrying the barcode is a good day rather than a failure. The values here are those same numbers, so a caller who passes nothing gets the behaviour that already existed; what changes is that the numbers can now be said out loud by someone whose data disagrees with them.

Each field judges a DIFFERENT question, and they are asked in this order: is the sample big enough to say anything, is the hit rate distinguishable from coincidence, is it big enough to map from, and do the hits sit at one place in the read. A verdict of INDETERMINATE names which question it was that could not be answered.

__post_init__()[source]¶

Refuse thresholds that cannot decide anything.

A threshold out of range does not fail loudly at the point it is set; it produces a report in which every table is PRESENT, or every table is ABSENT, and the run looks like it worked. Checking here means the complaint names the setting rather than the data.

Raises:: ValueError – when a threshold cannot produce a real verdict.

spacr.barcode_search.annotate_read(read, tables, orientations=None)[source]¶

Locate every barcode inside one read so a display can colour them.

The reads a search looked at are the evidence behind its verdicts, and the quickest way for someone to believe or disbelieve a verdict is to see the barcodes sitting in the reads. This returns the stretches to paint, left to right, with overlaps resolved in favour of whichever table is listed first.

Parameters:

read – the read sequence.
tables – the reference tables to look for.
orientations – a mapping from table label to the orientation to search in; both orientations are searched for a table not named there.

Returns:

a tuple of hits ordered by their position in the read.

spacr.barcode_search.expected_chance_rate(table, read_lengths)[source]¶

Return the share of reads expected to contain a barcode by coincidence.

A read is treated as a run of independent bases drawn evenly from the four of them, so one barcode of a given length matches at one position with a probability of one over four raised to that length, and a read offers one more position than the difference between its length and the barcode’s. The probability that no barcode of the table matches anywhere is the product over every barcode and every position of the probability that it does not match there, and the answer is one minus that product. Reads of different lengths are averaged.

The even base assumption is the weak point and it was checked rather than assumed. On a real run the tables that were genuinely absent matched within two tenths of a percentage point of the rate this returns, which is close enough for the ratio between observed and expected to be trusted.

Parameters:

table – the reference table whose coincidence rate is wanted.
read_lengths – a read length, an iterable of them, or a mapping from read length to how many reads had it.

Returns:

the expected share of reads holding at least one barcode.

spacr.barcode_search.infer_barcode_role(path)[source]¶

Guess whether a reference file holds row, column or guide barcodes.

The guess reads the file name only, because the tables themselves carry no field saying what they are. Guide tables are recognised first, since a guide file name mentions neither rows nor columns, and column names are recognised before row names.

Parameters:: path – the path of the reference table.
Returns:: one of the role names, or None when the name settles nothing.

spacr.barcode_search.iter_annotated_reads(fastq_file, tables, orientations=None, limit=DEFAULT_CHUNK_READS, only_matching=False)[source]¶

Yield reads together with the barcodes found inside them.

This feeds the window that shows one read per line with its barcodes picked out, so it stays bounded in the same way the search is and never walks a whole file.

Parameters:

fastq_file – the sequencing file to read.
tables – the reference tables to look for.
orientations – a mapping from table label to the orientation to search in; both orientations are searched for a table not named there.
limit – how many reads to take from the file at most.
only_matching – whether to skip reads in which nothing was found.

Yields:

each read and the hits inside it.

spacr.barcode_search.iter_barcode_search(fastq_files, tables, max_reads=DEFAULT_SAMPLE_READS, chunk_reads=DEFAULT_CHUNK_READS, anchor=None, thresholds=None)[source]¶

Search for barcodes a chunk at a time, reporting after every chunk.

The interface shows this running, so the search hands back a complete report after each chunk rather than only at the end. Every report has the same shape as the final one and the counts inside it only ever grow, so the rates settle rather than jump and a verdict that was undecided becomes decided as the evidence arrives. Stopping early is legitimate and leaves a report that says how many reads it rests on.

Parameters:

fastq_files – a sequencing file, a sequence of them, or a mapping from the label each should carry to its path.
tables – the reference tables to look for, of any number.
max_reads – how many reads to take from each file at most.
chunk_reads – how many reads each step takes from each file.
anchor – an optional fixed sequence, such as the one the mapping run anchors its window on, searched alongside the tables so that offsets can be expressed relative to it.
thresholds – a SearchThresholds saying what counts as present, absent and undecided, or None for the defaults. Every verdict in every yielded report is judged by these.

Yields:

a report after each chunk, and one report when there is nothing to read.

Raises:

ValueError – when no file was supplied or the budgets are not positive.

spacr.barcode_search.iter_fastq_reads(path, limit=None)[source]¶

Yield read sequences from a sequencing file without loading it.

The files this runs against are commonly several gigabytes on a network share, so records are pulled one at a time and the caller decides when to stop. A record whose four lines are cut short at the end of the file is dropped rather than raising, because a truncated final record is a normal consequence of copying a file that is still being written.

Parameters:

path – the path of the sequencing file, gzip compressed or plain.
limit – how many reads to yield at most; every read when omitted.

Yields:

the sequence of each read, upper cased.

spacr.barcode_search.load_barcode_table(path, name=None, role=None)[source]¶

Read a reference table of barcodes from a comma separated file.

The file needs a header naming a sequence column and, for readable output, a name column. Sequences are upper cased and stripped, blank rows are skipped, and a sequence repeated under two names keeps the first name so that the table stays a mapping.

Parameters:

path – the path of the comma separated reference table.
name – a label for the table; the file stem is used when omitted.
role – the barcode role this table fills; inferred from the file name when omitted.

Returns:

the loaded table.

Raises:

ValueError – when the file carries no sequence column or no rows.

spacr.barcode_search.load_barcode_tables(paths)[source]¶

Read several reference tables at once.

Any number of tables may be supplied, so a run that carries more than the three barcode kinds spaCR grew up with is handled the same way as one that carries fewer.

Parameters:: paths – paths of the reference tables, or a mapping from the label a table should carry to its path.
Returns:: a tuple of loaded tables in the order given.

spacr.barcode_search.propose_map_barcodes_settings(report, base_settings=None, reference_file=None)[source]¶

Turn a finished search into settings for a barcode mapping run.

The run joins both mates into one sequence, using the direction of the first mate. Every position below is given in that direction.

A table found in the second mate as it is stored is therefore reported as needing to be reverse complemented. That is what the run will need in order to recognise it.

A role that no table established is left alone rather than guessed at. It is named among the unresolved roles, so the caller can say so instead of starting a run that will map nothing.

The extraction window is derived from where the barcodes actually landed. Its start is the earliest offset any established barcode occupied and its end is the furthest any of them reached, both measured against the anchor when one was searched for, since the mapping run locates its window by finding that anchor first.

Parameters:

report – the report from a search.
base_settings – settings to start from, left unmodified; an empty dictionary when omitted.
reference_file – the label of the file whose orientation is the frame; the first file of the search when omitted.

Returns:

the proposal, holding the settings and what did not fit in them.

Raises:

ValueError – when the report holds no files.

spacr.barcode_search.reverse_complement(sequence)[source]¶

Return the reverse complement of a nucleotide sequence.

Upper case and lower case are both handled.

Any letter that is not one of the four bases is returned unchanged, so a read with an uncertain position is still reversed rather than refused.

Parameters:: sequence – the nucleotide sequence to flip.
Returns:: the reverse complement, as a string.

spacr.barcode_search.sample_fastq_reads(path, limit=DEFAULT_SAMPLE_READS)[source]¶

Read a bounded sample of sequences from a sequencing file.

Parameters:

path – the path of the sequencing file, gzip compressed or plain.
limit – how many reads to take from the start of the file.

Returns:

a tuple of read sequences.

spacr.barcode_search.search_barcodes(fastq_files, tables, max_reads=DEFAULT_SAMPLE_READS, chunk_reads=DEFAULT_CHUNK_READS, anchor=None, thresholds=None)[source]¶

Search a bounded sample of reads and return the finished report.

This is the whole of iter_barcode_search() run to its end, for callers that want an answer rather than a running display.

Parameters:

fastq_files – a sequencing file, a sequence of them, or a mapping from the label each should carry to its path.
tables – the reference tables to look for, of any number.
max_reads – how many reads to take from each file at most.
chunk_reads – how many reads each step takes from each file.
anchor – an optional fixed sequence searched alongside the tables.
thresholds – a SearchThresholds, or None for the defaults.

Returns:

the report from the last chunk.