Precision-first title / not-title at 65 million unique strings

Aside / field note. Written 2026-08-28. Plan, not a shipped scorer. Canonical git copy: validation-research-handbook (PLAN-65M-PRECISION-CASCADE.md). This post is the same argument in one place so a later session can pick it up without the chat.

L1 only: is this string a job title. Not L2 (could it exist at this company). Not L3 (does this person hold it now).

The ask

A database of about 65 million unique values from a job-title field: real titles, garbage, profanity, company names, departments, the lot. Exportable in ~50k batches. Need a confidence that it is a title or not.

Constraints that actually decide the design:

• Precision is the goal, not coverage theatre.
• Fast. Cannot sit behind a big LLM at serving time.
• No human in the loop on 65 million rows.
• Teacher access is Copilot Studio with Sonnet, Fable, and GPT-5.6. No GitHub Copilot. Microsoft 365 is basic chat only, not Copilot Studio Pro / M365 Copilot licensed as a bulk API.

What was offered as “the method”: AI distillation, or embeddings, because those sound like they scale.

They do not, not as the system.

What already failed on this line

Matching is not validation. Frozen embedding top-1 cosine as accept/reject: AUC 0.535. Lexical fuzzy: AUC 0.426 (worse than chance). Negatives built by recombining real title vocabulary (Senior Google Engineer Berlin) score higher on token overlap than real titles.

A 140M mmBERT with a toxicity head hit 100% on four training templates and 4.2% on unseen carriers. Job-title precision of 1.000 was a holdout with zero negatives. A published 0.4% false-accept rate on Norwegian company names was 28% on English names (company / university had been derived into the head list).

A weighted sum let "Regional fuck Coordinator" land at 0.485 uncertain until profanity became a veto clamp, not a vote. token_set_ratio('sales department', 'department sales manager') is 100.

The Evidence Ladder rule still stands: do not train a classifier for “is this a title” on circular labels. There is no non-circular label source at 65 million rows. Distillation is a teacher for the leftover middle, not a license to treat the lake as positives.

An LLM-as-judge on 600 rows did beat rules on departments and places (precision 0.979 vs rules 0.957 at the 0.65 cut). It ran at 0.82 items/s. At that rate 65 million is years. Copilot Studio will not beat physics.

Throughput, so the architecture is forced

50k is a good export unit for a local cascade. It is not a Copilot prompt.

50,000 titles at ~30–40 characters is on the order of 1.5–2 MB and hundreds of thousands of tokens. Sonnet’s context window will not swallow a 50k file as “label all of these,” even if Copilot Studio lets you attach it. Basic M365 chat will not either.

65 million / 50k = 1,300 files. That is a checkpointed local job, not 1,300 Copilot threads.

Output contract

Three labels. HOLD is not a human queue.

Report three numbers, never one accuracy: precision of USE, precision of DROP, HOLD rate. A 2% HOLD on 65 million is 1.3 million strings you do not sell as titles. That is the product.

Confidence is which layer fired, not a softmax. Attestation hit → the string exists as a title form. Grammar VALID → it looks like one. Student 0.91 → it looks like the teacher’s USE class. Do not mix those into one calibrated-looking float.

The method: cascade. Distill only HOLD.

Later layers never un-DROP a veto.

0. One frozen normalizer

NFKC, hyphen/punctuation policy, casefold. One function for attestation keys, hash splits, and the student. Two different normalizers on this host already put 805 of the same SEC companies on both sides of a train/holdout cut.

1. Vetoes, and they win

Microsecond, closed, short-circuit: profanity as a word-boundary regex (not a token set — "engineer (fuck)" tokenizes to "(fuck"), email/URL/phone, placeholders (n/a, see remarks), mash, digits, no letters, emoji. Run profanity/email on the raw string so decoration cannot hide them.

Do not distill “is this profanity.”

2. Attestation = high-precision USE

Exact dict lookup after a closed cleanup list (not fuzzy): strip parentheticals and trailing ids, expand sr→senior / mgr→manager / …, peel a closed modifier set, try X, Y as Y X.

Hit in ESCO ∪ O*NET ∪ UK SOC ∪ in-house accepted → USE. Novel real titles (CJK, missing German compounds, messy vacancy strings) go to HOLD, not USE. That is the precision.

Forbidden: promoting "n/a | Chief Executive Officer" via compound-split after the whole string failed as a placeholder.

3. Grammar on the rest

Role-head grammar: [SENIORITY]* [DOMAIN]* HEAD [SCOPE]*. Named negative class → DROP. Score ≥ bar and a real head → USE. Else HOLD. Split on separators, not \w (Python \w shreds Indic/Thai/Arabic). Do not let organisational nouns (company, university) into the head inventory. Agent suffixes shorter than 3 characters are banned (Equinor / Berlin became role nouns).

Low score is HOLD, not DROP.

4. Distill only HOLD

Rules already own the easy mass. A student trained on the lake relearns n/a and CEO and stays weak on Sales Department vs Sales Manager.

Teacher: Copilot Studio (Sonnet + GPT-5.6, Fable as a third vote if it follows the schema). 2–5k rows, stratified from HOLD, not from USE/DROP.

Student: logistic regression or linear SVM on character n-grams (2–4) plus a few rule features. Not mmBERT on CPU. Not ONNX-int8 from titlebert.

Cut: lowest threshold that hits a frozen gold precision (e.g. 0.98 USE). Below the cut stays HOLD. The student cannot override a veto.

Optional later, not v1: competing n-gram LMs (title vs company vs person vs department vs junk); decide on argmax plus margin. Company as a first-class hypothesis is how The Coca-Cola Company stops surviving a head match on Company.

How 50k exports and Copilot Studio actually fit

The database export is the input to Layers 1–3, locally (or on this VPS). Each 50k file is seconds of veto+attest+grammar, then you write use / drop / hold sidecars and go to the next file. 1,300 files, resumable, no model.

Copilot Studio is the teacher for a few thousand HOLD rows, not a 50k classifier.

Practical Copilot Studio setup with the models you actually have:

1. One agent, generative orchestration off for this topic. A single topic with a fixed schema:

   TITLE | DEPARTMENT | PERSON | COMPANY | PLACE | CREDENTIAL | JUNK
   

   plus a 0–100 you throw away for training except as a disagreement flag. No tools, no web, no “looks valid-ish.”

2. Attachment size: 40–100 titles per turn, one column CSV. Not 50k. A Power Automate / Copilot flow that loops 50k → 500 calls of 100 is how you could brute-force a batch, and it is still the wrong spend. Use it to label the stratified HOLD sample only.

3. Two-teacher disagreement (this is the point of having Sonnet and GPT-5.6):

   • Same 100-row CSV, same prompt, both models.
   • Agree TITLE → student positive.
   • Agree not-TITLE → student negative.
   • Disagree → HOLD forever, not a training row.

   Fable: run it if it reliably emits the seven labels and nothing else. If it chatters or invents categories, drop it. Do not average three softmaxes.

4. Microsoft 365 basic chat is a scratchpad. It is not a batch labeler, has no agent file loop you can trust at this volume, and must not be the teacher of record.

5. Do not upload USE or DROP rows “for completeness.” That is how the student copies the vetoes and looks accurate.

Suggested HOLD sample, by shape, not random (~200–400 per bucket, cap ~3k total): 2-token English; … Department / … Team; person-name-ish; company-suffix; non-Latin; long vacancy sentences; occupational surnames (Baker / Porter); anything that looks profane but survived Layer 1 (those are bugs).

Eval that will not retract

Gold is protocol rows, not “it was in the title column” and not “it was in ESCO.” Print Layer-1-only and majority baselines beside the student. Negatives must include departments, companies, places, occupational surnames, and recombined title-vocabulary garbage. Report USE/DROP/HOLD per bucket. Metamorphic tests on raw strings: inserting profanity must not raise USE; prepending Senior must not lift asdfgh.

Do not quote titlecheck 0.0% FA, titlebert 1.000, or titlevalidate 0.4% as evidence this cascade works on 65 million.

What not to build

Build order

1. Freeze normalizer + Layer 1 vetoes + raw-string tests.
2. Attestation table. Score one 50k export end-to-end. Emit USE/DROP/HOLD counts and the 20 ugliest HOLD strings. Stop if HOLD is tiny or is obviously junk — distillation may not be worth 3k labels.
3. Grammar on HOLD. Re-count. Then the rest of the 1,300 files, checkpointed.
4. Protocol + Copilot Studio labels on 2–5k remaining HOLD (Sonnet × GPT-5.6).
5. Char-ngram student, precision-gated cut, freeze 500 gold.
6. Full lake. Publish USE precision, DROP precision, HOLD rate, per-bucket table.

Canonical plan on git: https://git.binary.ovh/brahm/validation-research-handbook/src/branch/main/PLAN-65M-PRECISION-CASCADE.md