# Inter-Rater Reliability (IRR) — blind second-rater pass
**Roman Economic & Production Centers / SEDZ** · 2026-07-07 · audit remediation A3 (the codebook's outstanding item)

## Design
An **independent second rater** (fresh agent, no access to the original scores) scored the three **convergence** axes (bounded_jurisdiction, allocated_exploitation, fiscal_monopoly_apparatus) for all **28** cases from a **blinded worksheet** (`_audit/blinded_worksheet.json` — each entry's `evidence[]` with the `sedz_assessment` stripped) applying only the anchored rules in `schema/SCORING-CODEBOOK.md`. Rater 1 = the original project scores (`_audit/gold_scores.json`); Rater 2 = the blind rater. Genus = (all three convergence axes ≥ 1).
*Caveat: two further blind raters stalled on environment instability, so this is a **2-rater** κ (original vs one blind), not a 3-rater α. A second independent rater would tighten the estimate.*

## Result — the instrument is reliable
| Axis | exact agreement | within-1 | linear-weighted Cohen's κ |
|---|---|---|---|
| bounded_jurisdiction | 89% | 100% | **0.87** |
| allocated_exploitation | 86% | 100% | **0.84** |
| fiscal_monopoly_apparatus | 89% | 100% | **0.82** |
| **genus (binary)** | **93%** | — | **0.85** |

All three convergence axes score **κ ≥ 0.82** ("almost perfect" agreement, Landis & Koch), and the two raters **never diverge by more than one point** on any axis. This substantially upgrades the rubric's standing: it is no longer single-rater expert coding but a **criterion-explicit instrument that an independent rater reproduces at κ ≈ 0.85** working from the codebook alone.

## The two genus disagreements — both at the flagged fragile boundary
Only **2 of 28** cases flip genus membership, and both flip on the **bounded_jurisdiction 1↔0** line the codebook already marked fragile — and both are thesis-critical recent additions:

| Case | original | blind | the call |
|---|---|---|---|
| **sisapo** | 1/2/2 → genus **✓** | 0/1/2 → genus **✗** | Does a state commodity-monopoly regime (refining ban, sealed ore, fixed price) *within a functioning city* count as bounded jurisdiction (BJ 1) or ordinary civic territory (BJ 0)? |
| **laurion** | 1/2/1 → genus **✓** | 0/2/1 → genus **✗** | Does the Athenian mining-law regime (poletai leases) over civic-territory workings count as a special jurisdiction (BJ 1) or ordinary civic administration with disputes in ordinary *dikai metallikai* (BJ 0)? |

The codebook is genuinely ambiguous here: its "1 = a special legal regime that does **not** displace civic government" clause supports BJ 1, while its "0 = ordinary civic territory" clause supports BJ 0. The blind rater took the strict-civic reading. **Both readings are defensible.**

## Consequences for the argument (now reflected in D2)
- The instrument's **reliability is high for the clear cases** — every uncontested member (Vipasca, Mons Claudianus/Porphyrites/Simitthus/Dokimeion, the modern zones) and every clear non-member (garum ×4, La Graufesenque, Baetica, portorium, Ptolemaic, Aphrodisias, figlinae) agree exactly. Luna's corrected genus-fail and Noricum's 1/1/1 both **replicate** blind.
- The two arguments that most depend on the borderline — **Sisapo** as the "spectrum-breaking" market-pole Roman member, and **Laurion** as the closest "Greek cousin" — rest on a **contestable BJ call**. They must be presented as *calibration-fragile*: on the strict-civic reading both fail the genus, which would (a) restore finding (i)'s "high-imperial members are dirigiste" without the Sisapo exception, and (b) demote Laurion from cousin-in-the-genus to a near-miss. On the special-regime reading they qualify and the spectrum/cousin claims stand.
- **Net:** the reliability pass validates the instrument *and* correctly localizes its one soft spot — the BJ 0/1 boundary for state-regime-within-a-city cases — which is precisely where the paper's most interesting (and most contestable) claims live. That is the honest place for the argument to rest.

## Codebook update (applied)
Added a tie-breaker to `SCORING-CODEBOOK.md` for the Sisapo/Laurion pattern (special commodity/mining-law regime inside a functioning civic community): record **both** the strict-civic (0) and special-regime (1) readings and treat the case's genus membership as *conditional* until the boundary is adjudicated by a domain panel. The fragile set is now: sisapo, laurion, luna, proconnesus, karystos, noricum.