GRASS methods companion: notation, calibration detail, and auxiliary diagnostics

Austin Semmel

Rachel Gidaro

2026-07-03

Scope of this companion

This vignette accompanies the GRASS methods paper. It is a parallel reference: the paper is self-contained for review, while this companion carries the procedural detail and auxiliary diagnostics a practitioner running grassr may want at hand. Section numbering does not parallel the paper.

Content moved here from earlier paper appendices during the 2026-05-04 appendix consolidation. The paper does not cite this vignette; this is a free-standing artifact for grassr users.

---

1. Notation glossary

Symbols used throughout the package and the paper:

• Y — the $N \times k$ binary rating matrix; rows are subjects, columns are raters.
• pi_hat — observed marginal positive rate.
• tau2_hat — between-subject variance of the per-subject mean call rate (sample-side, computed from Y).
• Se_j, Sp_j — per-rater sensitivity and specificity, inferred from the rating matrix, not against external truth.
• q — diagonal-DGP rater quality; $\mathrm{Se}_j = \mathrm{Sp}_j = q$ for every rater under the diagonal case the surfaces are calibrated on.
• q_hat — internal scaffolding only. Used by position_on_surface() to select the sampling distribution from which the percentile is read; not exposed on the Report Card.
• delta_hat — cross-coefficient surface-percentile spread.

---

2. Recovering $\hat q$ from an observed coefficient

For each agreement coefficient, position_on_surface() inverts the closed-form expectation curve to recover $\hat q$ at the observed coefficient value.

• PABAK: algebraic inverse, $\hat q = \tfrac{1}{2}(1 + \sqrt{\mathrm{PABAK}})$.
• Fleiss $\kappa$, Krippendorff $\alpha$, AC1, ICC: 501-point $q$-grid lookup with linear interpolation between the two nearest grid points where the curve crosses the observed value.
• Out-of-range observations: clamped to the boundary $q$ value with a clamped = TRUE flag.

---

3. ICC and the subject-prevalence distribution

The Report Card prints ICC alongside PABAK, AC1, and Fleiss $\kappa$, but with a [distribution-sensitive] marker, and ICC is excluded from the cross-coefficient stability signal $\hat\delta$ by construction.

The agreement-family coefficients each have a closed-form expectation that depends only on the panel marginal $\hat\pi_+$ and the rater quality $q$. ICC depends additionally on the full distribution $F$ of subject-level positive probabilities $p_i$ across the rated subjects — the shape of how the underlying positive rate varies from one subject to the next. Two studies can both have an overall positive rate of 30% and yet differ on whether that 30% comes from a clear class of obvious-positives (high between-subject variance) or from a mass of borderline-50/50 cases (low between-subject variance). The agreement-family coefficients can’t tell those two studies apart at the panel level. ICC can, and so its reference surface must commit to a specific $F$.

The bundled grassr reference covers a 48-point logit-normal grid in $(\mu, \tau^2)$ plus four discrete-mixture profiles (uniform, bimodal-low-high, skew-rare-positive, and similar shapes). A panel whose true $F$ sits cleanly inside that grid reads ICC’s surface percentile faithfully. A panel whose $F$ sits between grid points or under a different family carries a distribution-mismatch cost that the agreement-family percentiles do not. This is why ICC carries a [distribution-sensitive] marker on the Report Card and does not enter $\hat\delta$.

Variant selection (ICC(1,1), ICC(2,1), ICC(3,1)) follows the Koo and Li (2016) decision tree on both the inter-rater and intra-rater axes. The package exposes each variant through the metric argument of position_on_surface(). The four-question decision tree (same-rater-set / rater sampling / type / definition) is reproduced with worked examples per branch in the package help files.

---

4. Coefficient recovery: the full diagnostic

The paper’s App C carries the lead recovery table plus the asymmetry-cost figures. This companion expands the same dataset across the full grid.

---

5. $\hat\delta$ thresholds and operating characteristics

The paper’s Appendix~D carries the per-$(k, N)$ calibrated threshold table, the adopted-thresholds at the modal applied design, the TPR-by-asymmetry-magnitude tables for both caution and divergent flags, and the shared-bias vs split-bias detection comparison. This section populates the practitioner-facing detail that didn’t make the paper.

How the thresholds are derived. At each $(k, N)$ design point, the cuts are size-$\alpha$ calibrated against the diagonal-symmetric null distribution of $\hat\delta$ following the Neyman-Pearson construction: pick the smallest $t$ such that $P(\hat\delta > t \mid \text{symmetric panel at } (k, N)) \leq \alpha$ on the calibration grid, with $\alpha = 0.05$ for caution and $\alpha = 0.01$ for divergent. The thresholds are therefore a function of $(k, N)$, not a fixed pair. At the modal applied design ($k = 5$, $N = 200$) the cuts are $(9.25, 10.75)$ pp; at smaller designs they shift upward (the snapped pair at $k = 5$, $N = 50$ is $(10.25, 16.00)$ pp).

Small-design behavior. At $k = 2$ and at $k = 3$ with $N \leq 30$, the diagonal-null distribution of $\hat\delta$ has heavy tails that no size-$\alpha$ cut controls. The lookup snaps such panels to the nearest fully-calibrated cell on the $(k, \log_{10} N)$ grid and surfaces the snap on the Report Card; this is an optimistic lower bound rather than a calibrated cut at the user’s design. At $k = 2$ the snap collapses to $k = 3$ at the same $N$, because the panel-level coefficient battery at $k = 2$ (PABAK, AC1, Cohen’s $\kappa$) measures effectively the same agreement structure under both the symmetric null and the asymmetric alternatives. This is why the framework’s $k = 2$ prescription routes to direct pairwise reporting and identifiable bounds on per-rater $(\Se_j, \Sp_j)$ rather than relying on $\hat\delta$ to fire.

What $\hat\delta$ targets. $\hat\delta$ targets cross-coefficient surface-percentile inversion — the asymmetry pattern in which different chance-correction baselines respond differently to per-rater heterogeneity. Independently-drawn-sign asymmetry produces this inversion; uniform shared bias, conditional dependence, and other panel pathologies that move all coefficients together do not. Paper Appendix~D documents the gap: at $A = 0.20$ the divergent flag’s TPR plateaus near $0.40$–$0.46$ under split-bias asymmetry, while shared-bias panels move all three coefficients together and leave $\hat\delta$ nearly unmoved; the percentile, not $\hat\delta$, carries the shared-bias signal.

---

5b. Pairwise PABAK and pooled-reference recovery

When the divergent flag fires, the Report Card abandons the panel-aggregate summary and routes to a pairwise PABAK matrix plus per-rater pooled-reference $(\widetilde\Se_j, \widetilde\Sp_j)$.

Why PABAK at the pair level. PABAK is the framework’s pairwise coefficient because under symmetric raters its closed-form expectation reduces to $E[\mathrm{PABAK}] = (2q-1)^2$, a function of rater quality $q$ alone — independent of prevalence and of the subject-prevalence distribution $F$. Pairwise reading needs that property because every rater pair has its own observed marginal $\hat\pi_{+, ij}$, and those per-pair marginals can move substantially with rater asymmetry. Cohen’s $\kappa$ at the pair level is prevalence-attenuated (the $\kappa$ paradox of Byrt, Bishop, and Carlin (1993), the original motivation for PABAK); AC1’s chance term carries a milder but non-zero prevalence dependence; ICC at $k = 2$ is degenerate because one pair of raters per subject leaves the variance decomposition under-identified.

Pooled-reference identifiability. The pooled-reference estimates are label-flip clean conditional on the panel-majority direction matching the truth direction: the reference is the observable panel majority rather than a latent class, so the estimates do not depend on which mode an EM solver lands in. They read each rater’s profile relative to whichever direction dominates the panel. Under a 2:3 panel where the minority direction happens to be the truth direction, the pooled-reference inverts and reads each rater’s profile backwards.

Empirical recovery (54-cell Monte Carlo). A sweep over $k \in \{3, 5, 7\}$, $\pi \in \{0.10, 0.30, 0.50\}$, asymmetry magnitude $A \in \{0.10, 0.20, 0.30\}$, and both shared-bias and split-bias configurations at $N = 200$ with $500$ replications per cell:

• Pooled-reference recovers each rater’s true bias direction in $94\%$ of split-bias in-majority-direction rater-replicates (rate improving in $k$ from $90\%$ at $k = 3$ to $97\%$ at $k = 7$).
• Correctly inverts the direction in $98\%$ of minority-direction rater-replicates — exactly as the conditional predicts.
• Mean bias on $\widetilde\Se_j$ is $0.06$ and on $\widetilde\Sp_j$ is $0.03$ at $A \geq 0.20$.

The pooled-reference summary is meaningful only at $k \geq 3$ (no rest-of-panel to pool against at $k = 2$) and is noisier on subjects where the panel is split. The package reports the per-rater pool size $\widetilde n_j$ alongside each pooled estimate so the reader can see how much information backs each one.

---

6. Worked code examples per Report Card branch

A small helper generates an N x k rating matrix under the paper’s diagonal DGP at a stated seed, k, N, per-rater Se/Sp, target prevalence pi, and logit-normal spread parameter F_sigma2:

library(grassr)

simulate_panel <- function(seed, k, N, Se, Sp, pi, F_sigma2 = 0.25) {
  stopifnot(length(Se) == k, length(Sp) == k)
  set.seed(seed)
  mu_logit <- qlogis(pi)
  p_i <- plogis(rnorm(N, mu_logit, sqrt(F_sigma2)))
  C   <- rbinom(N, 1L, p_i)
  Y   <- matrix(0L, N, k)
  for (j in seq_len(k)) {
    Y[, j] <- rbinom(N, 1L, ifelse(C == 1L, Se[j], 1 - Sp[j]))
  }
  Y
}

Aligned case (paper §4.1)

# Se = Sp = 0.85, k = 5, N = 1000, seed = 5
Y_aligned <- simulate_panel(
  seed = 5L, k = 5L, N = 1000L,
  Se = rep(0.85, 5), Sp = rep(0.85, 5),
  pi = 0.08
)
card_aligned <- grass_report(Y_aligned)
card_aligned
#> GRASS Report Card
#> 
#>   sample      = 5 raters, N = 1000, pi_hat = 0.21
#>   PABAK        = 0.49  ->  46th percentile  (decisive)  <- primary
#>   AC1          = 0.62  ->  46th percentile
#>   Fleiss kappa = 0.23  ->  47th percentile
#>   ICC          = 0.33  ->  50th percentile  [distribution-sensitive]
#>   delta       =   1 pp (aligned)
#>   thresholds  = (8.50, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   Notes:
#>     - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500).
#>     - Fitted-ICC F_key picked via glmer: mu_hat=-1.716, tau2_hat=1.633 -> F_key tau2=1.0000, mu=-1.386.
#>     - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=-1.386_tau2=1.0000, k=5, N=1000 (family=logit_normal, M1=0.239).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.

The three agreement coefficients land within ~1 pp of each other on their reference surfaces; the panel is aligned and the primary coefficient (PABAK) is safe to cite. ICC is reported alongside with [distribution-sensitive] since its reference surface depends on the full subject-prevalence distribution rather than on (q, pi_hat) alone.

Divergent case (paper §4.2)

# op_strong panel: R1,R3,R5 at Se=0.95/Sp=0.75; R2,R4 inverted
# k = 5, N = 1000, seed = 6, balanced prevalence
Y_divergent <- simulate_panel(
  seed = 6L, k = 5L, N = 1000L,
  Se = c(0.95, 0.75, 0.95, 0.75, 0.95),
  Sp = c(0.75, 0.95, 0.75, 0.95, 0.75),
  pi = 0.50
)
card_divergent <- grass_report(Y_divergent)
card_divergent
#> GRASS Report Card
#> 
#>   sample      = 5 raters, N = 1000, pi_hat = 0.50
#>   PABAK         = 0.49  ->  40th percentile  <- primary
#>   AC1           = 0.49  ->  58th percentile
#>   Fleiss kappa  = 0.49  ->  41st percentile
#>   ICC           = 0.62  ->  50th percentile  [distribution-sensitive]
#>   panel-agg.  = suppressed (divergent)
#>   delta       =  18 pp (divergent)
#>   thresholds  = (8.50, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   pairwise PABAK / surface percentile (lower / upper):
#>     R1     R2     R3     R4     R5    
#>   R1    --     25%    79%    43%    98% 
#>   R2    0.47   --     97%    90%    17% 
#>   R3    0.51   0.41   --     98%    96% 
#>   R4    0.48   0.52   0.42   --     62% 
#>   R5    0.55   0.46   0.54   0.50   --  
#> 
#>   per-rater vs panel-majority of OTHER raters (pooled-reference):
#>     R1   Se_tilde = 0.94  Sp_tilde = 0.77  (n_pos = 434, n_neg = 471, excl = 95)
#>     R2   Se_tilde = 0.71  Sp_tilde = 0.95  (n_pos = 466, n_neg = 439, excl = 95)
#>     R3   Se_tilde = 0.95  Sp_tilde = 0.71  (n_pos = 433, n_neg = 485, excl = 82)
#>     R4   Se_tilde = 0.74  Sp_tilde = 0.95  (n_pos = 468, n_neg = 441, excl = 91)
#>     R5   Se_tilde = 0.96  Sp_tilde = 0.78  (n_pos = 428, n_neg = 476, excl = 96)
#> 
#>   per-rater (latent-class fit; alongside pairwise):
#>     R1   Se = 0.95  (0.21, 0.97)   Sp = 0.78  (0.05, 0.82)
#>     R2   Se = 0.73  (0.05, 0.77)   Sp = 0.95  (0.27, 0.97)
#>     R3   Se = 0.96  (0.27, 0.98)   Sp = 0.72  (0.04, 0.76)
#>     R4   Se = 0.75  (0.05, 0.79)   Sp = 0.95  (0.25, 0.97)
#>     R5   Se = 0.96  (0.22, 0.98)   Sp = 0.78  (0.04, 0.82)
#> 
#>   Notes:
#>     - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500).
#>     - Fitted-ICC F_key picked via glmer: mu_hat=0.005, tau2_hat=5.278 -> F_key tau2=4.0000, mu=0.000.
#>     - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=+0.000_tau2=4.0000, k=5, N=1000 (family=logit_normal, M1=0.500).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.

All three agreement coefficients return values around 0.49 (uniformly moderate by Landis-Koch) but the surface percentiles spread by ~18 pp: AC1 lands materially higher than the κ-family. The divergent flag fires; the panel-aggregate summary is suppressed; the Report Card emits the pairwise PABAK matrix and per-rater pooled-reference (Se_tilde, Sp_tilde) against the panel-majority of the other k - 1 raters.

$k = 2$ counterexample (paper §4.3)

# High-quality k = 2 panel: Se = Sp = 0.97, pi = 0.08, N = 200
Y_k2 <- simulate_panel(
  seed = 17L, k = 2L, N = 200L,
  Se = rep(0.97, 2), Sp = rep(0.97, 2),
  pi = 0.08
)
card_k2 <- grass_report(Y_k2)
card_k2
#> GRASS Report Card
#> 
#>   sample      = 2 raters, N = 200, pi_hat = 0.11
#>   PABAK  = 0.89  ->  50th percentile
#>   AC1    = 0.93  ->  55th percentile  (decisive)  <- primary
#>   delta       = 5.4 pp (aligned)
#>   thresholds  = (8.25, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   Notes:
#>     - (k = 2, N = 200) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 3, N = 200).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.

Single-pair high-quality panel; the bundled k = 2 surface (densified to 12 q-points in v0.5.2) places the panel near the median of its (k, N, pi_hat) reference cohort.

---

7. Audit pass-rate detail

The paper’s App B (Reproducibility, grids, and DGPs) summarizes audit pass rates at Bonferroni-corrected $|z| < 4.80$. This companion lists the per-cell audit results for reviewers wanting to verify any specific corner.

---

8. Confidence-interval methods detail

The paper App B carries the load-bearing CI numbers (Monte-Carlo error on percentile, nearest-neighbour cost, bootstrap coverage at one calibration point). This companion adds the implementation detail.

---

References

The paper’s bibliography is the authoritative source. Key methodological references this companion leans on:

• Byrt, Bishop, Carlin (1993) — PABAK derivation.
• Cohen (1960), Fleiss (1971) — $\kappa$ family.
• Gwet (2008) — AC1.
• Krippendorff (2011, 2019) — $\alpha$.
• Koo & Li (2016) — ICC variant selection.
• Hui & Walter (1980) — $k = 2$ identifiability bounds.
• Dawid & Skene (1979) — latent-class fit.
• Neyman & Pearson (1933) — size-controlled testing.

(Full citations in references.bib next to the paper source.)