Skip to contents

Per-rater Se/Sp asymmetry diagnostic

Source: R/asymmetry.R
check_rater_asymmetry.Rd

check_rater_asymmetry() is the rater-level companion to check_asymmetry(). It takes per-rater sensitivity and specificity estimates, computes the per-rater gap |Se_j - Sp_j|, reduces them to a scalar delta_hat = mean_norm |Se_j - Sp_j| (or max), and assigns the three-tier model-safety regime that governs whether q_hat (the GRASS operating-quality projection onto the Se = Sp diagonal) is trustworthy as the primary summary.

Usage

check_rater_asymmetry(
  se,
  sp,
  rater = NULL,
  threshold_caution = 0.05,
  threshold_unsafe = 0.1,
  summary = c("max", "mean")
)

Arguments

se

Numeric vector, one per rater. Per-rater sensitivity estimates in [0, 1].

sp

Numeric vector, one per rater. Per-rater specificity estimates in [0, 1]. Same length as se.

rater

Optional character vector of rater labels, same length as se. Defaults to "R1", "R2", ... .

threshold_caution

Boundary between Tier 1 (ok) and Tier 2 (caution). Default 0.05.

threshold_unsafe

Boundary between Tier 2 (caution) and Tier 3 (unsafe). Default 0.10.

summary

How to reduce per-rater gaps to the scalar delta_hat. "max" (default, conservative tripwire – any single rater above the threshold triggers escalation) or "mean" (panel-average asymmetry). The framework uses "max"; "mean" is provided for sensitivity checks.

Value

An S3 object of class grass_asymmetry with fields:

  • per_rater: data.frame of rater, se, sp, and gap = |se - sp|

  • delta_hat: the scalar delta_hat summary

  • summary: which summary statistic was used ("max" or "mean")

  • tier: integer tier (1, 2, or 3)

  • regime: character regime label ("ok", "caution", or "unsafe")

  • thresholds: named list of the caution and unsafe cutoffs

Details

This function is appropriate when the user already has per-rater Se/Sp estimates – typically from a Hui-Walter / Dawid-Skene latent-class fit (see latent_class_fit()), a simulation with known truth, or a reference-standard comparison. It is the function called inside the divergent branch of the new check_asymmetry() / grass_report() workflow when per-rater (Se, Sp) become available.

For ratings-matrix input (when only the N x k binary ratings are available, no per-rater Se/Sp), use check_asymmetry() instead – it computes the cross-coefficient percentile spread without requiring identifiability of per-rater (Se, Sp).

Tier thresholds follow the three-tier architecture:

  • Tier 1 – ok (delta_hat < 0.05): diagonal default. Report q_hat +/- SE plus EDR and EMR_panel without per-rater disclosure.

  • Tier 2 – caution (0.05 <= delta_hat < 0.10): diagonal + diagnostic. Report q_hat +/- SE with a caution flag and the delta_hat value.

  • Tier 3 – unsafe (delta_hat >= 0.10): full latent-class. q_hat is withheld as primary; report per-rater (Se_j, Sp_j) via a Hui-Walter / Dawid-Skene fit. PABAK's prevalence-flatness was conditional on Se = Sp; at Tier 3 that condition is visibly broken.

se and sp are not identifiable from a single 2-rater binary table without external ground truth. Supply them from: (a) a simulation with known truth, (b) a Hui-Walter / Dawid-Skene latent-class fit under k >= 3 and prevalence heterogeneity, or (c) a reference-standard comparison. Passing raw table() off-diagonals as se / sp is wrong and the function cannot detect the error.

See also

check_asymmetry() for the ratings-matrix companion (panel-level percentile spread). latent_class_fit() for the recommended source of (se, sp) estimates.

Examples

# Tier 1: symmetric raters -- safe to use q_hat as primary
check_rater_asymmetry(se = c(0.86, 0.88, 0.84), sp = c(0.85, 0.87, 0.86))
#> grass rater asymmetry diagnostic (per-rater |Se - Sp|)
#>   raters              : 3
#>   per-rater gaps |Se-Sp|
#>     R1        Se = 0.860   Sp = 0.850   gap = 0.010
#>     R2        Se = 0.880   Sp = 0.870   gap = 0.010
#>     R3        Se = 0.840   Sp = 0.860   gap = 0.020
#>   delta_hat (max)    : 0.020
#>   thresholds           : caution = 0.050,  unsafe = 0.100
#>   tier                 : 1  (ok)
#>   reporting guidance   : Diagonal default: report q_hat plus EDR / EMR_panel.

# Tier 2: one rater pressing the Se = Sp assumption
check_rater_asymmetry(se = c(0.90, 0.88, 0.92), sp = c(0.82, 0.86, 0.85))
#> grass rater asymmetry diagnostic (per-rater |Se - Sp|)
#>   raters              : 3
#>   per-rater gaps |Se-Sp|
#>     R1        Se = 0.900   Sp = 0.820   gap = 0.080
#>     R2        Se = 0.880   Sp = 0.860   gap = 0.020
#>     R3        Se = 0.920   Sp = 0.850   gap = 0.070
#>   delta_hat (max)    : 0.080
#>   thresholds           : caution = 0.050,  unsafe = 0.100
#>   tier                 : 2  (caution)
#>   reporting guidance   : Diagonal + diagnostic: report q_hat with caution flag and delta_hat.

# Tier 3: within-rater Se-favoring regime -- requires latent-class fit
check_rater_asymmetry(se = c(0.95, 0.93, 0.94), sp = c(0.78, 0.80, 0.79))
#> grass rater asymmetry diagnostic (per-rater |Se - Sp|)
#>   raters              : 3
#>   per-rater gaps |Se-Sp|
#>     R1        Se = 0.950   Sp = 0.780   gap = 0.170
#>     R2        Se = 0.930   Sp = 0.800   gap = 0.130
#>     R3        Se = 0.940   Sp = 0.790   gap = 0.150
#>   delta_hat (max)    : 0.170
#>   thresholds           : caution = 0.050,  unsafe = 0.100
#>   tier                 : 3  (unsafe)
#>   reporting guidance   : Full latent-class: withhold q_hat as primary; report per-rater (Se_j, Sp_j).