Skip to contents

Pairwise reliability for a divergent panel

Source: R/pairwise_agreement.R
pairwise_agreement.Rd

Computes pairwise PABAK between every pair of raters in an \(N \times k\) binary rating matrix and places each entry on the \(k = 2\) reference surface at the pair's observed marginal. Also returns per-rater pooled-reference sensitivity and specificity — each rater's call rate against the panel-majority of the other \(k - 1\) raters — which uses the larger panel's information about rater behavior rather than discarding it to a strictly pairwise comparison.

Usage

pairwise_agreement(ratings, axis = c("inter", "intra"))

Arguments

ratings

An \(N \times k\) binary rating matrix (rows = subjects, columns = raters), or a data.frame whose columns are 0/1 / logical / two-level factor. Same input conventions as grass_report.

axis

Character; "inter" (default) or "intra". The intra-axis path treats the columns of ratings as per-rater viewing pairs; see paper Sec.3.3 intra-rater section.

Value

An object of class c("grass_pairwise", "list") with fields:

  • pabak_matrix — \(k \times k\) symmetric numeric matrix; \([i, j]\) is \(\mathrm{PABAK}_{ij}\), the diagonal is 1.

  • percentile_matrix — \(k \times k\) symmetric numeric (0–100); \([i, j]\) is the surface percentile of \(\mathrm{PABAK}_{ij}\) on the \(k = 2\) reference at the pair's observed marginal. Diagonal is NA.

  • marginal_matrix — \(k \times k\) symmetric numeric; \([i, j]\) is \(\hat\pi_{+, ij}\). Diagonal is NA.

  • band_matrix — \(k \times k\) character matrix; four-band label for each pairwise percentile. Diagonal is NA.

  • qualifier_matrix — \(k \times k\) character; decisive / moderate / weak per pairwise percentile.

  • pooled_per_rater — data frame with \(k\) rows, one per rater, columns rater, se_tilde, sp_tilde, n_pool_pos, n_pool_neg, n_pool_excluded (subjects with tied panel majority).

  • sample — list with k, N, pi_hat, tau2_hat, axis.

  • notes — character vector with caveats, if any (e.g., undefined pooled-reference at \(k = 2\)).

  • call — the matched call.

Details

This function is the recommended primary deliverable when grass_report() flags the panel as divergent; the panel-aggregate coefficients no longer summarize the panel adequately, but the pairwise matrix exposes the panel's structure directly (uniform inconsistency, sub-group clustering, or single- rater outliers).

Examples

set.seed(6)
Se <- c(0.95, 0.75, 0.95, 0.75, 0.95)
Sp <- c(0.75, 0.95, 0.75, 0.95, 0.75)
truth <- rbinom(200, 1, 0.5)
Y <- sapply(seq_along(Se),
            function(j) ifelse(truth == 1,
                               rbinom(200, 1, Se[j]),
                               rbinom(200, 1, 1 - Sp[j])))
pw <- pairwise_agreement(Y)
pw
#> GRASS Pairwise Reliability
#> 
#>   sample      = 5 raters, N = 200, pi_hat = 0.51, tau2_hat = 0.117, axis = inter
#> 
#>   Pairwise PABAK (lower triangle = PABAK_ij; upper triangle = surface percentile):
#> 
#>    R1     R2     R3     R4     R5    
#> R1     --   75%    31%    79%    50% 
#> R2   0.40     --   38%    38%    69% 
#> R3   0.61   0.47     --   69%    56% 
#> R4   0.41   0.47   0.52     --   75% 
#> R5   0.49   0.39   0.50   0.40     --
#> 
#>   Per-rater behavior against pooled panel-majority:
#> 
#>  rater Se_tilde Sp_tilde n_pos n_neg n_excl
#>     R1     0.98     0.72    88    93     19
#>     R2     0.71     0.95    95    83     22
#>     R3     0.99     0.77    89    94     17
#>     R4     0.75     0.90    97    86     17
#>     R5     0.90     0.74    91    91     18
#>   (Se_tilde, Sp_tilde are calls vs panel-majority of OTHER raters,
#>    not against external truth. n_excl: subjects with tied majority,
#>    excluded from the per-rater pool.)