2031. Inter-coder Reliability of ICD9 Assignment: Flaws in an Important Diversity Assessment Tool
JF Hurdle, VA Salt Lake City Health Care System, CR Weir, Salt Lake City Health Care System, JR Nebeker, Salt Lake City Health Care System, JM Hoffman, Salt Lake City Health Care System

Objectives: During a large-scale, HSRD-funded study of adverse drug events (ADEs), we found evidence that ICD9 coders were non-uniform in assigning codes that we conclusively had shown to be documented in the chart.  Concerned about overall ICD9 reliability, the objective of this study was to measure reliability across coders at several different hospitals for all diagnoses in a set of cases.

Methods: We randomly selected 80 admissions from 237 (34%) known to have one or more ADEs in our previous study (we limited the study to ADE cases as a convenience sample for a later study). We hired three certified (RHIA or CCS) coders employed separately at nonVA hospitals to independently review the 80 admissions (for a total of four coders including ours). We calculated inter-case agreement (three-digit code match) and aggregate statistics. Coders were limited to a maximum of 10 codes per case (following local practice).

Results:  The four coders logged a total of 1,994 ICD9 codes (mean 6.2 codes/case). Overall, the sum of unique codes across all cases was 901; of these 168 (18.6%) agreed 4-ways (unanimous); 141 (16.5%) agreed 3-ways; and 171 (19%) agreed two-ways. The remaining 45.9% were codes that only one coder found in a case. The overall average agreement was very low at 2.2 codes/case.

Conclusions: ICD9 codes affect diverse populations of veterans in profound ways. For an admission, they are arguably THE most important metric of clinical outcome and resource consumption, usually reported by population subgroup. Decisions about a host of issues for these subgroups range from billing-reimbursement to resource allocation to clinician productivity profiling to DRG assignment to station- and VISN-performance measures to VA versus nonVA care provision, all hinging on ICD9 code sets. While many studies have assessed how accurate a set of codes is for a given admission, or show how certain diagnostic groups defy accurate coding, this study is unique in focusing on coding reliability between institutional personnel.

Impact: These results suggest that a concerted effort needs to reconcile cross-institutional differences in this crucial metric; at a minimum such an effort will clarify the quality and quantity of care provided to diverse veteran populations.