HSR&D » For_researchers » Measurement » Practice

  

Enter your search text Search HSR&D: Button to start search

Measurement in Practice: Annotated Bibliography

Please note that this section is an archive and is no longer being updated.

Studies Evaluating the Accuracy of Diagnostic Coding using the International Classification of Diseases

Prepared by Kimberly Raiford Wildes, DrPH, MA

Overview
The systematic classification of diseases has always intrigued patients and their healers. Western societies developed an interest in disease classification in the seventeenth and eighteenth centuries when they began to track the causes of sickness and death among their citizens. The International Classification of Diseases (ICD), now in its ninth and soon to be tenth iteration, is now the most widely used system for statistical classification of diseases in the world. Applications of the ICD codes have also evolved from classifying morbidity and mortality information for statistical purposes to reimbursement, administration, epidemiology, and health services research. The application of ICD codes to purposes other than what the classifications were designed has resulted in increased attention on code accuracy. Since the accuracy of the classifications influences each application differently, accuracy is a complicated issue. To facilitate code users’ ability to evaluate ICD code accuracy for their particular application, the Measurement Excellence Initiative included a repository of abstracts on articles that relate to ICD code accuracy.

Article 1: Accuracy of diagnostic coding for Medicare patients under the prospective-payment system. Hsia DC, et al. 1988

Article 2: Administrative databases' complication coding in anterior spinal fusion procedures. Faciszewski T, et al. 1995

Article 3: Accuracy of coding for cardiac catheterization and percutaneous transluminal coronary angioplasty at a Department of Veterans Affairs Medical Center. Mendelsohn AB, at al. 1996

Article 4: Inaccuracy of the International Classification of Diseases (ICD-9-CM) in identifying the diagnosis of ischemic cerebrovascular disease. Benesch C, et al. 1997

Article 5: Quality of data regarding diagnoses of spinal disorders in administrative databases. A multicenter study. Faciszewski T, et al. 1997

Article 6: Community-acquired pneumonia: can it be defined with claims data? Whittle J, et al. 1997

Article 7: Accuracy of ICD-9-CM coding for the identification of patients with acute ischemic stroke: effect of modifier codes. Goldstein LB, et al. 1998

Article 8: Agreement between administrative files and written medical records: a case of the Department of Veterans Affairs. Kashner TM, et al. 1998

Article 9: Accuracy of ICD-9-CM codes in detecting community-acquired pneumococcal pneumonia for incidence and vaccine efficacy studies. Guevara RE, et al. 1999

Article 10: Positive predictive value of the diagnosis of acute myocardial infarction in an administrative database. Peterson LA, et al. 1999

Article 11: Effect of discharge letter-linked diagnosis registration on data quality. Prins H, et al. 2000

Article 12: Stroke: who's counting what? Reker DM, et al. 2001

Article 13: Accuracy of computerized outpatient diagnoses in a Veterans Affairs general medicine clinic. Szeto HC, et al. 2002

Article 14: Self-reported awareness and use of the International Classification of Diseases coding of inflammatory bowel disease services by Ontario physicians. Farrokhyar F, et al. 2002

Article 15: Can administrative data be used to compare postoperative complication rates across hospitals? Romano PS, et al. 2002

Article 1

Background: The authors wished to study the accuracy of coding for diagnosis-related groups (DRGs) in hospitals receiving Medicare reimbursement. A two-stage cluster method was used to sample 239 hospitals and 7,050 medical records. The authors concluded that “creep” (mis-specification, miscoding, and resequencing) does occur in DRG coding, which results in hospitals being overpaid for Medicare patients. Results suggested a DRG error rate of 20.8 percent, distributed equally across physicians and hospital administration. Smaller hospitals had higher error rates (p <0.0001), and 61.7 percent of coding errors favored the hospital (causing the mean hospital’s case-mix index—measure of illness complexity in a hospital—to increase by 1.9 percent).

Method to establish accuracy: Only acute care, short-stay hospitals were included and hospitals in the states of New York, New Jersey, Massachusetts, and Maryland were excluded. In addition, hospitals included had to have contributed data on discharged patients to the DRG categories at the start, and had to have participated as a Medicare provider. Patients included in the sample were similar to the larger Medicare population in age, sex, and average length of stay. Medical record DRGs were validated by technicians from the Health Data Institute of Lexington, Mass. Blinded technicians, supervised by a registered record administrator, reviewed each chart and translated each diagnosis and procedure into ICD-9-CM codes. Coders were told to ignore marginal problems or honest differences in judgment. Re-abstracted codes were processed into correct DRGs by use of the GROUPER computer software. If DRGs from the two times differed, a physician panel member (blinded to the assigned code) evaluated the record. If the coding discrepancies represented complex classification issues, then a committee of board certified physicians with experience in ICD-9-CM coding resolved the case. To perform reliability checks and quality performance, five percent of the sample received a second, blinded coding by a different technician. This revealed no significant discrepancies.

back to top

Article 2

Background: This study sought to evaluate the accuracy of hospital ICD-9-CM complication coding in spinal procedures. The sample consisted of 310 consecutive patients (mean age=44 years, 55% female) who underwent anterior spine fusion at a tertiary spine care center. With a physician’s abstraction as the gold standard, sensitivity of ICD-9-CM codes was 44%, specificity was 86%, positive predictive value was 66%, and negative predictive value was 71%. The authors concluded that spinal procedure complication studies that use hospital administrative data might be intrinsically flawed because the data available to researchers may be inaccurate.

Method to establish accuracy: A research technician reviewed all medical records 3-4 months before the scheduled surgery. One research technician, a certified medical records technician with 15 years experience, abstracted the complications data. An independent physician reviewer, who was blinded to the ICD-9-CM discharge abstract, retrospectively reviewed the medical records which served as the gold standard. In addition, the physician added specific inclusion and exclusion criteria for certain complications. The hospital then provided reports of the ICD-9-CM diagnostic codes used during the patients’ hospital stay. From that, codes that met criteria for complications as established by Deyo et al. were identified. The medical records information was compared with the ICD-9-CM coded complications in the hospital databases.

back to top

Article 3

Background: The authors wished to examine the accuracy of coded data for cardiac catheterization (CATH) and percutaneous transluminal coronary angioplasty (PTCA) in a Pittsburgh VA hospital, using Patient Treatment File (PTF) data. The researchers identified patients discharged between 1988-1991 who were coded in the PTF with CATH (n=2325), PTCA (n=201), or coronary angiography. Patients were mostly male, mean age=61 years, and white. Errors in coding for CATH and PTCA were compared among groups classified by fiscal year, length of stay (LOS), vital status, race, and whether a coronary artery bypass graft surgery (CABG) had been performed. They also reviewed a convenience sample of 45 discharge face sheets as compared to PTF data. Sensitivity and positive predictive value (PPV) varied by hospital characteristics. Coding accuracy increased over 4 years, and was lower in patients whose hospital stays were longer. In general, a code for CATH in the PTF correctly identified 96% of patients; sixty-six percent of patients who underwent PTCA had correct PTF codes.

Method to establish accuracy: Clinical data taken from several sources were used as the reference standard for which to compare PTF data: cardiology section log of all patients treated in the catheterization lab, log of all PTCAs performed, alphabetized cardiology file of summary dictations for all CATHs and PTCAs, and the standard medical record (only used when other data sources were inadequate). Errors in coding were compared for the aforementioned subgroups. Chi-square tests were used to determine if differences in error rates were statistically significant overall and among patients who actually had the procedures. Stratified analyses were used to compare overall error rates. The authors reviewed a sample of 45 discharge face sheets to assess if errors in PTF data corresponded with omission of recording the procedure on the discharge sheet, miscoding, or inaccurate data entry into the database.

back to top

Article 4

Background: This study assessed the accuracy of ICD-9 codes for cerebrovascular disease by comparing codes in administrative databases with clinical findings from medical record abstractions (gold standard). The sample included patients from five academic medical centers who had ICD-9 codes of 433 through 436 in administrative databases. The study concludes that ICD-9 coding may be inaccurate in classifying patients with ischemic cerebrovascular disease, as accuracy depends on the specific code being used. Limiting the identifying ICD-9 code to the primary position increased agreement with the medical record.

Method to establish accuracy: Patients who were coded as 433 through 436 in administrative databases were interviewed, medical records were reviewed by trained abstractors, and clinical conditions and history of cerebrovascular symptoms were recorded. Patients were classified into three groups: 1) stroke according the WHO definition, 2) TIA but not stroke according to WHO definition, and 3) asymptomatic for cerebrovascular disease. These results were compared to the codes from the administrative databases.

back to top

Article 5

Background: The investigators aimed to evaluate accuracy of administrative database data regarding six major categories of spinal disorder diagnoses, using eight different spine center hospitals. The sensitivity of hospital coding ranged from 28% to 100% depending on diagnosis, and specificity ranged from 94% to 100%. Accuracy of coding depended on the diagnosis, and most errors were related to the low sensitivity of coding for previous spinal operations (28% of such diagnoses were correct). The authors concluded that the accuracy of spinal disorder diagnosis recorded in administrative databases depends on the largely on the specific condition being evaluated.

Method to establish accuracy: Medical records specialists at the hospitals used the St. Anthony’s ICD-9-CM Code Book as the coding reference, and the records were sent to a clinical coordinator who reviewed the charts. Records of 189 patients were independently reviewed by a spine surgeon who was blinded to the original ICD-9-CM data in the discharge abstract. The physician then assigned the appropriate ICD-9-CM diagnostic codes, which were used as the gold standard. A second blinded physician reviewed charts and assigned codes for a random sample of 20 of the 189 records; no disagreements between physicians were found for primary diagnoses, but minor disagreements were found for secondary diagnoses. The specific codes listed by the physician were collapsed into six major diagnostic categories for comparison. The codes assigned by the physician were then compared with codes assigned by personnel in the medical records department of each of the hospitals.

back to top

Article 6

Background: Researchers wished to compare discharge diagnoses of community-acquired pneumonia (CAP), based on clinical chart versus three administrative approaches. The sample included 144 discharges from Presbyterian University Hospital (52% male, median age=66 years, median length of stay=10 days, inhospital mortality 15%). The administrative approaches included diagnoses based on Diagnosis Related Groups (DRGs), a pneumonia diagnosis code in the principal diagnosis position, and a diagnosis and procedure based algorithm developed by the authors. These methods were compared to medical chart review for level of agreement. The authors also compared the length of stay (LOS) and mortality among CAP groups identified in different ways. Agreement between the clinical chart review and the three administrative approaches ranged from 86% to 80%, with DRG classification performing the worst. Kappa values (k) were as follows: algorithm (0.69), principal diagnosis (0.68), and DRG (0.59). Further, the groups identified by administrative approach had shorter LOS (p<0.01) than those identified by chart review. The authors conclude that each approach had good agreement with the clinical review. Using the principal diagnosis position gave results similar to the algorithm and was less complex. However, using these administrative approaches to identify CAP may underestimate LOS.

Method to establish accuracy: Clinical review was used as the gold standard. Researchers examined records using a standard protocol based on implicit and explicit criteria to determine which discharges were for CAP. Using explicit criteria for CAP or HAP, a physician assistant (PA) and registered nurse (RN) reviewed charts and decided if the discharge was for CAP. If they agreed, an internist gave it a final review. If the internist agreed, a final assignment was made; however, if he disagreed a further review was performed. If a discharge did not meet explicit criteria for CAP or HAP, each reviewer again examined the documents using implicit criteria. If they agreed on a classification, then it was accepted. If agreement could not be reached, then the full medical record was obtained. The internist made a classification based on this and prepared a synopsis of the case, which was then classified by two additional internists. The majority classification between the three internists was considered correct. The clinical review was compared to three administrative approaches for agreement: an algorithm based on the discharge abstract, classification based on four DRGs judged to be CAP, and a pneumonia diagnosis code in the principal position.

back to top

Article 7

Background: The study aimed to determine if the accuracy of coding for ischemic stroke using the primary ICD-9-CM codes is improved by modifier codes and how specific codes may reflect stroke subtypes. Hospital charts of 175 discharged patients from the Durham VAMC with ICD-9-CM codes for ischemic stroke in the first position were reviewed. Although modifier codes were used, 15%-20% of patients coded as acute ischemic stroke by primary ICD-9-CM codes actually had other conditions. The proportion of patients with acute stroke increased from 61% to 79% when modifier codes were used, but the use of modifier codes did not have a significant effect on coding accuracy if patients with code 433 were excluded. Assignment of stroke subtype was inaccurate.

Method to establish accuracy: The primary discharge diagnosis was verified by review of information in the discharge summary, and then a presumed stroke subtype was assigned by the investigator into one of five categories based on criteria developed for the TOAST (Trial of ORG 10172 in Acute Stroke Treatment). The sensitivities and specificities of each code were calculated.

back to top

Article 8

Background: The study examined the reliability of VA Patient Treatment Files (PTF) and Outpatient Care Files (OCF) concerning demographics, use of care and diagnoses in comparison to medical charts and administrative records (MR). All diagnoses were coded using the first three digits of the ICD-9-CM. The study utilized a national random sample of 1,356 outpatients and 414 inpatients drawn from all VAMCs. Demographics, length of stay and selected diagnoses showed adequate reliability. Determining the treating bedsection or clinic showed inadequate reliability. Patient Treatment Files/Outpatient Care Files resulted in higher estimates of disease prevalence and outpatient costs when compared to medical charts. PTF/OCF agreed with MR as follows: demographics (average k=0.92, range=0.87-0.98), inpatient principal diagnoses (average k=0.75, range=0.54-1.0), inpatient secondary diagnosis (average k=0.62, range=0.39-0.82), inpatient bedsection locale (average k=0.53, range=0.01-0.85), and outpatient clinic stops (average k=0.46, range=0.01-1.0). The authors advise researchers to construct health and utilization data from several sources, including the patient and collaterals, administrative files, and medical record.

Method to establish accuracy: Records were abstracted at the VA Medical Care Cost Recovery (MCCR) national office by a focus group of two utilization review nurses and 24 medical record coders who were unaware of the administrative file data. Abstractions were performed using structured forms according to VA regulations concerning the VA databases. The focus groups met in teams to discuss cases and ensure uniform coding across sites. Data reliability was determined by comparing each PTF or OCF with an entry from the MR, and calculating percent agreement and kappa statistics. Group averages calculated using PTF/OCF were compared to those based on MR in order to determine data bias.

back to top

Article 9

Background: The authors assessed the accuracy of coding data to identify laboratory-confirmed pneumococcal pneumonia by determining sensitivity, specificity, positive predictive value, and negative predictive value. The information was collected using an Ohio community-based pneumonia incidence study, of which patients were hospitalized with community-acquired pneumonia. Sensitivity, positive predictive value, and negative predictive value of individual and groups of codes differed by case definition or pneumococcal pneumonia. The authors conclude that incidence and vaccine efficacy studies that are able to validate diagnosis by medical chart review can use a combination of ICD-9-CM codes to maximize sensitivity.

Method to establish accuracy: Patients’ medical charts were abstracted and patients were interviewed using standardized questionnaires. The authors used specific case definitions to categorize patients as definite, probable, possible, or no pneumococcal infection. Sensitivity, positive predictive value, and negative predictive value were determined by comparing frequencies of ICD-9-CM codes for patients with or without evidence of pneumonia.

back to top

Article 10

Background: The study was undertaken to determine the positive predictive value of ICD-9-CM codes for acute myocardial infarction (AMI) and cardiac procedures. They examined VA administrative data (Patient Treatment File, PTF) and compared it to chart abstracted data (used as standard) for a national sample of 5,151 discharged male veteran patients with a primary ICD-9-CM diagnosis of AMI. The positive predictive value of coding AMI was 96.9%, sensitivity of coding catheterization was 96% while specificity was 99%, coding for coronary artery bypass graft surgery had a sensitivity of 95.7% and a specificity of 100%, and lastly coding for percutaneous transluminal coronary angioplasty yielded sensitivity of 90.3% and a specificity of 99.7%. The authors concluded that coding for AMI and related procedure using ICD-9-CM codes in the PTF yielded positive predictive value that was comparable to or better than previous reports of administrative databases. Positive predictive value is high when using an algorithm to refine AMI cases; patients who were hospitalized less than three days because of “rule-out” myocardial infarction were excluded.

Method to establish accuracy: Once the random sample was generated, they used the Cooperative Cardiovascular Project structured review instrument and criteria to confirm the AMI diagnosis. Four registered nurses, who were not blinded to the PTF coding, collected medical record information. They evaluated data based on patient symptoms, electrocardiographic data, and cardiac enzyme values. This medical record abstraction was used as the gold standard in comparison to the PTF ICD-9-CM codes. Positive predictive value, sensitivity, and specificity were calculated to assess the accuracy of the PTF ICD-9-CM codes.

back to top

Article 11

Background: The authors redesigned the form-based diagnosis registration at a pediatric center in an academic medical center, so that pediatricians provided diagnoses with codes in a heading of the discharge letter instead. The purpose of this study was to compare the quality of this discharge letter-linked diagnosis registration with the quality of the previous form-based registration. The sample included 60 randomly selected admissions for both registration periods; mean age=4.5 years and length of stay=8.8 days for the old format, and mean age=5.2 years and length of stay=7.2 days for the redesigned format. The study was retrospective with blinded pre- and post-measurements. “Re-abstracted diagnosis descriptions of the text of discharge letters” were used as the gold standard, and completeness and accuracy were the outcome measures. At the three-digit level of the ICD-9-CM, completeness of the old format was 51% (95% CI: 44-58%) and accuracy was 65% (95% CI: 58-72%), while completeness of the new discharge letter-linked diagnosis registration was 54% (95% CI: 47-60%) and accuracy was 67% (95% CI: 60-74%). Discharge letter-linked diagnosis registration was neither more accurate nor more complete than form-based. It also did not have significantly higher specificity.

Method to establish accuracy: The text of the discharge letter was used as the gold standard by gathering electronic versions of the letters and removing the medical registration heading for blinding. One pediatrician highlighted text that referred to relevant diagnoses, and decided what category (primary or secondary diagnosis, reason for admission) each diagnosis belonged to. A blinded medical record coder checked if each diagnosis was recorded in the hospital information system (HIS) and whether it could be recognized in the corresponding letter. The expert coder followed formulated rules. Completeness was defined as “the proportion of marked diagnoses in discharge letters that, at a three-digit level of ICD-9-CM, are coded in the corresponding admission records of HIS.” Accuracy was defined as “the proportion of diagnosis codes in HIS that, at 3-digit level of ICD-9-CM, have matching diagnosis descriptions in corresponding discharge letters.” Specificity was calculated as “the proportion of accurate HIS codes that contain, as far as possible with the local 6-digit codes, all additional diagnostic information in text of corresponding discharge letters.”

back to top

Article 12

Background: This study assessed the accuracy of ICD-9-CM codes in identifying patients with stroke. The sample consisted of 279 patients with new stroke and 392 non-stroke patients from 11 VAMCs, identified by medical record review. The prospective portion of the sample was identified by clinician research assistants as having a recent stroke event, while the retrospective portion was identified as such by using administrative data (Patient Treatment File, PTF) and ICD-9 codes based on a high-sensitivity scoring algorithm. Accuracy of ICD-9-CM codes in identifying patients diagnosed with stroke varied, confirming what previous literature has found. The high-sensitivity coding algorithm yielded a sensitivity of 91%, specificity of 40%, positive predictive value of 52%, and negative predictive value of 86%. The high-specificity coding algorithm yielded sensitivity of 54%, specificity of 87%, positive predictive value of 75%, and negative predictive value of 73%. The authors suggest caution in using ICD-9 coded administrative data to identify patients with stroke.

Method to establish accuracy: For the prospective portion, the gold standard for diagnosis of new stroke was determined by reviewing the medical record and the patient record to make sure a diagnosis of stroke had been documented. In the retrospective portion, the gold standard was review of patients’ medical records at each study site by the same clinician research assistants or physician site investigators to determine presence or absence of a stroke diagnosis. Sensitivity and specificity were assessed using individual ICD-9 codes and two coding algorithms (the high-sensitivity algorithm and a high-specificity algorithm).

back to top

Article 13

Background: The authors sought to determine the accuracy of outpatient primary care diagnostic information as recorded in electronic administrative and clinical problem list files in the VA VISTA database system (which uses ICD-9 codes) compared with medical charts (used as the gold standard). The study was a cross-sectional chart review of 148 VAMC general medicine patients in California attending a half-day clinic module. Mean patient age was 64 years and 96% were men. Charts were reviewed for nine diagnoses that could affect the drug choice for treatment of hypertension. Sensitivity and specificity were estimated for both sources of electronic data in detecting medical chart diagnoses. Administrative file sensitivity was greater than 80% for 8 of 9 diagnoses, and specificity ranged from 91% to 100%. Clinical problem list sensitivity was 49% and specificity ranged from 98% to 100%. The administrative approach yielded higher sensitivity but less specificity than the clinical problem file. The authors concluded that electronic, computerized administrative data across several visits may be accurate and efficient at ascertaining chronic medical diagnoses, while using clinical data is more specific but less sensitive.

Method to establish accuracy: One reviewer abstracted medical chart notes and recorded the absence or presence of the 9 diagnoses. A second reviewer re-abstracted all charts a second time. The results were compared and discrepancies reviewed by a third reviewer, then again by the first and second reviewers until consensus was reached. VISTA codes were then compared to the medical chart review codes (gold standard) and specificity and sensitivity were determined.

back to top

Article 14

Farrokhyar F, McHugh K, Irvine EJ. Self-reported awareness and use of the International Classification of Diseases coding of inflammatory bowel disease services by Ontario physicians. Can J Gastroenterol. 2002 Aug;16(8):519-26. PMID: 12226679

Background: The study objective was to estimate the awareness, use, and accuracy of ICD-9 codes for Crohn’s disease (CD) and ulcerative colitis (UC) by physicians. Rates showed that ICD-9 coding was used more frequently among physicians who graduated after 1981, and they did not differ by sex. Gastroenterologists used these codes more frequently than all other physicians. More than 75% of physicians used ICD-9 inflammatory bowel disease (IBD) codes always or frequently when they billed for IBD services (this was used as the self-reported accuracy criterion). These codes were used by 10% of physicians to bill for non-IBD services. The authors conclude that there is acceptable accuracy and use of ICD-9 diagnostic coding for UC and CD services.

Method to establish accuracy: A questionnaire was mailed to all Ontario gastroenterologists, and a 10% random sample of pediatricians, internists, pediatric or general surgeons, and family physicians, in order to estimate the frequency and confidence intervals of using codes 555 (CD) or 556 (UC) when billing for CD and UC services. Chi square (c2) tests were used to compare physician groups. Therefore, accuracy was determined as self-reported accuracy by physicians.

back to top

Article 15

Background: The objective of this study was to determine the accuracy of reporting postoperative complications in administrative data, whether it varied across hospitals, and whether serious complications were more consistently reported. This was a retrospective cohort study of 991 randomly selected adults, at 30 nonfederal acute care hospitals in California, who underwent elective lumbar diskectomies. Hospitals with low or high complication rates were over sampled, as well as patients who experienced complications. Hospital reported complications yielded a sensitivity of 35%, specificity of 98%, positive predictive value of 82%, negative predictive value of 84%, and a kappa statistic of 0.42. Complications reported with adequate accuracy of at least 60% sensitivity were: reoperation, bacteremia or sepsis, post-op infection, deep vein thrombosis, cardiac arrest, hypoglycemia, acute renal failure, and pulmonary embolus. ICD-9-CM complications were underreported among the patients seeking lumbar diskectomy, and even more so at hospitals with fewer complications than expected (versus hospitals with more expected complications). More than 50% of the difference between these groups of hospitals was due to reporting variation. The authors conclude that the validity of using coded complications to compare provider performance is questionable.

Method to establish accuracy: Post-op complications were defined by literature review and consultation with clinical experts. Sixteen categories of complications were identified and mapped to ICD-9-CM codes by two coding professionals. They then reviewed ICD-9-CM Volume I and the complication lists developed by DesHarnais, Iezzoni and Deyo (cited within article) to identify complications that the panel may have missed. Medical records were abstracted using an instrument developed by the authors and the advisory panel, which served as the gold standard. Each record was reviewed by an accredited records technician who was certified as a Coding Specialist. The specialist was blinded to the original abstract and had at least 10 years of experience. The ICD-9-CM codes were verified and entered into the database by a nurse or physician reviewer, who then abstracted clinical data. Supervisors checked 5% of the records. Complications reported by hospitals were compared with the independent recoding of the same records.

back to top