In 2003, the VA MS Centers of Excellence created a VA MS National Data Repository as a part of their clinical, administrative, and research mission. This repository, containing over 40,000 unique cases, is one of only two population based repositories for MS in the United States, contains the largest collection of information on men with MS in the world, and is the only national source of information on the care of veterans with MS within VHA.
The original purpose of the VA MS National Data Repository was case identification. It was created from VA demographic, service utilization, pharmacy, and cost databases using broad inclusion criteria in order to locate all individuals receiving care for MS with VHA. However, not every individual coded as receiving care for MS actually has MS. Reasons for misclassification are many, and include the fact that MS has a complex presentation, and the fact that VA encounter/prescription files also contain errors. Finally, recent advancements in the understanding of MS have greatly improved the validity and reliability of MS diagnoses.
In 2004, VISN 5 conducted a small preliminary study in which a single nurse practitioner conducted a chart review of cases in that VISN with an ICD-9 code of 340.0 and found that as many as 40% of individuals in the VHA MS National Data Repository would not meet criteria for having definite MS. It also established that an algorithm using readily available patient treatment information could be used to identify 'true' cases of MS. This early work was important in that it underscored the need for case validation within the repository, yet its generalizability was limited due to the fact it was conducted at one time, by a single individual, at a single predominantly-urban location, and without comparison to a face-to-face clinical examination by an MS specialist.
The next logical step in the development of the VA MS National Data Repository would be to continue to identify the best and most efficient methods for validating the cases contained within it. There is a widely accepted gold standard for diagnosing, and confirming a diagnosis of MS, known as the McDonald criteria. There is little evidence, however, to suggest that these standards are routinely or uniformly applied to all veterans within VHA. Over the past two years, the VA MS Centers of Excellence have developed an electronic template designed to systematically review these diagnostic criteria for record review or a face-to-face encounter. The feasibility and utility of this electronic template, however, remain to be established. Accurate identification of individuals with a confirmed diagnosis of MS within the repository would increase its clinical value by helping to clarify national treatment needs, its administrative value by providing more accurate estimates of service utilization and cost, and its research value by highlighting individuals who might be recruited for future clinical trials.
A) To classify individuals who are included in the Veterans Administration (VA) Multiple Sclerosis (MS) National Data Repository and have been seen for care in VISN 20 into three categories: 1) MS, 2) not MS, and 3) possible MS using electronic medical record review and calculate validity statistics comparing this review to a face-to-face diagnosis by an MS specialist and B) To generate inter- and intra-observer reliability statistics for medical record reviewers with four different levels of professional training C) To estimate costs associated with validation of the entire national repository and D) To validate specific cases in VISN 20.
This study consisted of two study populations: a convenience sample of the first fifty veterans who attended a MS Clinic and consented to be in the study (the validity group) and a simple random sample of fifty veterans found in the VA MS National Data Repository who had ever sought care in VISN 20 (the reliability group). The validity of chart review vs. in-person evaluation of MS diagnosis was examined through examination of sensitivity and specificity. Kappa was used as the measure of inter- and intra-observer reliability.
Of the veterans seen in the MS Clinic 90% fulfilled the diagnostic criteria for MS when evalulated by a MS specialist. Two percent of veterans were classified as not MS and 8% as possible MS. When the people with possible MS were classified as MS, the sensitivity for the four different reviewers ranged from 0.90 to 0.86. When the criteria were relaxed to allow patient report of clinical and laboratory tests, the sensitivity ranged from 1.00 to 0.90. The specificity was either or 0 or 1.00. The intra-observer reliability ranged from 0.79-0.43 while the inter-observer reliability ranged from 0.83-0.31.
Results provide information about the reliability and validity of a chart review template to confirm the diagnosis of MS in the VA MS National Data Repository, provide preliminary data examining algorithms of existing administrative data associated with a higher likelihood of MS diagnosis, and provide rough estimates of the cost necessary to validate the entire VA MS National Data Repository.
- Turner AP, Hawkins EJ, Haselkorn JK, Kivlahan DR. Alcohol misuse and multiple sclerosis. Archives of physical medicine and rehabilitation. 2009 May 1; 90(5):842-8.