2006 HSR&D National Meeting Abstract
1065 — Development of the QUASII (QUality Assessment of Studies in Informatics Implementations) Instrument
Phansalkar S- (George E. Wahlen VAMC, Salt Lake City, UT)
Staggers N (College of Nursing, University of Utah, Salt Lake City, UT)
Weir CR (George E. Wahlen VAMC, Salt Lake City, UT)
We are conducting a meta-analysis of the efficacy of clinical informatics solutions (CIS) implementations. Quality assessment of studies in informatics poses problems peculiar to this domain, including: failure to specify the independent variable (e.g. the intervention and confounders of the implementation process), failure to model the relationships between variables, reporting only significant variables, inappropriate unit of analysis, and a lack of generalizability to other interventions. In this study, we describe the development of an instrument, QUASII (QUality Assessment of Studies in Informatics Implementations), designed to detect biases specific to CIS implementations.
Current quality assessment instruments were reviewed. A modified Delphi approach was used to identify pertinent elements. Each item is scored on a 7-point Likert scale ranging from ‘Completely Inadequate’ to ‘Completely Adequate’. Inter-rater reliability for the weighting of the constructs to be included in QUASII, was evaluated.
Nine quality assessment instruments were used to develop an initial list of 52 items. Following expert consensus, the final instrument included 9 constructs specific to informatics: Confounders of CIS Implementation, Clinical Unit of Analysis, Clinical Applicability, Unintended Organizational Co-Interventions, Equality of User Groups, Construct Validity of CIS Intervention, Causal Model of CIS Effects (e.g. underlying mediating and moderating variables), Specification of Dependent Variables (e.g. publishing only significant results), and Appropriate Generalizability of Findings. Intraclass correlation coefficient (ICC) between three experts was 0.78; (95% CI: 0.56, 0.90).
QUASII is the first instrument for the assessment of study quality specific to informatics. Further, QUASII describes biases that are peculiar to CIS implementations and uses a weighted scale for the measurement of its constructs. This enables the measurement of change that is attributable to the CIS alone and prevents inferences that may lead to false estimates of the impact of a CIS intervention.
VA has a strong culture of deploying CIS for improving patient care. The development of an instrument to critically appraise the quality of informatics studies can help derive proper conclusions about their applicability to VA hospitals and the dissemination of research findings in CIS implementations into clinical practice.