Talk to the Veterans Crisis Line now
U.S. flag
An official website of the United States government

Health Services Research & Development

Veterans Crisis Line Badge
Go to the ORD website
Go to the QUERI website

IIR 06-260 – HSR&D Study

New | Current | Completed | DRA | DRE | Portfolios/Projects | Centers | Career Development Projects

IIR 06-260
Identifying and Characterizing High Performing VHA Nursing Homes
Michael Shwartz PhD MBA BA
VA Boston Healthcare System Jamaica Plain Campus, Jamaica Plain, MA
Boston, MA
Funding Period: July 2008 - September 2011

As part of its commitment to provide the highest quality to those in VA Community Living Centers (CLCs), the VA Office of Geriatrics and Extended Care (GEC) monitors and provides performance data to CLCs on 28 quality indicators (QIs) calculated from the Minimum Data Set (MDS). In addition, GEC launched a major effort to implement Patient Centered Care (PCC) practices, which it monitors using the Artifacts of Culture Change Tool.

1) To develop composite measures of quality from the 28 MDS-based QIs and use these to identify high and low performing facilities; 2) To analyze the relationship between quality and job satisfaction, workgroup functioning and organizational culture as assessed by the All-Employee Survey (AES); 3) To analyze the relationship between quality and the extent of implementation of PCC practices; and 4) To analyze the relationship between quality and CLC costs.

We selected 112 of the 132 CLCs in which, in FY07, at least one-third of their residents were long-stay (over 90 days) and there were at least 10 long-stay residents. We primarily focused on FY05 to FY07 MDS data, although we also used data from FY03-FY07 to develop risk-adjustment models and data from FY08 to analyze PCC implementation.

We considered two sets of QIs: 1) The MDS-based 28 QIs, which were our main focus; and 2) 5 incidence-based risk-adjusted QIs calculated from risk-adjustment models developed from MDS variables. We used several approaches to calculate composite measures. First, we used opportunity-based weights (OBWs) to combine observed rates for each indicator into a composite measure. OBWs, the approach used by the Centers for Medicare and Medicaid Services (CMS) in its pay-for-performance programs, gives equal weight to each type of QI event measured by the QIs. Second, we used a Bayesian multivariate normal model to calculate "shrunken" rates for each of the QIs, which were then combined using OBWs. Shrunken rates adjust for differences across QIs and facilities in the reliability of observed rates due to differences in sample sizes. Third, we used two Benefit-of-the-Doubt (BOD) approaches, simple linear programming models (SLP) and Data Envelopment Analysis (DEA). These approaches, which are based on the assumption that a facility's performance on individual QIs indicates its revealed preferences about the relative importance of the indicator, adjust weights within specified constraints to optimize performance. We profiled facilities using these different approaches, concentrating on those facilities identified as being in the top and bottom quintiles. Using the shrunken rate composite (for which interval estimates are available), we were able to identify a set of high performing facilities that were statistically different than a set of underperforming facilities (significance was based on the absence of overlap between their respective 95% Bayesian credible intervals).

GEC uses the Artifacts Tool (developed by CMS) to measure implementation of PCC. Although the individual items that make up this instrument have face validity, scoring of the Artifacts Tool has not been validated. We considered 3 scoring methods: 1) the current method; 2) a method that sums standardized domain scores; and 3) a method that assigns optimal weights to domains to maximize the PCC/quality relationship. We examined the relationship between MDS-based quality and PCC using both multiple regression models and latent variable models. To analyze the quality/CLC cost relationship, we merged data from the VA Decision Support System with MDS data. We examined several generalized estimating equations (GEE) models with dependent variable total facility costs (for FY05, FY06 and FY07) and the following independent variables (by year): total patient days, RUGs score (a case-mix severity measure), year and the shrunken composite measure of quality.

1) In FY2007, CLC composite scores, which reflect the chance of any QI event, ranged from 0.066 to 0.190 (0.068 to 0.184 using shrunken rates). Facilities in the top and bottom quintile were quite similar (agreement on 19 or more of 22 facilities) using the observed-rate composite, the shrunken-rate composite and the SLP BOD composite. Based on the shrunken rate composite, there were 27 high performing CLCs that were statistically distinguishable from 28 underperforming facilities.
2) Prediction errors when using shrunken rates were generally (though not exclusively) lower than when using observed rates.
3) Using each of the Artifacts scoring methods, the coefficient linking PCC implementation to quality was statistically significant. When the Artifacts Total Score was calculated from standardized domain scores, CLCs one standard deviation (SD) below the mean had a prevalence of 8.2/1000 more QI events than CLCs one SD above the mean. Using the latent variable model and standardized domain scores, there were 13 high performing facilities in terms of PCC implementation that were clearly distinguishable from 18 underperforming facilities.
4) In the GEE model, the coefficient linking quality to cost was 1.09 (95% CI 0.174 to 2.011). A facility with a quality score one SD below the mean (high quality) had about 5% lower costs than a facility with a score one SD above the mean.
5) Five incidence-based risk-adjustment models were developed (validated c statistics in parentheses): pressure ulcers (0.75), functional decline (0.67), behavioral decline (0.69), 6-month mortality (0.75) and 6-month preventable hospitalization (0.75).
6) We found no consistent relationship between any of the composite measures and any of the AES scales or individual questions we examined.

VA is currently implementing PCC in CLCs. By demonstrating that more extensive implementation of PCC practices is associated with better quality and that better quality is associated with lower costs, our findings provide empirical support for the VA PCC movement. Over the longer term, by identifying a set of high performing facilities in terms of quality and PCC implementation, we provide a basis for learning and sharing best practices; by identifying a set of underperforming facilities, we highlight areas for targeting interventions in order to improve quality of care.


Journal Articles

  1. Shwartz M, Peköz EA, Burgess JF, Christiansen CL, Rosen AK, Berlowitz D. A probability metric for identifying high-performing facilities: an application for pay-for-performance programs. Medical care. 2014 Dec 1; 52(12):1030-6.
  2. Sullivan JL, Shwartz M, Burgess JF, Peköz EA, Christiansen CL, Gerena-Melia M, Berlowitz D. Person-centered care practices and quality in Department of Veterans Affairs nursing homes: is there a relationship? Medical care. 2013 Feb 1; 51(2):165-71.
  3. Setodji CM, Shwartz M. Fixed-effect or random-effect models: what are the key inference issues? Medical care. 2013 Jan 1; 51(1):25-7.
  4. Shwartz M, Cohen AB, Restuccia JD, Ren ZJ, Labonte A, Theokary C, Kang R, Horwitt J. How well can we identify the high-performing hospital? Medical care research and review : MCRR. 2011 Jun 1; 68(3):290-310.
  5. Lischko AM, Burgess JF. Knowledge of cost sharing and decisions to seek care. The American journal of managed care. 2010 Apr 1; 16(4):298-304.
  6. Shwartz M, Burgess JF, Berlowitz DR. Benefit-of-the-doubt approaches for calculating a composite measure of quality. Health Services and Outcomes Research Methodology. 2010 Feb 26; 9(4):234-251.
  7. In H, Pearce EN, Wong AK, Burgess JF, McAneny DB, Rosen JE. Treatment options for Graves disease: a cost-effectiveness analysis. Journal of the American College of Surgeons. 2009 Aug 1; 209(2):170-179.e1-2.
  8. Ryan AM, Burgess JF, Tompkins CP, Wallack SS. The relationship between Medicare's process of care quality measures and mortality. Inquiry : A Journal of Medical Care Organization, Provision and Financing. 2009 Jan 1; 46(3):274-90.
  9. Shwartz M, Ren J, Peköz EA, Wang X, Cohen AB, Restuccia JD. Estimating a composite measure of hospital quality from the Hospital Compare database: differences when using a Bayesian hierarchical latent variable model versus denominator-based weights. Medical care. 2008 Aug 1; 46(8):778-85.
Conference Presentations

  1. Sullivan JL, Adjognon Y, Engle RL, Shin M, Afable MK, Rudin WM, White RA, VanDeusen-Lukas C. Factors affecting implementation of a patient-centered non-institutional long term care grant program in the VA. Paper presented at: VA HSR&D / QUERI National Meeting; 2015 Jul 9; Philadelphia, PA.
  2. Sullivan JL, Shwartz M, Burgess JF. Examining the Relationship between a Minimum Data Set Composite Quality Indicator and the Artifacts of Culture Change Tool. Poster session presented at: Gerontological Society of America Annual Scientific Meeting; 2011 Nov 20; Boston, MA.
  3. Burgess JF, Shwartz M, Stolzman KL. Analyzing the Relationship Between Cost and Quality Using a Bayesian Shrinkage Composite Measure of Quality. Paper presented at: Decision Sciences Institute Annual Meeting; 2011 Nov 19; Boston, MA.
  4. Shwartz M, Sullivan JL, Burgess JF, Pekoz E, Christiansen C. Patient-Centered Care Practices and Quality in Department of Veterans Affairs Nursing Homes: Is There a Relationship? Paper presented at: Decision Sciences Institute Annual Meeting; 2011 Nov 19; Boston, MA.
  5. Loveland S, O'Brien W, Shwartz M, Borzecki A, Shin M, Cevasco M, Hanchate AD, Rosen AK. Socio-demographic and Clinical Characteristics of Early vs. Late Readmissions in the VA: A Study. Presented at: AcademyHealth Annual Research Meeting; 2011 Jun 13; Seattle, WA.
  6. Holmes SK, Cohen AB, Restuccia JB, Horwitt J, Shwartz M. Strategies for Improving Hospital Quality and Efficiency: Lessons from Eight Hospitals. Paper presented at: AcademyHealth Annual Research Meeting; 2010 Jun 28; Boston, MA.
  7. Shwartz M, Burgess JF, Berlowitz D. Benefit-Of-The-Doubt Approaches for Calculating a Composite Measure of Quality. Poster session presented at: AcademyHealth Annual Research Meeting; 2010 Jun 28; Boston, MA.
  8. Shwartz M, Cohen AB, Restuccia JB, Ren Z, Labonte A, Theokary C, Kang R, Horwitt J. How Well Can We Identify the High-Performing Hospital? Poster session presented at: AcademyHealth Annual Research Meeting; 2010 Jun 28; Boston, MA.
  9. Sullivan JL, Shwartz M, Gerena-Melia M, Berlowitz D, Burgess JF. Examining the relationship between minimum data set quality indicators and artifacts of culture change tool. Poster session presented at: VA Geriatric and Extended Care Leadership Conference; 2010 Jun 9; Rochester, NY.
  10. Shwartz M, Burgess JF, Berlowitz D, Sullivan JL, Gerena-Melia M, Pekoz E, Christiansen C, Kader B, Stolzman KL. Composite Measures of Quality in VA Community Living Centers. Paper presented at: VA HSR&D Field-Based Long-Term Care Meeting; 2010 Jun 6; Rochester, NY.

DRA: Aging, Older Veterans' Health and Care, Health Systems
DRE: Epidemiology
Keywords: Management, Organizational issues, Quality assessment
MeSH Terms: none

Questions about the HSR&D website? Email the Web Team.

Any health information on this website is strictly for informational purposes and is not intended as medical advice. It should not be used to diagnose or treat any condition.