BACKGROUND/RATIONALE:
The project sought to develop an assessment tool that could be used by programs to evaluate their level of awareness and quality in addressing PTSD/SUD. The assessment, originally titled VA PTSD/SUD Program Assessment (VPPA), was modeled on existing trauma-informed assessment tools that are designed to help gauge programs' level of trauma awareness. The measure was renamed the Best Practices for Trauma and Addiction-Program Assessment to be more user-friendly. The measure is being sent separately by email to the program officer. We retain use of the abbreviation "VPPA" below to be consistent with the original grant.

OBJECTIVE(S):
There is no existing tool thus far that focuses on PTSD/SUD comorbidity or on VA programs in particular. Such an assessment can provide a mechanism to build self-awareness among programs and their staff to increase sensitivity and knowledge of PTSD/SUD treatment. This project had two aims: (1) To develop a VA PTSD/SUD Program Assessment (VPPA) based on expert input, as well as program administrator and clinician input, using a mixed-methods development process that emphasized both qualitative and quantitative methods. (2) To conduct a pilot evaluation of the VPPA at two VA sites (Boston and Bedford), in a sample totaling 40 clinicians and 5 program administrators, and 12 Veterans. The pilot included ratings of content, satisfaction, feasibility, attitudes toward PTSD/SUD, and knowledge of PTSD/SUD treatment principles.


METHODS:
AIM (1) To develop a VA PTSD/SUD Program Assessment (VPPA) based on expert input, as well as program administrator and clinician input, using a mixed-methods development process that emphasized both qualitative and quantitative methods.

This aim was achieved fully. The study PI collected expert input from study team members Drs. Najavits, Ellison, and Carlson. Feedback was elicited from members of the SUD/QUERI workgroup and SUD/QUERI leadership team, and posted to the SAMHSA trauma listserve for feedback from the field. The PI also researched comparable measures for program assessment in the trauma field and SUD field. Further, the measure development was informed by the PI's in-person 1-hour interviews with 11 veterans who had been enrolled at some point in VA treatment for PTSD, SUD or both. The interview was a qualitative open-ended set of questions focused on their experiences as patients, with questions such as the following:
-How helpful was your treatment in VA for your PTSD, your SUD?
-Were you given attention to both issues, if you had both?
-Was there any difference in staff attitudes toward your PTSD, your SUD?
-How can the VA improve its approach to treatment in VA for veterans who have PTSD, SUD?
-What were you most / least satisfied with regarding your PTSD, SUD treatment in VA?
-What would you like staff to know about what it's like to be a veteran seeking treatment for PTSD, SUD in VA?
-Were you asked/assessed about problem areas that often co-occur with PTSD, SUD in veterans, such as: chronic pain, domestic violence, head injury (TBI), need for housing?

The final measure (Najavits, 2014) was developed as a result from the above efforts. Some items and some formatting were drawn from "Trauma-Informed Care for Women Veterans Experiencing Homelessness: A Guide for Service Providers" at http://www.dol.gov/wb/trauma/, which is a public domain measure. The final VPPA measure is a 93-item self-report scale in which the clinicians rate their level of agreement regarding program staff knowledge on seven domains related to PTSD/SUD awareness. Responses were measured with a Likert Scale from 0 (Strongly Disagree [this rarely or never happens]) to 3 (Strongly Agree [this happens most of the time]) with the options to respond "Do not know" and "Not applicable to my program". The measure is structured consists of two parts: (a) Staff Awareness, which refers to respondents' ratings of knowledge about various topics related to PTSD, SUD and veterans. (b) Program Elements, which refers to the respondents' ratings of how successfully the program carries out treatment strategies routinely for all patients with PTSD and SUD. Each section also had subsections per below.
Part 1 Staff Awareness
(a) PTSD (16 items)
(b ) Addiction (16 items)
(c) Military/veterans (12 items)
Part 2 Program Elements
(a) Assessment (15 items)
(b) General treatment strategies (13 items)
(c) Evidence Based treatments (9 items)
(d) Policies (12 items).

AIM (2) To conduct a pilot evaluation of the VPPA at two VA sites (Boston and Bedford), in a sample totaling 40 clinicians and 5 program administrators, and 12 Veterans. The pilot included ratings of content, satisfaction, feasibility, attitudes toward PTSD/SUD, and knowledge of PTSD/SUD treatment principles.

This aim was mostly achieved (our sample was slightly smaller). Data collected for this study occurred at the Veterans Administration Boston Healthcare System (VA Boston) and Edith Nourse Rogers Memorial Veterans Hospital in Bedford, MA (VA Bedford). The clinicians were recruited through emailing VA program listservs, publicized at employee meetings, and posting an announcement on VA computer screensavers. The only inclusion criterion for the clinician sample was that the clinician or program administrator worked with PTSD/SUD clients. The only inclusion criterion for patient sample were that the veterans had at some point been enrolled in a treatment for PTSD or SUD in VA. The study target sample was 40 clinicians, 5 program administrators, and 12 veteran patients, but due to several challenges in recruitment and delays, the total study sample was 26 clinicians, 2 program administrators, and 11 veterans. Hereafter in this report the term "clinicians" is used for the n=28 staff (26 clinicians and 2 program administrators) as the two program administrators were also clinicians who were currently treating veterans with both PTSD/SUD. Due to small sample sizes, we did not analyze the program administrators separately from the clinicians.
The study encountered several challenges in staff recruitment due to the measures having to be completed onsite yet to be compensated for their time they needed to do the measures outside of their tour of duty hours. These parameters limited the availability of clinicians and program administrators for study participation. Also, subject funds at the Bedford site were on hold from May-July 2014 due to the site's fiscal office issuing a determination that funds could not be used for non-veteran participants. The fiscal office later retracted this decision after consulting with the site's research compliance officer and funds were available from July-mid September. This delay resulted in study staff having to recruit beyond the original deadline of finishing recruitment by June. Also, during this time, there was a reduction in work hours for some of our research staff.

The following measures were used:
(a) Background Questionnaire (based on Najavits, 1996), which obtained sociodemographic information from the clinicians and program administrators and also included professional characteristics (e.g., setting, years experience, degree).

(b) Alcoholism and Attitudes of Mental Health Professionals (Schwartz & Taylor, 1989), a 23-item questionnaire developed in VA to evaluate clinicians' views on substance abuse treatment, such as whether SUD is a disease, whether all clients must attend 12-step groups, etc.

(c) Attitudes toward PTSD patients (Bras et al., 2012). This questionnaire addressed clinicians' attitudes toward PTSD patients. This measure by Bras et al., 2012 had the advantages of being short (14 items), psychometrically studied, designed for military-related PTSD, and designed to be completed by clinicians. We slightly changed the wording of item 9 to "I'm able to tolerate" rather than "I'm able to live through" (the latter is from the Croatian language of the measure and is a mistranslation into English). Also the scaling was changed from 'agree/disagree' to scaling as follows, to allow more variability in responses: 0=not at all, 1=a little, 2=a lot).

(d) VA PTSD/SUD Program Assessment (VPPA). The final measure is described earlier under Aim 1.

(e) Feedback on the VA PTSD/SUD Program Assessment. This 7 item measure briefly queried for views of the new VPPA. It consisted of 3 items that clinicians rated on a 0-2 scale (0=not at all, 1= a little, and 2= a lot): the usefulness of the measure, whether it increases awareness of the needs of PTSD/SUD patients, and how relevant the VPPA measure is to VA care for PTSD/SUD patients. The remaining 4 items were open responses for clinicians to provide feedback on the measure and length of time it took to complete the measure.

The first three measures were given before the participant was exposed to the VPPA. The VPPA was then provided and the respondent was asked to sit and complete it with our project manager present (so as to answer questions, make sure the respondent is actually reading it, and to obtain any qualitative comments that arise). After the VPPA, "b" and "c" were re-administered a week later to see whether there may be any change in attitudes or knowledge toward SUD. A week time lapse was used both for practical purposes (it would have been too much of a burden to complete the same measures twice at one sitting) and also because we expected that change in attitudes/knowledge based on the VPPA needed time to develop a bit (e.g., the clinician would think about issues raised by the VPPA during that week). The measures a, b, c was administered to clinicians via the Inquisite system, which is behind the VA firewall (thus secure) and allowed for direct data entry by respondents. Clinicians were sent links of the measures or completed them at our VA office location.

Data Analysis
Descriptive statistics addressed satisfaction ratings of the VPPA; basic findings on the total and subsections of the VPPA; and characteristics of the sample (program type, years experience, etc.). Inferential statistics were paired-samples t-tests and chi-squares to determine whether changes from Time 1 to 2 in clinician's attitudes or knowledge related to PTSD/SUD. One-tailed t tests were used as the hypothesis was that after being exposed to the VPPA, clinicians' knowledge and attitudes would improve, and therefore the data was tested in one direction.
Per the original grant proposal, a larger sample size would be needed to assess psychometric characteristics of the measure in a statistically valid way (e.g., internal consistency, interrater reliability, item discrimination, and latent variable identification via factor analysis).

FINDINGS/RESULTS:
RESULTS
Veteran Sample
Our Veteran sample consisted of 11 patients with 72.7% (n=8) from the Boston VA and 27.3% (n=3) from the Bedford VA. The Veteran sample was 45.5% (n= 5) female and 54.5% male (n=6). Slightly over half of the Veteran sample was Caucasian (n=6) and close to half African American (n=5).

Clinician Sample
The clinician sample was 28 clinicians from VA Boston (42.86%, n = 12) and Bedford (57.14%, n=16). They were selected for having had experience working with PTSD/SUD clients.

From the Background Questionnaire, the clinician sample was 82.1% (n=23) female and 17.9% (n=5), with a mean age 39.32 years (sd= 13.63) . Most clinicians (71.4%, n=20) were Caucasian, with 3.6% (n=1) African American, 14.3% (n=4) Asian, 3.6% (n=1) Hispanic, and 7.1% (n=2) multiracial. Overall, the clinicians had a mean of 8 years clinical experience (sd= 10.16), and training consisted of 64.3% (n=18) bachelor's degree, 64.3% (n=18) master's degree, 39.3% (n=18) PhD or equivalent, 3.6% (n=1) MD. 3.6% (n=1) of the sample had one of the following degree or certification: Associate Degree in Nursing, Certified Peer Specialist, clinical respecialization PhD, and Nurse.

The clinicians treated PTSD/SUD clients in the following VA treatment settings (with some endorsing more than one): most (82.1%, n=23) were at the VA outpatient clinic, with (7.1%, n=2) CBOC outpatient clinic (10.7%, n=3) Inpatient, Detox (25%, n=7) Residential, (14.3%, n=4) Day Treatment, and (17.9%, n=5) "Other" (described as Emergency Room Psych, Home Visits, Internship, College Campus, or Private Practice.)

Clinicians reported an average 19.75 hours (sd= 10.16) directly treating patients and 17.93 hours (sd= 8.75) non-patient contact (e.g. teaching, supervising, research). On a scale from 0-100, clinicians rated 70.18 overall effectiveness as a therapist (sd= 25.48), 65.54 effectiveness as a therapist with PTSD patients (sd= 27.09) , and 58.32 effectiveness as a therapist with PTSD/substance abuse patients (sd= 27.61) .

Clinicians reported they used 2.82 treatment manuals (sd= 2.14) in their career thus far. On the scale of 0 (never) to 10 (always) they reported a mean score 3.93 (sd= 2.11) for the question, "How much of your clinical practice do use manual-based treatment." Also, clinicians were asked to provide a percentage of which therapy models were used in their clinical practice, with results as follows: 12.32% Psychoanalytic/psychodynamic (sd= 13.44), 3.86% 12-Step (AA,NA,CA) (sd= 9.93), 36.25% Cognitive-Behavioral (sd= 25.12), 13.64% Humanistic (sd=13.94), 3.32% Psychopharmacologic (sd= 9.62), 5.75% Family Systems (sd= 10.76), 3.93% No Model (sd=15.24), and 18.21% "Other" (sd=27.33) which was described as interpersonal, client-centered, peer support, transpersonal, integrative orientation.

Overall, we were able to recruit a diverse sample. We had an almost 30% rate of minority clinicians and 50% veterans. The clinicians were primarily outpatient and cognitive-behavioral, although there was also representation across all levels of care at VA and many different theoretical orientations.

A. VPPA Results

The VPPA measure is described in Aim 1 above. It was scored from 0 (Strongly Disagree/This rarely or never happens) to 3 (Strongly Agree/This happens most of the time). For data analysis purposes, "Do Not Know" and "Not Applicable to my program" were labeled as data missing and were omitted from the measure's section scores. One participant partially completed the measure due to a technical difficulty with the online assessment platform.

1- Mean scores for total and subsections of the VPPA

The mean score on the measure 2.19 (sd=.33, n=28). Higher scores indicate better results, e.g., stronger knowledge, program succss, etc. The range is 0-3 for all variables.
Part 1 Staff Awareness
(a) PTSD (16 items) mean = 2.18 (sd=.49, n=28);
(b ) Addiction (16 items) mean = 2.18 (sd=.46, n=28);
(c) Military/veterans (12 items) mean = 2.31 (sd=.36, n=27);
Part 2 Program Elements
(a) Assessment (15 items) mean = 2.24 (sd= .41, n=26);
(b) General treatment strategies (13 items) mean = 2.31 (sd = .36, n = 27);
(c) Evidence Based treatments (9 items) mean = 2.14 (sd = .43, n = 27);
(d) Policies (12 items) mean = 2.06 (sd = .43, n = 27);

The clinician sample reported moderate to high score across all sections of the measure, with relatively low standard deviations. As higher scores represent more positive ratings, the overall finding was that clinicians viewed


2- Satisfaction with the VPPA

Feedback on the VA PTSD/SUD Program Assessment: for this measure, higher scores indicate better results on the 0-2 scale. This measure addressed the usefulness of the measure, its ability to increase awareness of the needs of PTSD/SUD patients, and how relevant the VPPA measure is to VA care for PTSD/SUD patients.

Clinicians rated the measure .59 (sd=.75) for usefulness overall; .85 (sd=.72) for whether the VPPA helps increase awareness of the needs of PTSD/SUD patients; and .96 (sd=.71) relevance to VA care for PTSD/SUD patients. As indicated by each the standard deviations, there was variability on the clinician ratings. Overall, these ratings indicate low satisfaction with the measure, although the feedback measure was not psychometrically validated so findings may also reflect measure problems.

The open response questions (qualitative data) also provided information. Positive comments indicated that clinicians appreciated that the VPPA measure highlighted important issues related to PTSD/SUD treatment and awareness of policies within a program. Specifically, "It brought up issues that may be relevant to SUD and PTSD". "I felt I learned more about potential questions to ask during screening/assessment. I also realized that there are some things I don't know about the program, policies in particular, that I should definitely know early on in my employment." "I like that it provides details on various factors that I should consider, as a VA provider, with regards to the assessment and treatment of PTSD/SUD."
There were also several areas for improvement identified by the clinicians: for the assessment questions to be more inclusive of other co-morbid disorders and clinicians' professional experience in their program. Several clinicians commented "...the answer choices on certain pages (i.e., not at all, a little, or a lot) felt a bit restrictive when it came to certain questions"; "disliked that there was no option for indicating the difference between what I do as a provider, and what my larger program tends to do"; "perhaps additional items to assess for impact of co-morbid conditions". Other clinicians said the length of the measure could be shortened but did not provide further detail which questions should be omitted.

B. Over-time changes

Changes between Time 1 and Time 2 were measured with one-tailed paired samples t- tests and chi-squares.

1- Attitudes toward PTSD patients (Bras et al., 2012): higher scores indicate better results on the 0-2 scale. There was no significant difference between Time 1 (M=20.3, SD=6.4) and Time 2 (M=18.9, SD=2.4); t (1.00), p=.16.

2- Alcoholism Attitudes of Mental Health Professionals(Schwartz & Taylor, 1989)
Paired- samples t-tests for continuous data and chi-square tests for categorical data were conducted on each of the 23 questions in this scale to evaluate possible changes in attitudes about alcoholism by clinicians. The measure has no subscales or total, so each item was analyzed separately. Higher scores indicated better results for all continuous variables, which were scaled from 1 (Never) -5 (Always). Categorical data were yes/no responses measuring clinician understanding of SUD related concepts as enabling, AA book, and relationship between SUD and other mental health disorders .

Of the 23 analyses, 22 were not significant, and 1 was a trend. The trend item was: Question#14 To what extent do you feel most patients can be helped to stop drinking? [Time 1 (M=3.6, SD=.50) and Time 2 (M=3.8, SD=.7); t (-1.3), p=.10)]. However, given the large number of tests run and the merely trend nature of the one result, we are not interpreting this finding.


Overall, there was no difference between Time 1 and Time 2. Possible explanations are as follows: 1. The sample had many years of clinical experience (M=8, sd = 10.16) and has high ratings at the start. In more novice clinicians, the VPPA might be more useful. We did not have adequate sample size to evaluate the latter question, however, as the study was not powered to evaluate subsample differences. 2. The lack of adequate statistical power (n=26) from Time 1 to Time 2 may help explain the lack of any difference between these two timepoints. 3. The measures used to evaluate Time 1 versus Time 2 may not have been sensitive enough. 4. Last but not least, the VPPA itself may not actually impact attitudes toward PTSD and/or SUD. The VPPA may serve primarily as an exploratory tool to highlight areas in need of more training, and it well may be the case that in-depth training is needed to change attitudes for such complex disorders as PTSD and SUD.

IMPACT:
Conclusion Notes
Both study aims were met. The VPPA measure was developed and we were able to recruit our target for Veteran participants (n=12) and most of the clinician sample (26 out of 40 clinicians), and 2 out of 5 program administrators. The sample was diverse in ethnicity, gender, and professional characteristics.
Main study findings were that: (a) ratings on the VPPA were moderate to high, indicating the clinicians viewed their own knowledge and their program's staff knowledge as moderate-to-high in terms of addressing the needs of veterans with PTSD and SUD. (b) The VPPA was seen as positive by some clinicians, but overall the feedback ratings were not as strong as expected. The measure may be too long in its current form, for example. (c) There were no differences in attitudes toward PTSD or SUD between Time 1 and 2. This may indicate methodology issues (e.g., low statistical power, measures that were not sensitive) and/or that the VPPA may not actually impact attitudes toward PTSD and/or SUD. The VPPA may serve primarily as an exploratory tool to highlight areas in need of more training, and it well may be the case that in-depth training is needed to change attitudes for such complex disorders as PTSD and SUD.

The VPPA measure has been sent by email to the program officer for this project.

---

External Links for this Project

---

PUBLICATIONS:

None at this time.

DRA: Mental, Cognitive and Behavioral Disorders
DRE: none
Keywords: none
MeSH Terms: none