Community Ment Health J DOI 10.1007/s10597-014-9714-y
BRIEF REPORT
Measuring Wraparound Fidelity to Make Quality Improvements Joan B. Kernan
Received: 22 January 2013 / Accepted: 16 February 2014 Springer Science+Business Media New York 2014
Abstract Wraparound care coordination has offered an individualized family approach to providing services to families in our county for the past decade. Instigated by our county mental health department interested in providing quality improvements to mental health services for families, we assessed the wraparound care coordination process through use of the Wraparound Fidelity Index (WFI), which measures adherence to established principles of wraparound. This article reports on findings from WFI data collected over three summers and reported back to the management team and family organization. Improvements were decided through consensus and included recommendations of parents and care coordinators. The goal of our study was to identify which activities and supports of wraparound were in need of improvement, enact quality improvements to address these areas and determine whether we were doing a better job over time. Keywords Wraparound process Treatment fidelity Care coordination Children’s mental health Wraparound Fidelity Index
Introduction Our community has invested a great deal to provide individualized services to children, youth and families in need of mental health services through the wraparound care coordination process. Wraparound is a team-based approach to providing for the needs of a child and family with serious emotional challenges. Guided by a philosophy which J. B. Kernan (&) Department of Family Medicine, University at Buffalo, 77 Goodell St., Rm. 220C, Buffalo, NY 14203, USA e-mail:
[email protected]
includes a set of principles and administered through a ‘child and family team’ (CFT), wraparound is designed to provide for the individualized needs of the child and family (Burns and Goldman 1998). As part of the care management process, community services and natural supports are developed with the family to help achieve a positive set of outcomes (Walker and Schutte 2004). The wraparound facilitator or care coordinator makes sure the process happens by ensuring that the team develops and implements a plan of care which addresses the needs of the family. This individualized approach is attractive to families and providers and has increasingly been implemented throughout the US (Bruns et al. 2010, 2011). Wraparound implementation, because it is a process based on general principles, can vary widely and be difficult to monitor its’ effectiveness (Walker et al. 2003). The National Wraparound Initiative (NWI) was begun by national experts in human services in response to the increased use of wraparound; the NWI has helped define standards and compile research and strategies for conducting high-quality wraparound. As identified by the NWI, the process of wraparound follows ten principles: family voice and choice, team based, natural supports, collaboration, community-based, culturally competent, individualized, strengths based, persistence, and outcomes based elements. The activities of wraparound can be grouped into four phases including engagement, plan development, plan implementation and transition (Walker and Bruns 2012). The ‘Wraparound Evaluation and Research Team’ (WERT), is a part of this initiative to develop and perfect a tool to measure adherence to the principles of wraparound (Walker and Bruns 2013). This tool, the Wraparound Fidelity Index 4.0 (WFI), aims to assess the wraparound process from the perspectives of the wraparound facilitator (care coordinator), parent/caregiver, youth, and team member resulting in a fidelity score on each scale and an
123
Community Ment Health J
overall fidelity score. The WFI has been shown to associate high fidelity scores to an improvement in the child or youth’s needs and to positive child and family outcomes (Bruns et al. 2005, 2008; Effland et al. 2011). Yet there is a lack of research on the use of the WFI or any other tool to measure program improvement over time. Our study will be helpful to other communities interested in assessing the quality of wraparound and care planning for children with complex needs and their families. The Current Study Wraparound care coordination has offered an individualized family approach to providing services to families in Erie County for the past decade and our families have been supportive of this approach as an alternative to traditional case management services. Erie County, in Western New York State, has a population of about 950,000 including the city of Buffalo, a rust belt, low income, mid-size city. Starting with a pilot program in 2000 and serving about 40 families a year, the County’s Department of Mental Health provided limited flexible wraparound services to children and families at high risk for out-of-home placement. By 2004, the county had received a Substance Abuse and Mental Health Services Administration (SAMHSA) System of Care grant award of 9 million dollars to implement a wraparound service delivery model within and across the child-serving systems of care for children and families. Now nine years later, our county is serving nearly 450 families a year utilizing the wraparound service delivery model. The county had invested both money and manpower in contracting with social services agencies to provide wraparound care coordination services to youth and families with serious emotional challenges (Kernan et al. 2009). Determining whether the process of wraparound was occurring, which has been referred to as measuring fidelity was part of the strategic plan for the County’s contracts with the agencies providing services (Bruns et al. 2004). In addition, the family organization, a not-for-profit begun with funding from the SAMHSA award, had begun participating in the quality improvement process and made recommendations for service delivery changes and enhancements. The goal of our study was to identify which activities and supports of wraparound were in need of improvement and enact quality improvements to address these areas.
Methods Wraparound Fidelity Index The WFI 4.0 measures adherence to the principles and primary activities of the wraparound process (Bruns et al.
123
2004). It was chosen for use because of its reliability and validity and support from the WERT which developed the WFI (Bruns et al. 2004, 2005). WERT provided us with all of the WFI forms, a user’s manual, information on training interviewers and managing data, data entry files and a power-point slide with notes for use in training interviewers (Walker and Bruns 2013). And most importantly, we could compare our WFI scores to national norms allowing us to benchmark our progress at implementing high quality wraparound. We were able to receive training materials and the survey at minimal cost. The WFI is completed through brief, 20–40 min confidential telephone interviews with four types of respondents: (1) parents or caregivers, (2) youths (11 years of age or older), (3) wraparound facilitators (also referred to as care coordinators) and (4) team members. The interview items focus on the wraparound process and services received by an identified child or youth and his/her family. Each type of respondent is asked to respond from their point of view. We utilized three respondent forms: the caregiver (CG) form , the youth (Y) form, and the wraparound facilitator (WF) form. The WFI is designed to assess the extent to which both the principles and activities are being implemented in service delivery. The interviewer rates agreement with the WFI items by assigning a score of 0 (low fidelity), 1 (moderate) or 2 (high fidelity) to each of 40 items on the CG and WFs and to 32 items on the Y form. Items are based on the ten principles of wraparound and organized according to the four phases of the wraparound process. Subscale scores are calculated for each phase of wraparound for each respondent type and a total WFI score by respondent type. The scores are converted to percentages, 0–100 %, for easy review by families, staff, and supervisors.
Procedures Three adult interviewers, chosen because of their personal experience with the wraparound process, were given a 1 day training based on a training protocol provided by the WERT team to ensure ratings were consistent. Training included background on the wraparound process itself, how to introduce the interview to parent/CGs, practice interviews with the evaluator, and feedback after the interviewers’ were monitored during their first ‘live’ interview with a family member. Because of the low response rate for youth in 2007 and 2008, in 2009 we trained a young adult, age 19 and with personal experience receiving mental health services, to conduct youth interviews and we offered youth a $10 gift card to participate. The interview is conversational in nature and the interviewer is trained to probe for information from the respondent. For example, the CG
Community Ment Health J
is asked ‘Does your wraparound team get your child involved with activities she or he likes and does well? Please give two examples of those activities.’ Respondents may need probing by the interviewer to obtain an accurate assessment of the item. Interviewers did not know any of the families or care coordinators they interviewed (Walker and Bruns 2013). We conducted a convenience sample of families currently receiving wraparound services for at least 3 months during the summers of 2007, 2008, and 2009. By 3 months in services CFTs should be well established, the plan of care has been developed and implemented and transition to community-based services has been discussed by the team. Since our community expects wraparound to be time-limited with most families transitioning after 12 months in services, we are very concerned that the process of wraparound begin immediately to address the needs of the youth and family. Six agencies provided wraparound care coordination services and we aimed to interview at least 20 % of each agency’s caseload. A county database of demographic information on each family receiving services was used to create listings of families enrolled at least 3 months. Each interviewer was given a list of family names, phone numbers, and wraparound facilitators to call. Of 350–400 families being served at any one time approximately 250–300 have been enrolled in services for at least 3 months and would be eligible for interviews. Interviews were conducted over a span of 3–4 months allowing new families to be added to listings as they reached enrollment of 3 months. The number of families and/or youth that refused to participate was not documented. As soon as 20 % of the caseload was reached we stopped calling that agency’s families. We planned to interview a total of at least 90 families. Calls were made to the parent/CG and if they consented, the interview was completed immediately over the phone or at a time convenient for them. If the child receiving services was age 11 or older, permission was obtained from the parent/CG to interview the youth. If the family had more than one youth receiving services only one child or youth was the ‘identified’ youth for purposes of the interview questions. The wraparound facilitator (our community uses the term care coordinator) assigned to the family was only called if the parent/CG agreed to an interview. Consent of CG was obtained over the phone as well as permission to interview the youth. Due to budget constraints wraparound facilitators were not interviewed in 2009. IRB approval was obtained from the University at Buffalo Social and Behavioral IRB board. We conducted the WFI during the summers of 2007, 2008, and 2009 and reported our findings back to the management team and to the family organization each fall following each summer’s data collection.
Results We conducted 105 parent/CG and care coordinator interviews in 2007, 107 parent/CG and care coordinator interviews in 2008, and 91 parent/CG interviews in 2009. Thirty-three youth were interviewed in 2007, 22 in 2008 and 56 in 2009. We attribute the increase in youth interviews in 2009 to the youth interviewer we purposely hired to increase the number of youth participants. The length of time the youth was enrolled in services at the time of the interview ranged from 3 to 24 months in 2007 (M = 6.6, SD = 4.5, mode = 4), from 3 to 24 months in 2008 (M = 7.7, SD = 3.1, mode = 6), and 3 to 13 months for 2009 (M = 6.3, SD = 2.8, mode = 4). The race/ethnicity of the youth in services generally matched the population served (about 25 % African American/Black, 58 % White, 10 % Hispanic and 7 % other. The average age of youth who interviewed was 14.5 and 64 % were male, 36 % female. In tests of reliability, Cronbach’s alphas of the 40 care coordinator items were .67 and .76 for the 2007 and 2008 data, respectively, shows moderate consistency across items. The 40 parent/CG items were .84, .90 and .89 for 2007–2009 data respectively and the 40 youth items were .83, .90 and .84 for 2007–2009 respectively, showing strong consistency across items. These data suggest that for parent/CGs and youth especially, the responses reflect dependable perceptions of the care coordination process and activities. Intraclass correlation coefficients for all three respondents for total score data from 2007 was .43, for 2008 .82 and for the CG and youth from 2009 data .43 indicating fair to good agreement across the different respondents. WFI mean phase scores and total WFI scores by respondent type were calculated for each data collection period: 2007, 2008, and 2009 and by agency. The magnitude of change between 2007 and 2009 for parent/CGs and youth and between 2007 and 2008 for care coordinators was determined using independent samples t tests. For purposes of comparison to national benchmarks, high fidelity scores or those that are closest to wraparound fidelity, are mid to high 80th percentile based on a 100 percent scale, 80–84 % are above average fidelity, 75–79 % are average, 70–74 % below average and below 70 % is non-wraparound (Bruns et al. 2008). The WFI results showed significant improvements in the wraparound process from 2007 to 2009 as perceived by the care coordinators and parent/CGs. The total WFI score improved significantly from 2007 to 2009 for both parent/ CGs (from 76 to 84, p = .000) and for care coordinators (from 87 to 90, p = .006). Total WFI scores for youth increased from 74 in 2007 to 78 in 2009, although not significantly (p = .199).
123
Community Ment Health J
We also analyzed total mean scores by wraparound phase (engagement, planning, implementation, and transition) for each respondent type. Care coordinators rated themselves at high fidelity ([85) for all phases of wraparound in both 2007 and 2008 except transition, which in 2007 they rated as 80 (above average fidelity). In 2008 they rated Transition as 86 which was a significant improvement from 2007 (p = .02). Parent/CGs ratings improved significantly from 2007 to 2009 for all phases of wraparound and were in the high fidelity range in 2009 for each phase except transition. In 2007 parent/CGs rated transition at 63, considered nonwraparound; and in 2009 they rated it 72, a significant improvement but still considered below average fidelity. Youth ratings of the engagement phase improved slightly from 72, below average, to 77 or average fidelity. Their ratings of the planning phase improved significantly from 72, below average, to 79 or average fidelity. The implementation phase rating by the youth remained constant at 82, above average fidelity, from 2007 to 2009. The transition phase improved little from 62 in 2007 (nonwraparound) to 70 in 2009, and although moving in the right direction these scores are considered borderline wraparound. We also compared scores by agency. Findings across the six agencies were not significantly different by total score or respondent group suggesting that all of the agencies have similar strengths and weaknesses. Application of Results Following each summer’s data collection in 2007 and 2008, WFI results were reported to the system of care’s management team. Our management team includes representatives from all child-serving agencies, family members, care coordination supervisors and youth. The management team decides by consensus of the group to make changes in service delivery or training. Reports by agency and overall totals were distributed to each of the six agencies providing wraparound services. Results of the 2007 data collection showed undesirable scores in fidelity for the transition phase of wraparound by the parent/CGs and youth. The transition phase prepares the family and youth for passage from formal to natural or community supports. This sparked our management team to develop case transition training and education programs for care coordinators, and a requirement that transition planning be included on the agenda of all child and family team meetings. Youth scores for the engagement phase were also identified in need of immediate attention. Results were disseminated to a group of families and youth who made suggestions for improvements. As a result, the orientation workshop, conducted by the family organization,
123
and provided to families newly enrolled in wraparound services, began to include a discussion about the transition phase of the wraparound process so families would begin to understand that the process of wraparound would eventually wind down and services would transition to natural supports. To help improve youth engagement, youth were invited to attend the orientation workshop and were asked to identify a friend to include on their CFT. Quality improvements included: •
• •
•
Training of care coordinators, service providers, and families and youth includes an explanation and discussion of the transition phase of wraparound. Transition is included on the agenda at each child and family team meeting. The orientation workshop held for all families enrolled in wraparound services and conducted by the family organization includes a discussion of the transition phase. All trainings are to be scheduled more frequently and at different times so all families are able to attend.
Discussion Perception of fidelity to wraparound care coordination varies greatly between the care coordinator, parent/CG and youth in our community. Care coordinator mean scores were higher than parent/CGs and youth for all phases of wraparound and were also in the high acceptable range. Important to note is that many care coordinators had multiple interviews, an average of 3.4 in 2007 and 2.2 in 2008, and may have grown tired of being interviewed. Although care coordinators were informed of the confidentiality of the interview in the consent process, they may have believed their scores would affect their employment status in some way. This may have influenced their responses and caused rater drift. With the care coordinators rating themselves at high fidelity for all phases of wraparound by 2008, it was decided that we would not interview care coordinators during 2009 and instead would devote more resources to interviewing the youth. As mentioned above, we trained a young adult to conduct interviews solely with the youth. The length of stay the families had been receiving services also varied greatly, from 3 to 24 months. By 3 months into services the wraparound care coordination process should be fully implemented. To determine if families enrolled for a longer period at the time of the interview resulted in higher scores, we analyzed only the families enrolled at least 5 or more months. We found only a one point difference in scores for all respondents except for three points lower on the 2007 youth total score and two
Community Ment Health J
points higher for the parent/CG 2009 total score. The perception of wraparound fidelity by the respondents would seem to be inherent by the third month of enrollment. The good news is that by the third summer of data collection, 2009, fidelity scores by parent/CGs and youth, were acceptable or above acceptable for the planning and implementation phases. Transition scores as rated by the parent/CGs and youth improved by 2009 to the borderline range of fidelity from non-wrap scores in 2007. Parent/CGs and youth had very different mean scores for the engagement and planning phases but were more in agreement on the implementation and transition phases. It is interesting to note that the implementation phase is the longest phase, time-wise, of the wraparound process because this is the time when family and team members are meeting regularly to work towards goals and implement the plan. It generally can last up to 9–18 months. Engagement and planning lasts about 1–3 weeks total and transition planning should be ongoing from enrollment in services to discharge from services. After the third year of data collection, youth were rating all phases except the transition phase as acceptable fidelity. Yet, youth rated the engagement phase significantly lower than the parent/CGs (77 vs. 90). This is concerning since this may indicate that the youth is not involved from the start, perhaps does not feel a part of the process or that they can trust the process. The transition phase was rated as borderline fidelity by parent/CGs and youth. This suggests general agreement that transition planning is not happening according to wraparound principles and is an area in need of immediate attention. Comparable WFI results could not be found in the literature however, two presentations at national conferences have reported total WFI scores which are consistent with our findings. Sather et al. (2011) report findings from data collected from over 4,000 teams in 50 WFI user sites from 2006 to 2011. Similar to our findings, they report above average fidelity scores, [80 %, by the wraparound facilitator compared to the CG and youth. Our CG scores for 2007 (76) are similar to theirs from 2006 (75); our CG scores increase to 84 in 2009 while theirs increases to 79 in 2011. Also similar to us, their youth scores were lowest of the three respondent groups, starting at 73 in 2006 rising to 76 in 2011. Our youth scores were 74 in 2007 and 78 in 2009. Sather et al. (2011) point out that mean fidelity scores have increased since the WFI was introduced in 2006 and may be a result of several factors from a better understanding of the wraparound process to better training or coaching. And to decrease the burden of conducting the WFI, which can take 20–40 min per interview, the WERT is developing a brief version of the WFI, the WFI-EZ (Walker and Bruns 2013). Our community believes that showing the results of the WFI interviews is helping to
improve the process and practice of wraparound. In another presentation, Walton et al. (2010) total WFI scores were similar to our 2007 data collection: their wraparound facilitator score was 81 compared to ours of 87 (both above average fidelity). Their CG total score was 79 compared to ours at 76; their youth total score was 76 compared to our 74. Quality improvement efforts undertaken by the individual wraparound care coordination agencies were not analyzed by this study. Although agencies often share creative strategies during management team meetings, it would be useful, in the future, to document what particular efforts have worked or not. During management team meetings, we discussed which respondent’s results were most related to outcomes for families. Various studies have shown different results, from both CG and care coordinator perceptions of wraparound being significantly associated with outcomes, to other studies finding that the CG and youth reports of wraparound are related to program and system level conditions that support high-quality wraparound (Bruns et al. 2005). The answer is not an easy one and while the research continues on the use of fidelity instruments to measure adherence to wraparound principles, our community continues to value the input gleamed from all of the participants in the wraparound process.
Conclusions This article describes how the WFI, a fidelity assessment tool, was used to measure adherence to the wraparound process in one community. WFI findings have been a useful tool for our management team in identifying our strengths and areas for improvement. Total scores and scores by respondent type by agency are distributed to the team for analysis. Indeed, during management team meetings where WFI results were reviewed, members of the team referred to lower scores as ‘opportunities’. People began to ask questions about the data and discussed ways to improve for example, youth engagement. The tool’s organization mirrors the wraparound process and thus, allows one to pinpoint specific activities which need work. The WFI could be used as a training tool to help care coordinators and parents better understand the process of wraparound. Results of this study show that there has been an increase in attention to wraparound quality and fidelity over time. And importantly, it allows for input from families and youth currently being served. As a result of this study, youth engagement has become a priority area for improvement by the system of care. The review of data at management meetings should include the question of what needs to be done based on these data, so that action items
123
Community Ment Health J
are developed and the results discussed at the next meeting. In addition, the WFI results have been used to engage the family organization in developing recommendations for improvements to care coordination. There are several limitations to this study. Although all the interviewers received a standardized training protocol developed by the WFI developers, internal consistency checks were not made between interviewers and may have affected the quality of the data collected. Only CGs who answered the phone call and agreed to participate were included in the sample. If the CG did not participate or could not be reached, their youth was not asked to participate. Because we had a limited timeframe in which to conduct interviews (summer months), we were limited to calling families receiving services during those months. There was no mechanism to offer participation to families who did not have a phone or missed the phone call from the interviewer. And finally, we did not interview ‘team members’, those individuals who are natural supports to the child and/or family and whom may have offered a unique perspective to the wraparound process. Acknowledgments The author thanks the youth, parents, caregivers, and care coordinators for their input and participation in this project which has improved services to all families. The Substance Abuse Mental Health Services Administration, Children’s Mental Health Services Grant SM-03-009 and Erie County, NY Department of Mental Health funded this work.
References Bruns, E. J., Burchard, J. D., Suter, J. C., Leverentz-Brady, K., & Force, M. M. (2004). Assessing fidelity to a community-based treatment for youth: The wraparound fidelity index. Journal of Emotional and Behavioral Disorders, 12(2), 79–89. Bruns, E. J., Sather, A., Pullmann, M. D., & Stambaugh, L. F. (2011). National trends in implementing wraparound: Results from the state wraparound survey. Journal of Child and Family Studies, 20(6), 726–735. Bruns, E. J., Suter, J. C., Force, M. M., & Burchard, J. D. (2005). Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies, 14(4), 521–534.
123
Bruns, E. J., Suter, J. C., & Leverentz-Brady, K. (2008). Is it wraparound yet? Setting quality standards for implementation of the wraparound process. The Journal of Behavioral Health Services & Research, 35(3), 240–252. Bruns, E. J., Walker, J. S., Zabel, M., Matarese, M., Estep, K., Harburger, D., et al. (2010). Intervening in the lives of youth with complex behavioral health challenges and their families: The role of the wraparound process. American Journal of Community Psychology, 46, 314–331. Burns, B. J., & Goldman, S. K. (Eds.). (1998). Promising practices in wraparound for children with serious emotional disturbance and their families ( Vol. IV, p. 14). Washington, D.C.: Center for Effective Collaboration and Practice, American Institutes for Research. Effland, V. S., Walton, B. A., & McIntyre, J. S. (2011). Connecting the dots: Stages of implementation, wraparound fidelity and youth outcomes. Journal of Child and Family Studies, 20, 736–746. Kernan, J., Pagkos, B., & Grieco, J. (2009). Family voices network of Erie County: One community’s story of implementing system reform. In E. J. Bruns & J. S. Walker (Eds.), The resource guide to wraparound. (Chap. 5b.2). Portland, OR: National Wraparound Initiative. Sather, A., Pullmann, M., & Bruns, E. (2011). What predicts wraparound fidelity? Findings from the wraparound fidelity index. Poster presented at the 24th Annual Systems of Care Conference in Tampa, FL. Retrieved February 19, 2014 from http://depts.washington.edu/wrapeval/docs/What_Predicts_Wrap around_Fidelity_handout.doc. Walker, J. S., & Bruns, E. J. (2012). Wraparound basics: What takes place during the wraparound process? Retrieved January 10, 2013 from http://www.nwi.pdx.edu/wraparoundbasics.shtml. Walker, J. S., & Bruns, E. J. (2013). Wraparound fidelity assessment system. Wraparound evaluation and research team. Retrieved January 10, 2013 from http://depts.washington.edu/wrapeval/ index.html. Walker, J. S., Koroloff, N., & Schutte, K. M. (2003). Implementing high-quality collaborative individualized service/support planning: Necessary conditions. Portland, OR: Portland State University, Research and Training Center on Family Support and Children’s Mental Health. Walker, J. S., & Schutte, K. M. (2004). Practice and process in wraparound teamwork. Journal of Emotional and Behavioral Disorders, 12(3), 182–192. Walton, B., Effland, V. S., & Bruns, E. (2010). Relationship of fidelity to wraparound and outcomes for youth and families. Paper presented at the 6th Annual CANS Conference, San Francisco, CA. Retrieved February 19, 2014 from http://depts.washington. edu/wrapeval/docs/WFI-Outcomes_CANS_conf_4-20-10_FINAL. pptx.
