Journal of Child and Family Studies, Vol. 7, No. 2, 1998, pp. 161-170
Reliability of the Wraparound Observation Form: An Instrument to Measure the Wraparound Process Michael H. Epstein, Ed.D.,1,8 Madhavi Jayanthi, Ed.D.,2 Janet McKelvey, B.S.,3 Erin Frankenberry, B.S.,4 Ron Hardy, M.S.,5 Kathy Dennis, M.S.,6 and Karl Dennis, M.S.7
Within the past decade, the wraparound approach has gained significant popularity in providing services to children with challenging social and family needs. While a plethora of wraparound programs have been developed and studies have been conducted to assess their effectiveness, the need to develop instruments that measure the implementation of wraparound services is clear. The purpose of the present study was to evaluate the reliability of a scale that measures wraparound services. In this study, the Wraparound Observation Form (WOF), was developed to evaluate the implementation of the wraparound process in treatment planning meetings. The WOF includes 34 closed-ended items that requires the respondent to note the occurrence or non-occurrence of specific events or behaviors at treatment planning meetings. In the present study, two data collectors attended planning meetings and independently completed the WOF. The inter-rater reliability was 95%. The WOF appears to be a reliable instrument and be appropriate in evaluating wraparound services. KEY WORDS: wraparound measure; instrument reliability; evaluation measure. 1William Barkley
Professor, Department of Special Education and Communication Disorders, University of Nebraska, Lincoln, NE. 2Adjunct Professor, Department of Special Education, University of Texas, Austin, TX. 3Research Associate, Center for Educational and Social Services, DeKalb, IL. 4Doctoral Student, Department of Psychology, DePaul University, Chicago, IL. 5Director, Satellite Family Outreach Program, Kaleidoscope, Chicago, IL. 6Director of Training, Kaleidoscope, Chicago, IL. 7Executive Director, Kaleidoscope, Chicago, IL. Correspondence should be directed to Michael H. Epstein, Department of Special Education and Communication Disorders, Barkley Center, University of Nebraska, Lincoln, NE 68583. 161 1062-1024/98/0600-0161$15.00/0 © 1998 Human Sciences Press, Inc.
162
Epstein et al.
Wraparound programs were developed in response to the absence of individualized services for children with serious emotional disturbance and have been conceptualized as an important component within the system of care (Stroul & Friedman, 1986). Wraparound programs have been viewed as central to the delivery of services to children with challenging social and family needs and who require services from child welfare, mental health, special education, juvenile justice, and other service delivery components (Clark & Clarke, 1996). VanDenBerg and Grealish (1996) described wraparound as "a philosophy and overall approach that mandates that services be tailored to the specific needs of all children and families, even when services are delivered as part of a categorical service program" (p. 8). The key characteristics of the wraparound process include: (a) services must be based in the community; (b) services and supports must be individualized to meet the needs of the children and families, and not the priorities of the service care agency; (c) parents must be included in every stage of the treatment process; (d) services must be provided on an unconditional basis, i.e., if the needs of a child or family change, the services being provided must be changed and the child or the family must not be discharged from the service care agency; (e) outcomes of all services must be measured; (f) agencies implementing the services must have access to non-categorized funding; (g) services must be built on the unique values, strengths, and social and racial make-up of children and families being served; and (h) services must be implemented on an inter-agency basis (VanDenBerg & Grealish, 1996). The push to rapidly implement wraparound programs has resulted in a plethora of service models that vary widely with respect to their philosophies, services, and mode of implementation of programs. Recently, several outcome studies were conducted to evaluate some of the exemplary models in existence (e.g., Clark, Lee, Prange, & McDonald, 1996; Eber, Osuch, & Redditt, 1996; Yoe, Santarcangelo, Atkins, & Burchard, 1996). Outcomes examined included emotional and behavioral adjustment of children and youth and placement restrictiveness and stability. However, these research initiatives are not an end in themselves, but the beginning of a long line of studies needed to further define, refine, and measure the wraparound process. Rosenblatt (1996) in his critical examination on the current status of the wraparound programs and research initiatives identified several areas that are in need of further study. One of these research areas was related to the development of instruments needed to measure the various elements of the wraparound process. Specifically, the research questions included, "What outcome measures and indicators best tap the wraparound process? What new measures are needed? Are newly developed measures valid and reliable?" (Rosenblatt, 1996, p.111).
163
Satellite Observation Form
Indeed, there is a need for the development of additional instruments to measure the various processes and practices of the wraparound programs. Identification of such process-related information has several benefits. For instance, identification of the key processes and elements of exemplary wraparound programs will help in the establishment of best practices that other agencies and communities will be able to replicate. Furthermore, outcomes of successful wraparound programs can be better understood within the context of such process-related information. Even with these benefits, minimal efforts have been undertaken to develop and determine the psychometric characteristics of measures that assess wraparound services. In one of the few published studies, Singh, Curtis, Wechsler, Ellis, and Cohen (1997) reported on the development of the Family Assessment and Planning Team Observation Form (FAPT) that was constructed to assess the family friendliness extended to families by professionals at planning meetings. The FAPT assesses important aspects of meetings including meeting management, case presentation, formulation of services, and professional tactfulness. The authors reported that the FAPT achieved a satisfactory level of inter-rater reliability. In an attempt to extend the research on measures, we developed an instrument, the Wraparound Observation Form, to evaluate the implementation of the wraparound process in treatment-planning team meetings. Specifically, the purpose of this study was to examine the inter-rater reliability of the Wraparound Observation Form.
METHOD Setting This study was conducted at Kaleidoscope Inc., a not-for-profit child welfare organization in Chicago, Illinois. Kaleidoscope provides unconditional, comprehensively individualized services that are based on the unique circumstances presented by each of the 600 children and families served. Kaleidoscope received a grant from the Annie E. Casey Foundation to evaluate its family support environment referred to as the Satellite Family Outreach Program. Satellite has a no-reject admission policy, an unconditional care intervention philosophy, and provides services based on a wraparound approach. This study was one component of that overall program evaluation.
164
Epstein et al.
Treatment Planning Team Meetings An evaluation of the wraparound process can be targeted on the planning of professional services or the implementation of professional services. The present study focused on the planning of wraparound services. As part of the Satellite program, Master Rehabilitation Service Plans (MRSP) meetings are held within three weeks of intake and every six months thereafter that a family receives services. The purpose of MRSP meetings are to identify the strengths and needs of the family, to develop a comprehensive treatment plan for the family consisting of goals, objectives, services, and measurable outcomes, and to review the current functioning of the family. MRSP meetings are usually attended by the family members, Satellite team members serving the family, a representative from the Illinois Department of Children and Family Services, other external providers (e.g., foster care representatives, clinical psychologists), and natural supports to the family (e.g., neighbors, other family members). Instrument The Wraparound Observation Form (WOF), was used to collect information on the implementation of the wraparound process at treatment-planning team meetings. The 34 items in the WOF elicited information on the following eight elements of the wraparound process: (a) community-based services (e.g., team chooses community placements for children rather than out-of-community placements); (b) individualized services (e.g., strengths of family members are identified and discussed at the meeting); (c) involvement of family members (e.g., the parent is asked what problems he/she would like to work on); (d) inter-agency cooperation (e.g., professionals from other agencies or facilities have an opportunity to provide input); (e) unconditional care (e.g., for severe behavior challenges, discussion focuses on solutions rather than discharge); (f) measurement of outcomes (e.g., the service plan goals are discussed in objective, measurable terms); (g) management of team meetings (e.g., key participants are invited to the meeting); and (h) cooperation of team members (e.g., team members develop goals/outcomes and solve problems together). All 34 items were closed-ended; individuals completing the WOF were required to select one answer from the following three answers: Yes, No, and Not Applicable. In addition, the WOF also elicited the following information: (a) number of persons attending the meeting, their relationship to the client, and if applicable, the agency they represented and (b) life domain areas addressed in the treatment plan (e.g., residence, medical, legal). The response
Satellite Observation Form
165
format for these two items was open-ended. The WOF was designed to be completed mainly while observing the treatment-planning team meetings. However, some of the items (e.g., convenient arrangements for family's presence at meeting are made [i.e., time, transportation]) require the individual completing the WOF to obtain information on those items by talking to a social worker or a parent before or after the team meeting. The Wraparound Observation Form was adapted from the Family Assessment and Planning Team Observation Form (FABT), a 42-item instrument used for measuring the individualized treatment-planning process for children in Virginia's system of care for children and adolescents with emotional and behavioral disorders (Singh, Curtis, & Wechsler, 1995: Singh, Curtis, Wechsler, Ellis, & Cohen, 1997). The WOF was collaboratively developed by the evaluation team members and the Satellite administrators. First, the evaluation team members reviewed the literature on the wraparound process and identified the key features of the wraparound process. Second, they modified the FABT to reflect the key features of the wraparound process. Third, the adapted form was evaluated by two administrators of the Satellite program for appropriateness of content, clarity, and readability. Fourth, the evaluation team members modified the form in accordance with the feedback given by the administrators. Fifth, the evaluation team members piloted the form at two treatment-planning team meetings. Sixth, on the basis of the information obtained from these two pilots, the evaluation team members made changes in the form. These changes included deleting or clarifying certain items and finalizing the wording of the operational definitions of each item and the response format of the form. Finally, a user manual for the Wraparound Observation Form was prepared, which included (a) operational definitions for each of the 34 items and (b) instructions for completing and scoring the form. Data Collection Data were collected at 10 treatment-planning team meetings. At each team meeting, two members of the evaluation team collected data on the WOF by observing the team meetings as well as talking to the social worker or the parent immediately following the team meetings. Both data collectors were trained in completing the WOF by the first author. This included discussions of the operational definitions of the 34 items, the response format, and examples and non-examples of the items. Then, the two evaluation team members practiced using the WOF at two treatment planning team meetings. Following the two practice sessions, the data collectors met with the senior author to discuss any disagreements and answer any questions.
166
Epstein et al. RESULTS Inter-rater Reliability
Inter-rater reliability was calculated on a point-by-point basis using the formula: number of agreements divided by the number of agreements plus number of disagreements and multiplied by 100. The mean reliability across the 34 items was 95.3% (range = 70%-100%; median = 100%). In Table 1, the inter-rater reliability for each of the 34 items listed in the WOF is presented. The inter-rater reliability of the WOF was evaluated by initially having two members of the evaluation team score the WOF at 10 team meetings and then by calculating the extent to which they agreed or disagreed while scoring the WOF. A reliability of (a) 100% was obtained on 24 items, (b) 90% was obtained on five items (i.e, item number 10, 17, 25, 27, and 30), (c) 80% was obtained on four items (i.e., item number 1, 3, 21, and 29), and (d) 70% was obtained on one item (i.e., item number 32). For the five items with 90% reliability, the two data collectors had one disagreement per item. All their disagreements were due to human error. For the four items with 80% reliability, the two data collectors had two disagreements per item (i.e., a total of eight disagreements). Of these eight disagreements, one was due to human error; seven were due to differences in the way the data collectors initially interpreted the operational definitions and the scoring instructions for these items. DISCUSSION Data from this study indicate that the Wraparound Observation Form is reliable. Given the widespread applicability of the wraparound approach and the dearth of instruments available to measure the various processes of the wraparound programs, the importance of this instrument cannot be ignored. Instruments that help measure wraparound processes and practices are clearly needed for a couple of reasons. One, process-related instruments will help identify the best practices of exemplary wraparound programs in existence. This information will not only help in refining and defining the wraparound processes delineated by VanDenBerg and Grealish (1996), but also in establishing a "gold standard" for wraparound services that can be built upon or modified depending on the accumulating knowledge base (Rosenblatt, 1996). Two, process-related instruments are also needed to determine if the wraparound programs in various communities are being executed accurately and consistently. Given the proliferation of wraparound programs around the country, such information
Satellite Observation Form
167
Table 1. Inter-rater Reliability on the 34 Items of the Wraparound Observation Form Item Number
1. Information about support services in the area is offered to the 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27.
parent/team. Plans include at least one public and/or private community service. Team choose community placements for child(ren) rather than out-of-community placements whenever possible. Individuals (non-professionals) important to the family are present at the meeting. The parent is asked what treatments or interventions he/she felt worked/didn't work prior to Satellite. Satellite staff advocates for services and resources for the family (e.g., identifies and argues for necessary services). All services needed by family are included in plan (i.e., no services were rejected. The steps needed to implement the service plan are clearly specified by the team. Strengths of family members are identified and discussed at the meeting. Convenient arrangements for family's presence at meeting are made (e.g., time, transportation). The parent/child is seated or invited to sit where he/she can be included in the discussion. Family members are attended to in a courteous fashion at all times. The family's perspective is presented to professionals from other agencies. The family is asked what problems he/she would like to work on. The parent is asked about the types of services he/she would prefer for his/her family. Family members are involved in designing the service plan. In the plan, the family is assigned tasks and responsibilities that facilitate their independence (e.g., accessing resources on own, budgeting, maintaining housing). The team plans to keep the family intact or to reunite the family. Professionals from other agencies who care about or provide services to the family are at the meeting. Professionals from other facilities or agencies (if present) have an opportunity to provide input. Problems that can develop in an interagency team (e.g., turf problems, challenges to authority) are not evident or are resolved. Services are not terminated because of the multiplicity or severity of the child's/family's behaviors/problems. For severe behavior challenges (e.g., gangs, drugs) discussion focuses on solutions (e.g., services and stuff to be provided) rather than discharge. The service plan goals are disussed in objective, measurable terms. The criteria for discharge of services is discussed. Objective information on child and parent functioning is used as outcome data. Key participants are invited to the meeting (i.e., family members, DCFS worker, teacher, therapist, other significant to the family).
Reliability %
80 100 80 100 100 100 100 100 100
90 100 100 100 100 100 100 90 100 100 100
80 100 100 100 90 100
90
168
Epstein et al.
Table 1. Continued Item Number 28. Basic information about the family is gathered prior to the meeting. 29. All meeting participants introduce themselves. 30. The family is informed that they may be observed during the meeting. 31. A service plan is completed at the meeting. 32. A service plan is agreed on by all present at the meeting. 33. Team members are supportive of other Satellite staff (e.g., share information, respond to each other's ideas, offer to follow through on specific tasks). 34. Team members develop goals/outcomes and solve problems together.
Reliability % 100
80 80 100 70 100 100
is of critical importance as it can be used to determine if the gold standard is being implemented with sufficient strength and integrity to orchestrate the intended outcome, namely provision of individualized services. Third, administrators who offer training to staff on wraparound services may use the WOF as one way to measure how well staff engage in these skills. The WOF can be completed by administrators during supervisory sessions or by staff as a type of self-evaluation. Finally, without adequate definition or measures it will be difficult if not impossible to adequately assess the impact of wraparound services (Rosenblatt, 1996). If additional research indicates that the WOF, or a modification of the WOF, is a reliable and valid measure of wraparound services, the WOF can serve as an operational definition of wraparound services as they are being planned and implemented in social service agencies. To this end, professionals can begin to build an empirical data base on the implementation and outcomes of the wraparound process. For the item (32) with 70% reliability, the two data collectors had three disagreements in all. These three disagreements were due to the way in which the data collectors interpreted the scoring instructions for this item. The two data collectors differed in the way they scored this item as they had trouble determining if the behavior had or had not occurred. Following the completion of this study, the manual accompanying the WOF was revised to further clarify the operational definitions and the scoring instructions for items with 80% or less reliability. Certain limitations of this study must be acknowledged. One, the WOF was designed to measure the wraparound processes only in team meetings. Thus the procedure does not assess how well wraparound services were
Satellite Observation Form
169
implemented in the field, face-to-face with children, family members, and other professionals. In the future, investigators need to extend this line of research to the implementation of wrapraround services in community settings. Second, the instrument was evaluated only at one family support program which restricts its generalizability. Clearly, the reliability of the instrument needs to be assessed in other settings. Third, 100% reliability was not obtained for all 34 items which raises questions regarding the consistency with which data collectors use the instrument. However, as the instrument was revised following the study to clarify the operational definitions and scoring instructions for items with 80% or less reliability, the impact of this limitation may be diminished when used in the future. The final, and perhaps major limitation, is that the scale does not measure an important element of the wraparound process, namely cultural competence. In identifying the key elements of wraparound, VanDenBerg and Grealish (1996) stated that "the process must be culturally competent and build on the unique values, strengths, and social and racial make-up of children and families" (p. 9). Several items to assess cultural competence were included in a preliminary version of the WOF, but because of the inability to actually observe these behaviors in planning meetings they were deleted from the final version of the scale. The purpose of this study was to examine the inter-rater reliability of the WOF, an instrument used for measuring the various elements of the wraparound process when implemented in team meetings. There are several areas in need of further study. For instance, the study needs to be replicated to determine if the changes in the operational definitions and scoring instructions help in raising the inter-rater reliability score. Furthermore, systematic replications of the study at other family support programs will help in refining the instrument and making it more usable across a variety of settings. Finally, further studies are needed to assess the test-retest reliability of the instrument as this information would enhance the predictability and stability of the instrument. ACKNOWLEDGMENTS
This study was supported in part by the Annie E. Casey Foundation, Evaluation Grants Program. The opinions and statements in this article are those of the authors and in no way represent positions of the Annie E. Casey Foundation. The authors wish to thank the Kaleidoscope administrators for their support in the current evaluation, the Satellite staff members for their assistance in collecting the data, the families who allowed their meetings to be observed and Dr. Cynthia Guy, Senior Research As-
170
Epstein et al.
sociate for the Annie E. Casey Foundation, for her continued support and assistance with the overall evaluation. The Wraparound Observation Form can be obtained from the senior author. REFERENCES Clark, H. B. & Clarke, R. T. (1996). Research on the wraparound process and individualized services for children with multi-system needs. Journal of Child and Family Studies, 5, 1-6. Clark, H. B., Lee, B., Prange, M. E, & McDonald, B. A. (1996). Children lost within the foster care system: Can wraparound service strategies improve placement outcomes? Journal of Child and Family Studies, 5, 39-54. Eber, L., Osuch, R., & Redditt, C. A. (1996). School-based applications of the wraparound process: Early results on service provision and student outcomes. Journal of Child and Family Studies, 5, 83-100. Rosenblatt, A. (1996). Bows and ribbons, tape and twine: Wrapping the wraparound process for children with multi-system needs. Journal of Child and Family Studies, 5, 101-116. Roizner, M. (1996). A practical guide for the assessment of cultural competence in children's
mental health organizations. Boston, MA: Judge Baker Children's Center. Scannapieco, M. (1994). Home-based services program: Effectiveness with at risk families. Children and Youth Services Review, 16, 363-377. Singh, N. N., Curtis, W. J., & Wechsler, H. A. (1995). Family Assessment and Planning Team
Observation Form: Manual. Richmond, VA: Commonwealth Institute for Child and Family Studies, Medical College of Virginia, Virginia Commonwealth University. Singh, N. N., Curtis, W. J., Wechsler, H. A., Ellis, C. R., & Cohen, R. (1997). Family friendliness of community-based services for children and adolescents with emotional and behavioral disorders and their families: An observational study. Journal of Emotional and Behavioral Disorders, 5, 82-92. Stroul, B. A, & Friedman, R. A. (1986). A system of care for children and youth with severe
emotional disturbances. (Revised edition). Washington, DC: Georgetown University Child Development Center, CASSP Technical Assistance Center. U.S. General Accounting Office. (1993). Foster care: Services to prevent out-of-home placement are limited by funding patterns. Washington, DC: General Accounting Office, GAP/HRD - 93 - 76. VanDenBerg, J. E., & Grealish, E. M. (1996). Individualized services and supports through the wraparound process: Philosophy and procedures. Journal of Child and Family Studies, 5, 7-21. Yoe, J. T., Santarcangelo, S., Atkins, M., & Burchard, J. D. (1996). Wraparound care in Vermont: Program development, implementation, and evaluation of a statewide system of individualized services. Journal of Child and Family Studies, 5, 23-38.