Administration and Policy in Mental Health, Vol. 29, No. 2, November 2001 ( 2002)

CONSUMER SATISFACTION AND INCARCERATION AFTER TREATMENT John A. Pandiani, Steven M. Banks, and Lucille M. Schacht

ABSTRACT: Consumer satisfaction is increasingly used to measure community mental health program performance. Understanding the relationship between consumer satisfaction and treatment outcomes is only beginning. This article adds to this understanding by reporting on an assessment of the relationship between consumer evaluation of community mental health services and incarceration after treatment in a statewide system of care. Results indicate that satisfaction with services is related to incarceration after treatment, with satisfied consumers having lower incarceration rates. These results support the use of self-reported consumer satisfaction as a measure of mental health program performance. KEY WORDS: consumer satisfaction; incarceration; mental illness; treatment outcomes.

Since the early 1990s, evaluators of human services programs have been increasingly called upon to provide quantitative indicators of community mental health program performance. Measures of consumer satisfaction with services and treatment outcomes are among the most widely advoJohn A. Pandiani, Ph.D., is Chief of Research and Statistics in the Vermont Department of Developmental and Mental Health Services in Waterbury, Vermont. Steven M. Banks, Ph.D., is Senior Mathematician at The Bristol Observatory in Bristol, Vermont. Lucille M. Schacht, Ph.D., was Senior Research and Statistics Analyst at the Vermont Department of Developmental and Mental Health Services. An earlier version of this article was presented at the 49th Annual Conference on Mental Health Statistics, May 30 to June 2, 2000, in Washington, DC. This project was supported in part by supplemental technical assistance support to Grant No. 4 HR1 SM46203-06-3 from the Substance Abuse and Mental Health Services Administration’s Center for Mental Health Services (SAMHSA/CMHS). The contents are solely the responsibility of the authors and do not necessarily represent the official views of the SAMHSA/CMHS. This work could not have been completed without the assistance of Janet Bramley, Erin Turbitt, and the staff of the Vermont Department of Developmental and Mental Health Services. The authors also express their appreciation to the consumers of community mental health services in Vermont for their evaluation of the performance of these programs. Address for correspondence: John A. Pandiani, Ph.D., Chief of Research & Statistics, Vermont Department of Developmental & Mental Health Services, 103 South Main Street, Waterbury, VT 056711601. E-mail: [email protected].

145

 2002 Human Sciences Press, Inc.

146

Administration and Policy in Mental Health

cated (Pandiani, Murtaugh, & Pierce, 1996). Today, information on consumer evaluation of and satisfaction with community mental health services is becoming increasingly available to mental health program administrators and policy makers. The 1999 State Mental Health Profiling System (National Association of State Mental Health Program Directors Research Institute, 2000) indicates that 13 of the 47 responding state mental health authorities had completed statewide consumer surveys; 11 were in progress (7 on a statewide basis); and 21 states were planning consumer surveys. Only one state said it had no plans to implement a consumer survey. One consequence of this growth in availability of consumer evaluations of mental health programs has been a resurgence of interest in the relationship between consumers’ subjective evaluation of services and the ultimate outcome of care. This relationship has been a major issue in evaluation of mental health services for at least two decades (Edwards, Yarvis, Mueller, & Langsley, 1978; Lebow, 1983; Pekarik & Wolff, 1996; McLellan & Humkeler, 1998; Holcomb, Parker, Leong, Thiele, & Hidgon, 1998). Most recently, Roland, Langbehn, and Rohrer (2000) found significant relationships between consumer satisfaction and treatment outcomes in an outpatient setting. People who were more satisfied with their treatment were significantly more likely to report having positive outcomes.

There is a long tradition of skepticism about interpreting measures of consumer satisfaction in both health and mental health program evaluation. Our study is designed to measure the relationship between consumers’ evaluation of the services they receive from community mental health programs (the independent variable) and subsequent incarceration in a correctional facility (the dependent variable). We will explicitly test the hypothesis that higher consumer satisfaction is related to more positive outcomes. Unlike much of the literature, this test will be sensitive to the temporal ordering of events; satisfaction measured during one period will be correlated with a treatment outcome that occurs during a subsequent period. In addition, distinct and independent procedures will be used to measure satisfaction with services (a survey of consumers) and subsequent incarceration (an administrative database). There is a long tradition of skepticism about interpreting measures of consumer satisfaction in both health and mental health program evaluation. In his classic review of the literature, Jay Lebow (1983) noted that

John A. Pandiani, Steven M. Banks, and Lucille M. Schacht

147

the satisfaction research of the day lacked both “theoretical basis and programmatic plan.” Five years later, Cleary and McNeil (1988) noted that “there is no consensus within the medical profession on the role satisfaction should play in the assessment of quality of care.” Fifteen years later, a similar concern was noted by Beigel and Torre (1998) who cited “professional suspicions about the use of such [satisfaction] measures in program evaluation.” Despite this uncertainty, the measurement of consumer satisfaction with services has been a mainstay of mental health program evaluation for two decades. The continuing popularity of consumer satisfaction can be traced to at least three underlying assumptions. For many, consumer satisfaction is seen as an important value, a good in and of itself, which deserves to be monitored and attended to (Campbell, 1997). Consumer satisfaction is also presumed to be a consequence of good treatment practices, a kind of proxy measure of the appropriateness of care and perhaps the accessibility of treatment (Morris, 1998). Finally, consumer satisfaction may be considered a correlate of continued participation and a predictor of favorable treatment outcomes (Sullivan & Spritzer, 1997).

Consumer satisfaction is presumed to be a consequence of good treatment practices. Rates of incarceration of mental health consumers are among the most widely accepted measures of community mental health treatment outcomes. Two factors have contributed to this concern. First, the population of state mental hospitals in the United States has been steadily decreasing, while the number of people in correctional facilities has been steadily increasing (Steadman, Monahan, Duffee, Hartstone, & Clark Robbins, 1984). Much of this discussion, which has been organized around the catch phrase, “the criminalization of mental illness” (Torrey et al., 1992), focuses on inappropriate incarceration of people who are exhibiting symptoms of mental illness. Second, the emergence of managed care in mental health services has increased interest in this area. The Mental Health Consumer Network, for instance, speaks clearly of fears that incarceration will be used as a cost shifting mechanism under capitated reimbursement systems (Van Tosh, 1996). Both the National Alliance for the Mentally Ill (1995) and the Bazelon Center for Mental Health Law (Koyanagi, 1995) call on states to publish rates of incarceration for clients of mental health programs. This growing interest in the relationship between mental health and criminal justice caseloads is beginning to stimulate new evaluation research in this area (Pandiani, Banks, & Schacht, 1998a, 1998b).

148

Administration and Policy in Mental Health

METHOD The research reported here consisted of two distinct components: a statewide consumer survey, and a comparison of treatment outcomes for the most satisfied and the least satisfied survey respondents. The survey asked people, who had been served by community mental health programs for adults with severe and persistent mental illness, to evaluate various aspects of the services they had received. The treatment outcome under examination was the rate at which these consumers of community mental health services were incarcerated in local and state correctional facilities during the year after the consumer survey. Consumer Survey

During the last 6 months of 1997, the Vermont State Division of Mental Health mailed consumer satisfaction questionnaires to all people enrolled in the state’s Medicaid program who had been served by a state-designated community mental health program for adults with a severe and persistent mental illness during the first 6 months of that year (Pandiani, Schacht, & Banks, 1998). The satisfaction instrument is a slightly modified version of the satisfaction items of the Mental Health Statistics Improvement Project Consumer Oriented Report Card survey (Ganju et al., 1996). One followup was mailed to people who had not responded to the first mailing within 3 weeks. A total of 1,157 consumers, more than 50% of the people in the targeted population, responded to the survey. Older people were more likely to respond to the survey than younger people, and people who received more services were more likely to respond than people who received fewer services. There was little difference in the response rates of men and women or the response rates of people in different diagnostic categories. The responses were treated as personal/confidential information, but they were not anonymous. A clearly identified code on each questionnaire allowed the research team to link the responses with information in the Medicaid-eligibility and paid-claims files. For purposes of this study, a summary measure of overall satisfaction with community mental health services was derived from the 21 fixed alternative questions. Responses to each completed item were coded as favorable (1 or 2 on a 5-point scale) or not, and the proportion of favorable responses was calculated for each respondent. On the basis of this score, each person was classified as most satisfied (at least 90% favorable responses), least satisfied (less than 70% favorable), or average. The most satisfied group included 555 respondents (48%), the least satisfied group included 437 respondents (38%), and the average group included 165

John A. Pandiani, Steven M. Banks, and Lucille M. Schacht

149

respondents (14%). Individuals in the most satisfied and least satisfied groups were selected for comparison. The 992 individuals whose incarceration rates were determined included more women (58%) than men (42%). Two thirds of these respondents were 40 years of age or older. People with a diagnosis of schizophrenia were the largest diagnostic group (42%) followed by people with affective disorders (31%). Incarceration Outcomes

The second component of this study involved the determination of rates of incarceration after the community survey for people in our two satisfaction categories. The methodology used in this part of the study was based on procedures used to routinely measure the outcomes of community mental health services in Vermont (Pandiani et al., 1998a). This procedure involves measurement of the overlap between the caseloads of community mental health programs and the state correctional authority for specified time periods. Caseload overlap may be measured for the client population as a whole, or for different client groups (e.g., age, gender, diagnosis). In this study, incarceration rates were determined for the most satisfied and the least satisfied clients, overall, and for gender and age groups. The information used for this analysis was abstracted from the survey response data set and the Facilities Database of the Vermont Department of Corrections. Both data sets included the date of birth and gender of each person represented and a data-set-specific unique person identifier. The two data sets do not share a unique person identifier. When a unique person identifier is shared across data sets, a simple matching of the data sets can provide the number of people in both data sets, and rates of incarceration can be derived directly. When a unique person identifier is not available, or the concerns about personal privacy and confidentiality of medical records limit the availability of personal identifiers, similar results can be derived using Probabilistic Population Estimation (PPE) (Pandiani et al., 1998b). PPE is a statistical procedure that uses information on the number of dates of birth represented in a data set in conjunction with knowledge of the distribution of birth dates in the general population to determine the number of people represented in the data set (Banks & Pandiani, 1998; Banks & Pandiani, 2001). In order to probabilistically determine the number of people shared across data sets that do not include a common person identifier, the sizes of three populations are determined and the results are compared. The number of people in the incarceration data set and the number of people in a satisfaction data set provide the sizes of the first two populations.

150

Administration and Policy in Mental Health

The third data set is formed by combining the two original data sets and determining the number of people represented in the combined data set. In the current study, the number of people represented in the combined (concatenated) data set was determined using PPE because no unique person identifier is shared by the two original data sets. The number of people who are shared by the two data sets is the difference between the sum of the numbers of people represented in the two original data sets and the number of people represented in the combined data set (Whitehead & Russell, 1927). This result occurs because the sum of the number of people represented in the two original data sets will include a double count of every person who is represented in both data sets. The number of people represented in the combined data set does not include this duplication. The difference between these two numbers is the size of the duplication between the two original data sets, the size of the caseload overlap. The Relationship Between Satisfaction and Incarceration

Rates of incarceration subsequent to treatment were determined for consumers in both the most satisfied and the least satisfied group by using the PPE method described above. The incarceration rate for the most satisfied group was determined by measuring the caseload overlap between the data set that included only the most satisfied consumers and the data set that included all individuals who were incarcerated in Vermont during the year after the satisfaction survey. Similar procedures provided incarceration rates for the least satisfied consumers. All incarceration rates are accompanied by 95% confidence intervals and levels of statistical significance for comparing the most and least satisfied respondents in Table 1. In the interest of readability, all incarceration rates in the text are accompanied by the symbol (±) to indicate that the rates are estimates. RESULTS The consumers who were most satisfied with the services they had received from their local community mental health program were significantly less likely to be incarcerated during the following year than the consumers who were least satisfied with the services they had received. Of the 437 consumers in the least satisfied group, 8.9% (±) were incarcerated during the year after treatment. Of the 555 consumers in the most satisfied group, only 4.3% (±) were incarcerated during the year after satisfaction with community services was measured. The least satisfied consumers were

John A. Pandiani, Steven M. Banks, and Lucille M. Schacht

151

TABLE 1 Risk of Incarceration After Treatment by Satisfaction with Services

Total Rate 95% CI (Number) Male Rate 95% CI (Number) Female Rate 95% CI (Number) Young Rate 95% CI (Number) Old Rate 95% CI (Number)

Most Satisfied

Least Satisfied

Significance

4.3% 3.3–6.8% (555)

8.9% 7.5–11.9% (437)

p<.01

6.3% 4.1–12.4% (219)

14.7% 12.0–21.0% (202)

p<.02

3.1% 2.2–4.1% (336)

3.8% 2.7–5.4% (235)

ns

9.3% 6.0–17.3% (148)

12.3% 10.0–18.6% (182)

ns

2.6% 1.8–4.3% (407)

6.4% 4.7–8.9% (255)

p<.01

Note. CI=confidence interval; ns=nonsignificant.

more than twice as likely as the most satisfied consumers to be incarcerated during the year after satisfaction was measured. Men in both satisfaction groups were substantially more likely than women to be incarcerated during the year after the survey. Men who were most satisfied with services were significantly less likely to be incarcerated during the following year than men who were in the least satisfied group. Of the 202 men in the least satisfied group, 14.7% (±) were incarcerated during the year after treatment, compared with 6.3% (±) of the 219 men in the most satisfied group. The difference in incarceration rates between the most satisfied and the least satisfied women were not significantly different, although the estimated incarceration rate for the 336 most satisfied women (3.1% ±) was lower than the rate for the 235 least satisfied women (3.8% ±).

152

Administration and Policy in Mental Health

The least satisfied consumers were more than twice as likely as the most satisfied consumers to be incarcerated during the year after satisfaction was measured. Younger consumers (39 years old or younger) in both satisfaction groups were substantially more likely than older consumers to be incarcerated during the year after the survey. Older consumers in the most satisfied group were significantly less likely to be incarcerated during the following year than older consumers in the least satisfied group. Of the 255 older consumers in the least satisfied group, 6.4% (±) were incarcerated during the year after treatment, compared with 2.6% (±) of the 407 older consumers in the most satisfied group. The difference in incarceration rates between the most satisfied and the least satisfied younger consumers was not significant, although the estimated incarceration rate for the 148 most satisfied younger consumers (9.3% ±) was lower than the rate for the 182 younger consumers in the least satisfied group (12.3% ±).

DISCUSSION This research has demonstrated that consumer satisfaction has a clear relationship to an important treatment outcome. This finding contributes to the growing literature on consumer satisfaction and should help to ground discussion of the role of consumer participation in program evaluation. We support the use of consumer self-reports as part of a diverse set of methods for evaluating community mental health programs. Consumers have a unique perspective on the delivery of care that is important to recognize and consider. In addition, service providers can learn a great deal from consumer evaluations that can be useful in the design and implementation of treatment programs. Finally, we believe that consumers of community mental health services, like other consumers, deserve a voice in the evaluation of services they receive. Critics of the use of consumer self-report in the evaluation of mental health programs are frequently skeptical about the relationship between consumer satisfaction and other measures of program performance. Does consumer satisfaction contribute to or inhibit progress in terms of clinical and/or functional outcomes? The results of this analysis indicate that consumer satisfaction is related to one very important and widely recognized functional outcome. As mental health program evaluation continues to expand to include multiple measures of treatment outcomes, the relation-

John A. Pandiani, Steven M. Banks, and Lucille M. Schacht

153

ship between consumer satisfaction and other outcome measures will become apparent. The nature of the relationship between consumer satisfaction and rates of incarceration, however, is not clear. It could be that both satisfaction and incarceration are related to some third factor. The quality of clinical care, for instance, may be related to both satisfaction and incarceration after treatment. If this were the case, improvement in the quality of care would be likely to have an impact on both satisfaction and rates of incarceration. This observation suggests a need for research designed to identify factors that contribute to consumer satisfaction (or dissatisfaction) with services. This project identified two factors (age and gender) that were related to both the levels of satisfaction and the probability of incarceration, and it tested the degree to which these factors accounted for the observed relationship. Age and gender did not account for the relationship between satisfaction and incarceration. Future research should also (a) systematically address the relationship between consumer satisfaction and other treatment outcomes, such as distress and symptomatology, and (b) include a broad range of both positive and negative functional outcomes. Positive functional outcomes could include employment and continuing education. Negative functional outcomes could include rates of hospitalization and mortality rates. Exploration of the possibility that other client and/or program characteristics could explain these relationships should attend to client diagnostic categories, program practice patterns, and indicators of the relative placement of clients in the treatment process (e.g., recently entered treatment, long history of treatment, about to exit treatment).

Consumers have a unique perspective on the delivery of care that is important to recognize and consider. We believe that community mental health program evaluation needs to continue to broaden its focus to include multiple service sectors and explore the relationships among various indicators of program performance. Program evaluation also needs to focus on the relationships among consumer satisfaction, treatment outcomes, practice patterns, and access to care. We hope our modest effort will encourage other service system researchers and program evaluators to expand the focus of their work to include multiple service sectors and multiple indicators of program performance on a routine basis. Fortunately, administrative databases that include complete information on the caseloads of a variety of community service programs are widely available. These databases usually include information on all episodes of

154

Administration and Policy in Mental Health

care over several years. In combination with data sets from mental health service providers, these data sets can provide information on the movement of clients among service sectors, which is critical to the evaluation of treatment programs that operate in increasingly complex community service networks. Where unique person identifiers are shared across data sets, direct linking of records can be used to integrate databases for the purpose of tracking movement among service sectors. When unique person identifiers are not available, or concerns about personal privacy limit their availability to researchers and evaluators, PPE can be used to identify patterns of movement among service sectors. PPE provides valid and reliable measures of the unduplicated number of individuals shared by data sets. Because PPE relies on existing data resources, it is much less expensive than methodologies that rely on original data collection. Because PPE does not require unique personal identifiers, the personal privacy of individuals and the confidentiality of medical records are protected.

REFERENCES Banks, S.M., & Pandiani, J.A. (1998). The use of state and general hospitals for inpatient psychiatric care. American Journal of Public Health, 88(3), 448–451. Banks, S.M., & Pandiani, J.A. (2001). Probabilistic population estimation of the size and overlap of data sets based on date of birth. Statistics in Medicine, 20, 1421–1430. Beigel, A., & Torre, C. (1998). Asking the right questions: A rational approach to measuring client satisfaction. In K. Coughlin (Ed.), 1999 Medicaid managed behavioral care sourcebook (pp. 359–367). New York, NY: Faulkner & Gray. Campbell, J. (1997). How consumers/survivors are evaluating the quality of psychiatric care. Evaluation Review, 21(3), 357–363. Cleary, P.D., & McNeil, B.J. (1988). Patient satisfaction as an indicator of quality care. Inquiry, 25, 25–36. Edwards, D.W., Yarvis, R.M., Mueller, D.P., & Langsley, D.G. (1978). Does patient satisfaction correlate with success? Hospital and Community Psychiatry, 29(3), 188–190. Ganju, V., Callahan, N., Dailey, W., Dumont, J., Hornik, J.A., Johnson, J.R., Kaufmann, C., Manderscheid, R., McKinney, J., Minsky, S., Mulkern, V., Robinson, G., Ross, K., Schomberg, T., Sherman, P., Teague, G.B., & Tippett, M. (1996). The MHSIP Consumer-Oriented Mental Health Report Card. Rockville, MD: Center for Mental Health Services. Holcomb, W.R., Parker, J.C., Leong, G.B., Thiele, J., & Higdon, J. (1998). Customer satisfaction and self-reported treatment outcomes among psychiatric inpatients. Psychiatric Services, 49(7), 929–934. Koyanagi, C. (1995) Managing managed care for publicly financed mental health service (pp. 34–35). Washington, DC: Bazelon Center for Mental Health Law. Lebow, J.L. (1983). Research assessing consumer satisfaction with mental health treatment: A review of the findings. Evaluation and Program Planning, 6(3–4), 211–236. McLellan, A.T., & Humkeler, E. (1998). Patient satisfaction and outcomes in alcohol and drug abuse treatment. Psychiatric Services, 49(5), 573–575. Morris, J., (Ed.). (1998). Preserving quality and value in the managed care equation: Final report on the 1997 Santa Fe summit on behavior health. Pittsburgh, PA: The American College of Mental Health Administration. National Alliance for the Mentally Ill. (1995). Principles for managed care. Arlington, VA: Author. National Association of State Mental Health Program Directors Research Institute. (2000). 1999 state mental health agency profiling system: MHSIP consumer survey. Available: http://www.nasmhpd.org/ nri/consurv1.cfm

John A. Pandiani, Steven M. Banks, and Lucille M. Schacht

155

Pandiani, J.A., Banks, S.M., & Schacht, L.M. (1998a). Using incarceration rates to measure mental health program performance. The Journal of Behavioral Health Services and Research, 25(3), 300–311. Pandiani, J.A., Banks, S.M., & Schacht, L.M. (1998b). Personal privacy versus public accountability: A technological solution to an ethical dilemma. The Journal of Behavioral Health Services and Research 25 (4), 456–463. Pandiani, J.A., Murtaugh, M.L., & Pierce, J. (1996). The mental health care reform debate: A content analysis of position papers. Journal of Mental Health Administration, 23(2), 217–225. Pandiani, J.A., Schacht, L.M., & Banks, S.M. (1998). Consumer evaluation of community rehabilitation and treatment programs in Vermont: 1997. Waterbury, VT: Vermont Department of Developmental and Mental Health Services. Pekarik, G., & Wolff, C.B., (1996). Relationship of satisfaction to symptom change, follow-up adjustment, and clinical significance. Professional Psychology: Research and Practice 27(2), 202–208. Rohland, B.M., Langbehn, D.R., & Rohrer, J.E. (2000). Relationship between service effectiveness and satisfaction among persons receiving Medicaid mental health services. Psychiatric Services 51, 248– 250. Steadman, H.J., Monahan, J., Duffee, B., Hartstone, E., & Clark Robbins, P. (1984). The impact of state mental hospital deinstitutionalization on United States prison populations, 1968–1978. The Journal of Criminal Law and Criminology, 75(2), 474–490. Sullivan, G., & Spritzer, K.L. (1997). Consumer satisfaction with CMHC services. Community Mental Health Journal, 33(2), 123–131. Torrey, E.F., Stieber, J., Ezekiel, J., Wolfe, S.M., Scharfstein, J., Noble, J.H., & Flynn, L.M. (1992). Criminalizing the seriously mentally ill: The abuse of jails as mental hospitals. Washington, DC: Public Citizen’s Health Research Group and the National Alliance for the Mentally Ill. Van Tosh, L. (1996). Consumer Managed Care Network platform for action. Washington, DC: Consumer Managed Care Network. Whitehead, A.N., & Russell, B. (1927). Principia mathematica (Vol. 1, 2nd ed.). Cambridge, England: Cambridge University Press.