AIDS and Behavior, Vol. 1, No. 2, 1997
Recruiting a Cohort for the HIV Vaccine Trial: Sensitivity and Specificity of a Screening for Sexual and Substance Use Risk Acts Debra A. Murphy,1,3 Mary Jane Rotheram-Borus,1 Shobha Srinivasan,1 William K. Hunt,1 and Leonard Mitnick2 Received May 31. 1996; revised Sept. 11, 1996; accepted Sept. 12. 1996.
Brief screening procedures are needed that identify persons at risk for HIV among high-risk groups, in order to begin preparing for future HIV/AIDS vaccine trials. The sensitivity and specificity of brief screening procedures were evaluated among 481 African-American and Latina women (both English- and Spanish-speaking) and 143 gay men to identify people at risk for HIV who may benefit from participation in an HIV/AIDS vaccine trial. There were significant ethnic and population differences in sensitivity and specificity. Overall, the brief screening instrument has a sensitivity rate of 94.8% and specificity rate of 88.6%; these rates are significantly better than earlier screening procedures employed to assess risk acts. KEY WORDS: HIV vaccine; HIV risk; risk screening.
vaccine efficacy (Hoff, 1994; Nelson et al., 1994). To identify a cohort with a 3% seroincidence rate and to allow assessment of vaccine efficacy in a reasonable time frame, pretrial screening that is reliable, valid, and brief is critical (Temoshok, 1994). Pretrial screening involves conducting a very brief interview—one that only takes a few minutes—in which individuals are asked a set of questions designed to determine whether they are at high risk for contracting HIV If a person is determined to be at high risk, then a more in-depth interview to assess specific types of risk behaviors, their frequency, and the settings in which they occur can be conducted. In order to be of optimal use, a screen must have sensitivity and specificity. Sensitivity of such a screen is defined as the proportion of persons who are truly at risk who are identified as at risk by the screening interview (true positive/sum of true positive and false negative). Specificity is the proportion of persons truly not at risk who are so identified by the screening test (true negative/sum of true negative and false positive). If a screening procedure were 100% reliable and valid, and the eligibility rate was 25% among a subpopulation, 40,000 people would have to be screened
INTRODUCTION In the United States, there has already been widespread preparation for the AIDS vaccine efficacy trials. Although researchers are continuing work to develop a more efficacious vaccine, in 1994 there were approximately 12 prophylactic candidate AIDS vaccines currently in phase I clinical trials, two of which were being evaluated in phase II studies (Johnston et al., 1994). In order to test the efficacy of potential HIV/AIDS vaccines, candidate vaccines will need to be tested among high-risk populations with a seroincidence rate (annual rate of seroconversion) of approximately 3%; therefore, it may be necessary to recruit a cohort of up to 10,000 participants in order to have sufficient statistical power to confirm
1Division
of Social and Community Psychiatry, Department of Psychiatry, University of California, Los Angeles. 2 NationaI Institute of Mental Health, Rockville, Maryland. 3Correspondence should be directed to Debra A. Murphy, Ph.D., Neuropsychiatric Institute, UCLA, Health Risk Reduction Projects, 10920 Wilshire Blvd., Suite 1103, Los Angeles, California 90024.
75 1090-7165/97/0600-0075$1150/0 © 1997 Plenum Publishing Corporation
76 to identify the cohort of 10,000 persons for a vaccine efficacy trial. With a false-negative rate of 18%, and assuming a false positive rate of 15%, at least 13,200 [40,000 x (0.18 + 0.15)] extra persons (i.e., a total of 53,200 people) will need to be screened to recruit the vaccine trial cohort. Every percent point increase in the sensitivity and specificity of the screening instrument will result in 800 fewer people needing to be screened with a 25% eligibility rate (i.e., 25% of the population is at risk for HIV). With only a 10% eligibility rate, percent point increases in the screening instrument would result in 2,000 fewer people needing to be screened. Therefore, the lower the eligibility rate of a population, the more savings as sensitivity of the screening is improved. So, although screening may appear to be but a small part of a vaccine trial, it is of critical importance to develop a brief and reliable method of screening for HIV risk. Enrolling cohorts for HIV vaccine efficacy trials will require that potential participants disclose personal high-risk behaviors that put them at risk for contracting HIV such as sexual and injection drug use behaviors. Typically, high-risk cohorts have been recruited through brief screening procedures. Unfortunately, previous attempts to screen have generated false negative rates ranging from 18% to 29%. For example, Doll et al. (1994) investigated nondisclosure of HIV-risk behavior on a brief screen used with gay men in sexually transmitted disease (STD) clinics. On the brief self-administered or interviewer-administered measure of demographics and sexual risk behaviors, 29% of the sample of 1,063 reported no risk on the screening measure, but did disclose risk during a detailed, face-to-face behavioral and psychosocial interview administered approximately 8 days after the screening. Following Doll et al.'s work, McKirnan et al. (1994) modified several screening procedures to attempt to develop a brief screening instrument to decrease nondisclosure, which was assessed with sexually active gay men recruited from nonclinical and clinical settings. Modifications included: slightly longer screens; assessment of general rather than only highrisk sexual behaviors; and administration of all screens by trained study personnel. McKirnan reported that 48% of the sample reported unprotected anal sex on the interview. Of those, 18% had reported no risk on the screen. This rate of nondisclosure on screening is lower than that reported by Doll et al. (1994), but remains a high rate of nondisclosure or false negatives. The goal of this study was to assess the sensitivity and specificity of a brief screening interview for
Murphy et al. two populations: women and gay men in communitybased settings. The Centers for Disease Control (CDC) has reported median seroprevalence rates in STD clinics for women of 4.5% and for young gay men of 25.5% (Centers for Disease Control and Prevention, 1994). In Baltimore's sexually transmitted disease (STD) clinics, seroprevalence of HIV increased more than 20-fold, from 0.23% to 5.35% from 1979 to 1989, primarily among heterosexual men and women; the greatest rates of increase have occurred among women, adolescents, and African Americans (Quinn et al., 1992). Over the 10 years of the Baltimore study, the most dramatic change in epidemiology was the declining male-to-female ratio. Among gay men, African Americans and Latinos have higher rates than White males (median prevalence rates of 43.5%, 27.9%, and 18%, respectively). Similarly, higher rates have been found among African-American women than among Latina women, and higher among both of those groups than among White women (Sweeney et al., 1992). In addition to ethnicity, HIV seroprevalence varies substantially based on geographic regions: the highest seroprevalence rates are in urban inner cities. Given the seroprevalence rates among women and gay men, it is anticipated that African-American, Latina, and White women, as well as gay men—particularly in urban, inner-city areas—will be candidates for future AIDS vaccine trials, and supports inclusion of women and gay men as potential populations to participate in the vaccine trials. Although participants who screen positive for risk and then at an in-depth interview report no risk behaviors add some costs to an HIV/AIDS vaccine trial, they have not been missed in terms of access to recruitment for the trial. However, individuals who screen negative but at in-depth interview report risk behaviors are vaccine candidates who would not be recruited when only brief screening occurs. Improving sensitivity of screening becomes critical. The primary focus of this paper will be on the subsample who screen negative; the accuracy of the screening will be calculated by looking at the number of true and false negatives. METHOD Participants Women and gay males were recruited for this study. Women were recruited from three primary care clinics in the Los Angeles area. Each women's
Screening for Sexual and Substance Use Risk Acts clinic (1) reports between 8,000 and 11,000 unduplicated patient visits per year, (2) serves low-income women, (3) serves primarily African-American, Latina (including Central American refugees and immigrants), and White women, and (4) provides services for both English- and Spanish-speaking women. Gay men were recruited from a community service center providing multiple services for approximately 3,000 gay and lesbian adolescents and adults. Recruitment Each person sitting in the waiting room of a clinic/agency was approached by an interviewer. In waiting rooms with so many patients that all of them could not be approached by interviewers, a random approach method was used. Each person was asked whether he or she was interested in hearing about a study provided by UCLA and the agency, and the purpose of the study, the brief screening, and the indepth interview were described. If the individual was willing to participate, he or she went to a private location with the interviewer, informed consent was obtained, and the brief screening was conducted. The in-depth assessment interview was then scheduled, for which participants returned within a 7-day time period. Each participant was given $2 for their participation in the brief screening interview and $15 for their participation in the in-depth interview. Screening and Interview Assessments Screening The NIMH Multisite HIV Prevention Trial screening instrument was utilized in this study. The brief screen interview includes basic demographic information such as birth date, gender, race, education level, marital status, and socioeconomic level. During the brief screening, participants were questioned by an interviewer to determine if they had sexual intercourse during the past 90 days, and if so, how often they had used a condom. Two sets of risk questions were presented in a grouped question format—participants were first asked if in the last 90 days they had done any of the following: had sex with a new partner; had sex with more than one partner; had sex with someone they knew was having sex with someone else during the past 90 days. Par-
77
ticipants did not have to indicate which one had happened, but rather whether any of the behaviors had occurred. Then a five item list of things that could have happened in the past 90 days was read to the participant, and s/he indicated if any of the risk indicators had happened. The list asked if the participant (1) had or was treated for an STD infection or was there that day to be treated, (2) had unprotected sex with someone who had an STD, (3) had injected drugs, (4) had sex with someone who was injecting drugs during the past 90 days, and (5) had sex with someone they knew had the virus that causes AIDS. A "positive" screen indicated the person had engaged in at least one risk behavior; screening "negative" indicated that the person did not engage in any of the risk behaviors. A few final general sexual behavior questions followed the risk questions, including number of partners in the last year and lifetime, and whether the participant had ever had an STD.
Interview Following the brief screening interview, all participants were scheduled for an in-depth, 1½-hr interview. (In addition, participants who had screened positive were told about the National Institute of Mental Health's Multisite HIV Prevention Trial due to ethical considerations that they should be provided with a referral for a risk reduction program.) The in-depth interview instrument used for this study was the NIMH Multisite HIV Prevention Trial assessment, which includes: demographic questions; alcohol and drug use questions; in-depth STD treatment history, sexual risk behavior history, and HIV testing history; and current birth control practices. All risk questions on the NIMH assessment are specific (i.e., they are not grouped format as were the items on the screening). A total of 481 women and 143 men were screened and interviewed; the mean age of each group was 30.7 years (SD = 8.8) and 33.8 years (SD = 7.1), respectively. Among the women's sample, 57.4% were African American and 42.6% were Latina. Of the Latina women, 64.9% were English-speaking and 35.1% were Spanish-speaking. In the gay men's sample, 19.6% were African-American, 35.0% Latino, 42% Anglo, and 3.5% other ethnicities.
78
Murphy et al. Table I. Percentages of False-Positive and False-Negative Participation Rates Screening interview Positive Baseline interview Women African American Latina English-speaking Spanish-speaking Gay men
Negative
Overall positive
True positive
False positive
Overall negative
True negative
False negative
Total N
68.2 81.6 52.8 63.0 25.7 20.3
90.5 89.8 92.2 91.8 94.4 96.5
9.5
31.8 18.5 47.2 36.1 75.0 79.7
93.5 90.2 94.6 95.8 96.2 93.0
6.5 9.8 5.4
481 276 205
4.2 5.5 7.0
133 72 143
10.2
RESULTS Sensitivity and specificity rates were calculated; for the following definitions, see the brief calculation format below. Sensitivity is the proportion of persons who are truly at risk who are identified as at risk by the screening test, i.e., true positive/sum of true positive and false negative, a/(a+c). Specificity is the proportion of persons truly not at risk who are so identified by the screening test, i.e., true negative/sum of true negative and false positive, d/(d+b).
Screening: positive Screening: negative Total
Interview: positive
Interview: negative
True positive (a) False negative (c) a + c
False positive (b) True negative (d) b +d
The results of our study indicate that among those who screen negative on the brief screening (42.8%), this instrument appears adequate, yielding a false-negative rate of 6.7% for the sample. Table I depicts the base rates for the screening and interview instruments for the entire study. Table II presents the sensitivity and specificity rates. Among those who are not at risk (i.e., those who screen negative), AfricanAmerican women have a false-negative rate of 9.8%, Latina women 5.4% (English-speaking 4.2% and Spanish-speaking 5.5%), and gay men 7%. Chi squares were conducted to determine if there were significant gender or ethnic differences in the sensitivity and specificity rates. Women and gay men differ significantly on sensitivity (x2 = 16.5, p < .001) and specificity (x2 = 11.9, p < .001), with women having a higher sensitivity rate (97%) than gay men (78%), but women having a lower rate of specificity (82%) than did gay men (99%). Among the women, there was no significant difference on sensitivity, except for among Latina women (x2 =
7.8 8.2 5.6 3.4
10.9, p < .001). English-speaking Latina women had a higher sensitivity rate (98%) than did Spanishspeaking women (85%). African-American women had a significantly lower rate of specificity (66%) than Latina women (92%; x2 = 20.4, p < .01), and English-speaking Latina women had a lower rate (87%) of specificity than Spanish-speaking women (98%; x2 = 8.7, p < 0.01). Analyses of differences in level of risk between women who fell into the true-negative and falsenegative categories were conducted from the interview risk assessment data; this was also done for the sample of gay men. Women in the true-negative category reported a significantly lower number of partners in the last 3 months, M = 1.0 (SD = 0.2) than those in the false-negative category, M = 1.8 (SD = 1.0), F(1, 116) = 50.6, p < .0001. There was no significant difference in number of partners for the true-negative and false negative gay men. However, true negative gay men reported an average of 18.7 (SD = 22.7) acts of anal intercourse in the last 3 months, while false-negative men reported a much higher average of 37.7 (SD = 33.1), F(l, 65) - 4.0, p < .05. Additionally, the percent of protected acts for the true-negative gay men was 96.5 (SD = 12.8); the false negative group was significantly lower, 85.7 (SD = 15), F(l, 61) = 3.8, p < .05.
Table II. Sensitivity and Specificity Rates
Women African American Latina English-speaking Spanish-speaking Gay men
Sensitivity (%)
Specificity (%)
97 98 95
82 66 92
98 85 78
87 98 99
Screening for Sexual and Substance Use Risk Acts DISCUSSION As the debate continues as to if and how soon data will support an efficacy trial of an HIV/AIDS vaccine, basic questions such as validity of screening instruments need to be addressed in order for investigators to be ready to mount such a complicated trial. As noted, previous attempts to develop a reliable, valid, brief screening instrument have generated false-negative rates with samples of gay men of 18-29%; the screening instrument in the current study with gay men yielded a false-negative rate of only 7%. The overall false-negative rate for the women's sample was approximately the same (6.5%), although African-American women had a higher false-negative rate (9.8%) than did English-speaking and Spanish-speaking Latina women (4.2% and 5.5%, respectively). Chi-square analysis results indicate that Spanish-speaking Latina women are more likely than English-speaking Latina women to report no risk at screening; this is consistent with anecdotal reports that Latina women who are less acculturated are less likely to disclose personal information to researchers or clinic staff whom they do not know well. It is also consistent with a more conservative sexual culture. While Spanish-speaking interviewers are of course used with a Spanish-speaking population, further investigation as to whether matched cultural interviewers (e.g., Hispanic, Puerto Rican, etc.) would reduce false negatives in this population could be conducted. However, the rates are relatively low at this point; this may be the optimal screening that can be obtained. One obvious limitation of this study is that the "standard" to which the brief screening is held is selfreported risk behavior from the in-depth, face-toface interviews. Certainly other, more private methods of assessment, such as paper/pencil or computer-assisted interviews, may result in increased reports of sensitive behaviors. While we acknowledge the limitations of this method, we have followed the methods utilized by Doll et al. (1994) and of McKirnan et al. (1994), who defined part of the "gold standard" for accurate reporting of risk at screening as a detailed face-to-face behavioral interview. A second limitation of this study is that it does not present sensitivity and specificity of the screening instrument for inner-city men—specifically heterosexual or injection-drug using men—populations that also are likely to be at risk.
79 The risk reported by the two samples varied greatly, with 68.2% of the women reporting risk initially and only 20.3% of the gay men reporting risk. This is likely due to the sites chosen for study recruitment: women were recruited from a primary care clinic, while men were recruited from a community service setting in which risk reduction education and training is provided. As noted earlier in the introduction, a very large number of people will need to be screened for risk to obtain sample sizes needed for vaccine trials, and the lower the eligibility rate of a population, the more savings as sensitivity of the screening is improved. Doll et al. (1994) and McKirnan et al. (1994) both reported that nondisclosers (those who did not disclose risk at the screen, or false negatives) had more stringent sexual norms. Doll et al. found these men to have a greater identification with the gay community, and suggested that strong social desirability pressures may be operating. While we did not measure these variables, it is clear from the differences that we found in level of risk among the truenegative and false-negative groups—for both women and gay men—that the non-disclosers or false negatives are engaged in much higher risk levels than the true-negative group. This finding would certainly fit with Doll et al.'s hypothesis. Since a brief screening is conducted by an interviewer who is a stranger to the participant, and takes place within a few minutes, people at high risk may be reluctant to admit risk. Only later in a longer interview designed to gradually move to higher risk level questions, and when more adequate rapport can be established by the interviewer, may that group feel more comfortable in disclosing risk. The false-negative rate for the screening instrument evaluated in the present study is only 6.5% for women and 7% for gay men, compared to Doll et al.'s rate of 29% and McKirnan's rate of 18%. While those two investigators had much larger samples than did this study, they only evaluated it with one population—gay men. The false negative rate across the two populations we assessed lends support to the viability of this instrument. Given the finding that the highest risk people—and therefore the individuals most wanted for recruitment in a vaccine trial—are more likely to be missed at screening due to a falsenegative report, it is imperative that a reliable and valid brief screening instrument be utilized. The screening instrument evaluated in the present study appears to meet this criterion.
80 ACKNOWLEDGMENTS This paper was completed with the support of the National Institute of Mental Health Grant No. 5 R01 MH49958 to M.J.R.-B. The work benefited from the assistance of the group facilitators and interviewers staff of the Health Risk Reduction Project at UCLA, and from the NIMH Consortium Investigators.
REFERENCES Centers for Disease Control and Prevention. (1994). National HIV serosurveillance summary: Results through 1992, Vol. 3. Atlanta, GA: U.S. Department of Health and Human Services. Doll, L. S., Harrison, J. S., Frey, R. L., McKirnan, D. J., Bartholow, B.M., Douglas, J., Joy, D., Bolan, G., and Doetsch, J. (1994). Failure to disclose HIV risk among gay and bisexual men attending STD clinics. American Journal of Preventive Medicine, 10, 125-129. Hoff, R. (1994). Preparations for HIV vaccine trials: Moving from baseline studies to efficacy trials. AIDS Research and Human Retroviruses, 10 (Supplement 2), 191-193.
Murphy et al. Johnston, M. I., Noe, J. S., and Killen, J. Y. (1994). Recent advances in AIDS vaccine research and development. AIDS Research and Human Retroviruses, 10 (Supplement 2), 317-323. McKirnan, D. J., Doetsch, J., Vanable, P., Buchbinder, S., Douglas, J. M., and Judson, F. (1994). Developing brief valid screening instruments for HIV-related sexual risk behavior among gay and bisexual men. AIDS Research and Human Retroviruses, 10 (Supplement 2), 285-289. Nelson, E. E., Vlahov, D., Galai, N., Astemborski, J., and Solomon, L. (1994). Incident human immunodeficiency virus (HIV) infections in a cohort of injection drug users in Baltimore, Maryland. AIDS Research and Human Retroviruses, 10 (Supplement 2), 201-205. Quinn, T. C., Gruseclose, S. L., Spence, M., Provost, V., and Hook, E. W. (1992). Evolution of the human immunodeficiency virus epidemic among patients attending sexually transmitted disease clinics: A decade of experience. Journal of Infectious Diseases, 165, 541-544. Sweeney, P. A., Onorato, I. M., Allen, D. M., Byers, R. H., and the Field Services Branch. (1992). Sentinel surveillance of human immunodeficiency virus infection in women seeking reproductive health services in the United States, 1988-1989. Obstetrics and Gynecology, 79, 503-510. Temoshok, L. R. (1994). Behavioral research contributions to planning and conducting HIV vaccine efficacy studies. AIDS Research and Human Retroviruses, 10 (Supplement 2), 277-280.
