TIP 14 — TIPs — Documents — Home

Chapter 4—Outcomes Monitoring Methods

This chapter discusses the methods and principles of designing instruments, gathering data, and sampling from patient populations and programs for an outcomes monitoring system (OMS). In the next chapter of this Treatment Improvement Protocol (TIP), the data needs of an OMS are described, and suggestions are given to help the director and staff of a single State agency (SSA) decide which data are relevant for the agency's specific monitoring purposes.

On the next page is a content outline for the chapter to guide the reader.

As OMSs are designed, decisions have to be made about how many patients and programs to monitor, which questions to include, which instruments to use, and when and how often to follow up patients. Conflict can arise between advocates of scholarly research design and advocates of a more pragmatic approach. SSAs, treatment providers, and other stakeholders must weigh alternatives carefully when deciding what system will best fit their needs. Chapter 2 provides suggestions for considering a variety of perspectives while reaching consensus.

The fundamental principles of OMS design are feasibility, applicability, and utility. Since these principles can be used to guide all OMS-related decisions, they are briefly explained.

Feasibility

Feasibility refers to the extent to which planned actions can actually be carried out. OMS planners must take into account the demands of the system on State agency resources, treatment providers, and patients. Whatever the amount of funding available to support the OMS, these resources will have to be allocated carefully. Putting all the money into instrument design will leave nothing for equally important aspects of the OMS, such as data analysis and report writing. Using instruments that require a great deal of time for treatment program staff and patients to complete will detract from time needed for other clinical responsibilities and patient needs and will build resentment toward the project.

Applicability

Applicability refers to the extent to which the OMS is designed to fit local needs. Since States differ in their treatment service delivery system structures and programs, as well as in the treatment populations they serve, their OMSs will differ in specific details. Designers of the OMS will need to take the specific characteristics of their State and programs into account.

Utility

An OMS must be useful; it must serve some worthwhile end. Data collection for its own sake cannot be justified. Utility refers to the usefulness of an OMS. The purpose of each question and each procedure must be clear in all participating programs. If participants cannot be convinced of the utility of the OMS, it will fail. Chapters 1 and 2 address broad goals for an OMS and the process for achieving support for the system; Chapter 8 discusses putting the findings to practical use.

Scientific Rigor

Scientific rigor is the ideal to be pursued in developing an outcomes monitoring system. However, there are substantial costs associated with scientific rigor, and it is unlikely that single State agencies will be able to deploy the resources required to develop an OMS as rigorous as the ideal. Nevertheless, if planning teams in single State agencies concentrate on limited and specific service system questions, they can achieve an acceptable degree of scientific rigor without crippling their capacities. Ways to maintain scientific rigor while respecting budgeting constraints are discussed. Doing a limited number of things very well is preferable to attempting too much with sloppy results. Some general approaches to minimizing costs are also discussed, such as starting small and adding increments in design over time or collecting only minimal data on the universe of programs and patients and comprehensive data on only a representative sample. Therefore, while this TIP advocates for rigorous design, practical constraints are acknowledged throughout.

Standardization

The ability to compare results across treatment programs and groups of patients is fundamental to the proper design of an OMS. For purposes of valid comparisons, the standardization of data collection elements and procedures is recommended for all programs and all localities within the State. Beyond State boundaries, national standardization of data elements would expand the capacity to compare outcomes in different parts of the country where there is a great deal of diversity in the provision of AOD services.

Standardization of data elements, or variables, means consistency in these elements, whatever the variable sources. For OMS purposes, the data to be standardized include patient predictor variables, baseline and outcome measures, and treatment variables, as described in Chapter 5. While standardization of data elements does not necessarily mean that all the same data must be collected at every program on every patient, it does mean that when information is collected on specific data elements, those data elements are defined clearly and consistently and contain the same response choices.

For example, suppose it is agreed that frequency of alcohol use is a required data element. A variety of different variables has been used in existing instruments. Suppose one program were to use a variable that classified alcohol use as daily, at least once a week, at least once a month, less than once a month, or never; and another program were to use a variable that asked for the number of days in the previous 30 in which alcohol was consumed. While disagreement may exist as to which variable is preferable, the advantages of standardization are obvious. Unless all participants in the system use the same variable, patient responses cannot be meaningfully compared across programs. Standardization also ensures that similar treatment services, defined consistently, are similarly coded.

The need for standardization also applies to data collection procedures whenever possible. For instance, if certain information is to be elicited through an interview with the patient, this method should be consistently used in all the programs participating in the OMS. If inpatients are to be interviewed within 7 days of admission but only after the effects of recent alcohol and/or drug ingestion have cleared, this timeline must be adhered to consistently in all programs where data are being collected. If treatment service data are to be recorded by a primary counselor in one program, primary counselors should be data recorders in all programs.

In some parts of the country, a great deal of progress has been made toward standardization. The Federal Client Minimum Data Set is a standardized set of commonly used demographic and other intake variables. Because this data set is mandated for providers receiving Federal block grant or other State agency funds for treatment, it is described in detail in the next chapter (see Exhibit 5-2). This client data set is recommended as the foundation for the State OMS.

Standardization of patient assessment variables is exemplified by instruments such as the Addiction Severity Index (ASI) (Longabaugh, 1991), also discussed in the next chapter. Cost-conscious planners and designers of an OMS will build on existing data collection strategies. Integration of standardized data elements across levels of government and other agencies with oversight or evaluation responsibilities will also help eliminate extra work for provider agencies, which in many cases report data in different formats for local, State, and Federal agencies.

While the benefits of standardization are clear in terms of producing higher quality data and making valid comparisons possible, there may be some objections to standardization. Programs that focus on special issues or serve special populations may find that standard descriptors do not provide sufficient information about the unique needs of their patients or aspects of their programs. Flexibility must be considered to respond to special needs, but it can be offered as an adjunct to standardization rather than as a substitute for standardization. For instance, standardization could be limited to a core set of variables required for patients in all programs. At the discretion of the individual agency, a supplemental set of variables could be added to address questions of greatest interest to the special programs and populations. The advantage of such a flexible plan is that it would provide comparable data for systemwide decisions related, for example, to resource allocation, while still allowing specialized programs to address their own particular needs. Adaptations of the Federal Client Minimum Data Set and the Addiction Severity Index have been incorporated by some States into their outcomes monitoring systems. These OMSs provide examples of modifications to meet local needs and are illustrated in Chapter 5 and Appendix B.

The balance between standardization and program flexibility should be addressed during the OMS planning process. Balancing the needs of various stakeholders is one reason it is essential to have their involvement at the earliest stages of planning (see Chapter 2). Respecting the needs and wishes of various groups will enhance the OMS and the buy-in from providers. With sufficient flexibility built into the system to meet their needs, provider agencies are likely to respond positively and appreciate the potential of the OMS to address State policy questions and their own service evaluation objectives.

While standardizing core elements from all data domains is an ideal rather than a reality for most systems, the vision of integrating elements within the OMS to meet local, State, and Federal reporting mandates should guide the process. In practice, all systems are developed iteratively, and the development of an OMS is also likely to be an evolving, one-step-at-a-time process in most States. Standardization across systems or throughout different levels of government, therefore, will probably arise as an outcome of systems development iterations.

Validity and Reliability of Instruments

Validity and reliability are close cousins to standardization. Standardization is one method of achieving validity and reliability. Validity refers to usefulness, and reliability refers to accuracy. Validity means that an instrument measures what it purports to measure (this is technically known as construct validity). A valid inventory on depression among patients must measure depression as distinguished from anxiety disorders or grief reactions.

A reliable instrument is an accurate instrument; it provides repeatable, consistent results. Reliability means that the question or instrument will elicit the same answer from the same respondent regardless of the interview conditions or other factors (except an actual change in the respondent's condition). For example, if a question about physical abuse is vague and abuse is not defined, the respondent might answer "yes" to the question at one point and "no" at another; this question would not produce reliable results.

Consistency of responses at different points in time is known as test-retest reliability. Consistency of responses to different interviewers asking the same questions or consistency of observer ratings of patient behavior, is known as interrater reliability.

Standard procedures exist to design questionnaires and structured interviews to ensure validity and reliability. These procedures include pilot testing the instruments and conducting a series of data analyses on the responses, followed by refining the instruments and repeating the process.

Instrument design is a science in itself, and sound instruments are expensive to develop and test. While the level of rigor of the OMS design will depend on available resources, measures and instruments used must have the highest levels of validity and reliability possible. Results from instruments that have gained wide acceptance in the research community will be less open to challenge than those from untested instruments.

External validity is another important consideration; external validity is commonly called generalizability. Generalizability refers to the extent to which an instrument that has been found valid for one group is valid for other groups as well. For example, an instrument developed for use with adult white males in the United States, as many AOD-related instruments have been, cannot be assumed to be valid for adolescents, females, persons of color, or persons from other countries or cultures. For an instrument to be recommended for widespread use, it must have undergone testing on a wide variety of populations. State OMS planners must keep this principle in mind as they select instruments, particularly when treatment populations of interest include special subgroups such as American Indians, Southeast Asian or Mexican immigrants, pregnant women, people with mental illness, or people with little formal education, to name a few. Many of these populations have not been included in development of existing instruments; at best these instruments would need to be pilot tested with samples of these groups or, at worst, used with great caution.

Principles of generalizability also apply to study findings as a whole. A well-designed study may find that a specific program with a unique array of services produced excellent outcomes for its patients. A careful review of the study design and data analyses might find the results to be internally valid. However, these results are not necessarily generalizable to other groups of patients with different characteristics, histories, and needs. To ensure generalizability, the same program would have to be delivered to a wide spectrum of different patient groups in different locations at different times.

Experimental Design

As noted in Chapter 1, the strongest type of research design is the experimental design because of its capacity for demonstrating causal relationships between interventions and outcomes. In an experimental design, patients are randomly assigned to two or more groups. One group receives the conventional treatment, while the other receives the experimental treatment or no treatment at all. (Withholding or delaying treatment poses serious ethical problems and thus is not considered an option for AOD treatment studies.) While random assignment can be an extremely effective tool for research purposes, it has its own limitations, which are often overlooked. Patients must grant informed consent for random assignment to alternative treatments; those willing to be part of such an experiment may not necessarily be representative of the group as a whole (for instance, they may be more compliant or more altruistic), thus producing a biased sample.

Experimental designs are not appropriate for an OMS; by definition, an OMS records findings from the system as is; it does not manipulate the treatment system. This is not to say there is not a place for experimental design in treatment research, however. Such projects could be developed and applied on a smaller scale to address questions raised by OMS results or not covered by the OMS.

Posttest-Only Design

A posttest-only design is the simplest to develop and implement. Data are collected from patients at some point following treatment and then analyzed to determine if certain groups of patients have had better outcomes than other groups receiving the same services. This design can provide accurate information on the status of patients following the intervention, for example, if they are currently using AODs, if they are employed, or if they are physically healthy. A posttest-only design cannot, however, provide information on whether patients improved since treatment. Without baseline measures for comparison, most outcome measures are meaningless (Allo et al., 1988; Sobell et al., 1987). The exceptions are those that have no meaningful baseline, such as treatment satisfaction ratings or the use of community recovery resources after treatment.

Pretest/Posttest Design

A pretest/posttest design is recommended for an OMS because it balances scientific rigor with practicality. A pretest/posttest design allows for the measure of change over a period of time. "Pretest" and "posttest" are analogous to "before" and "after" or "baseline" and "outcome." "Pretest" and "posttest" are generally used to refer to overall study or system design and the process of collecting comparable measures before and after the intervention or treatment. The terms "baseline" and "outcome" are typically used to refer to the measures or variables themselves.

The limitation of pretest/posttest designs is that they do not prove a causal relationship between patient outcomes and treatment. Even when significant improvements can be documented, the possibility exists that factors other than treatment could account for the changes. Changes in patients' behavior or functioning may have occurred even without the intervention, for example, as a result of community changes in law enforcement efforts, family crises, influences of friends, change in employment status or job satisfaction, maturation (especially in the case of adolescents), or a host of other potential influences. Unless the effects of other factors can be ruled out, or controlled for in statistical analyses, cause-and-effect relationships cannot be proven.

Pretest/posttest designs can provide satisfactory results when sample sizes are large enough and populations sufficiently diverse that comparison groups can be derived from the database. Patients can be matched on many characteristics and factors so that differences found can reasonably be assumed to be related to differences in the services received (California Department of Alcohol and Drug Programs, 1994).

An OMS should incorporate data collection at four periods in time:

• At intake
• During treatment
• At discharge or other transition points
• After treatment.

At Intake

A patient intake form should collect basic data on every patient in the system. The Federal Client Minimum Data Set, for example, can easily be incorporated into an OMS intake form. (Chapter 5 addresses this point in greater detail.) Intake items are generally factual and straightforward. The advantage of collecting at least minimal information immediately is that even if the patient leaves treatment soon after starting, descriptive information will be available.

During Treatment

Two kinds of data should be gathered during treatment: patient data and treatment service data. The patient data will be more comprehensive than that collected at intake. It is recommended that more detailed historical data and that which requires patients' evaluation of their problems be delayed until the initial effects of AOD ingestion have cleared and the patient is not in acute distress. If programs are asked to wait a week or so to collect a certain set of patient data, it will be lost for those patients who drop out of treatment early in the process. Nonetheless, the likelihood of increased patient cooperation and increased accuracy of information make this timing preferable.

Treatment service data can be collected at various points during the treatment process. The next chapter describes two instruments that can be used to collect a weekly record of services received by each patient and discusses the variety of treatment information that may be useful to collect. Appendix B also illustrates how some States have incorporated treatment service data into their OMSs.

At Discharge or Transition Points

A patient discharge form should include at minimum the date of discharge and discharge status (treatment completed, left against staff advice, etc.). A list of discharge referrals could also be included. A discharge form could also include service information, such as the type of services received and the duration of treatment (total days for inpatients or total hours for outpatients). Costs of treatment might also be recorded here. This form should be completed on the day of discharge to ensure accuracy.

Discharge from an inpatient or residential setting may be the end of a treatment episode. It has become more common, however, for inpatient treatment to be followed by outpatient treatment, sometimes called a "step-down" model. In some situations, patients who begin treatment in outpatient settings may need to be transferred to inpatient or residential treatment. For administrative purposes and recordkeeping, transfers to different settings are often treated as discrete admissions and discharges even when they are part of the same treatment episode. Whether or not the change in setting or intensity of services is considered a formal discharge, these transition points are an ideal time to collect the same kind of information recommended for inclusion in a discharge form.

AOD treatment may well evolve in the future into a continuing care model in which a patient receives care on an "as needed" basis, as in the case of many other illnesses. Even now, many programs offer continuing care (for instance, a session per week) for months or even a year or longer after completion of a formal treatment episode. In Minnesota, where this continuing care is common, the end of the more intense phase of treatment (typically 1 or 2 months) is treated as the "discharge" point for purposes of outcomes monitoring, and attendance at continuing care sessions is recorded during the 6-month followup interview. While the concept of "treatment completion" has clear benefits for ease of data collection and analysis, changing clinical realities may blur this concept in the future, a possibility that must be taken into account in the design of OMSs.

After Treatment

The essence of an OMS is outcomes information. While patient changes during treatment can be considered preliminary outcomes (Manu et al., 1994; Wickizer et al., 1994), posttreatment data are necessary to measure sustained change. Because followup data collection is more difficult than intreatment data collection, it will be addressed separately later in this chapter.

Data to be included in an OMS can come from a variety of sources and can be gathered by one or a combination of several techniques. The primary source will be the patient. Other possible sources include spouses, partners, significant others, other family members, treatment providers, and other official records. In designing the system, the possibility of multiple information sources and multiple data-gathering techniques should be considered.

There are several different methods of getting data from these sources. Best practice in the field may be a mix of several of these methods, and the best mix will be determined by system resources and goals. The methods of obtaining data include the following:

• Self-administered questionnaires
• Staff-administered interviews
• Researcher-administered interviews
• Chart review
• Biochemical AOD testing.

Self-Administered Questionnaires

Some instruments are completed by patients without the interaction of a staff member. These self-administered questionnaires offer the advantages of economy and efficiency. They can sometimes be administered to a group, and administration can be arranged at the patient's convenience rather than the provider's or program staff's convenience. Self-administered questionnaires can also be completed by family members or significant others to obtain a collateral perspective on patient functioning and behavior.

There are disadvantages to self-administered questionnaires. They may require a level of literacy that some patients or collateral informants do not have; if program staff are not available to answer questions, the questionnaires may be incomplete or inaccurately filled out. Some patients may need prompting and encouragement to complete forms and may fail to do so without staff support.

These problems can be addressed to a certain extent by requiring that a staff member or research assistant review the questionnaire with the patient after completion. This help provides an environment of supervised administration and will help ensure that complete and usable data are obtained from each questionnaire. However, if this assistance requires a great deal of staff time, the advantages of self-administration are negated.

Questionnaires mailed to patients for followup purposes fall into this category. Mail surveys are inexpensive and make minimal use of project staff time. However, they have a number of significant drawbacks. Mailing a survey form to a patient's residence may compromise his or her confidentiality (see Chapter 6 for a discussion of methods used to protect against this risk). There is also no way of verifying who actually fills out the survey and no way to determine how thoughtfully or carefully the questions have been answered.

Mail surveys often have poor response rates because they depend upon the patient to initiate the filling out and returning of the questionnaire, an action that even optimally functioning people are often unlikely to perform. In general, because of low response rates and problems regarding quality assurance, mail surveys are not recommended. Followup methods will be discussed in more detail later in this chapter.

Staff-Administered Interviews

Staff-administered interviews offer a number of advantages over self-administered questionnaires. Interviews share the burden of work between staff and patient. The interaction between patient and staff is an important component of this method of data collection. When an interviewer establishes good rapport with a patient, the likelihood is increased that the patient will respond truthfully. The data collector can probe certain responses for clarification or expansion and can also verify the patient's understanding of the question's intent. Questionnaire content is not limited by literacy levels of the patient population. Staff can receive prior training in accurately coding responses so that errors are likely to be fewer than with self-administered questionnaires. The most respected interviews in the AOD field now are staff-administered structured interviews such as the Addiction Severity Index.

The need for staff training may be considered a drawback of staff-administered interviews (Longabaugh, 1991). Staff training can be costly and, depending on staff turnover and the complexity of the interview, may require constant updating. Although the use of structured interviews may demand a fair amount of staff time, if the data collected are valid and/or used for multiple purposes, the end results may justify the investment in staff time.

Researcher-Administered Interviews

Another alternative is to have in-treatment interviews conducted by a researcher assigned to the OMS project rather than by treatment program staff. The major obstacle to this type of interview is cost. Most States would not consider this a feasible option. However, using research staff to conduct followup interviews has major advantages. This use of staff is discussed in the section devoted to followup later in this chapter.

Chart Reviews

Patient charts represent a potentially rich source of information about clinical assessments, treatment services, and patient responses to these services. Chart review is traditional in clinical practice, a concept that is well understood and accepted by most treatment providers. Advantages of using chart reviews to gather data for an OMS include the fact that chart reviews do not require input from the patient, and they can be performed at the convenience of the reviewer/data collector.

However, relying on chart review has some disadvantages. Reviewing patient files is labor intensive and, therefore, costly. The subjective and qualitative nature of most of the information in charts may lead to inconsistent coding. Charts also do not provide the same information for each patient, leading to missing or ambiguous data for some patients. For an OMS to most effectively use data from chart review, it is recommended that a standard instrument be adopted or developed statewide to document patient characteristics and treatment services.

An alternative to reviewing charts to obtain a description of treatment services is to develop a standardized form on which to record weekly information listing type and frequency of services provided. This relatively simple type of recordkeeping can greatly improve comparisons between patients and across programs. Two such forms are described in Chapter 5.

Extracting records from other sources can also be used as a method of data collection. Potential sources of information are other medical records, criminal justice records, and social service agency records. See Chapter 6 for information on patient confidentiality protections.

Biochemical AOD Testing

Biochemical alcohol and drug testing is objective, and this methodology has become widely accepted among AOD treatment systems. The most common tests are urine tests for drug use and blood alcohol concentrations for alcohol use. Hair analysis, a relatively recent (and expensive) technology, can reveal drug use over a longer time interval than more widely used drug tests. Saliva testing is also used in some regions.

Biological testing has a high level of accuracy. Tests can verify or refute patients' self-reports about AOD use. They also can serve a deterrent function in keeping drug free those patients who know they will be tested. However, most biochemical tests provide information only about very recent use. They are also obtrusive and, unless required as part of the treatment regimen, may be resisted by patients.

Other problems with urine, blood, and other biological tests are that staff time is required to administer the tests and the laboratory analysis is expensive. The purpose of drug testing must be carefully weighed in light of its potential costs. If the purpose is primarily therapeutic (the deterrent effect), costs may be deemed justifiable whether or not the data are to be incorporated into an OMS. Some treatment programs already administer drug tests routinely. On the other hand, if the testing is to be instituted primarily for purposes of an OMS, its relative costs may not be justifiable in light of its added value to other data sources.

Patient Consent and Successful Contact

While patient consent is not technically required for contact after treatment, it is recommended. Chapter 6 addresses the legal issues related to patient consent for followup, and interviews with collateral sources of information. A sample consent form is also provided.

Irrespective of the number or interval of followups, contacting patients is the key to a successful OMS and valid results. The fewer patients contacted, the less generalizable the results, particularly when contacts are skewed in favor of higher functioning, easier-to-reach patients (Gerson et al., 1985; Harrison and Hoffmann, 1989; Stinchfield et al., 1994a).

The benefits of seeking informed consent are increased patient investment in the process and better information about locating the patient after treatment. Explaining the purpose of the posttreatment contact and the value of outcome information can boost patients' cooperation. The consent form should include detailed information on how to locate the patient after treatment. Patients can also be asked to provide names, addresses, and telephone numbers of others who would know patients whereabouts if they moved. Patients should be reassured that calls to these other contact persons will not reveal that the patient was in treatment (unless, of course, patients expressly consent to interviews with others); these other sources will be used only to try to secure a new address or telephone number for the patient. Potential sources of information include close family members and friends, social or financial workers, probation or parole officers, and anyone else with whom the patient expects to keep in touch.

Followup Methods

While the patient is in treatment, staff-administered interviews are typically conducted in person. At followup, however, in-person or telephone interviews can be used. Telephone interviews are preferred for followup, primarily because of their relatively low cost. In-person interviews either require former patients to come to the interview site—a burden on them that may not elicit compliance without an incentive—or interviewers to visit patients' homes, a very costly method.

Telephone interviews have been shown to produce valid results and have been used extensively in AOD treatment outcome studies (Hoffmann and Ninonuevo, 1994). However, telephone interviews are of marginal use with transient or homeless patients or others without a telephone; because this population is disproportionately low income and socially unstable, its members are at higher risk for relapse. Not obtaining outcomes information on these patients will not only bias results, it will neglect groups for whom treatment improvements are probably most important. Some studies attempt to arrange for followup interviews for these patients at a location the patient may visit on a regular basis: a social or financial worker's office, the office of a parole or probation worker, a shelter, or a similar site.

With prior patient consent (see Chapter 6), structured interviews can also be conducted with patients' family members or significant others to elicit collateral information regarding the severity of patients' problems and level of functioning. Followup interviews are sometimes conducted with significant others to verify patient self-report. While reports from these collateral sources are frequently assumed to be more accurate than patients' self-reports, this is not always the case. People serving as collateral sources can also distort reality, minimize or deny problems, forget events, or simply be unaware of some aspects of the patient's behavior. In one large patient-followup registry, patients were as likely to report AOD use after treatment when collaterals did not as collaterals were to report it when patients did not (Hoffmann and Harrison, 1988).

Patient followup interviews can be conducted either by treatment program staff or research agency staff under contract for this purpose. Patient followup is very time consuming, much more so than inexperienced planners typically estimate. Because of the difficulty in locating patients, the need for numerous callbacks, and the length of the interview itself, a great deal of time should be allotted to followup. When program staff are assigned this duty, it conflicts with their clinical responsibilities. Even well-motivated program staff are not likely to be as successful as research staff who are experienced in this process and have no competing responsibilities (Longabaugh, 1991).

Minnesota used program staff followup for about 5 years. Programs were required to attempt followups for a specified number of patients and document those attempts. Contact success was very uneven across programs. The followup sample that resulted was heavily biased toward those patients with less serious problems and more social stability (Harrison, 1992). With the introduction of its Treatment Accountability Plan in 1993, the Minnesota State agency dropped the requirement that providers follow up with patients and budgeted funds for an independent research agency to conduct the followup interviews.

In addition to the skills and time necessary for successful followup contact and interviewing, other factors argue against having treatment staff conduct followup. These are referred to as demand characteristics and therapist bias.

Demand characteristics refer to those characteristics of an interviewer that may "demand" a certain response from the interview respondent. These characteristics may be extremely subtle and not even in the conscious awareness of the interviewer. But treatment staff are not neutral and objective with respect to treatment outcomes. They are heavily invested in good outcomes. The former patient knows this and may want to please rather than disappoint the interviewer. This attitude can lead to the patient's minimizing or denying AOD use or other problems.

Therapist bias also results from the lack of neutrality of program staff. They may want to hear only the best and disregard the rest, and they may not even realize this is the case. While objective, behaviorally based questions can minimize these effects, they can still influence the interviewer's interpretation of the former patient's responses.

Followup Intervals

In order to compare patient functioning before and after treatment, comparisons have to be made over the same duration of time for different patients. The timing of posttreatment followup contacts is one of the major decisions confronting designers of an OMS. Typical followup periods are 1 month, 3 months, 6 months, 12 months, 18 months, and 24 months. While conducting multiple followup contacts might constitute an ideal design (Longabaugh, 1991), doing so would be cost prohibitive for most systems.

Different followup intervals have different advantages and disadvantages, though they are relative rather than absolute. OMS designers need to weigh the relative pluses and minuses to make the best choices for their needs.

At the 1-month mark, when it is probably easiest to locate patients, few are lost to followup, thereby minimizing the effects of attrition or contact bias. Less memory distortion and forgetting among patients probably occur over such a relatively short period. The drawback is that the interval may be too short to measure any but the most immediate treatment outcomes. The 3-month interval has similar strengths and weaknesses.

The 6-month followup interval has a great deal to recommend it. The contact point at the sixth month is not so far removed from the time of treatment discharge that too many patients are lost. The interval is of sufficient length for the patient to have established some meaningful measures of functioning. Many treatment outcome studies have found that the vast majority of relapses occur during the first 6 months after treatment (McLellan et al., 1992a). Most patients who remain abstinent for the first 6 months are likely to continue with a sustained recovery (Hoffmann and Harrison, 1988). Thus, even though good arguments can be made for longer term followup, selecting sixth-month followup offers a reasonable compromise and has been used extensively in treatment outcome studies (McLellan et al., 1992a).

Followup contacts at 12, 18, or 24 months (or even longer) have the advantage of providing information on long-term status posttreatment. When this approach is used, patients are typically contacted at several contact points, rather than the interviewer's attempting the first contact at 12 months, for example. Contacting the patient every 6 months can protect somewhat against sample attrition; however, the longer the time between contact and treatment, the greater will be the proportion of the sample that is lost. Contact bias must be considered in interpreting long-term results. At all contact points, better functioning patients are more likely to be contacted than poorer functioning patients; the bias may grow in magnitude the farther from treatment the followup extends.

Followups at 6 and 12 months would probably be ideal for most outcomes monitoring systems. These are the intervals Tennessee has chosen for its OMS. If only one followup is feasible within budget constraints, the 6-month contact point is probably preferable because of the higher contact rate associated with the shorter interval. This is the choice made for the Iowa OMS.

Regardless of the duration of the followup interval or intervals selected, the duration of time for which pretreatment functioning is assessed must be the same. That is, the intake assessment must cover the same period of time as the interval selected for followup. For instance, if the only contact is at 6 months posttreatment, questions at intake must address the 6 months preceding treatment. If the only contact is at 12 months posttreatment, the baseline assessment should also be 12 months.

Establishing a 6-month pretreatment and followup interval does not rule out using shorter windows for some measures. The Addiction Severity Index, for example, has some useful scales that are based on 30-day windows. An OMS could be designed to include both 6-month and 30-day measures. This is the approach taken by the Minnesota State agency (see Chapter 5 for more details). With this approach, one set of questions covers the 6 months before and after treatment, and another the 30 days before treatment and the 30 days before the 6-month followup interview.

Incentives

To attempt to increase contact success rates, some projects have printed postcards to be given to patients at discharge from treatment so that they can notify researchers of a new address; others provide a toll-free phone number for the same purpose. One possible though controversial strategy to reach a greater number of patients is the use of financial or other incentives. Incentive fees typically range from $2 to $25, depending on the length of the interview and whether or not a drug test is required. Alternative incentives to cash payments include bus tokens, food coupons, or other items of value to the patient.

Gathering Data From Other Systems

Patient pretreatment and posttreatment data can be compiled from other sources. Other systems may contain information on patients' use of medical care, detoxification admissions, driving offenses or other arrests, or public assistance. Such information can be used as primary outcome data or as collateral data to verify patient self-report. Chapter 6 discusses legal issues related to using data from collateral sources.

Participating Patients and Programs

In designing the OMS, planners will also have to make decisions about the numbers of treatment programs and patients to be monitored. While it might be ideal to monitor all patients, such a course is probably not feasible. The next section will examine incremental and hierarchical designs that may achieve desired results without overburdening available resources. Hierarchical and incremental designs can both be part of a long-term, feasible, OMS implementation process.

Incremental and Hierarchical Approaches

Incremental designs are those that are phased in over time. For example, a State with no previous AOD OMS experience might start by introducing a patient intake form. After it is working well, a discharge form might be introduced. Eventually, a more comprehensive patient assessment tool or a services record might be added. Finally, posttreatment followup could be included.

Hierarchical designs are those that collect some minimum set of data on all patients and programs in the treatment system and more comprehensive data on a subset. The Minnesota State agency has taken this approach in its OMS (see Appendix B). Intake, history, and discharge forms are required for every patient. This process is ongoing. The Minnesota Treatment Accountability Plan adds data collection components for a subset of patients at each program: a modified ASI and a Treatment Services Record is required. This subset of patients is also interviewed 6 months after treatment. This approach combines the advantages of having some information on all patients and programs with having sufficient information to answer the State's questions about the best services for different groups of patients.

In a hierarchical approach, intensive data collection would not need to be ongoing. To minimize costs, data collection could be accomplished within a specified time frame and then discontinued while the results were analyzed and policy changes and program improvements put into effect. Then the data collection could be reinstituted and the process repeated. An iterative design such as this—maybe 1 year on and 2 years off—has some appeal if resources available for an OMS are greatly limited. Treatment providers would also get a break from extensive data collection demands.

With any approach that does not collect the same data on all patients, some method of sampling must be used. Sampling is a means of studying a representative segment of a population to gain knowledge of the whole. In the context of outcomes monitoring, samples can be taken of patients, programs, or both. How extensively sampling is used will depend on a system's budget and human resources constraints, but it is unlikely that any system will have the resources to measure outcomes for the universe of patients or programs, particularly on an ongoing basis. The need for sampling is a pragmatic reality, and sampling will probably be used in most outcomes monitoring systems.

Usually there are enough similarities among groups of programs or patients that properly designed samples will provide sufficient information to draw generalizable conclusions. In most cases, following up the universe of patients or programs will constitute an unnecessary and unjustifiable expense in terms of dollars and staff time.

The most important principle of sampling is that it be done so that the sample is accurately representative of the whole. One such method, called a convenience sample, might be simply using consecutive admissions until a target number is reached, or, if this method presents too great a burden for collecting data, using every second, third, or fourth admission is an option. Minnesota is a State that uses this approach. Another option is to retrospectively select a random sample from an admission or discharge list. Tennessee and Colorado employ this approach. Whatever method is used, it is critical that it be applied consistently. Colorado's OMS does a random selection from its discharge list. It is important to guard against any method that would allow programs to select patients for a sample, since this selection could lead to a biased sample that might emphasize a program's achievements and minimize its problems.

It is also important that the sample not be limited to those patients who complete treatment. Consent for participation should be sought shortly after admission so that even patients who drop out of treatment early can be interviewed later. A sample solely of treatment completers will not be representative of the totality of patients who enter treatment. Obtaining information from patients who do not complete treatment is essential to program and systemic improvement.

Stratified sampling involves an attempt to assure sufficiently large samples of subgroups for data analysis. A random method may not produce a sufficient number of persons of color or cocaine abusers, for example, in areas where these populations are relatively small. Similarly, a large enough sample of pregnant women may not be generated by a random selection of general admissions. Depending on the questions to be addressed by the OMS, sampling methods may have to be adapted to ensure that questions of great interest can be answered.

Instrument Design and Selection

The specific kinds of information recommended for an OMS are described in Chapter 5. A State has four broad options for meeting its identified information needs: 1) design its own set of instruments; 2) use existing instruments; 3) modify existing instruments; or 4) use some combination of new, existing, and modified instruments.

The advantage of new instruments is that they can be tailored to the unique needs of each State's OMS. The disadvantages are the costs and time associated with instrument development and validation.

Existing instruments have many advantages, provided they have established validity and reliability for the populations being studied. If the instruments are accepted within the research community, findings will be less open to challenge. All the expenses associated with instrument development have been paid for by other sources. To the extent that the same instruments are in use elsewhere, valid comparisons can be made across States and studies. Many widely used instruments are in the public domain. (The content for any instrument developed with public monies is by law in the public domain; however, there may be costs associated with reproduction of forms or software-licensed administration packages.) The content of instruments in the public domain can also be modified without permission. The use of copyrighted instruments is typically much more expensive, and in some cases costs may be prohibitive for the number required for a statewide OMS. On the other hand, the expense of copyrighted instruments may be justifiable if the instrument delivers exactly what the planners want and there is no comparable instrument in the public domain.

Modifying instruments offers some of the benefits of designing new ones and some of the benefits of using existing ones. Yet, modifications must be made with extreme caution. The validity and reliability associated with existing content cannot be assumed to transfer to a modified instrument. However, in some cases, sections of an instrument can be deleted with no harm done; a few new items could be pilot tested and added. Instrument modifications should be done only by experienced instrument developers.

Many States may look to a combination of existing and new instruments because it is unlikely everything they want has already been developed. Whatever package is assembled, a field test should be undertaken to try out the instruments and procedures on a small number of cases and programs to determine whether the package will be satisfactory for the State's purposes.

A few questions will be useful in guiding the evaluation and comparison of existing instruments.

• What is the overall purpose of the instrument? Is it a patient assessment tool? Does it contain an adequate history of AOD use? Does it include other important domains (physical and psychological health, criminality, family and social relationships, vocational functioning)? What is missing that should be covered?
• What time interval is addressed? Can the instrument be used at admission and at posttreatment followup?
• Is the instrument designed to be self-administered? Is the reading level appropriate for the patient population? How long will it take the average patient to complete? Are the instructions to the patient clear, and is the layout designed for ease of completion?
• Is the instrument a structured interview? How long will the average interview take? How much staff training is required?
• Does the instrument measure treatment services? Are the categories and questions appropriate to local needs? Do the response choices allow for adequate characterization of similarities and differences? Is the instrument a one-time summary or week-by-week record? Who completes the instrument—the patient or staff?
• Is the instrument in the public domain or copyrighted? How much does it cost?
• Where else has this instrument been used? Are those users satisfied with the results they got? Are there any modifications they would recommend?

Without knowing the purpose of a specific OMS, it is risky to recommend specific instruments. The next chapter suggests some legitimate starting points, however. Different instruments are better for different purposes and needs. The challenge is to weigh all the needs identified by OMS planners and to determine how best to meet those needs by capitalizing on others' experience and instrument development whenever possible. Some States are ahead of others in this process, and their examples can provide guidance by illustrating both pitfalls and successes.

Field Testing

Treatment providers must be involved in the OMS at the design stage since treatment staff will bear the burden of most of the in-treatment data collection. Flaws that appear during the field test should be addressed immediately and the appropriate modifications made. The field-testing phase requires honest acceptance of whatever flaws may appear and a commitment to respond as necessary and not remain wedded to the original design.

The field test need only involve a small number of programs, but they should be selected so that their diversity reflects the range of programs that will be involved in the OMS as a whole.

Staff Training and Implementation

Successful implementation of the OMS will involve training and supervision of personnel who will collect the information, and the facilitation of conditions under which accurate information can be collected.

Studies that have been done with AOD patients show that accurate information can be obtained both during treatment and in followup interviews. Self-report information from AOD patients tends to be more accurate under the following circumstances: 1) when the measures are standardized, 2) when the data gatherers have been trained to use the instruments consistently, 3) when patients are drug free and are not undergoing withdrawal or emotional distress, and 4) when patients are motivated to cooperate with the information gatherer (Litten and Allen, 1992).

Training is an important part of ensuring that those who will be involved with the OMS are invested in its success, and personnel should take the training very seriously. Because of staff turnover, training should be available for replacement staff as well those on board when the project is initiated.

Training needs will vary from system to system and will depend upon the involvement of outside contractors and the expertise level of State personnel.

For outcomes monitoring purposes, providers should be trained to collect patient baseline and treatment service data in accord with rigorous procedures. Each provider agency should have at least one representative who receives direct training in the implementation of the OMS.

Three phases of training are necessary:

• Training to orient providers to the overall implementation of the OMS. This basic orientation should be directed not only toward data collectors but toward a broad audience of providers and single State agency personnel.
• Technical training for data collectors, including instruction in using the instruments that will be part of the outcomes monitoring process. A procedures manual and instrument completion manual are essential.
• Followup or refresher training for training new personnel and assuring continuous quality assurance.

Training can be provided by qualified single State agency personnel, by an independent contractor, or through some other available form of technical assistance. While trained provider personnel can, to some extent, pass on what they have learned to colleagues who will also be involved in implementing the OMS, this "train-the-trainer" model can result in dilution of rigor and consequent reduced effectiveness. This reduction can be a real concern in the context of data collection and research, since the communication of mistakes or misinterpretations and the incorporation of these errors into the use of the OMS could substantially corrode the research effort. As much as possible, direct training should be available for all participating providers.

Training is more feasible if all programs do not start data collection at the same time. A training schedule can be set up to stagger individual program participation in the data collection process.

Last Updated 11-7-02