Performance-Based Monitoring in Narcotic Addiction Treatment

In devising a strategy to answer these questions, MTQAS staff consulted an advisory panel of providers, researchers, quality assurance experts, representatives from national health professional organizations, and Federal agency representatives. Staff then developed a client-level data collection instrument (Client Assessment Profile or CAP) that was field tested in Phase I. The first part of the field test was conducted in five narcotic addiction treatment clinics. Based on inter-rater reliability tests and clinician ratings, the items in the CAP proved to have both high validity and reliability.

The second part of the field test involved a controlled 6-month test of the performance reporting system in 25 narcotic addiction treatment clinics in 16 States and the District of Columbia. Program staff collected data on approximately 1,200 patients and provided one feedback report to each participating clinic. The feedback report contained descriptive data about each program's patients and, based on case-mixed data, provided quintile ranks for each outcome. MTQAS staff met with program directors to obtain feedback about their experience participating in MTQAS and about the usefulness of the information provided to them.

Phase I Findings

Phase I resulted in good information about which outcomes form the basis for a performance monitoring system and which patient-level information is necessary to case-mix the data. Additionally, the Phase I program directors had a significant impact on the structure of Phase II, particularly the operational issues associated with MTQAS and the structure and format of the performance feedback.

Phase II: Testing the System

Phase II was a full-scale assessment of the MTQAS system. Phase II's goals were to:

MTQAS Phase II was innovative for several reasons. First, Phase II was implemented in 7 States and approximately 80 clinics, over a third of which were private. Nearly 80 percent of clinics in the States participated to some degree. Second, the study involved a partnership among providers, the States, and the research community. Clinic staff collected intake and quarterly follow-up information on all patients in narcotic addiction treatment over a period of 18 months. Each State processed the data as it was received from its clinics, and sent a data file to RTI for analysis and production of the Quarterly Performance Feedback Reports. Finally, CSAT provided technical assistance to a sample of the participating clinics whose outcomes were below expectations.

The MTQAS study design allowed for comparisons across clinics, across funding and regulatory environments, and across time. Seven States participated: Arizona, Colorado, Georgia, Massachusetts, North Carolina, Pennsylvania, and Washington. MTQAS was implemented statewide in each, in both public and private clinics. Pennsylvania had some difficulty implementing this approach because of changes in the public health care system, and eventually discontinued data collection. Reassessments were ongoing, with quarterly performance feedback reports provided to the clinics. Ten clinics also received technical assistance (TA) through CSAT to translate the MTQAS feedback into action. A standardized, on-site assessment was conducted at clinics selected for TA by an experienced narcotic addiction treatment provider to develop the TA plan. The TA delivery occurred in early 1998.

The MTQAS study fostered discussion among the narcotic addiction clinics, State offices, CSAT, NIDA, and the research community. Such communication is important to determine how the performance-based outcome information can best be used for improving the quality of narcotic addiction treatment.

In addition to producing the feedback reports, RTI staff also assessed the implementation of MTQAS through quarterly calls to State staff. RTI staff were interested in learning what challenges were encountered and how they were being addressed during the MTQAS implementation. Study staff are to visit each of the States at the conclusion of the study to meet with providers and State staff. These meetings are designed to obtain a better understanding of what it took to implement MTQAS, the usefulness of the feedback reports, including both the value and limitations of the information, and how the system might be improved.

Description of MTQAS Assessments

The performance feedback reports were based on patient-level data collected by clinic staff. Most items necessary for the performance feedback reports were items that a clinician would ask as part of a routine assessment. The experience in Phase I demonstrated that if MTQAS were to be successful, the system needed to be embedded within clinic operations. Clinics sent their data to their State office, which acted as a clearinghouse for the data. In each State office, a contact person was responsible for shipping the data to RTI on a monthly basis. The MTQAS assessment schedule for each patient included the following:

The periodic reassessments, including completing the record data, took approximately 10 minutes per patient.

The Case-Mix Adjustment Process

One important reason that outcomes may differ across clinics is that clinics serve different types of patients. Case-mix adjustment is a way of leveling the playing field when comparing outcomes across clinics that have different patient populations.

Case-mix adjustment is used to look at patient outcomes in a wide range of service settings: hospitals, nursing homes, home health agencies, ambulatory care settings, and mental health clinics. In MTQAS, this process was used to adjust for different patient populations when comparing or ranking clinics according to their patients' outcomes.

Case-mix adjustment involves a statistical analysis in which patient characteristics and baseline behaviors are used to predict patient outcomes. For each outcome, a different statistical model is used. For example, one of the outcomes is "no arrests." How well each clinic does on this outcome is estimated, while adjusting for a number of factors that may affect patients' arrest rates. These factors include patients' age, gender, race, current criminal justice status, and arrest

history.

MTQAS Indicators/Outcomes

MTQAS used a variety of outcomes as part of the performance reporting system. An important part of the study's development was determining which outcomes would be most appropriate to include in a performance feedback system. The MTQAS outcomes included self-reported drug-using behaviors and results from urine tests, as well as social functioning, physical and mental health, utilization of medical services, the patient's satisfaction with services, and retention in treatment. MTQAS staff selected these outcomes on the basis of MTQAS Phase I results, discussions with the participating State staff and advisory panel members, and a review of pertinent literature. Indicators were selected for the core MTQAS data set. In addition to this core set of outcomes, many States have opted to add other items to their data set. These include HIV-risk behaviors, social support, and the use of other drugs, such as methamphetamine and alcohol.

Application of MTQAS to Other Treatment Modalities

Although MTQAS has been developed for and is being tested in narcotic addiction treatment clinics, many of this system's attributes make it applicable to other treatment modalities. This new system provides data on measurable outcomes, tracks performance over time, and uses indicators related to the treatment process. Furthermore, some of the outcomes are associated with changes in costs and may be useful in a cost-offset model.

The indicators used in MTQAS are likely to be valuable both internally to clinic staff and externally to payers and patients. MTQAS will assist the treatment field in determining how best to implement a performance-based system and how the feedback may be used to implement changes in clinic practices.

Last Updated