​Performance Outcomes and Quality Improvement (POQI): History & Legislation - Introduction

Table of Contents

1. History of DMH Performance Outcomes

2. Performance Outcome Legislation

History of DMH Performance Outcomes

Over the past several years, there has been a growing trend, both nationally and in California, towards the measurement of consumer outcomes and cost effectiveness in mental heath service systems. California's Bronzan-McCorquodale Act of 1991, also known as "Realignment", specifically mandated that counties report data on established performance measures to the director of mental health.

There has also been a trend towards greater involvement of consumers in the services they receive and in the operations of the mental health service system. In fact, Realignment calls for consumer empowerment in a variety of ways and in a variety of programs. For example, "The program (residential care) should encourage the participation of the clients in the daily operation of the setting in development of treatment and rehabilitation planning and evaluation." Other passages relative to outpatient services, call for consumer participation as an integral part of assessment, intervention, and evaluation. Finally, Realignment specifically requires that each county develop a local managed mental health care plan that includes, among others, a beneficiary satisfaction component.

The interest in measuring consumer outcomes and consumer evaluation of the mental health system did not begin with this legislation. Rather, it is the result of an evolving mental health service delivery culture that preceded any legislation and continues to develop. This culture, involving consumer empowerment, consumer involvement, and consumer perceptions of care, among other features, is a consumer-centered system. A consumer-centered system involves the consumer, not just as a recipient of services, but central to the mental health system and his or her own provision of services.

Working within this model, the California Department of Mental Health (DMH), in a cooperative relationship with the California Mental Health Planning Council (CMHPC) and the California Mental Health Directors Association (CMHDA), adopted a performance outcome measurement system that brings together the views of the clinical team, consumer, and where appropriate, family members of the consumer. Not only is the consumer involved in providing factual information, but is also providing his or her perception of the system and services with which he or she is involved.

This outcome measurement model is supported in the literature. The research indicates that consumer satisfaction has a role in evaluating primary health care and explaining health-related behavior. PACs (1983) noted, "Patient satisfaction can serve as an outcome measure of the quality of health care and provides a consumer perspective that can contribute to a complete, balanced evaluation of the structure, process, and outcome of services." Further, "patient satisfaction should be considered as one of several sources of information for program planning and evaluation."

Prior to the development of the California Performance Outcome System, the decision to collect outcome and consumer satisfaction information was at the discretion of each of the 59 local mental health entities. The Bronzan - McCorquodale Act, in its mandate to establish a consumer outcome system for California's Mental Health System, centralized the process with the intent of developing statewide uniformity: all of California's 59 local mental health entities would be required to use the same surveys for collecting performance outcome data.

While the Bronzan-McCorquodale Act mandates a consumer-based performance outcome information system, it specifically identifies a public mental health service system for a target population of persons who are seriously mentally ill which is "client-centered, culturally competent, and fully accountable." The Act requires the development of a uniform comprehensive statewide consumer-based information system that includes performance outcome measures. Counties are to report data on performance measures to DMH which, in turn, is to make that data available to the California Legislature, local mental health boards and commissions, CMHPC, and national stakeholders.

The first attempt at collecting performance outcome data was based on a custom designed survey, the Adult Performance Outcome Survey (APOS), developed by DMH in conjunction with county and consumer representatives. This custom survey was designed to be administered to a sample of severely mentally ill (SMI) adult consumers at a beginning time, 6 months later, and then again one year later. Several issues that emerged during the study include the difficulties of maintaining a representative sample and the lack of comparability of the data.

Based upon the results from APOS, the California Mental Health Director's Association (CMHDA), California Mental Health Planning Council (CMHDA), and DMH established several criteria for the Children/Youth Performance Outcome System which include:

• Data should be useful to clinicians for treatment planning
• Data should be useful to counties for quality management purposes
• Data must meet the requirements of the state for performance outcome data
• Data should be comparable with data from other states/entities

The model of using a battery of widely recognized assessment instruments to meet these criteria was developed by Abram Rosenblatt of the University of California, San Francisco Child Services Research Group, and a past contract evaluator for the Systems of Care counties in association with Don Kingdon of Ventura County and Norm Wyman of Santa Cruz county. They submitted a recommendation, which was adopted by the CMHDA, that a model similar to that used in System of Care counties be used statewide for collecting data related to performance outcomes. This proposal was agreed upon by the CMHPC, which would meet oversight responsibilities, and was accepted by DMH.

The 1999-2002 Children & Youth Performance Outcome System was developed based on the Children's System of Care evaluation model. To come into compliance with the legislation, CMHPC and DMH committed to using this system as the first step in an evolutionary process. The system was evaluated and modified in planned intervals, taking into account the effort required to modify and design new county systems and procedures, making the system more effective and informative for all constituency groups while seeking to minimize cost. For more information on the 1999-2002 Children and Youth Performance Outcome System [click here].

The California Adult Performance Outcome System (APOS) was implemented on July 1, 1999. APOS was developed in collaboration with the California Mental Health Directors Association, California Mental Health Planning Council, and the California Department of Mental Health. APOS was intended to ensure accountability for the expenditure of public behavioral healthcare dollars and to ensure high quality and effective care to adult (ages 18 through 59) mental health consumers. To learn more about the1999-2002 Adult Performance Outcome System [click here].

During fiscal year 2003-04, California focused efforts towards improving its data collection methodology for each of the Performance Outcomes Systems. Until 2003, the California Department of Mental Health (DMH) used a longitudinal method to collect consumer-specific performance outcome survey data. That is, mental health consumers were surveyed with regard to their perception of care upon entry into the mental health system (i.e., intake), on an annual basis, and/or at discharge from services. Through the examination of service utilization patterns, DMH determined that the collection of information through a point-in-time survey process would yield as much useful data as was collected using the legacy method. The point-in-time method targets all consumers receiving face-to-face mental health services, case-management, day treatment and medication services from county-operated and contract organization providers during a two-week sampling period semi-annually.

Consistent with its commitment to the quality and improvement process, DMH revised the performance outcome data collection instruments to ensure that quality indicators of specific relevance to California’s public mental health system would be measured, and to ensure data comparability with national quality benchmarks. Through the assistance of a Performance Outcomes Steering Committee, with representation from the California Mental Health Planning Council (CMHPC), California Mental Health Directors Association (CMHDA), county program management, county evaluation/quality improvement personnel, and consumer and family members, DMH adopted the most recent version of the national Mental Health Statistics Improvement Program (MHSIP) Consumer Survey, as well as the Youth Services Survey (YSS) for Youth and Youth Services Survey for Families (YSS-F). Additionally, Performance Outcomes Steering Committee members recognized the importance of collecting quality of life data as a mental health outcome for adults, as well as older adults, and advocated for the development of two somewhat different Quality of Life (QOL) measures, tailored to the specific needs of each population. Collectively, these instruments assess consumers’ perceptions of quality and outcomes of care, and are currently being used for broad-based evaluation of California’s community-based mental health services. All instruments are currently available in English and Spanish, and translations of the surveys into other languages (Tagalog, Chinese, Korean, etc.) are underway in order to accommodate the language needs of California’s diverse mental health consumer population.

An additional improvement was the use of on-line, internet-based data capture methods that allow direct key-pad data entry and provides a paper-form scanning and verification option for larger volume direct data submission. This new data entry and submission technology provides flexibility for system users, while increasing data uniformity and accuracy.

Data that are transferred to DMH via the new technology are housed in a single database, and are therefore quickly available for centralized data analysis, and for return to counties for local processing. Quick data analytic turn-around time allows DMH, other oversight entities, and interested stakeholders to maintain a “pulse” on the mental health system’s performance, and to make administrative decisions/apply quality improvement strategies in a timely manner.

DMH continues to perceive the performance outcomes measurement process as being tied to a continuous quality improvement process and, consequently, data elements and methods of evaluation are necessarily subject to change. As such, this new technology provides low-cost flexibility to accommodate changes over time.

DMH envisions applicability of the system to numerous future data capture endeavors. These include performance outcome indicators derived through national stakeholder processes (e.g., requirements for Federal Block Grant Performance Partnership reporting), collaborative performance measurement activities between DMH and other State departments, (e.g., Department of Rehabilitation, Department of Alcohol and Drug Programs, etc.) and special studies designed to evaluate specialty mental health programs and/or integrated system services for targeted mental health populations (e.g., Children’s System of Care, Older Adult System of Care, etc.).

For additional information regarding the collection of DMH performance outcomes data unit, please contact staff from the Performance Outcomes and Quality Improvement Unit.