Loading


IEE Brief No. 8, October 1993

Benchmarking in Education: Tech Prep, a Case in Point

Morton Inger


A benchmark is a standard of excellence or achievement used to compare and measure similar things. Benchmarking is a process by which an organization compares its practices, processes, and outcomes to standards of excellence in a systematic way. Benchmarking is often called best-in-class benchmarking because establishing the standard of excellence is a search for the "best." How you define "best" becomes the benchmark, that is, the standard you will attempt to match or exceed.

The Benchmarking Process

The process has four essential steps:

1. Determining what to benchmark and establishing internal baseline data. This first step is an assessment of your needs and current status. What results do you want to achieve? By when? This step in the process requires that you start with a vision of what "doing it well" means; in other words, that you understand what you would consider superior performance. Second, you must know your own practices and processes down to the smallest detail. For example, when a student fills out a particular form, what becomes of the form, what use is made of it, why do you ask for this particular information, and what do you do with each piece of information on that form?

2. Finding out what the benchmark is. What is the standard of excellence for this particular practice or process? Who has achieved it? What is the difference between your results and the results of the best organization? For example, you may decide to benchmark L.L. Bean because they handle telephone requests faster, cheaper, and with fewer complaints than anyone else. You may not sell clothing or camping equipment, but you do have to deal with telephone requests.

3. Determining how that standard has been achieved, and comparing your organization's current practices with the way that the "best" organization does similar things. Does another organization do a similar task or process, or a part of a process, better than you do? Why does the way they do it produce better results in less time or at less cost? This analysis helps focus improvement efforts and sets expectations for what can be achieved through such efforts.

4. Deciding to make changes or improvements in order to meet or exceed the benchmark. If benchmarking is to be useful, the data must not be used for score-keeping (e.g., comparing one department to another). The purpose of the data gathering and, in fact, of the entire effort, is to take action—to allow your organization to make improvements and to think about its decisions in concrete terms.

The action plan that evolves must include a monitoring phase, specifying (1) how you will know whether your changes are succeeding, (2) what measurements you will use, (3) how you will collect the data, and (4) when. A still further step is a plan to re-evaluate the benchmark to see if it is still "the best," and, if not, to determine what is the new benchmark and to begin the effort to exceed the new one. This monitoring phase illuminates a critical aspect of benchmarking that further sets it apart from the usual way organizations try to improve: Benchmarking is not a one-time, quick-fix undertaking. It is a learning process, and it is done continuously.

Where Should You Look for Benchmark Models?

Although educators normally look only at other schools for practices to borrow, successful benchmarking involves looking outside of as well as within your own industry or field. In the business world, successful companies who have used benchmarking have borrowed ideas from companies far outside their own industry. For example, Convex Computer visited Disney World to learn about facilities management. Similarly, Xerox selected L.L. Bean as best-in-class in the warehousing and materials handling function; Federal Express for billing efficiency; and Cummins Engine for production scheduling—all outside of Xerox's industry. Corning Glass, which has a manufacturing unit designed to meet customers' emergency needs, visited best-in-class hospital emergency wards to understand how to organize teams for crises. Confining benchmarking efforts to organizations like your own—other schools or school systems—limits your goals and creativity: one's own field is so familiar that it is difficult to see it with a fresh perspective. Conversely, a search that includes organizations outside of your industry or field stimulates your creativity because it forces you out of habitual patterns of thought and ways of doing things.

Most organizations, whether schools or businesses, possess incomplete information about what is the best practice. That is because most improvement targets are set internally and are based on past performance. Ordinarily, for example, organizations compare one internal department to another, or this year's performance to last year's: Which department produces the most sales or reduces its expenses the most? Which one made the biggest increase in sales over last year's results? Which fourth-grade teacher raised the reading score of his or her students the most? This type of comparison avoids the question of whether any of the departments are as effective as they could be. Best-in-class benchmarking lets organizations break free of these self-imposed limits on performance.

Applying Benchmarking to Education

The idea of performance measures for practices is relatively novel in education. The field talks about "exemplary" or "best" practice, but how does anyone know if a practice is in fact "best"? We usually do not know because we lack plausible metrics for comparing practices. The field needs professionally based standards of best organizational, curricular, and pedagogic practice to complement outcome standards. Evaluations of practices can give educators the ability to diagnose the reasons for limited or poor learning outcomes and reveal what processes and products have led, elsewhere, to better outcomes.

In education we tend to borrow whole programs, but benchmarking forces borrowers to assemble their program from best-in-class instances wherever they exist, as Xerox borrowed best practices for different functions from several different companies. No one organization conducts all functions at best-in-class levels; they are good at some things but not at others. This point is particularly key to innovations as complicated as integrating vocational and academic education, which involve many organizational, curricular, and pedagogic functions.

Benchmarking bridges the gap between two different approaches to change in education. One, often called a "top-down" approach, involves the development of a generic model and the dissemination of it to educators at the local level with the expectation that it will be adopted because it is conceptually and practically sound. This approach does provide quality standards, but local conditions always affect how knowledge will be used, and local educators have difficulty implementing a model that was not tested with their particular situation in mind.

In the second approach, often called a "bottom-up" approach, local practitioners create innovations and invent solutions to cope with their particular problems. Although this approach does deal with the local situation, it lacks quality standards and becomes relevant only to a momentary problem. What is worse, it leaves each locality isolated, without the opportunity to learn from the successes and failures of others.

What is needed is a mechanism for the continuous evolution and improvement of practice. Many consider the networks of schools carrying out innovative programs established for the exchange of experiences to be such a mechanism. But these networks may perpetuate the diffusion of random practices that do not meet any quality standards. In evaluating whether to adopt a program and how best to implement it, educators need to know whether the organizational, pedagogical, and curricular practices are best-in-class and meet standards of quality. Benchmarking provides such an evaluation strategy because it allows for the diffusion of locally determined best practices that have been measured against other local practices. Borrowed ideas must be adapted to the circumstances of the borrower. But the basis for the adaptation is that the borrower—a school, for example—has established measures of its own practices and has compared these to the best practices elsewhere.

Benchmarking: Focusing on Outcomes

Thus far, this paper has emphasized the benchmarking of processes, but equally important is benchmarking's focus on outcomes rather than on the efforts to produce those outcomes. To illustrate the benefits of this focus, take the hypothetical case of an adult literacy program. Too often, organizations believe that they have made improvements because their expenditures in a program have doubled. Benchmarking's focus on outcomes would force an organization to ask how many adults who entered the program learned to read this year rather than how much was spent on adult literacy this year. Thus, the focus on outcomes gives an organization data on expenditures per output.

There are, of course, a variety of kinds of outcome benchmarks that can be used, and the following two examples from Oregon and North Carolina will illustrate that variety. The Oregon State Board of Education uses the following outcomes as benchmarks of its progress.

  • percent of high school students will graduate from high school by the year 2000
  • percent of high school students will be enrolled in the tech prep program by the year 2010
  • percent of high school students will complete at least one year of postsecondary education or training by the year 2000
  • percent of high school students will be enrolled in structured work experiences by the year 2000

Oregon has also established a number of other benchmark measures, starting in 1970 and every ten years through 2010. The benchmark measures for basic academic skills are:

  • Percentage of high school graduates proficient in at least one language other than English
  • Percentage of students who achieve basic skill levels in third grade, fifth grade, eighth grade, and eleventh grade

The benchmark beasures for comparative performance are:

  • Ranking on national assessments in geography, history, math, reading, science, and writing in fourth grade, eighth grade, and twelfth grade
  • Ranking on international reading, math, and science assessments

Before setting these benchmarks, the Board had to determine the current benchmarks—where they are now on these measures. By setting measurable standards, the Board avoids falling into the trap of relying on often-misleading anecdotes and testimonials ("The program is going great!").

The Richmond County (North Carolina) Schools' tech prep program uses the following outcomes as benchmarks of its progress.

  • The percentage, each year, of high school students taking algebra I and II and chemistry
  • The dropout rate each year
  • What students intend to do when they graduate, and how this changes each year
  • College board computerized adaptive test scores in reading, writing, mathematics, and algebra—and how these compare to the average scores of the students in college prep and in basic studies each year

In addition, Richmond County uses as benchmarks a follow-up survey of graduates as to whether they are attending a four-year college, attending community/junior college, in the military, or working/other. All of these benchmarks provide Richmond County concrete evidence about the outcomes of their efforts.

Applying Benchmarking to Tech Prep

Benchmarking can be applied to entire programs or, more likely, to the processes involved in the development and implementation of a program. Take a tech prep program, for example. Tech prep, an organizational alternative to the traditional secondary school college prep and general education program, is a planned sequence of integrated academic and technical courses embracing the last two years of high school and the first two years of college. A high school and a community college often formally coordinate their programs. Students who complete the sequence receive an associate degree and are qualified to enter a variety of technical careers. Because the emphasis is on continuity in education and learning how to learn, its graduates are also prepared for advanced education, and many go on to earn a four-year baccalaureate degree.

Tech prep has four general program components:

  • Development of an articulated curriculum
  • Program assessment and improvement
  • Career guidance
  • Marketing

(These components are not sequential; that is, tech prep programs reflect a mixture of each component operating at various stages of development.)

These four components include, of course, many important activities and processes, such as setting local priorities, staff development, and business and industry collaboration. In selecting an aspect of tech prep to benchmark, program staff would list the processes within each component and identify those that (1) are the most critical for the success of the program or (2) are discouraging enrollment, causing delays, leading to complaints, or otherwise interfering with the mission and goals of the program.

We can see how benchmarking can be applied to education by looking at two of tech prep's components—articulation and marketing—and examining how one would benchmark processes involved in these components.

Articulation in Tech Prep

Articulation is a central concept of tech prep. When a curriculum is articulated, secondary and postsecondary courses are coordinated and complementary to each other. As a result, the total sequence of courses becomes a functional whole, and completion of the program communicates to potential employers that the participant has demonstrated needed skills and abilities.

Articulation involves several important aspects and processes. These include:

  • The development of articulation agreements between the high school and the postsecondary institution
  • Vision and goal setting
  • System-wide policy making
  • Administration, including budgeting and project management
  • Program planning and development
  • Curriculum review, development, and integration
  • Staff orientation, training, and development
  • Program evaluation and outcomes assessment

Understanding what makes these processes work or fail is crucial to the success of tech prep.

The development of articulation agreements is essentially a negotiation process. To benchmark that process, program staff would identify the best-in-class for this process. One example outside of the education world would be strategic planners at companies who have been involved in mergers. Next, they would decide what metrics to use to assess the performance for this process. One metric could be how long it takes to resolve an issue to the satisfaction of the parties involved.

Vision and goal setting are key activities at many kinds of organizations. Nonprofit, care-giving organizations, for example, have to reassess their goals often in response to changing conditions. And as competitive pressures change, businesses have to reassess their vision and goals. To benchmark this process, program staff would have to identify organizations that are best-in-class for this process and determine what metrics they would use to assess their performance. For vision and goal setting, the metrics would perhaps be less clear-cut than for some other processes, such as administration and project management, but it would be just as essential to have the metrics. One might be how long it takes an organization to develop a corporate vision and a workable set of goals. Another series of metrics might be who and how many of the affected parties are involved in the goal-setting, how readily the affected parties "buy into" the vision and goals, and what the feedback mechanisms are to rethink the vision and the goals.

For each of these processes—the development of the articulation agreements and vision and goal setting—the program staff would then analyze how the best-in-class organizations conduct this process and why their approach leads to the high level of performance or satisfaction in the organizations being examined. Ultimately, staff would decide on an action plan for improving these processes, and this would be followed by monitoring and re-evaluation phases.

Marketing Tech Prep

To achieve its goals, tech prep requires several kinds of partnerships: between educators and businesses, between academic and technical faculties, and between secondary schools and postsecondary institutions. These partnerships and the concept that drives them are new and not readily accepted or understood. For example, tech prep is often seen as simply a continuation of vocational education, and negative attitudes toward vocational education constitute a significant barrier to the adoption and implementation of tech prep. Therefore, effective marketing plans are as critical to the success of tech prep as are the instructional programs.

Tech prep can be thought of as a new product or service. The concept and its benefits must be marketed internally to college and secondary school administrators and guidance counselors, boards of trustees, school board members, college faculty, and secondary vocational and academic faculty; and externally to local businesses, students, parents, local community groups, government agencies, and the general public. Any tech prep initiative, therefore, must include specific marketing plans for both internal and external audiences.

The marketing effort could clearly benefit from looking at how "the best" organizations conduct the various marketing processes, for example:

  • Businesses that introduce new products or services to the public
  • Non-profit organizations and businesses that introduce new ideas and concepts to internal audiences
  • Schools that have successfully adopted educational reforms
  • Schools that have begun to introduce tech prep

Marketing can be disaggregated into the following processes, each of which could have a different benchmark company or organization.

  • Identifying the target market
  • Developing a marketing plan
  • Developing promotional materials
  • Developing and implementing an advertising campaign
  • Gathering data—conducting surveys to determine audience response to the concept, the product, the service, and the message
  • Establishing mechanisms for receiving feedback—customer complaints, for example, or customer confusion or misapprehension about the concept, its implementation or impact
  • Brainstorming techniques to develop marketing ideas

With the massive changes required by tech prep, overturning traditions of curriculum, teaching methods, and even goals, tech prep has many barriers to overcome. While marketing can go a long way to address these barriers, benchmarking's emphasis on continuous improvement offers a key to gaining support for tech prep. For one thing, evaluation and benchmarking of practices would identify problems early enough to solve them before they hurt the program and lead to negative outcomes. Equally important, continuous improvement in staff performance, student performance, and employer and community college cooperation could be the most effective methods of overcoming barriers to the acceptance of tech prep.

Why Benchmark?

The objective is to aim high, to get results that other approaches have not been able to achieve. The results from benchmarking are far greater than those achieved by informal approaches because benchmarking provides focused and immediately useful data, not just anecdotal information, intuition, or opinion; creates a culture that values continuous improvement to achieve excellence; enhances creativity; encourages staff to be open to ideas even though they were "not invented here"; and, increases sensitivity to changes in the external environment.

Benchmarking provides an opportunity for an organization to see if it has met its own expectations, and to learn why or why not. Benchmarks enable an organization to think about its current operations and its future in concrete terms; set measurable goals—benchmarks to aim at; and, consider the costs and benefits of pursuing or not pursuing a given course of action.

Further, benchmarking has a strong dissemination potential. Others can see what you did, why you did it, what the results were, and what produced the results.

References

Altany, D. (1991, July 15). Share and share alike. Industry Week, pp. 12–17.

AT&T. (1992). Benchmarking: Focus on world-class practices. Indianapolis, IN: Author.

Bragg, D. D. (1992). Implementing Tech Prep: A guide to planning a quality initiative. Berkeley, CA: National Center for Research in Vocational Education, University of California, Berkeley.

Dornsife, C. V. (1992). Beyond articulation: The development of Tech Prep programs. Berkeley, CA: National Center for Research in Vocational Education, University of California, Berkeley.

First find your bench. (1991, May 11). The Economist, p.72.

Flaxman, E. (1993). Unpublished manuscript.

Xerox Corporation. (1990). Leadership through quality: A guide to benchmarking in Xerox. Stamford, CT: Author.

Mittelstaedt, R. E., Jr. (1992, Summer). Benchmarking: How to learn from best-in-class practices. National Productivity Review, 11 (3), 301–315.

Oregon Progress Board. (1991). Oregon benchmarks: Setting measurable standards for progress (Report to the 1991 Oregon Legislature). Salem, OR: Author.

Richmond County (NC) Schools. [Tech Prep results, 1986-1992]. Unpublished raw data.

Vaziri, H. K. (1992, October). Using competitive benchmarking to set goals. Quality Progress, pp. 81–85.

This publication was developed with support from the Ford Foundation and the National Center for Research in Vocational Education (NCRVE), funded by the Office of Vocational and Adult Education, U.S. Department of Education, authorized by the Carl D. Perkins Vocational Education Act.

 

 


 

Institute on Education and the Economy, Teachers College, Columbia University
525 West 120th Street, Box 174, New York, NY 10027
Phone: (212) 678-3091 | Fax: (212) 678-3699 | iee@columbia.edu