Number 4 / June 1992

THE EFFECTIVENESS OF NEW YORK CITY'S CAREER MAGNET SCHOOLS

Robert L. Crain, Amy Heebner, and Yiu-Pong Si

This Brief summarizes a report on the effects of a unique, large-scale, random assignment experiment in New York City's career magnet high schools. The schools that were studied accept half of their enrollment through a random-assignment lottery conducted by the Educational Testing Service (ETS). The random assignment of students, with a comparable group randomly excluded, allowed for an evaluation based on a rigorous experimental design. This is the only certain methodology for determining whether students who attend these schools benefit from doing so because of the school, rather than because of the characteristics that they bring to the school.

The report, the first of three, compares the ninth-grade performance of students who were randomly admitted to the career magnets with those who were randomly not offered seats in these schools. The research measured the effect of career magnets on students of average and low reading ability whose academic records were not strong enough to gain them admission to a selective magnet school. The next two reports will present data on these same students as they go through their sophomore and junior years.

New York City's Magnet Schools

More than one-third of all New York City high school students attend one of the career magnet schools, and nearly every high school has a career magnet program. But career magnets are only one of several types of magnet high schools in New York. The city's program is so extensive that most high school students attend magnets of one kind or another. To maximize the number of applicants to magnets, the school system offers every student the opportunity to choose their high school from the full range of New York City high schools.

All eighth graders are required to complete an application for high school, even to attend their neighborhood school. Each student can make up to eight choices. They are also given two opportunities during the year to change their minds. A surprisingly large number choose to attend a school other than their neighborhood high school, and nearly 60 percent get their first choice. Three-fourths of the students who apply to magnet schools are accepted.

New York City's career magnet high schools are not vocational schools; they combine career preparation with traditional college preparation. Some of the career magnets are in their own dedicated buildings; others are housed within regular neighborhood comprehensive schools as schools-within-a-school.

The Random Assignment Experiment

Originally, New York's career magnets were selective. But the random assignment process was developed to expand the opportunity to attend these schools to students who would ordinarily not be selected. The schools are now permitted to select only half of their students; the other half are randomly assigned.

To expand the opportunity even further, both the school-selected and the randomly-assigned groups must include fixed percentages of students with low, average, and high reading scores: sixteen percent from students with reading scores in the top sixth; sixteen percent from students with the lowest or missing reading scores; and the remainder from the large middle group of readers. (Students do not know whether they have been randomly-assigned or school-selected.)

Analysis

The analysis is based on results for 3,272 students with average reading scores and 986 students whose reading tests put them in the bottom 16 percent, or who had not taken the required reading test. (For technical reasons, we were unable to construct a legitimate experimental design for the top reading group). Ninety-one different school programs are included.

Criteria

We measured the impact of career magnets on five ninth-grade performance criteria:

• percentage of students who entered high school after the eighth grade;
• gain in reading scores;
• attendance;
• number of credits earned toward graduation; and
• percentage passing the advanced ("Regents") mathematics test.

A. The results for students randomly assigned to career magnets, compared to the results for students who lost the lottery. (Note: For simplicity, we use the terms "lottery winners" and "lottery losers" to refer to students who were randomly selected or randomly not selected. The New York City school system does not refer to the random assignment process as a lottery; nor does it call the students winners or losers.)

1. High school enrollment.

• Of the lottery winners with average reading scores, 89 percent enrolled in high school the following year, compared to 86 percent of similar students who were not offered seats.
• Among students with low or missing reading scores, 83 percent of the lottery winners enrolled in high school, compared to only 77 percent of similar students who were not randomly selected.

Not all of the students who do not enroll in high school are dropouts. Some lottery losers may enroll in a private high school or attend ninth grade in their local junior high, which would result in a slight overestimation of the enrollment effect of career magnets.

2. Gain in reading scores.

• After one year in the career magnet program, students with average reading scores showed a sizeable improvement in their reading scores. In fact, they gained over twice as much in reading as did similar students in comprehensive programs.
• Students with low or missing reading scores showed improvements in reading scores, but not as much as their counterpart lottery losers.

3. Attendance.

• For average readers, there was no statistically significant difference between lottery winners and losers on this dimension.
• Lottery winners with low reading scores had significantly more high school absences than comparable lottery losers. Many students who attend magnet schools have to travel long distances to other boroughs. This travel is not on regular school buses; students have to get themselves to school on public subways and buses--a situation conducive to "stopping off." This is a dropout-prone group; though they manifest this tendency in increased absences, they have not dropped out to the extent that lottery losers have.

4. Credits earned toward graduation.

• Lottery winners with average reading scores earned significantly more credits toward graduation than those who did not win the lottery.
• Lottery winners with low reading scores earned fewer credits than the lottery losers.

5. Passing advanced ("Regents") math test.

• Average readers who won the lottery did no better than the lottery losers.
• Poor readers who won the lottery showed a substantial, significant gain over the lottery losers. This is a clear indication that poor readers have been put into a more demanding curriculum than they would have received in a comprehensive school, where they may not even have taken the test.

B. Total, or free-standing, magnets compared with magnets located within comprehensive schools.

One can reasonably expect student outcomes to be better in free-standing magnets, where the whole school is devoted to a single career program or a cluster of career programs. A free-standing magnet can develop a strong career focus because it has its own principal and does not have to conform to school-wide comprehensive school policies or compete with the school's comprehensive program for resources.

1. High school enrollment.

• While eighth graders who are admitted to either free-standing or school-within-school magnets are more likely to enroll in high school than the lottery losers, the effect of career magnet admission on enrolling is much stronger in the free-standing career magnets.
• Average readers who were assigned to free-standing magnets were slightly more likely to enroll in high school than those. who were assigned to magnets located within comprehensive schools.
• The effect was especially strong for students with low reading scores. An extraordinarily high 90 percent of the poor readers who were assigned to free-standing career magnets enrolled in high school. In contrast, 80 percent of the poor readers who were assigned to magnets located within comprehensive schools enrolled in high school, virtually the same enrollment percentage as the lottery losers.

2. Gain in reading scores.

• In both kinds of magnets, students with average reading ability who won the lottery showed improvements in reading greater than that of lottery losers, but their gain was larger in magnets located within comprehensive schools.
• In contrast, students with low reading scores outperformed the lottery losers only in the free-standing career magnets. In magnets located within comprehensive schools, lottery winners with poor reading scores actually did less well than the lottery losers.

3. Attendance.

• Both lottery winners and lottery losers were absent more in high school than they had been in middle school. The increase in absenteeism was greater for lottery winners with low reading scores, and was slightly greater in the total career magnets than in the ones located within comprehensive schools.

A further study, not yet complete, indicates that some of the low readers feel less welcome in total career magnets. That plus the problem of traveling to school by themselves on subways, discussed above, may explain the increased absenteeism of students with low reading ability in these schools.

4. Credits earned toward graduation.

• Lottery winners with average reading ability earned significantly more credits than the lottery losers, but only in free-standing career magnets.
• Whether they attend free-standing or within-school magnets, poor readers earned fewer credits than similar students who lost the lottery.

5. Passing advanced ("Regents") math test.

Students in the free-standing career magnets appear to be taking a more advanced curriculum than their counterparts in magnets located within comprehensive schools.

• Average readers who won the lottery outperformed the lottery losers in this dimension only in the free-standing magnets.
• In both kinds of career magnets, poor readers who won the lottery outperformed the lottery losers. Especially in free-standing magnets, poor readers are much more likely to take and pass the advanced math test than their counterpart lottery losers.

Summary of Results

We can safely conclude that New York City's career magnets significantly benefit students. Career magnets attract potential dropouts to high school and expose them to a more demanding curriculum. For students with low reading scores, the results are mixed. They are much more likely to make the transition to high school, but their absentee rates go up higher than similar students who do not gain admission to the magnets. More of them pass an advanced math test, but those who attend magnets within comprehensive schools fall behind the lottery losers in reading.

Many of the effects of career magnet programs are significantly large:

• There was an extraordinarily high increase in high school enrollment of low readers who were assigned to free-standing magnets. They were much more likely to go on to high school than similar students who were not assigned to a magnet.
• In free-standing career magnets, low readers are two to four times more likely to pass the Regents math test than similar students in comprehensive schools.
• Average readers in magnet schools, whether in free-standing magnets or magnets-within-schools, increase their reading skills more than twice as much as comparable students in regular comprehensive schools.
• Average readers in free-standing magnets earn one-fourth more course credits than average students in comprehensive schools. Since lack of credits is a main cause of dropout, these students have considerably increased their chances of getting a diploma.
• Two negative effects are also significant. In both reading ability and the number of credits earned, poor readers who won the lottery did less well than comparable students who lost the lottery.

Discussion

Among the career magnets, there are significant differences in their effectiveness. Since each program is a separate experiment, we looked for common features and examined their effects on student outcomes. It seems that when career magnets work, it is because of their career focus. The programs that produce the most positive student outcomes are the ones that take their career commitment most seriously.

In both free-standing magnets and those within comprehensive schools, the most effective career magnet programs:

• separate their students from the regular curriculum and from students who do not participate in the program's career focus and
• provide students with special teaching and equipment.

There is also some evidence that career magnets that provide substantially more job placement activities are more effective, particularly in reaching poor readers.

These conclusions are based on the following findings:

1. In schools that offer magnet students many specialized classes, the dropout rate in the transition from middle school to high school decreased.

• In career magnet programs that had many separate classes, there was a very high increase in high school enrollment of poor readers who won the lottery compared to similar students who lost the lottery.
• In contrast, in career magnets that had fewer separate classes, poor readers who won the lottery were actually less likely to enroll in high school than the lottery losers.

Why do the career magnet programs with an emphasis on separate classes have such an impact on high school enrollment for poor readers? Schools with many separate and specialized classes gain a reputation for a sharper career focus. Students who search out a program with a clear identity or career focus may often be students with doubts about going to high school; for these students, winning or losing the lottery seems often to mean the difference between going to high school and dropping out.

2. Schools that provide many special sections of classes for magnet students are more successful at educating average readers than schools that provide few such classes.

• Lottery winners with average reading scores showed more improvement in reading than did comparable lottery losers. This is true only where the program separates its students into many specialized course sections. Average readers in programs with many special sections of classes improved their reading scores at over twice the rate of lottery losers in comprehensive schools. (Results for poor readers are similar but not statistically significant.)

There are several possible explanations for these results. Since half the students in these sections may be hand-picked, highly-motivated teachers may volunteer to teach them. It may also be that the school, being committed to producing a unique product with its student body, is striving harder and demanding more from each class. And, in programs where there is a strong commitment to a theme or a career, the lottery winners themselves may be more motivated. This effect would then be magnified, since the concentration of motivated students would create a favorable classroom climate.

3. Career magnet schools that offer specialized classes designed for only the students in the career magnet--that is, teaching the magnet students different subjects--produce favorable results for average readers but not for poor readers. To test these effects, programs that provided over eight career-specific one-semester courses in four years were compared with those that had fewer than eight.

• Average readers in programs that offered more such classes earned more credits toward graduation than magnet students in programs that offered fewer such classes.
• Poor readers, however, did not obtain normal amounts of course credit in the programs with more special classes. In contrast, poor readers in the programs that offered fewer special classes earned more credits.
• The increasing absenteeism of poor readers seems to occur only in programs that have more career-specific classes. Free-standing career magnets have more career-specific classes, and free-standing magnets frequently require a longer trip to school than the ones located within comprehensive schools.

4. Schools that provide hands-on computer experience may be more successful in motivating students to attend school.

• For students who are average readers, the presence of computer labs dramatically reduces absenteeism.
• For students with poor reading scores, this pattern does not appear, perhaps because computers still involve manipulation of words and poor readers are weaker in verbal skills.

5. Schools that offer more placement efforts for graduates reduce dropout rates.

• Poor readers have a much-reduced dropout rate (between middle school and ninth grade) in the career magnets that provide substantially more job placement activities.

In short, career magnets with a stronger career focus (1) motivate and improve the performance of students who are able to keep up with the higher work demands; (2) encourage students with low reading scores to enroll in high school; and (3) expose students of all levels of reading ability to a more demanding curriculum.

We note two caveats: The first is that educators in the career magnets are divided on how to deal with weaker students. Some feel that career magnets are harmed by the presence of these students. Other teachers in these schools believe it is possible to solve the problem by developing stronger remedial services. But career magnet programs do not get a larger budget than comprehensive schools and generally feel they need to spend whatever funds they have to provide special resources for their career focus. We do find that certain types of efforts are successful with students with low reading scores. For example, schools with strong employment placement programs are effective at reducing dropout rates among poor readers. More placement efforts could be adopted, but, given the limited budgets, this would involve trade-offs with other activities.

Second, the fact that New York's career magnets provide an effective secondary school education does not tell us whether the entire group of high school students in New York, including those who are academically strong enough to be selected by magnet programs and those who are left behind in comprehensive schools, have experienced a net benefit. This study did not measure effects on all students in New York.

Nonetheless, the positive results in the career magnets are impressive. Students in a predominantly minority urban school system whose records would not have been good enough to gain admission to a selective school made substantial progress. If further evaluations replicate these findings, New York City's experiment clearly shows, in an unbiased way, that:

• career magnet programs lead to higher achievement and lower dropout rates for inner-city youth;
• the random assignment process provides substantial opportunities for students who are usually turned away by magnet school admission committees.

In these important respects, New York City's career magnet program can be a model for the nation's schools.