A question often asked is “is my school district good?” Parents look for homes in “good” school districts, schoolteachers and administrators may want jobs in “good” school districts, and district employees may want to model their operations on the operations of other “good” school districts. Education researchers are also interested in identifying “good” school districts. There is a growing body of “district effectiveness research,” which studies the operations of school districts to identify practices that promote success in schools, such as providing sustained professional development. Frequently, however, these studies are critiqued for selecting districts unsystematically and/or for focusing on single year “snapshot” research, rather than examining change over time.
The critiques of prior district effectiveness research led Alex Bowers of Teachers College, Columbia University to ask “to what extent can school districts be [systematically] identified from all the districts in a state that significantly outperform or underperform long-term performance trends across multiple indicators?” (Bowers, 2015, p. 44). Using Ohio school districts as a sample, Bowers addressed this question in a 2010 study in which he investigated the use of hierarchical linear growth modeling for site selection, specifically, for identifying significantly unusual school districts. In a follow-up study from 2015, Bowers improved and expanded on the previous study, providing a set of recommendations for how to identify, over time, significantly outperforming school districts.
The 2015 Ohio district effectiveness study used data collected over eight academic years, 2005-2006 through 2012-2013, and included all 610 school districts in Ohio. The Ohio Department of Education collected and reported the outcome data, the Ohio Performance Index (PI) Score, in a uniform manner for all eight years, which provided a strong frame for the analysis. The Ohio PI score is a weighted average of student standardized test performance in grades 3, 8, and 10 across multiple subject tests, including mathematics, reading, writing, science and social studies. Figure 1 displays the trajectory of each Ohio school district’s PI score over twelve academic years, 2001-2002 to 2012-2013. Although the 2015 study focused on the academic years of 2005-2006 to 2012-2013, Figure 1 includes data from both Bowers’ 2010 and 2015 studies of Ohio school districts to allow for a more comprehensive presentation of the PI score trajectories.
Bowers used a two-level hierarchical linear growth model to identify school districts in Ohio that significantly outperformed or underperformed long-term performance trends. The model grouped districts with similar community demographics and followed their performance over time. Next, the model identified districts that significantly outperformed or underperformed the average for the districts in their group. Figure 2 presents the districts’ levels of performance. The districts above the top line in the plot were identified as outperforming, and districts below the bottom line in the plot were considered underperforming. The districts between the two lines were identified as at the norm, and provided a baseline for comparison. In his study, Bowers listed the names of the 15 school districts identified as outperforming and nominated these districts to be the subjects of future research. Speaking with the teachers, school administrators, and district leaders in these outperforming districts could help researchers understand how these districts consistently outperform their peers.
Interestingly, Figure 2 shows that some districts were identified as outperforming even though the Ohio Department of Education had assigned them a failing letter grade. State-assigned grades are important, but this study demonstrated the value of using a method other than state-assigned grades for guiding site selection for district effectiveness research. The hierarchical linear growth model is an improvement over previous research methods. The model controlled for district contexts, provided a baseline for comparison, and focused on change over time, presenting a more comprehensive depiction of district performance. Furthermore, Bowers’ model was the first to account for variation in school performance within districts. The model identified whether the district relied on a single high-performing school to raise its PI score or if a district improved evenly across all its schools. The model therefore highlighted districts that may have “instructional coherence” through a district-level leadership effect across schools.
In the 2015 study, Bowers demonstrated the ability of hierarchical linear growth modeling to identify outperforming and underperforming school districts. His method used data the Ohio Department of Education had already collected, suggesting that it can be employed in other states. The method’s application, however, should be adapted to suit the state’s policies. For example, the 2015 study used data collected while the “No Child Left Behind” Act was in place, but the “Every Student Succeeds” Act has recently been enacted and may necessitate adjustments to the application of Bowers’ method of site selection. That method, however, is robust and flexible and can help researchers select sites for district effectiveness studies in myriad contexts.
The full study is in Bowers, A.J. (2015). Site selection in school district research: A measure of effectiveness using hierarchical longitudinal growth models of performance. School Leadership & Management, 35(1), 39-61. doi: 10.1177/1942775116659462. Open Access Preprint: http://dx.doi.org/10.7916/D8P84B2J
The initial full study from 2010 is in Bowers, A.J. (2010). Toward addressing the issues of site selection in district effectiveness research: A 2-level hierarchical linear growth model. Educational Administration Quarterly, 46(3), 395-425. doi: 10.1177/0013161X10375271. Open Access Preprint: http://dx.doi.org/10.7916/D8W66HQ2