Data Analytics and Decision-making in Education

In the recently published chapter in the book Handbook of Contemporary Education Economics (Open Access Version), Tommaso Agasisti and Alex J. Bowers outline the importance of using data that are increasingly available to guide decision-making in education institutions, ranging from the federal and state policies at the system level to pedagogical and instructional decisions in schools and classrooms.

Distinction between Educational Data Mining (EDM), Learning Analytics (LA) and Academic Analytics (AcAn)

Educational Data Mining (EDM) applies data mining techniques to data about the determinants of the learning process (e.g., data collected from the knowledge management systems of online learning) to understand the learning patterns and digital trails. Learning Analytics (LA) focuses on using data mining models and other advanced techniques to explore the determinants of student achievement, the output of the learning process, to better inform instructional decisions. Academic Analytics (AcAn) uses statistical analysis and predictive modeling to understand the organizational-level determinants with the purpose of providing guidance for principals and other school leaders on decisions related to the operations and managerial processes of institutions. The main differences of these three approaches are summarized in Table 1.

Table 1. Main differences between educational data mining, learning and academic analytics – a classification

Source: Authors’ elaborations, originally inspired by Romero & Ventura (2010) and Siemens & Long (2011).

Agasisti and Bowers point out that the classification of these three approaches must “be intended as provisional, indicative and not prescriptive”, and may need to be revised as the literature evolves. In addition, the three approaches are difficult to separate in practice as they share the analytics techniques, research questions, policy implications and the “loop of data” that describes the process of using quantitative data to guide decision-making.

As shown in Figure 1, the first step “Collection and acquisition” involves identifying and collecting the relevant datasets to use. Then datasets constructed within institutions (“Storage”) and are cleaned through reconstruction and wrangling (“Cleaning”). Multiple datasets may need to be integrated together in order to explore the research problems from a more comprehensive perspective (“Integration”). The next step is to apply the statistical, econometric and data mining techniques to the datasets (“Analysis”). The patterns and results identified from the analysis are then synthesized and visualized in order to facilitate the transition to informing knowledge, such as through policy-makers, school and district leaders and teachers (“Representation and Visualization”). Lastly, the results of the analysis can be used to inform the instructional and managerial decisions (“Action”). The “loop of data” only illustrates a typical process of data use. Note that in practice it can be iterative and recursive.

Agasisti and Bowers recommend that an education data scientist can play a key role to bridge the data analysts and educational practitioners in the “loop of data”, as the education data scientist “owns the technical skills to collect, analyze and use quantitative data, and at the same time the managerial and communication skills to interact with decision-makers and managers at the school level to individuate good ways of using the information in the practical way of improving practices and initiatives”.

Data and tools of data analysis (and analytics) in educational arena

Agasisti and Bowers introduce multiple types of data analytics and tools that are used in the educational arena. First, at the policy level for implementing, managing and evaluating policy interventions, studies based on large-scale international assessment data on student achievement, such as PISA and TIMSS, are particularly informative, as they allow researchers to explore which practices and policies are working in different countries holding other factors constant. There is also a growing number of rigorous “field experiments” that aim to provide evidence on “what works” to improve student learning and school performance. Second, at the level of school practice for principals to improve school operations and for teachers to improve teaching effectiveness, the focus of research is on data-driven decision-making in K-12 schools. Data on student achievement, school administration and digital learning environments are collected and analyzed to help teachers and principals identify the problems and understand the patterns. Research on data use has shown that teacher’s and leader’s data literacy and assessment literacy determines whether they can successfully use data to inform their decision-making.

To illustrate how data analyses and analytics can improve student performance, the authors describe four examples. First, for the analyses of system-level determinants of instructional results, PIRLS (Progress in International Reading Literacy Study) , TIMSS (Trends in International Mathematics and Science Study), and PISA (Programme for International Student Assessment) are widely used to assess the effectiveness of national educational systems. Second, the Strategic Data Project at Harvard University and best practices among British Schools are demonstrated as good practices on how to use school-level information to help school leaders to understand students’ learning patterns. Third, the tool Course Signal developed by Purdue University shows how to use course-specific data to provide timely feedback to teachers for instructional improvement. Fourth, Degree Compass is one of the interfaces that utilize learning analytics to manage individual student data.

Barriers and impediments to the use of analytics in education

Agasisti and Bowers summarize four barriers that impede the use of data analytics in education and propose potential solutions to lower these barriers. The first concern is the potential threat to student privacy, as many tools are built upon tracking student information. The authors argue that “open code and open access standards must be used when data analytic or machine learning algorithms are used to inform evidence-based improvement cycles in schools, or to the extent that Learning Analytics algorithms make recommendations for content and instruction for student learning”. Second, to address the complexity of data, researchers are recommending using data warehouses for data reporting and analytics. Third, the creation of an adequate platform for data analysis can be costly. However, open access code publication will facilitate the sharing of code and reduce the cost to build such platforms. Fourth, it is challenging to develop methodologies that “present the results without excessive simplification (providing an awareness of the complexities of the learning process) but with enough clarity to make the information understandable, and thus usable”.

A way forward: systemic change and the role of education data scientists

To conclude, the authors recommend that education data scientists as a new profession would play a key role to improve data use in schools and higher education. Education data scientists are expected to “facilitate the communication between three worlds: (i) one of technical experts in data analyses and analytics, (ii) that of decision-makers at various levels (policy analysts, school principals, institutions’ mangers) and (iii) the community of teachers, engaged in frontline instruction”. The authors call for more commitment and resources to train this new expertise as there is currently a lack of training and capacity building in this field.

Agasisti, T., Bowers, A.J. (2017) Data Analytics and Decision-Making in Education: Towards the Educational Data Scientist as a Key Actor in Schools and Higher Education Institutions. In Johnes, G., Johnes, J., Agasisti, T., López-Torres, L. (Eds.) Handbook of Contemporary Education Economics (p.184-210). Cheltenham, UK: Edward Elgar Publishing. ISBN: 978-1-78536-906-3 http://www.e-elgar.com/shop/handbook-of-contemporary-education-economics

Open Access Version: https://doi.org/10.7916/D8PR95T2