NSF Education Data Analytics Collaborative Workshop

Workshop Overview

On December 5 and 6 of 2019, the National Science Foundation (NSF) Education Data Analytics Collaborative Workshop was held at Teachers College, Columbia University in New York City in the innovative multimodal learning space, the Smith Learning Theater. Approximately 80 educators, administrators, data scientists and education researchers from New York and beyond gathered for a two-day workshop. This workshop was a part of the final phase of the collaborative NSF funded research project (NSF #1560720) "Building Community and Capacity for Data-Intensive Evidence-Based Decision Making in Schools and Districts", which is a collaborative partnership on data use and evidence-based improvement cycles in collaboration with Nassau County Long Island BOCES (Nassau BOCES) and their 56 school districts in Nassau County Long Island New York.

As the final phase of the collaborative NSF grant, the purpose of this two-day event in 2019 was to bring together teachers, school data practitioners, and administrators partnered with national-level education data scientists and education researchers to collaboratively work together. These “datasprint teams” and the event overall were designed to help understand the needs of educators around data use and data visualization, and then iteratively build data visualizations in open source software (such as R or Python) that address these needs and could be useful for supporting evidence-based improvement cycles in schools.

This collaborative workshop event built on the success from the ELDA Summit 2018, an initial event in 2018 with over 120+ attendees to discuss the issues and needs of the emerging field of Education Leadership Data Analytics (ELDA). For the NSF collaborative workshop meeting in 2019, we decided to extend the original 3-hour long session from 2018 into a two-day workshop, mainly focusing on building collaborative interactions between members of “datasprint teams” made up of teachers, school and district leaders, and national-level education data scientists. As the capstone event of the NSF grant collaborative project, the 2019 meeting brought together aspects from the ELDA Summit 2018 and new learnings and collaborative opportunities around the goal of enhancing evidence-based decision practice in schools.

A central call from the ELDA Summit 2018 was to have a greater role for the voices of education practitioners (teachers and administrators) and to build stronger partnerships between researchers and practitioners in order to support instructional improvement in schools through improved data analytics, data dashboards, and ultimately teacher-led evidence-based improvement cycles. Towards that end, as demonstrated in this Figure 1 bar graph of the number of attendees, a majority of the participants were educators, which is attributable to the strong partnership and central role of Nassau BOCES and administrators and teachers from across Nassau County throughout the NSF collaborative grant. This type of collaborative participation between researchers and educators around data use is evident in in the Teachers College, Columbia University Newsroom story about the two-day event: “How Education Data Can Help Teachers Raise Their Game”.

Figure 1.
NSF Education Data Analytics Collaborative Workshop Post-event Survey self-identifier data analysis; Question: I attended the workshop as a… Select one.

Findings from the Workshop

Out of 77 participants who attended the NSF Data Collaborative Workshop, 74 participants filled out the post-event survey, providing a strong response rate of 95%. The following responses and analyses are based on the responses from the NSF Education Data Analytics Collaborative Workshop Post-event Survey.

The Best Sessions that Meet Participants’ Needs

We asked the participants the question “How well does each session that you attended meet your expectations?” to understand whether each session meets the expectations of the participants. There were in total five sessions, divided by the first day and the second day, as well as by morning and afternoon, with a special keynote lunch with Professor Richard Halverson from the University of Wisconsin - Madison on the first day.

During the two-day workshop with five sessions, the participants showed a high satisfaction with an average of 4.23 out of 5 for the entire workshop. Among the five sessions, however, the participants were most satisfied with the Day 1 Keynote Lunch presentation by Richard Halverson. This was an hour-long session during the lunch on the first day, a presentation successfully engaging both practitioners and researchers.

Richard Halverson, coauthor of Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in American Education

The Day 2 Afternoon session ranked as the most satisfying session after the keynote lunch. This session included a “Basecamp Journey” during the datasprint team collaborations. While the data scientists and education researchers were working on creating visualizations, educators and administrators had opportunities to “journey” around the event to visit with and learn from other teams and provide their thoughts and written feedback so that the each team could receive feedback and share ideas.

Creating “Travelers’ Comments” at the “Basecamp” (Photo: Seulgi Kang)

During the “Basecamp” Journey activity, an educator or administrator from each team would randomly pick a “destination” among ten other different teams. We then asked the remaining educators/administrators to welcome travelers and share their team’s working process – how, why, and what they are visualizing. After traveling to the other team, every traveler returned to the “Basecamp”, and the travelers were asked to provide either questions or opinions regarding the team they visited. We found this session to be a great opportunity to create deeper cross-team conversations, integrating ideas across Education Leadership, Data Science and Data Analytics, and Evidence-Based Improvement Cycles. Participants appreciated the second day’s afternoon session, and this offers an important implication on how the workshop succeeded in involving all participants, each of whom had different levels of knowledge and expertise in data science.

The Most Applicable Data Visualizations the Participants Found

We asked the following question to find out how participants reacted to the exposure to various new data visualization methods and conversations: “For the two-day event, please describe the data visualizations that you found most applicable to your context and role, and why.” With short-essay type answers, we created a word cloud to visualize the most commonly-mentioned words. Note that we exclude some generic words (‘data’, ‘teacher’, ‘student’), as well as the words that the question itself includes (‘visualize’, ‘applicable’, and ‘found’).

Figure 2.
NSF Education Data Analytics Collaborative Workshop Post-event Survey Data Visualization Analysis; A word cloud created by Qualtrics. This word cloud excluded the words: data, teacher, student, visualize, applicable, and found.

We found that the word “standard” appeared the most, and was frequently combined with words such as “group”, “test” and “year”.

Data visualizations that the participants mentioned including the word “standard” are (a) grouped standards for/by teachers – item analysis visualizations (b) multi-year GAP standard report and (c) non-standardized test data visualizations. In fact, these were some of the words that continuously appeared in the written open-ended responses from participants on the survey. A central finding across the post-event survey from participants is that the most applicable data visualizations were not complex, but rather visualized the needed information for teachers in a simple straightforward manner.

The Most Important Components of a Longitudinal Data System

The Post-event survey continued with the open-ended question, “What components of a longitudinal data system are needed to best meet the needs of superintendents, principals, and teacher leaders”. This question come from a survey study by Brocato, Willis, and Dechert (2014). As a reflection on the two-day event, this question sums up the needs of practitioners as well as the perceptions of researchers of educator data needs, especially given the collaborative conversations of the datasprint teams from the two day workshop. We also created a word cloud of the most frequent words, to examine the responses, excluding words that are either generic or appeared in the question itself.

Figure 3.
NSF Education Data Analytics Collaborative Workshop Post-event Survey Longitudinal Data Components Analysis; A word cloud created by Qualtrics. This word cloud excluded the words: data, teacher, student, system, longitudinal, and information.

Figure 3 clearly shows the needs of practitioners in a glance. The most common words the participants responded with were attendance, assessment, and demography. It reminds us that education practitioners have a range of data needs across a wide variety of data types. Overall, there was a frequent call for longitudinal student data in nearly all aspects, not just standardized test scores. One of the participants wrote as following:

“The data system should paint a full picture of each student - achievement, absences, tardiness, supports and interventions, parental engagement…. All elements of a child's being, performance, and needs should be tracked longitudinally to help give educators a full picture of who the child is and what the child needs to succeed.”

- from NSF Education Data Analytics Collaborative Workshop Post-event Survey (2019)

This quotation is a strong representation of participant responses to the question. On top of collecting information that gives a big picture of one student constantly, another distinctive call from the participants was a constant scale of assessment test scores. If the test scores are only applicable and interpretable in one school or district at a certain timeframe only, then the data cannot be shared or used beyond that single context.

Overall, the NSF Education Data Analytics Collaborative Workshop provided a meaningful opportunity to bring together educators and data scientists to build collaborative data visualizations together. For next steps in the project, educators and data scientists from the event are writing book chapter contributions about their perceptions of the event, the outcomes, and next steps for data use and data visualizations, with a goal to publish the final book soon.

References:

Bowers, A. J., Bang, A., Pan, Y., & Graves, K. E. (2019). Education Leadership Data Analytics (ELDA): A White Paper Report on the 2018 ELDA Summit. https://doi.org/10.7916/d8-31a0-pt97

Brocato, K., Willis, C., & Dechert, K. (2014). Longitudinal Data Use: Ideas for District, Building, and Classroom Leaders In A. J. Bowers, A. R. Shoho & B. G. Barnett (Eds.), Using Data in Schools to Inform Leadership and Decision Making (pp. 97-120). Charlotte, NC: Information Age Publishing.