Kristin Gorski

Dr. Kristin Gorski:

My dissertation looked at how students learn in digital, bilingual, multimodal spaces. Specifically, I looked at note-taking practices and how student's voice, gaze, gesture, and language use impacted that. My name is Kristin Gorski. I'm the Director of Operations of the Digital Futures Institute. My dissertation was part of my studies in the Department of Mathematics Science and Technology, and I matriculated in the communication media and learning technologies design program. My dissertation looked at the note-taking practices of emergent bilingual students in the New York City public school system.
They were middle school students in the seventh and eighth grades, and they were working on their social studies and science learning that it's part of the standard curriculum. I was lucky enough to be a research assistant on a grant funded by the Institute for Education Sciences, IES, and it was called Steps to Literacy. The note-taking practices that I looked at for the students were part of the ways that they constructed a standard academic English essay. What was unique about it is that we developed a multimodal bilingual web-based space with them. Then, they were then able to learn more about the topics of the Civil Rights movement and the Holocaust and theories of evolution by using bilingual multimodal resources in the web-based space, and by having a notepad that they could move around as they worked.
We gave them all sorts of different resources and we would populate the web-based spaces. We found new ones. They included everything from videos of marches for the Civil Rights movement. We had a good graphic novel introduction to the story of Anne Frank, which was another video that the students would watch. We gave them primary source documentation of speeches. We gave them lots of audio of speeches as well, transcripts, both in Spanish and English. Wherever we could find things in English and Spanish, we included them. Then, the students would open up the resources as we would direct them for the times that the teacher has taught them a lesson related to the topic.
Then, at other times, the students would open the resources on their own and they would open maps, and they would open infographics and things like this, and they would place them around on the screen as they wanted. They could review the audio and video as many times as they wanted. They could move the notepad around, and it was truly a creative space for learning that they could make on their own. They had a lot of choice and options for figuring out what exactly they wanted to learn more about and how those types of resources that we put in the system could support their learning. What I discovered in learning how the students took notes is that it's a very non-linear and complex process.
We saw students opening notes, closing notes, opening resources, closing resources, moving things around, talking with their friends, asking instructors questions. The notes were part of an ongoing conversation that they were having with their peers, their instructors, but also with the resources that they had options to access. Even though we were encouraging them to take notes so they could learn more on the topics and then formulate paragraphs, which would then be built into an academic essay, the learning process was very much their own pathway through it, even though the goal at end was something to find.
It was really interesting for me to see how these small moments of learning that we captured in our video capture software were able to be reviewed by me continually as I analyzed and worked to figure out what the student note-taking practices were like in this specific space. The dissertation form was ultimately a very traditional text-based dissertation. That being said, what I had to determine was, how do I take rich video data that showed two different viewpoints and then write that up and provide accompanying video stills and additional context to have it make sense to my dissertation committee and to anybody who would read about it. One thing that I think would be important to mention now is how the software worked.
It was a software called iShowU, and it did two things. It captured how the students moved all of the resources on the screen, so it was a live screen capture, and it was synced with the camera looking at the student. As a student would open the notepad and move it around the screen, open up different resources, look at them, read through the transcripts and the dialogue, look at the maps, they would often talk aloud. They would look around the room. They would look off in the distance. They would ask a question to their instructor, and it gave a much fuller picture of all of the things that they were thinking at that time.
The video challenge into the multimodal transcript was an interesting process because if I had access to a multimodal dissertation video aspect, I could have shown clips of the video and talk it through and have some annotations that would be in the video to explain what I was seeing and what I was noticing. But what I wound up doing in my dissertation was creating a frame-by-frame set of columns that had the timestamp, had a still image of the video, and then had different dialogue next to it, and then some other things that I noticed. It was a two-dimensional text-based version of some very rich multimodal data, and it's easy to cite and easy to read because it's in a traditional format.
But thinking back, there are ways that I could definitely have made it much more rich. I think I'm inspired by the other multimodal dissertations in this series and by others that I've been tracking now and just keeping aware of to see what else I could do to share data in a more multimodal way. The multimodal form of the dissertation and the analysis helped me gain some interesting insights into research in general. First of all, I think a qualitative approach to research in multimodal analysis is really important for its descriptive aspects. As I began to look at the video and see what was actually going on, I realized there was a very rich story there and a lot that I had to tell.
The challenge became, how do I transfer forms? How do I take modes from a video setup and put it into the mode of text plus still images in almost a diagrammatic format? There are references out there for multimodal transcripts, and there are software available like iShowU, but also ELAN, which is from the Max Planck Institute, and it's a linguistic annotator that's open source. We as a research team played around with all of these different software available at the time to see what could help us look at the different layers of gestures, look at the different layers of interactions between people. Look at instances where the students were trans-languaging between English and Spanish.
Both in their note-taking, also in their essay writing, which was outside of the scope of my dissertation, but we have data on that as well. Also, in the conversations that they had with each other and with their instructors. Watching the form continually morph as I thought of audience and as I realized I had to focus on a certain type of instance or a certain gesture or explanation or insight into the student's note-taking practices, I had to stay on my feet and just let myself think creatively and be open to what the data could tell me. One of the later questions that I thought about for this interview was, what's one thing that I would recommend for people?
I'll get back to that later, but it's to spend time with the data and really understand what you're seeing and to really appreciate what's happening on video and really appreciate the small moments. These micro moments of learning and insight that we can begin to see through the multimodal video capture. I realize that I didn't have all the answers on the first few number of passes. I began to get more insight the more I watched the video and the more I thought about it and reflected. I would also talk with my dissertation advisor. I would also read additional pieces. I would talk to other faculty members on my committee, and that's where I began to realize for qualitative data, especially multimodal data that's so rich.
It's important to give yourself a lot of time to really understand and appreciate what's there because it's not readily evident on the first or second pass. I think numerous passes and conversations related are key. Some of the challenges that arose from doing these different kinds of data collection were very down to earth. Did the hardware work that day? Did we have internet connection at the school? Were students that we had all the way through the project now suddenly not coming to school, so we did not have them participate any further? These are the types of things that can happen in any tech-based, school-based research projects.
A lot of different variables are happening that are very real time and real life. Some of the challenges that were more research and theoretically based were charting a path through a relatively uncharted territory about how to do multimodal work. There were definitely references that I could turn to at the time that helped me figure out how to proceed. I think at this point there are additional references and communities of dissertations now where people can really turn to peers and turn to examples and say, "This is a guiding light for me. This is something I can turn to and get ideas from." That being said, I think one of the most important aspects of the research framework is to be in an uncharted territory, and be open to what you're seeing that is not necessarily something that's been documented before.
Be open to new possibilities. Be open to new frameworks and new things that you would discover. It's important to keep an open mind on what you're seeing. I think there has to be a level of courage and there has to be a level of acceptance. That lack of comfort and just the unknown is a really important part of the process if you really want to see what's there. The advice that I would give to people who are embarking upon multimodal scholarship is, first, I think that guides any solid research, you have to determine your research questions in line with the modes that you are hypothesizing you'll see and that you're working with.
There are things that we know that are units of measurement, that are specific modes that we want to measure and that we want to account for. It's important to define those at the outset, so talk with your advisor. Make sure you know your research site. Make sure you understand the whole setting, and make sure you really understand what your participants are bringing to this. After that, I think the theoretical frameworks come in, and now that there is more published around multimodal research, make sure that you're getting the basics from the field, but also look at things that are specific to your mode that you may be focusing on.
Think widely about what modes mean, because the more we look at them, modes can have multiple levels and multiple meaning. Another piece of advice I would give to people embarking on this research would be to think about if you could bring the data back to your participants and ask them what they thought. You've recorded video or you've recorded audio, or you have their multimodal writing, and it would be very insightful for you as researcher to have a conversation with them to see what they actually meant or what they actually thought. Because there are modes that are evident to us on the video, but it may not reveal everything about what's going on in a learning process.
I think if that could be incorporated into the process of analysis, that could be very insightful. A final piece of advice I would give would be to spend time with the data. By spending more time reviewing and thinking about and talking about the data that you're analyzing, you're going to get additional insights into the learning practices of your students. You're really going to get a much fuller understanding of all the possible modes that are involved in those learning practices. When I first read the works of multimodal scholars and they introduced me to new notions of analysis and understanding students' learning, it was one of these aha moments for me because I thought, "Oh, first of all, people are researching this, so this is important."
Second of all, they formalized it in a way to share with others, and they are developing theoretical frameworks right now. It's happening as I am studying. It's happening right before I study, but I feel like I'm in the time period with these scholars, and that was very exciting. One thing that was a unique aspect of looking at note-taking in a multimodal online bilingual space is that a lot of note-taking is very personal, and it happens individually, and we don't really get to see it. Students take notes for themselves. It's in a notebook, it's with pen and paper. But being able to see it captured in this space in a research setting gave us additional insights into what was possible to expand outside pen and paper into a more interactive, engaging, multimodal way of taking notes.
I began to think what could be possible for people's learning if this became something that was common for students to be able to access? In citing the non-textual works in my dissertation, I think it's important for people to make sure that they're understanding multimodal transcript references. While my dissertation is really easy to cite because it's a two-dimensional dissertation, the different multimodal analysis aspects of it need to have more context added for people to fully understand all the different modes that are a part of the learning going on. Then, also make sure that they're understanding it's happening in a note-taking and formal learning environment in a classroom.
There's a lot of looking at how learning happens in the classroom and how learning happens as students just construct knowledge on their own with the resources that we give them. There are two projects that I'm working on currently. In October 2024, some colleagues and I presented our research on how artificial intelligence is disrupting higher education communities at the Association for Educational Communications and Technology Conference, AECT. As part of that research, we asked our participants to keep a multimodal diary and journal of their responses to AI according to prompts that we gave them. Some people responded in texts, some people responded on post-it notes.
Some people responded by going into the AI apps and generating some output and including that in their online journal. That was an interesting thing to see. It wasn't the centerpiece of our research, but it was definitely a part of our understanding and we would like to do more on that. I teach a course on the social and communicative aspects of information and communication technologies. In that course I help students understand the affordances and constraints of blogging. They get to choose a topic that is interesting to them, and they blog about it all semester. Throughout the semester I introduce a theory of multi-modality and we look at all the different expressive modes they can incorporate in order to optimally both express themselves, but also increase the audience member's learning about their topic.
This semester coming up, I'm going to be very focused on how students are working with AI and multi-modality in generating their blog posts. Those conversations are going to be ongoing, and I think it's going to be really interesting to see how my conversations with them around AI and multi-modality will form. It's completely undefined. I want to see what my students have to say, and it could point to some interesting future research. When I started at DFI, I was brought onto a project called Perspectives, which is our online digital publication featuring the voices of the staff and faculty affiliates at TC who work in all the different DFI mission areas. Joe Riina-Ferrie, who's the Associate Director of Educational Media, had set some groundwork on how this was going.
I was really happy to become a part of this. We're creating an online digital publishing space for everybody to continue adding to. We've got about six months since we first published a first post, and it's featured everything from art exhibits, interviews with staff members here who had an exhibit. We have people who have worked on different projects with faculty members, and they've shared their perspectives around different video processes and things like that. Joe and I will continue to co-lead it, and we're going to be bringing on some additional student workers in the fall, and we'll really roll out bringing additional voices to the Perspectives website.