Gaze and Brain Activity During Software Design


An Intelligent Tutoring System (ITS) is a computer system that provides immediate and customized instruction or feedback to learners without requiring intervention from a human teacher. An ITS emulates the qualities of skillful human teachers (including having empathy and creating an engaging presence). Doing that involves real-time awareness of students’ cognitive challenges and affective states (emotions and mood). Two technologies that can support providing an understanding of cognitive challenges and affective states to software systems are eye tracking for gaze behavior analysis and brain-computer interfaces for brain activity monitoring. Imagine having continuous real-time access to what a student is looking at and their brain’s activity while reading a page, solving a problem, or answering a quiz. How can we leverage such information (gaze and brain activity) to improve the depth and accuracy of machine learning modeling for the student and use such a model to drive an ITS’s behavior? In particular, we aim to leverage such capabilities into an ITS that supports students in learning Software Design topics. In that context, this summer’s undergraduate research project focused on

  1. exploring eye-tracking technology and its application in triggering tutoring behaviors, primarily examining the relationship between eye gaze and software models (diagrams), focusing on detecting the differences between skilled and unskilled students, and
  2. using a brain-computer interface (Emotiv Headset) to examine cognitive and emotional reactions to software structural models. Particularly, examine whether modular design affects levels of cognitive load or affect.



Funded by the Summer Undergraduate Research Program (SURP) of the College of Engineering (CENG). The purpose of the SURP program is to provide research experiences for our undergraduate students working alongside our faculty during the summer. This program also supports the scholarly efforts of our faculty through engagement with undergraduate students. Projects led to student-faculty-authored publications in venues appropriate to the area of research conducted during the summer. Further, the program invites industrial and other external sponsors to provide university-industry collaborative research and project activities.