Multimodal learning analytics on sustained attention by measuring ambient noise Jeffrey Pronk Responsible Professor: Marcus Specht (PhD) Supervisor: Yoon Lee (PhD candidate) Peer group members: Giuseppe Deininger (BSc student) Jurriaan Den Toonder (BSc student) Sven van der Voort (BSc student) I Abstract In this research, a learner’s sustained attention in the remote learning context will be studied by collecting data from different sensors. By combining the results of these sensors in a multi-modal analytics tool, the estimation of the learner’s sustained at- tention can hopefully be improved. This research will mainly focus on microphone recordings of ambient sound in a learners room. The main research question of this re- search was "How can ambient noise sensing aid in a multi-modal analytics tool to track sustained attention?". The multi-modal learning analytics tool, if accurate enough, could potentially be used by teachers to make their material more engaging and could help learner’s to keep their focus while performing a learning task (Schneider et al., 2015). The research resulted in a model with 61% accuracy. This percentage needs to be further researched, since because of the COVID situation, not enough data could be collected to train the model. Because of the relatively low accuracy of the model, it was found that ambient noise sensing can aid the multi-modal analytics tool to some extent by adding some data-points it is certain about, when the mobile movement tracking model does not detect a distraction. If the model improves in future research, the model could be able to help mobile movement tracking model, even if the mobile movement tracking model already predicts a distraction with bigger then 50% certainty. II 1 Introduction In this research, a learner’s sustained attention in the remote learning context was studied by collecting data from different sensors. It will be researched if a model can be constructed to predict if a learner is distracted using the various sensors and these models will be com- bined to try to improve the models accuracy in a multi-modal model. This research mainly focused on microphone recordings of ambient sound in a learners environment. The multi- modal learning analytics tool, if accurate enough, could potentially be used by teachers to make their material easier to learn and could help learner’s to keep their focus while per- forming a learning task (Schneider et al., 2015). In the current world where remote teaching is necessary, this research becomes more important as it is easier to see if someone is dis- tracted by seeing him/her before you in the class rather then behind a webcam. The main research question of this research was "How can ambient noise sensing aid in a multi-modal analytics tool to track sustained attention?". This main question was split into 5 different sub-questions to help answer the main research question. It was expected that by combining multiple sensors, a more accurate model could be achieved when com- pared to uni-modal models relying on a single sensor. Research on people being distracted by sound had already been done before (Ke et al., 2021; Lee, 2019; Morgan, 1917) and it was already known that people can get distracted by sound containing fluctuating frequen- cies. The interesting part of this research was to find out if a model could be created that automatically detects ambient sound and predicts if this sound is distracting. The research could focus on two different types of sound: vocal and non-vocal sounds. Because of the cocktail effect (Bronkhorst, 1999) it became hard to focus this research on vocal sounds. Vocal sounds are very subjective, someone will not immediately react to some- ones else’s name, but will notice his/her name for example. Therefore this research focused on non-vocal sounds. Multiple different sounds where used during the experiment, some sounds like machine noise and music that can occur in a (at home) learning environment, some sounds that should be relaxing and should help the learner focus and some sounds consisting of fluctuating frequencies. The research resulted in a model with 61% accuracy. The model seemed promising, though it needs to be further researched. Because of the COVID situation, not enough data could be collected to train the model. Unfortunately, in the end , two of the 3 other researchers where not able to provide a working model due to various reasons. Therefore the multi-modal model was only created with the ambient noise tracking model and the mobile movement tracking model (Deininger, 2021). Because of the relatively low accuracy of the model, it was found that ambient noise sensing can aid the multi-modal analytics tool to some extent by adding some data-points it is certain about, when the mobile movement detection model does not detect a distraction. If the model improves in future research, the model could be able to help phone movement detection model, even if it already predicts a distraction with bigger then 50% certainty. 1 2 Background In this research, a learner’s sustained attention in the remote learning context was studied by collecting data from different sensors. By combining the results of these sensors in a multi-modal analytics tool, it was tried to improve the estimation of the learner’s sustained attention. By studying the sustained attention of a learner, teachers can improve there study material and adapt their lectures, thereby improving the amount of material a learner learns. A multi-modal learning analytics tool, if accurate enough, can also be used by learners to stay focused while performing a learning task (Schneider et al., 2015). The research of this system was done by 4 different students, all focusing on different sensors and its data (Deininger, 2021, Toonder, 2021, van der Voort, 2021). In this specific research, the main focus was on measuring ambient noise and finding a correlation between learner’s losing focus and data from microphones recording ambient noise in the room the learner is situated in. Research to people being distracted by noise had already been done before and it had shown that people can be distracted by noise and that this distraction can be measured (Ke et al., 2021; Lee, 2019; Morgan, 1917). The interesting part about this study is to see if distracting sounds can be recorded by a microphone and if using this data it can be predicted if this sound is a distracting noise to help the multi-modal system to detect a distraction and its source. 2.1 Main research question: How can ambient noise sensing aid in a multi-modal analytics tool to track sustained attention? The main goal of this research was to find out if ambient noise sensing can aid a multi- modal analytics tool to track sustained attention. This multi-modal learning tool consists of the combined model created by the four separate researches. It was expected that ambient noise sensing can aid in finding possible distractions by measuring noises that will distract nearly everyone, however that it will not be possible to exclusively use noise measurements to detect a distraction. This is due to the huge variety of different sounds that can be measured and the different reaction people can have to these sounds (Lee, 2019; Min et al., 2016). This sensing will probably help the multi-modal analytics tool by confirming a possible distraction, detected by one of the other models. The main research question was divided into 5 different sub-questions that where an- swered during this research to work towards an answer for the main research question. The 5 sub-question are: 1. What should be measured on the noise (dB, Hz, Hz fluctuation or categorized sounds?) 2. Should background noise be filtered to reliably determine if a noise is distracting? 3. If the background noise needs to be filtered, how can this be filtered without losing possible noises that distract a learner? 4. Can a correlation between noise variation and distraction reliably be found in the microphone recordings? 5. How does the correlation between noise variation and distraction help the multi-modal analytic tool to track sustained attention? The sub-question will be further described in section 3. 2 3 Methodology To answer the main research question, first the 5 sub questions had to be answered. This was either done with a literature study, user test or by testing possible solutions. In this section the methodology to obtain answers will be described in the order in which the sub questions will be answered. 3.1 Sub-question: What should be measured on the noise (dB, Hz, Hz fluctuation or categorized sounds)? To know what metrics needed to be measured in the user study, it needed to be found what makes noise distracting and how this can be measured. To find this answer, a literature study was conducted. To find literature on this topic, the following search query was used: (auditory OR (ambient AND noise) OR noise OR sound) AND (attention OR distraction OR (sustained AND attention)). The main search engine used for this research was the TU Delft online library. Articles where only included if the research was conducted on non-speech sounds, no other inclusion and/or exclusion criteria where applied. During this literature study it was found that the distracting factor of sound is fre- quency fluctuation (Perham et al., 2007). Apart from this, sound categorization was also considered, however there are too many different sounds that have to be reliably categorized and implementing this and making it reliable enough to be used in the experiment is not possible within the time-scope of this project.