CHI2020: Layered Storytelling for Multimodal Learning Analytics

As a PhD student from 1988 at the University of York HCI group and Rank Xerox Cambridge EuroPARC (as it was called then), I found my intellectual community and cut my teeth at the British HCI conference, and ACM CHI. I then spun off into various other orbits, seeing HCI as my bedrock but enjoying smaller, more focused conferences (e.g. Hypertext, CSCW, Semantic Web, OER and then ed-tech). However, my current desire to see Learning Analytics become more human-centred in its design processes, and working with Roberto Martinez-Maldonado, has looped me back into the HCI community again, and I’m thoroughly enjoying reconnecting with old and new faces!

So, here’s our latest work, building on our CHI19 paper, which is for me a very satisfying convergence of multimodal analytics, collocated teamwork, visual analytics, pedagogy and my longstanding interest in narrative. It incorporates the doctoral work of Vanessa Echeverria (who has just submitted her thesis and is now at CMU HCII) and Gloria Fernandez-Nieto (who just passed her first year with flying colours).

The teaching and learning challenge is to give instant feedback to nursing students on how well they performed as a team in treating a patient in a simulation. The research question is how to make streams of multimodal data intelligible. Enjoy!

Martinez-Maldonado, R., Echeverria, V., Fernandez-Nieto, G. & Buckingham Shum, S. (2020). From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. Proc. ACM CHI 2020: Human Factors in Computing Systems (April 25–30, 2020, Honolulu, HI, USA), Paper 21, pp.1-15. [Open Access Eprint]

Abstract: Significant progress to integrate and analyse multimodal data has been carried out in the last years. Yet, little research has tackled the challenge of visualising and supporting the sensemaking of multimodal data to inform teaching and learning. It is naïve to expect that simply by rendering multiple data streams visually, a teacher or learner will be able to make sense of them. This paper introduces an approach to unravel the complexity of multimodal data by organising it into meaningful layers that explain critical insights to teachers and students. The approach is illustrated through the design of two data storytelling prototypes in the context of nursing simulation. Two authentic studies with educators and students identified the potential of the approach to create learning analytics interfaces that communicate insights on team performance, as well as concerns in terms of accountability and automated insights discovery.

Leave a Reply

You can use these XHTML tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>