visual analytics – Simon.BuckinghamShum.net

Congratulations Dr. Vanessa Echeverria!

It is with enormous pleasure that I applaud Vanessa Echeverria Barzola for completing her PhD here at UTS:CIC!

Moving with her family from Ecuador in 2017, Vanessa joined as one of our first doctoral researchers when we launched the Learning Analytics PhD Program, and has wrapped it up within her four years, demonstrating huge personal resilience on the way.

Read her thesis below, related CIC News stories, and browse her website for the extensive publications generated by her work. We wish her every success as she develops her academic career, and can’t wait to see what emerges!

Vanessa Echeverria (2020). Designing Feedback for Collocated Teams using Multimodal Learning Analytics. Doctoral Dissertation, Connected Intelligence Centre, University of Technology Sydney, Australia. http://hdl.handle.net/10453/140936

Abstract: The ability to communicate, be an effective team or group member and collaborate face-to-face are critical skills for employability in the 21st century workplace. Previous research suggests that learning to collaborate effectively requires practice, awareness of group dynamics and reflection upon past activities. However, although having a teacher closely supervising and providing detailed feedback to each group would be ideal, it may be unrealistic in practice. A promising way to approach this challenge could be to capture behavioural traces from group interactions in order to generate comprehensible and actionable feedback to support team reflection. In this sense, Multimodal Learning Analytics (MMLA) is a promising field, offering the potential to track learners’ activity across digital and collocated contexts, using emerging sensing and pervasive computing technologies. Most of the research in MMLA has been conducted in lab conditions, to help researchers validate learning theories or generate more comprehensive learner models. However, one of the most underexplored aspects of MMLA has been the generation of feedback to support teaching and learning, and moreover, in authentic locations and activities.

This thesis reports progress in tackling this challenge by designing and validating computer-based feedback, by means of visual representations and narrative, to support effective, guided reflection using multimodal learning analytics evidence. To achieve this, three contributions are presented. The first contribution is a human-centred design method to translate the informal outputs of co-design sessions with teachers and students, into more meaningful group work constructs with clear MMLA design requirements. The second contribution is a modelling approach to add meaning to low-level multimodal group data based on the characteristics of the context (domain expertise, theory, and the learning design). Finally, the third contribution is an approach for augmenting visual representations with data storytelling elements to facilitate the interpretation of group dynamics insights by educators and students. This thesis is developed in the context of two distinct, collocated group work settings, in the domains of collaborative database design and healthcare simulation. Using a Design-Based Research process, a set of interfaces (i.e. interfaces that communicate insights) was designed and validated with teachers and students. The thesis provides timely and necessary groundwork for researchers and practitioners to design visual representations capable of communicating actionable insights, using multimodal data in complex and authentic collaboration scenarios.

CHI2020: Layered Storytelling for Multimodal Learning Analytics

As a PhD student from 1988 at the University of York HCI group and Rank Xerox Cambridge EuroPARC (as it was called then), I found my intellectual community and cut my teeth at the British HCI conference, and ACM CHI. I then spun off into various other orbits, seeing HCI as my bedrock but enjoying smaller, more focused conferences (e.g. Hypertext, CSCW, Semantic Web, OER and then ed-tech). However, my current desire to see Learning Analytics become more human-centred in its design processes, and working with Roberto Martinez-Maldonado, has looped me back into the HCI community again, and I’m thoroughly enjoying reconnecting with old and new faces!

So, here’s our latest work, building on our CHI19 paper, which is for me a very satisfying convergence of multimodal analytics, collocated teamwork, visual analytics, pedagogy and my longstanding interest in narrative. It incorporates the doctoral work of Vanessa Echeverria (who has just submitted her thesis and is now at CMU HCII) and Gloria Fernandez-Nieto (who just passed her first year with flying colours).

The teaching and learning challenge is to give instant feedback to nursing students on how well they performed as a team in treating a patient in a simulation. The research question is how to make streams of multimodal data intelligible. Enjoy!

Martinez-Maldonado, R., Echeverria, V., Fernandez-Nieto, G. & Buckingham Shum, S. (2020). From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. Proc. ACM CHI 2020: Human Factors in Computing Systems (April 25–30, 2020, Honolulu, HI, USA), Paper 21, pp.1-15. https://doi.org/10.1145/3313831.3376148 [Open Access Eprint]

Abstract: Significant progress to integrate and analyse multimodal data has been carried out in the last years. Yet, little research has tackled the challenge of visualising and supporting the sensemaking of multimodal data to inform teaching and learning. It is naïve to expect that simply by rendering multiple data streams visually, a teacher or learner will be able to make sense of them. This paper introduces an approach to unravel the complexity of multimodal data by organising it into meaningful layers that explain critical insights to teachers and students. The approach is illustrated through the design of two data storytelling prototypes in the context of nursing simulation. Two authentic studies with educators and students identified the potential of the approach to create learning analytics interfaces that communicate insights on team performance, as well as concerns in terms of accountability and automated insights discovery.

Visual Analytics 2010 summer school

With great backup from my colleagues Anna De Liddo and Michelle Bachler, I spent last Thurs/Fri at the first UK Visual Analytics Summer School (VASS 2010), hosted at Middlesex University. Huge thanks to Simon Attfield for pulling together this week-long event. (3 Jan 2011: all the lecture videos are now replayable).

In my tutorial, as well as drawing on the foundational work of my long-term collaborators Jeff Conklin (gIBIS; Dialogue Mapping), Maarten Sierhuis and Al Selvin (Conversational Modelling; Knowledge Art), I drew on the pioneering work of Bob Horn and Tim van Gelder: Bob Horn from Stanford, developer of the Turing Test Great Debate Maps, was actually able to join the event on Friday thanks to fortuitous travel to UK, and it was great to see him present some of the latest work he has been doing with his large scale information murals visually synthesising outcomes of the World Business Council for Sustainable Development backcast analysis to a sustainable 2050 as a Pathway Mural:

Bob Horn’s Pathway Mural for the WBCSD

I also drew in my talk on the work of Tim van Gelder, who has done foundational work in articulating the discipline of Argument Mapping for real world visual deliberation:

Here’s the video of my lecture…

…and the slides [download PDF]

Visual Sensemaking for Contested Collective Intelligence

We then ran a hands-on session with Compendium, and gave an extended demo of Cohere, below [zipped hands-on exercises] – view the video replays:

The key message I took away was that the work that we are focusing on — visual analytics for scaffolding interpretations/debate about what a given element means — was recognised as distinctive and filling a potentially valuable niche, and we need to take the conversations further to connect this layer of meaning-making (in which the sensemaking is largely dependent on human insight making connections), with the many other powerful visual analytics tools that help automatically detect patterns in datasets and documents.

The “craft skill” of using computational tools for thinking/modelling is a long-term interest of mine. Tools for higher level thinking can rarely be “thrown over the wall” and put to immediate use by newcomers. As my colleague Al Selvin emphasises in his research on Knowledge Art, these are better thought of as musical instruments: they can enable virtuoso performances, but don’t expect this on Day 1. So, I found agreement that the third dimension in our framework requires more attention: understanding the nature of fluency with these tools, which includes the habits of mind and skillset required to use them well. Who are we designing these tools for, and how much does a given person need to know in order to gain value, and how soon?

Links to the following talks will take you deeper into some of the areas that I covered in the lecture, including the Online Deliberation Emerging Technologies workshop (demos of lots of tools in this part of the visual analytics space), and these recent talks provide many other examples of Compendium in use, and sensemaking frameworks: On Social Learning, Sensemaking Capacity, and Collective Intelligence and Software Agents in Support of Human Argument Mapping