Tag: multimodal

HuCETA: Human-Centered Embodied Teamwork Analytics

After about 7 years working on multimodal teamwork analytics (specifically in nursing simulations) with Roberto Martinez-Maldonado, and then PhDs with Vanessa Echeverria & Gloria Fernandez — and more recently in an ARC-funded project with Dragan Gasevic, Lixiang (Jimmie) Yan, Linxuan Zhao — we are moving towards theoretically grounded, open source infrastructure for analysing collocated teamwork. We’ve distilled the essence of all that we’ve learnt into a new conceptual framework called HuCETA:

Echeverria, V., Martinez-Maldonado, R., Yan, L., Zhao, L., Fernandez-Nieto, G., Gasevic, D., & Buckingham Shum, S. (2022). HuCETA: A Framework for Human-Centered Embodied Teamwork Analytics. IEEE Pervasive Computing, 1-11. https://doi.org/10.1109/MPRV.2022.3217454 [Open Access Eprint]

Abstract: Collocated teamwork remains a pervasive practice across all professional sectors. Even though live observations and video analysis have been utilized for understanding embodied interaction of team members, these approaches are impractical for scaling up the provision of feedback that can promote developing high-performance teamwork skills. Enriching spaces with sensors capable of automatically capturing team activity data can improve learning and reflection. Yet, connecting the enormous amounts of data such sensors can generate with constructs related to teamwork remains challenging. This article presents a framework to support the development of human-centered embodied teamwork analytics by 1) enabling hybrid human–machine multimodal sensing; 2) embedding educators’ and experts’ knowledge into computational team models; and 3) generating human-driven data storytelling interfaces for reflection and decision making. This is illustrated through an in-the-wild study in the context of healthcare simulation, where predictive modeling, epistemic network analysis, and data storytelling are used to support educators and nursing teams.

Analytics for “classroom proxemics”

In concert with making the best use of online learning at UTS, an enormous amount of learning and teaching still happens face-to-face — except when there’s a global pandemic on! The UTS learning.futures model underpins the massive redesign of our on-campus learning spaces.

The term “Classroom Proxemics” refers to how teachers and students use classroom space, and the impact of this and the spatial design on learning and teaching. While learning analytics is usually thought of in terms of capturing, analysing and feeding back activity that is mediated by a collaborative platform, one can now be teaching and learning face-to-face, whilst also generating digital traces from embodied activity (as well as being online via one or more platforms).

Roberto Martinez-Maldonado (who pioneered our work on multimodal learning analytics to support nursing teams in simulated wards) has developed a new strand of work, on the potential of tracking (with consent of course), how educators make use of learning spaces as they teach. Working closely with academics in the Science and Health Faculties, the results of which are now being published, and demonstrate:

  • the process we use to work closely with educators to co-design potentially invasive technologies in ethical ways
  • the potential educators see in being able to visualise their teaching practice in this way
  • the problems they perceive with such tools if they are used in appropriately

Martinez-Maldonado, R., Schulte, J., Echeverria, V., Gopalan, Y. and Buckingham Shum, S. (2020). Where is the Teacher? Digital Analytics for Classroom Proxemics. Journal of Computer Assisted Learning, to appear (Accepted March. 2020).

The term “Classroom Proxemics” refers to how teachers and students use classroom space, and the impact of this and the spatial design on learning and teaching. This paper addresses the divide between, on the one hand, substantial work on proxemics based on classroom observations and, on the other hand, emerging work to design automated feedback that helps teachers identify salient patterns in their use of the classroom space. This paper documents how digital analytics were designed in service of a senior teacher’s practice-based inquiry into classroom proxemics. Indoor positioning data from four teachers were analysed, visualised and used as evidence to compare three distinct learning designs enacted in a physics classroom. This paper demonstrates how teachers can make effective use of such visualisations, to gain insight into their classroom practice. This is evidenced by i) documenting teachers’ reflections on visualisations of positioning data, both their own and that of peers; and ii) identifying the types of indicator (operationalised as analytical metrics) that foreground the most useful information for teachers to gain insight into their practice.

Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C. and Buckingham Shum, S. (2020). Teacher Tracking with Integrity: What Indoor Positioning Can Tell About Instructional Proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
(UBICOMP): to appear (Accepted Jan. 2020).

Automatic tracking of activity and location in the classroom is becoming increasingly feasible and inexpensive. However, although there is a growing interest in creating classrooms embedded with tracking capabilities using computer vision and wearables, more work is still needed to understand teachers’ perceived opportunities and concerns about using indoor positioning data to reflect on their practice. This paper presents results from a qualitative study, conducted across three authentic educational settings, investigating the potential of making positioning traces available to teachers. Positioning data from 28 classes taught by 10 university teachers was captured using sensors in three different collaborative classroom spaces in the disciplines of design, health and science. The contributions of this paper to ubiquitous computing are the documented reflections of teachers from different disciplines provoked by visual representations of their classroom positioning data and that of others. These reflections point to: i) the potential benefit of using these digital traces to support teaching; and ii) concerns to be considered in the design of meaningful analytics systems for instructional proxemics.

Martinez-Maldonado, R. R., Elliott, D., Axisa, C., Power, P., Echeverria,  E. and Buckingham Shum, S. (2020). Designing translucent learning analytics with teachers: an elicitation process. Interactive Learning Environments: to appear (Accepted: 20 October 2019).

Learning Analytics (LA) systems can offer new insights into learners’ behaviours through analysis of multiple data streams. There remains however a dearth of research about how LA interfaces can enable effective communication of educationally meaningful insights to teachers and learners. This highlights the need for a participatory, horizontal co-design process for LA systems. Inspired by the notion of translucence, this paper presents LAT-EP (Learning Analytics Translucence Elicitation Process), a five-step process to design for the effective use of translucent LA systems. LAT-EP was operationalised in an authentic multimodal learning analytics (MMLA) study in the context of teamwork in clinical simulation. Results of this process are illustrated through a series of visual proxies co-designed with teachers, each presenting traces of social, physical, affective and epistemic evidence captured while teams of student nurses practised clinical skills in a simulated hospital setting.

Congratulations Dr. Vanessa Echeverria!

It is with enormous pleasure that I applaud Vanessa Echeverria Barzola for completing her PhD here at UTS:CIC!

Moving with her family from Ecuador in 2017, Vanessa joined as one of our first doctoral researchers when we launched the Learning Analytics PhD Program, and has wrapped it up within her four years, demonstrating huge personal resilience on the way.

Read her thesis below, related CIC News stories, and browse her website for the extensive publications generated by her work. We wish her every success as she develops her academic career, and can’t wait to see what emerges!

Vanessa Echeverria (2020). Designing Feedback for Collocated Teams using Multimodal Learning Analytics. Doctoral Dissertation, Connected Intelligence Centre, University of Technology Sydney, Australia. http://hdl.handle.net/10453/140936

Abstract: The ability to communicate, be an effective team or group member and collaborate face-to-face are critical skills for employability in the 21st century workplace. Previous research suggests that learning to collaborate effectively requires practice, awareness of group dynamics and reflection upon past activities. However, although having a teacher closely supervising and providing detailed feedback to each group would be ideal, it may be unrealistic in practice. A promising way to approach this challenge could be to capture behavioural traces from group interactions in order to generate comprehensible and actionable feedback to support team reflection. In this sense, Multimodal Learning Analytics (MMLA) is a promising field, offering the potential to track learners’ activity across digital and collocated contexts, using emerging sensing and pervasive computing technologies. Most of the research in MMLA has been conducted in lab conditions, to help researchers validate learning theories or generate more comprehensive learner models. However, one of the most underexplored aspects of MMLA has been the generation of feedback to support teaching and learning, and moreover, in authentic locations and activities.

This thesis reports progress in tackling this challenge by designing and validating computer-based feedback, by means of visual representations and narrative, to support effective, guided reflection using multimodal learning analytics evidence. To achieve this, three contributions are presented. The first contribution is a human-centred design method to translate the informal outputs of co-design sessions with teachers and students, into more meaningful group work constructs with clear MMLA design requirements. The second contribution is a modelling approach to add meaning to low-level multimodal group data based on the characteristics of the context (domain expertise, theory, and the learning design). Finally, the third contribution is an approach for augmenting visual representations with data storytelling elements to facilitate the interpretation of group dynamics insights by educators and students. This thesis is developed in the context of two distinct, collocated group work settings, in the domains of collaborative database design and healthcare simulation. Using a Design-Based Research process, a set of interfaces (i.e. interfaces that communicate insights) was designed and validated with teachers and students. The thesis provides timely and necessary groundwork for researchers and practitioners to design visual representations capable of communicating actionable insights, using multimodal data in complex and authentic collaboration scenarios.