Tag: participatory-design

Human-Centred Learning Analytics: 2019-2024 (BJET)

Buckingham Shum, S., Martínez-Maldonado, R., Dimitriadis, Y., & Santos, P. (2024). Human-Centred Learning Analytics: 2019-24. British Journal of Educational Technology, 55(3), 755-768. https://doi.org/10.1111/bjet.13442 (Editorial to a Special Section Human-Centred Learning Analytics)
BJET logo

In 2018, working with Rebecca Ferguson and Roberto Martínez-Maldonado, I co-edited a special section of the Journal of Learning Analytics introducing the thematic priority of Human-Centred Learning Analytics (HCLA), published in 2019. We called for LA to engage with the rich diversity of HCI theories, design processes and empirical methods, with explicit attention to meaningful engagement with educational stakeholders, to design LA tools that augment teaching practices and learner behaviour, and through this, illuminating the sociotechnical factors influencing the successful adoption (or rejection) of LA tools.

Since then, we have seen the HCLA community grow through annual international workshops linked to major conferences, and several literature reviews have now been published. So 5 years on, it seemed timely to bring together a new collection of work as a snapshot of the current state of the art, and we were delighted that the British Journal of Educational Technology chose from its call for special sections. In our editorial we reflect on the papers, how the field has developed, and what the next 5 years might hold. Here’s the abstract which is  now published for early online access, with the special section formally published in May:

Human-Centred Learning Analytics (HCLA) has emerged in the last 5 years as an active sub-topic within Learning Analytics, drawing primarily on the theories and methods of Human-Computer Interaction (HCI). HCLA researchers and practitioners are adopting and adapting HCI theories/methods to meet the challenge of meaningfully engaging educational stakeholders in the LA design process, evaluating systems in use, and researching the sociotechnical factors influencing LA successes and failures. This editorial introduces the contributions of the papers in this special section, reflects more broadly on the field’s emergence over the last five years, considers known gaps, and indicates new opportunities that may open in the next five years.

I hope you find this a provocative collection that inspires you to bring the voices of stakeholders more strongly and meaningfully into the design process.

Special Section Human-Centred Learning Analytics – papers and abstracts:

Campos, F., Nguyen, H., Ahn, J., & Jackson, K. (2023). Leveraging cultural forms in human-centred learning analytics design

In this article, we offer theory-grounded narratives of a 4-year participatory design process of a Learning Analytics tool with K-12 educators. We describe how we design-in-partnership by leveraging educators’ routines, values and cultural representations into the designs of digital dashboards. We make our long-term reasoning visible by reflecting upon how design decisions were made, discussing key tensions and analysing to what extent the developed tools were taken up in practice. Through thick design narratives, we reflect upon how cultural forms—recognizable cultural constructs that might cue and facilitate specific activities—were identified among educators and informed the design of a dashboard. We then examined the extent to which the designed tool supported coaches and teachers to engage in Generative Uncertainty, an interpretive stance in which educators manifest productive inquiries towards data. Our analysis highlights that attuning to cultural forms is a valuable first step but not enough towards designing LA tools for systems in ways that fit institutionalized practices, challenge instrumental uses and spur productive inquiry. We conclude by offering two key criteria for making culturally-grounded design decisions in the context of long-term partnerships.

Hilliger, I., Miranda, C., Celis, S., & Pérez-Sanagustín, M. (2023). Curriculum analytics adoption in higher education: A multiple case study engaging stakeholders in different phases of design

Several studies have indicated that stakeholder engagement could ensure the successful adoption of learning analytics (LA). Considering that researchers and tech developers may not be aware of how LA tools can derive meaningful and actionable information for everyday use, these studies suggest that participatory approaches based on human-centred design can provide stakeholders with the opportunity to influence decision-making during tool development. So far, there is a growing consensus about the importance of identifying stakeholders’ needs and expectations in early stages, so researchers and developers can design systems that resonate with their users. However, human-centred LA is a growing sub-field, so further empirical work is needed to understand how stakeholders can contribute effectively to the design process and the adoption strategy of analytical tools. To illustrate mechanisms to engage various stakeholders throughout different phases of a design process, this paper presents a multiple case study conducted in different Latin American universities. A series of studies inform the development of an analytical tool to support continuous curriculum improvement, aiming to improve student learning and programme quality. Yet, these studies differ in scope and design stage, so they use different mechanisms to engage students, course instructors and institutional administrators. By cross analysing the findings of these three cases, three conclusions emerged for each design phase of a CA tool, presenting mechanisms to ensure stakeholder adoption after tool development. Further implications of this multiple case study are discussed from a theoretical and methodological perspective.

Hutchins, N. M., & Biswas, G. (2023). Co-designing teacher support technology for problem-based learning in middle school science

This paper provides an experience report on a co-design approach with teachers to co-create learning analytics-based technology to support problem-based learning in middle school science classrooms. We have mapped out a workflow for such applications and developed design narratives to investigate the implementation, modifications and temporal roles of the participants in the design process. Our results provide precedent knowledge on co-designing with experienced and novice teachers and co-constructing actionable insight that can help teachers engage more effectively with their students’ learning and problem-solving processes during classroom PBL implementations.

Lawrence, L., Echeverria, V., Yang, K., Aleven, V., & Rummel, N. (2023). How teachers conceptualise shared control with an AI co-orchestration tool: A multiyear teacher-centred design process

Artificial intelligence (AI) can enhance teachers’ capabilities by sharing control over different parts of learning activities. This is especially true for complex learning activities, such as dynamic learning transitions where students move between individual and collaborative learning in un-planned ways, as the need arises. Yet, few initiatives have emerged considering how shared responsibility between teachers and AI can support learning and how teachers’ voices might be included to inform design decisions. The goal of our article is twofold. First, we describe a secondary analysis of our co-design process comprising six design methods to understand how teachers conceptualise sharing control with an AI co-orchestration tool, called Pair-Up. We worked with 76 middle school math teachers, each taking part in one to three methods, to create a co-orchestration tool that supports dynamic combinations of individual and collaborative learning using two AI-based tutoring systems. We leveraged qualitative content analysis to examine teachers’ views about sharing control with Pair-Up, and we describe high-level insights about the human-AI interaction, including control, trust, responsibility, efficiency, and accuracy. Secondly, we use our results as an example showcasing how human-centred learning analytics can be applied to the design of human-AI technologies and share reflections for human-AI technology designers regarding the methods that might be fruitful to elicit teacher feedback and ideas. Our findings illustrate the design of a novel co-orchestration tool to facilitate the transitions between individual and collaborative learning and highlight considerations and reflections for designers of similar systems.

Wiley, K., Dimitriadis, Y., & Linn, M. (2023). A human-centred learning analytics approach for developing contextually scalable K-12 teacher dashboards

This paper describes a Human-Centred Learning Analytics (HCLA) design approach for developing learning analytics (LA) dashboards for K-12 classrooms that maintain both contextual relevance and scalability—two goals that are often in competition. Using mixed methods, we collected observational and interview data from teacher partners and assessment data from their students’ engagement with the lesson materials. This DBR-based, human-centred design process resulted in a dashboard that supported teachers in addressing their students’ learning needs. To develop the dashboard features that could support teachers, we found that a design refinement process that drew on the insights of teachers with varying teaching experience, philosophies and teaching contexts strengthened the resulting outcome. The versatile nature of the approach, in terms of student learning outcomes, makes it useful for HCLA design efforts across diverse K-12 educational contexts.

What could Learning Analytics learn from HCI theory?

It’s good to share this chapter that’s been brewing for about a year now, with helpful feedback from quite a few colleagues en route, gratefully acknowledged. I go back to my roots and share some viewpoints from HCI on my current field of Learning Analytics. This preprint will appear (subject to minor production edits) in the forthcoming book:

Buckingham Shum, S. (In Press). What could Learning Analytics learn from Human-Computer Interaction theory? In: Kathryn Bartimote, Sarah Howard & Dragan Gašević (Eds.), Theory Informing and Arising from Learning Analytics. Springer Nature

Abstract: The design of Learning Analytics (LA) tools is an example of the general problem of designing interactive tools, which is the focus of Human-Computer Interaction (HCI) research and design practice. LA as a field must understand how to embed LA into organisations and the design of effective, trustworthy human-computer systems is where HCI theory and practice have much to offer. Consequently, this chapter argues that LA can learn from (i) the way that theory has evolved in HCI, (ii) the field’s methods for evaluating interactive systems at different scales, and (iii) HCI debates how established scientific theories and methods relate to design theories and methods. As a highly interdisciplinary applied field, LA (like HCI) faces the challenge of maintaining academic standards in the conduct and review of research from many disciplinary traditions. I propose that HCI offers inspiration for researchers seeking rigorous methods to design and evaluate LA in authentic contexts, including principles to maintain their intellectual rigour, which will also be of interest to LA journals and conferences seeking to maintain peer review standards.

[Update 11 Nov 2024] The book is due out soon, and includes in conversation chapters and podcasts:

“Theory Informing and Arising from Learning Analytics delves into the dynamic intersection of learning theory and educational data analysis within the field of Learning Analytics (LA). This groundbreaking book illuminates how theoretical insights can revolutionize data interpretation, reshape research methodologies, and expand the horizons of human learning and educational theory. Organized into three distinct sections, it offers a comprehensive introduction to the role of theory in LA, features contributions from leading scholars who apply diverse theoretical frameworks to their research, and explores cutting-edge topics where new theories are emerging. A standout feature is the inclusion of three “in conversation” chapters, where expert panels dive into the topics of ethics, self-regulated learning, and qualitative computation, enriched by accompanying podcasts that provide fresh, thought-provoking perspectives. This book is an invaluable resource for researchers, sparking debates on the evolving role of theory in LA and challenging conventional epistemological views. Published by Springer, it is an essential read for both aspiring and seasoned scholars eager to engage with the forefront of LA research.”

Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom

Here’s the open access preprint of a chapter to appear in a forthcoming book edited by Olga Viberg and Åke Grönlund. I was grateful to be invited to contribute to this, and it was an enjoyable reflective journey figuring out how to tell the story. I hope you enjoy exploring the four rooms!

Buckingham Shum, S. (2022). Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom. In Viberg, O. and  Grönlund, Å. (Eds.), Practicable Learning Analytics, SpringerNature.

Abstract: In this chapter, I describe and reflect on the last 8 years at an Australian public university, inventing, piloting and evaluating Learning Analytics tools, specifically focused on data-driven personalised feedback, leading in some cases to integration with the institution’s learning technology ecosystem, and accompanied by staff training and support. I will summarise this as conversations in the Boardroom, the Staff Room, the Server Room and the Classroom, reflecting the different levels of influence, partnership and adaptation required to introduce and sustain novel technologies in the complex system that constitutes a university, or indeed, any educational institution. This chapter is pragmatic, documenting aspects of our work that are typically not the focus in research papers, intending to make a practice contribution.

Keywords: Organisational Strategy, Innovation Diffusion, Personalised Feedback

Dec’22 update: I was invited by the Leiden-Delft-Erasmus Universities Centre for Education and Learning to share this work with them at their annual conference so here’s the replay [slides], together with the fab live drawing  by Mark van Huystee!

Live drawing by Mark van Huystee

AIED2021 Human-Centred Design session

The International Conference on Artificial Intelligence in Education has been running online all week. The theme this year is Mind the Gap: AIED for Equity and Inclusion, reflecting justified concerns in society at large, about the potential for data, analytics and AI to exacerbate societal inequities. Given the call for the community to work on “diversity, equity, and inclusion practices”, I was delighted to be asked to host a discussion session on Human-Centred Design.

Here we are in gather.town (worked nicely, and note the Covid-safe seating plan!) — below are the key points and links I posted to the chat during the session, which I’ve touched up slightly to make them intelligible… 

Firstly, from my perspective, designing AIED is a specific instance of the broader challenge of designing interactive systems that people value and use. So we can and should draw on the wealth of knowledge out there on how to do this well.

Secondly, I want to flag that HCD is about far more than nice user interfaces! Depending on the scale of lens you want to use, for me, we’re talking about understanding how socio-technical infrastructures get embedded into daily life.

So Informatics provides us with the broad lens needed to integrate computational artifacts into human ecosystems. Hence I frame my work as “Learning Informatics” https://simon.buckinghamshum.net/2020/09/why-learning-informatics

Human-Computer Interaction (HCI) aka Human-Centred Informatics provide a wealth of methodologies to study how people engage with interactive tools.

  • Dan Russell’s opening keynote emphasised the vital importance of an HCI skillset for designing effective AIED, especially when that intelligence is imperfect
  • Great book: “Ways of Knowing in HCI” edited by Judy Olson & Wendy Kellogg https://www.springer.com/gp/book/9781493903771
  • A great book on how the meanings and roles of “theory” have evolved in HCI (which might spark thoughts wrt how AIED is evolving) “HCI Theory: Classical, Modern, and Contemporary” by Yvonne Rogers https://doi.org/10.2200/S00418ED1V01Y201205HCI014
    • Note in particular that when HCI started out (the first ACM CHI conference was in the early 80s) we sought theory to get a grip on the dominant computing paradigm: individual in front of a computer. Cognitive psychologists believed they brought the concepts and tools needed to design and evaluate user interfaces, but that things have come a long way since.

The current important interest in FATE of AIED (https://doi.org/10.1007/s40593-021-00239-1) boils down to the trustworthiness of LA/AIED systems. This is about a lot more than opening black boxes. A diverse set of arguments underpins the claim that a system should be considered trustworthy https://simon.buckinghamshum.net/2019/11/black-box-learning-analytics

HCI is now coming into dialogue with Learning Analytics & AIED:

  • BJET special issue “Learning Analytics and AI: Politics, Pedagogy and Practices” https://onlinelibrary.wiley.com/toc/14678535/2019/50/6
  • Jnl Learning Analytics special issue “Human-Centred Learning Analytics”  https://learning-analytics.info/index.php/JLA/issue/view/463
    • Note: in the editorial we discuss briefly whether there are any features of education that make HCD different from other domains. I mentioned this:
      “In most HCI design contexts, stakeholders are treated as authoritative sources on how their work should be performed. Current work practices are studied to ensure that the envisaged software system does not inadvertently disrupt the human ecology of formal and informal activity. In sharp contrast, for HCLA, while learners are obviously able to speak with authority about their experiences of studying, they are

      • not expert learners whose work practices should necessarily be worked around;
      • not experts in the subject matter;
      • not expert educators whose views —e.g., about the design of a course, what counts as good feedback, or what analytics will help learning — can be treated as authoritative.”

There was some good discussion about whether there is anything distinctive about AIED systems that requires the invention of special HCD techniques to aid, for example, rapid prototyping.

  • We noted the use of Wizard of Oz
  • Paper prototyping, but ensuring that workshop participants are empowered to co-construct the designs
  • The tuning of existing techniques specifically for our domain, examples below

Specific examples of human-centred design for LA/AIED in recent CIC PhDs — Antonette Shibani, Vanessa Echeverria and Carlos Prieto-Alvarez:

 

We’re translating this into HCD training/resources for researchers and educators:

Backing out to the bigger picture, as researchers, we’re transitioning education into a new kind of “knowledge infrastructure” — the system of systems that interoperate technically and socially to generate, sanction and maintain knowledge about a field  https://simon.buckinghamshum.net/2018/06/icls2018-keynote

 

ICQE20 keynote: QE Visualizations as tools for thinking

We just wrapped up the 2nd International Conference on Quantitative Ethnography (open access proceedings from Springer), postponed from October due to the pandemic, in the hope that we might all yet meet up in February — alas it was not to be. However, the organisers did a really great job designing the program with a lot of informal interaction time, and the delegates threw themselves into it with a fantastic spirit, with everyone out to help everyone, as the crew figures out how to sail this recently launched ship!

If QE is new to you, it springs from the foundational work at the University of Wisconsin-Madison’s Epistemic Analytics Lab, led by the inspirational David Williamson Shaffer. The team’s publications are the source point, specifically, David’s QE book, which impressed me so much and drew me into this vision of how quant+qual could come together. In fact, I first met David in 2013 after I received a very hot tip that he was doing amazing work, relevant to a discourse analytics workshop I was chairing. His keynote was a revelation to me of his team’s long term research program, but it’s taken a while to figure out if and how to bring it into my own work.

Well, the first conference (ICQE 2019 / proceedings) was a great success, and so it was a real honour to be asked to give one of the keynote talks at this year’s conference. However, following the brilliant 2019 keynotes by Jim Gee, Dragan Gašević and Gol Arastoopour Irgens, I accepted with some trepidation to be honest, since I am far from a QE expert compared to those blazing this new trail. At the time of being invited, my team had not done any work with the main QE analysis approach, Epistemic Network Analysis, though we had drawn inspiration from its data modelling methodology in our multimodal learning analytics work.

So I thought long and hard about what I might bring to the party, with several false starts, which might have gone deeper into QE and Learning Analytics, or QE and Algorithmic Accountability. I decided in the end to go back to my roots — all the way back to my PhD in fact, focusing on the cognitive affordances of semiformal graphical representations, and what I’ve learnt since about what it takes to wield such tools in participatory design with fluency, developing open source visual hypermedia software for 20 years, and more recent work on data storytelling. What was particularly fun was bringing that into dialogue with what I was seeing in the QE webinars last year, specifically, how the community is telling its stories with visualizations. It was really enjoyable thinking what my journey might have to say to the QE community, the talk seemed to go down well, and I’m looking forward to seeing if/how these ideas take deeper root.

I am indebted to so many colleagues who have shaped those ideas, noting in particular, the Knowledge Art research and practice of my PhD student, colleague and friend, Al Selvin, tragically taken from us, far too early.

Quantitative Ethnography Visualizations as Tools for Thinking [pdf slides]

Abstract: All research must give form to data and insights. Visualizations serve as cognitive extensions that assist researchers not only in exploring their data, but in communicating findings to colleagues and broader audiences. Especially in data-intensive fields, widely used software tools define, and are defined by, research communities; you can’t fully participate in a community until you can wield its tools responsibly. In an emerging field like Quantitative Ethnography (QE), inventing its own tools, how we model and map the world are therefore defining characteristics, and merit critical reflection.

QE’s principles currently find fullest expression in Epistemic Network Analysis (ENA). It’s fair to say that the interest in ENA is attributable not only to the power of its data modelling and analysis, but also to the engaging, interactive visualizations it generates. Inspired by the ways I see ENA used, in this talk I bring my background in Human-Computer Interaction and the design of tools for working with conceptual structures, as a lens on ENA and other QEgenerated visuals. When we consider in detail how external representations serve as personal and shared cognitive tools, this illuminates current and future techniques for presenting QE analyses. A data-storytelling lens asks how the audience will engage with our insights, while participatory methods ask whether we cast them as passive recipients or active agents in validating those narratives. Moreover, as QE analyses begin to underpin new tools designed for people other than QE researchers, human-centred design should give voice to non-technical stakeholders. These lenses could point to a future in which visualization tools evolve to scaffold more participatory forms of sensemaking as an important hallmark of how QE models and narrates the world.

Why “Learning Informatics”?

One of the privileges of becoming a professor is to choose your title. Exciting but a challenge: encapsulate everything you’re passionate about in just a few words, which aren’t going to date too fast as thinking moves on.

I thought hard about this in 2014 when I was at The Open University UK. Learning Analytics was the hot new thing, but who knew how that was going to pan out? (very well as it happens!). But it  seemed too early to nail all my colours to this mast.

There was a bigger picture, but what was its name? My home-base was Human-Computer Interaction, with the ACM CHI and BCS HCI conferences my stamping ground as a PhD student and early postdoc. But I’d moved into a range of other communities since, and at the OU the focus was now firmly on the role of knowledge media in shaping the future of learning. Human-Centred Computing was too broad, so how about Human-Centred Educational Technologies? Knowledge Media? Learning Technologies? 

I reflected on which movements in HCI best expressed the richness of perspective that I found so exciting. And there it was staring me in the face: Informatics. 

That definition comes from Kristen Nygaard‘s invited address to the 1986 World Computer Congress, entitled Program Development as a Social Activity. Informatics was a longstanding term in Europe, and was spreading in the US and elsewhere (perhaps in part as an extension of the move to creating broad, rich iSchools — someone more familiar than me with that history might comment on this).

So, I married Learning + Informatics. With the launch last year of the Learning Informatics Lab at University of Minnesota, I was delighted to be invited by Bodong Chen to give this talk (but sadly that trip was cancelled). However, we finally put that right this week, and here it is: why in my view Learning Informatics offers the depth and breadth we need to design learning analytics and AI in truly human-centred ways.

Dedicated to the extraordinary life and work of Kristen Nygaard! You will see in his reflections on the shaping of participatory design methods with trades unions and management, and definition of informatics, prescient ideas that are as vital now as then.

Learning Informatics: AI • Analytics • Accountability • Agency

Slides [pdf]

Abstract: “Health Informatics”. “Urban Informatics”. “Social Informatics”. Informatics offers systemic ways of analyzing and designing the interaction of natural and artificial information processing systems. In the context of education, I will describe some Learning Informatics lenses and practices which we have developed for co-designing analytics and AI with educators and students. We have a particular focus on closing the feedback loop to equip learners with competencies to navigate a complex, uncertain future, such as critical thinking, professional reflection and teamwork. En route, we will touch on how we build educators’ trust in novel tools, our design philosophy of “embracing imperfection” in machine intelligence, and the ways that these infrastructures embody values. Speaking from the perspective of leading an institutional innovation centre in learning analytics, I hope that our experiences spark productive reflection around as the UMN Learning Informatics Lab builds its program.

Congratulations, Dr. Carlos Prieto-Alvarez!

I particularly enjoy writing these blog posts, celebrating the publication of a doctoral thesis. The blood, sweat and tears by doctoral researchers that goes into this moment is always immense, when one thinks about the learning curve they go through, the highs and the lows, and the balancing act of juggling this with the rest of life over 3-4 years. But here we are again, and I’m delighted to add a new one to these team posts!

Well done Carlos Prieto-Alvarez, as today sees the publication of one of the first PhDs in Learning Analytics devoted to the contributions of human-centred design methods (specifically, co-design) to give non-technical stakeholders such as educators and students a voice in shaping Learning Analytics. My thanks to co-supervisor Roberto Martinez-Maldonado, and also to Theresa Anderson who was on the team for the first year or so. Carlos is the third graduate from the CIC Learning Analytics PhD Program launched in 2016!

Prieto-Alvarez, C.G. (2020), Engaging Stakeholders in the Learning Analytics Design Process. Doctoral Dissertation, Connected Intelligence Centre, University of Technology Sydney, AUS. http://hdl.handle.net/10453/142525

The abstract and full dissertation are below, read the CIC news stories over the years from his work, and replay his final year thesis presentation:

Engaging Stakeholders in the Learning Analytics Design Process

ABSTRACT:

Learning Analytics (LA) is a new promising field that is attracting the attention of education providers and a range of stakeholders including teachers, learning designers academic directors and data scientists. Researchers and practitioners are interested in learning analytics as it can provide insights from student data about learning processes, learners who may need more help, and learners’ behaviours and strategies. However, problems such as low educator satisfaction, steep learning curves, misalignment between the analytics and pedagogical approaches, lack of engagement with learning technologies and other barriers to learning analytics development have already been reported. From a human-centred design perspective, these problems can be explained due to the lack of stakeholders’ involvement in the design of the LA tools. In particular, learners and teachers are commonly not considered as active agents of the LA design process. Including teachers, learners, developers and other stakeholders as collaborators in the co-design of LA innovations can bring promising benefits in democratising the LA design process, aligning analytics and pedagogy, and meeting stakeholders’ expectations. Yet, working in collaboration with stakeholders to design LA innovations opens a series of questions that are addressed in this thesis in order to contribute to closing the gap for effective co-design of LA innovations. The questions addressed in this thesis are the following:

  1. How can co-design techniques assist in the integration of diverse stakeholders in the LA design process?
  2. What are the roles of the co-design practitioner/researcher in the LA design process?
  3. What are the challenges in engaging stakeholders in the LA design process?

Based on co-design principles, and following a Design-Based Research process, this thesis explores the critical challenge of engaging educators and students, the non-technical stakeholders who are often neglected, but who should ultimately be the main beneficiaries of LA innovations. In this research work, three case studies have been used to test, analyse and verify various co-design techniques in diverse learning contexts across a university to generate a co-design toolkit and recommendations for other co-design practitioners: i) learners and educators engaged in simulation-based healthcare scenarios, ii) learners, educators and other stakeholders in a Data Science Masters program, and iii) educators interested in providing personalised feedback at scale.

This thesis presents three contributions to knowledge for effectively collaborating with educational stakeholders in the LA co-design process:

  1. Inspired by archetypal challenges reported in classic and contemporary co-design literature, and in current LA research, the thesis identifies, exemplifies and reflects on five key challenges for LA co-design: power relationships, surveillance, learning design dependencies, asymmetric teaching/learning expertise, and data literacy.
  2. By adopting and adapting well established co-design techniques, across the three case studies, the thesis provides empirical evidence of how these techniques can be used in LA co-design, reflecting on their affordances, and providing guidance on their usage. These detailed findings are distilled into a Learning Analytics Co-design Playbook, published under an open license to assist adoption and improvements.
  3. Recognising the importance of the co-design practitioner in ensuring that the design process is participatory, the thesis documents and discusses the key functions and skills that this position requires. The role is further complicated when the practitioner is not only a facilitator serving a project, but also a researcher of co-design. This motivates guidelines on the role of the co-design practitioner/researcher when working with stakeholders, and simultaneously studying the LA co-design process, tools and methods.

See Carlos’ ResearchGate site for full-text papers.

Conference Papers

  • Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2020). LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK2020), Frankfurt, Germany, March 2020, ACM, New York, NY, USA. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476
  • Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. (2018). Mapping Learner-Data Journeys: Evolution of a Visual Co-design Tool. Proceedings of the 30th Australian Conference on Computer-Human Interaction (OzCHI ’18), Melbourne, Australia, Dec. 2018, ACM, New York, NY, USA, pp. 205–214. DOI: https://doi.org/10.1145/3292147.3292168
  • Prieto-Alvarez, C.G, et al. (2018). Collaborative Personas for Crafting Learners Stories for Learning Analytics Design. Workshop Participatory Design for Learning Analytics, International Conference on Learning Analytics and Knowledge LAK’18. Sydney, Australia, ACM: 647-652. ISBN: 978-1-4503-6400-3

Book Chapter

  • Prieto-Alvarez, C.G., Martinez-Maldonado, R. Anderson, T. (2018). Co-designing learning analytics tools with learners. Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers, Taylor & Francis Groups: 93-110.

Workshops

  • Carlos G. Prieto-Alvarez et al (2018). Learning Analytics Design Cards (LA-DECK): Unpacking inter stakeholder co-design through strategic cards. Australian Learning Analytics Summer Institute. Melbourne, Australia. Website: http://ladeck.utscic.edu.au/events.html
  • Carlos G. Prieto-Alvarez et al (2018). Participatory design of learning analytics. International Conference on Learning Analytics and Knowledge LAK’18. Sydney, Australia, ACM. Website: http://pdlak.utscic.edu.au.

LA-DECK: A Card-Based Learning Analytics Co-Design Tool

Here’s a new tool (free download) and accompanying article from the doctoral research of Carlos Prieto Alvarez, in which we document the design rationale and empirical evaluation of a deck of cards specifically tuned to Learning Analytics. In LA-DECK sessions, participants ‘play a card’ and link it to others, to mark particular kinds of contributions to the conversation. This is a common approach to structuring the conversation, and in the process develops a tangible representation of the group’s work.

Carlos will be presenting this in a few weeks at LAK2020:

Prieto-Alvarez, C.G., Martinez-Maldonado, R. and Buckingham Shum, S. 2020. LA-DECK: A Card-Based Learning Analytics Co-Design Tool. Proceedings of the 10thInternational Conference on Learning Analytics and Knowledge, Frankfurt, Germany, March 2020 (LAK’20), ACM, New York, NY, USA. 10 pages. DOI: https://doi.org/10.1145/3375462.3375476 [Open Access ePrint]

ABSTRACT: Human-centred software design gives all stakeholders an active voice in the design of the systems that they are expected to use. However, this is not yet commonplace in Learning Analytics (LA). Co-design techniques from other domains therefore have much to offer to LA, in principle, but there are few detailed accounts of exactly how such sessions unfold. This paper presents the rationale driving a card-based co-design tool specifically tuned for LA, called LA-DECK. In the context of a pilot study with students, educators, LA researchers and developers, we provide qualitative and quantitative accounts of how participants used the cards. Using three different forms of analysis (transcript-centric design vignettes, card-graphs and time-on-topic), we characterise in what ways the sessions were “participatory” in nature, and argue that the cards succeeded in playing very similar roles to those documented in the literature on successful card-based design tools.

This is one case study from the broader work of Carlos’ PhD, investigating the challenges of introducing co-design methods that give a voice to non-technical stakeholders in Learning Analytics. Here’s a recent seminar in which he reviews the whole PhD. If you don’t have time to watch the whole thing (well worth it!), then jump to 19:21 for LA-DECK…

Human-Centred Analytics/AI in Education

Note: this page has been updated as these special issues were published.

A heads-up that three collections will hit the streets this year focused on how we can design so that human needs and values are well and truly centre-stage in educational tools powered by data, analytics and AI. It will be good to have detailed ‘insider accounts’ from researcher/developers who are reflecting deeply on how values are baked into their design practices and the infrastructures they are building, and how different stakeholders can engage meaningfully in shaping design. I’m excited about the papers shaping up for these volumes, so watch out for their releases mid- and end-2019…

Human-Centred Learning Analytics. Journal of Learning Analytics, 6(2), pp. 1–94 (Eds.) Simon Buckingham Shum, Rebecca Ferguson, & Roberto Martinez-Maldonado

What’s the Problem with Learning Analytics? Journal of Learning Analytics, 6(3), pp. 5-42. (Ed.) Simon Buckingham Shum.

Diverse reflections on an article by Neil Selwyn, based on his provocative keynote address to the 2018 International Conference on Learning Analytics & Knowledge. Commentaries from Carolyn Rosé, Rebecca Ferguson, Paul Prinsloo & Alfred Essa. [Replay the keynote]

Buckingham Shum, S.J. & Luckin, R. (2019), Eds: Learning Analytics and AI: Politics, Pedagogy and Practices. British Journal of Educational Technology (50th Anniversary Special Issue), 50, (6), pp.2785-2973.

While there is a growing chorus of justifiably cautionary voices about the dark sides of data, algorithms and machine intelligence when used uncritically in education, sometimes these are from commentators some distance from the ‘nuts and bolts’. This issue will provide accounts from insiders, all of whom have agreed to engage with the theme of “Politics, Pedagogy and Practices”, whose dynamics play out at many organisational scales:

Practices: We are seeking informed accounts of how these technologies come into being — the social and material practices of designing analytics and AI educational tools, and the related practices of educators and other stakeholders needed to deploy these tools.

Pedagogy: For some critics, analytics and AI equate to adopting a retrograde pedagogy from the industrial era. Any mention of quantification, or machine intelligence, evokes connotations of behaviourism or instructivism. Contributions to this issue will question such simplistic assumptions, illustrating a range of pedagogies and associated outcomes.

Politics:From international educational datasets gathered by governments and corporations, to personal apps, in a broad sense politics infuse any socio-technical infrastructure, because it mediates values and power. How do the researchers and developers of these tools frame their work in relation to concerns around values, ethics, and societal impact?

This issue will be written for a broad audience, introducing what is or soon will be possible, and describing strategies for taking into account data/algorithm/AI ethics. Written also for seasoned researchers, it will synthesise and clarify contemporary debates, providing a reference point for both teaching, teacher development and research.

Dec. 2020 update:

Momentum has continued to build around HCLA, leading to the First International Workshop on HCLA next April at LAK2021.

Pharmacy reflective practice: aligning learning design and analytics

The start of the new year is a good moment to distill the key ingredients of the 2 year collaboration around writing analytics (automated feedback to students on their reflective writing), that CIC has built with UTS academic Cherie Lucas from our School of Pharmacy. As with other collaborations (such as with Pip Ryan on her students’ legal writing), it exemplifies the co-design process that we initiate with academics, in which we iteratively seek to design an automated feedback tool that students can use to improve their drafts, prior to submission:

  • distill key insights from the scholarship into the teaching and learning of good reflective writing, to design a formal, implementable model that – in principle – should be applicable to a wide range of reflective writing contexts (learn more)
  • understand the academic’s specific educational challenge in context (e.g. students are struggling to  produce good reflective writing about their work experience placements)
  • establish the mapping to the features that our text analytics tool is able to detect
  • evaluate the performance of the parser (in close partnership with the academic)
  • co-design the feedback messages that students will receive depending on their writing
  • evaluate students’ reactions to the new tool
  • iterate…

In the video, Cherie Lucas describes the nature of the challenge, and what AcaWriter contributes to the learning experience. The interface looks like this, with the text editor frame on the left, and the Reflective Report annotation of the writing on the right, generated after a few seconds on clicking Get Feedback:

Zooming in on the automatically annotated student writing in the Reflective Report:

Not shown in the video is the Feedback Tab providing encouragement when there appear to be good features in the text, and actionable feedback for improvement, e.g.

For us, one of the hallmarks of a successful collaboration is that our academic partners’ own disciplinary community of educators recognise the advance they’ve made. Here’s a brief, very helpful introduction that Cherie wrote for her peers. Her work has excited significant interest with colleagues around the world, who are now initiating their own projects to install our software for piloting with their students.

Learn more about how we design Writing Activities with Writing Analytics through the integration of learning design, analytics, educator and student resources, and evaluation evidence…

Dive deeper…

Gibson A., Aitken A., Sándor Á., Buckingham Shum S., Tsingos-Lucas C. and Knight S. (2017), Reflective writing analytics for actionable feedback. Proceedings of LAK17: 7th International Conference on Learning Analytics and Knowledge, March 13-17, 2017, Vancouver, CA (ACM Press: NY). [Video] (AWARDED BEST PAPER)

Liu, M., Buckingham Shum, S., Mantzourani, E. and Lucas, C. (2019). Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. Proceedings AIED2019: 20th International Conference on Artificial Intelligence in Education, June 25th – 29th 2019, Chicago, USA. Lecture Notes in Computer Science & Artificial Intelligence: Springer.

Lucas C. (2016), The relationship between reflective practice, learning styles and academic performance in pharmacy education. Doctoral Dissertation, The University of Sydney, Australia. 2016

Lucas C. (2018), Accessorizing the Science Foundation with Internal Mirrors: A Novel Open Source Tool to Enhance Reflective Practice. Pulses. Currents in Pharmacy Teaching and Learning Scholarly Blog. August 28, 2018.

Lucas C, Gibson A. and Buckingham Shum S. (In Press), Utilization of a novel online reflective learning tool for immediate formative feedback to assist pharmacy students’ reflective writing skills. American Journal of Pharmacy Education.

Tsingos-Lucas C, Aitken A, Gibson A, Buckingham Shum S. (2017). Utilisation of a Novel Online Educational Toolto Assist Pharmacy Students to Self- Critique Reflective Writing Tasks. Proceedings of the 9th Pharmacy Education Symposium, Prato, Italy, 9-12th July 2017. (AWARDED BEST TEACHING INNOVATION POSTER)

ALASI2018: Co-designing learning analytics with LA-DECK

AUSTRALIAN LEARNING ANALYTICS SUMMER INSTITUTE

Learning Analytics Design Cards (LA-DECK): Unpacking interstakeholder codesign through strategic cards.

Carlos Gerardo Prieto Alvarez, Roberto Martinez-Maldonado and Simon Buckingham Shum (Connected Intelligence Centre, University of Technology Sydney)

This workshop introduces participants to LA-DECK — Learning Analytics Design Cards [preview the cards]. This workshop will work particularly well for a team who want to explore the provision of learning analytics for a system you all understand. In this way, you can assess the potential of LA-DECK for your context.

However, you can definitely come along solo, and then work in an ad hoc team. You’ll just have to agree either to work on a fictional learning analytics challenge we’ll provide (below), or one that a team member can easily explain from their context (but we want to minimise the time you spend deciding this).

Learning Analytics Challenge: You want to give personalized feedback to 1,000 students to help them improve their communication skills.

After a brief intro, the core of the workshop will be hands-on with the LA-DECK simulating a co-design process structured by the LA-DECK, concluding with a plenary debrief to share experiences, reflections and show examples.

JLA CfP: Human-Centred Learning Analytics

A reminder that you have until 31st Oct to submit to this exciting special section of JLA 🙂

GUEST EDITORS

Simon Buckingham Shum, University of Technology Sydney (Australia)

Rebecca Ferguson, The Open University (UK)

Roberto Martinez-Maldonado, University of Technology Sydney (Australia)

AIMS & SCOPE

An important feature of the learning analytics community is our interest in the human factors in learning analytics systems. When learning analytics tools are used, their success or failure must be judged not only on technical criteria, but also by their adoption and effectiveness in schools, universities and workplaces. Often this is where the gulf between hype and reality becomes apparent. The complexities of embedding innovative technology in authentic contexts open a range of critical challenges for the field. The theme of the 2018 Learning Analytics and Knowledge conference (LAK18) Towards User-Centred Design — how stakeholders can, or must, be engaged in the design, deployment and assessment of learning analytics. LAK18 also held its first Participatory Design workshop. We invite contributions to this special section that explore these issues in more depth.

TOPICS OF INTEREST

There are well-established research and design communities interested in human-centred design. The Human-Computer Interaction (HCI) community, for example, has worked hard to couple academic rigour with relevance in the fast-moving world of software design, evolving from an assemblage of disciplinary sciences towards practical design practices. Decades of work within communities under headings such as Participatory Design, User-Centred Design and Co-Design have led to many advances in theory, methodology and tools. ‘Human-centred’ can be defined at many levels, including the user interface, the impact on working practices, shifts in users’ power and control, and the values that are baked into the data models. The organisational obstacles to good user-centred design are well documented, since budget holders must be persuaded of the merits of investing money and effort in order to bring stakeholders into the design process. Most recently, of particular relevance to learning analytics, the user experience community has begun to engage with the specific challenges posed by interactive systems using machine learning. Building on decades of work in these communities, we need to apply and where necessary extend these approaches to the specific educational contexts in which we work. We therefore invite researchers and practitioners to submit theoretical, methodological, empirical and technical contributions including but not limited to:

  • Experiences deploying design processes that explicitly involve stakeholders (such as learners, educators, instructional designers, and leaders) in the co-design, co-creation or participatory design of analytics tools.
  • Evaluations of tools and techniques that have been effective in assessing how end-users make sense of, interact with, and act on analytics feedback.
  • Examples of how learning analytics systems can be made more transparent and accountable to different stakeholder groups.
  • Examples of how educational leaders can create the conditions for, or inadvertently undermine, human-centred learning analytics systems.
  • Examples of the benefits (and costs) that the adoption of human-centred design tools and techniques can bring to stakeholders.
  • Arguments/conceptual models/examples clarifying specific challenges of human-centred design for learning analytics, beyond those already well documented from other domains.

The special section will include two types of submission. First, invited extended versions of strong LAK18 papers on human-centred design. Second, additional papers submitted in response to this call. All submissions will undergo full peer review in accordance with JLA processes, in the context of their contributions to this call.

TIMELINE

Optional abstract pre-submission (see submission procedures): 1 June 2018 

Deadline for submissions: 31 Oct, 2018

Target publication date: Issue 2 (July), 2019

SUBMISSION PROCEDURE

Prospective authors are encouraged, though not required, to submit an abstract of at most 300 words to the special-section editors in advance in order to check fit with the special section: Simon.BuckinghamShum@uts.edu.au; rebecca.ferguson@open.ac.uk; Roberto.Martinez-Maldonado@uts.edu.au

The editors will reply to authors of submitted abstracts with feedback within two weeks.

Final submissions will take place through JLA’s online submission system at http://learning-analytics.info

When submitting a paper, select the section “Special Section: Human-Centred Learning Analytics”.

All submissions should follow JLA’s standard manuscript guidelines and template available on the journal website, and will undergo peer review.

Queries may be sent to the special section editors (emails above).