I’ve just spent last week with PhD students at the European Joint Technology-Enhanced Learning Summer School. This was no less than the 18th such event, and it’s clear why this has established itself as an annual fixture in so many doctoral researchers’ and mentors’ calendars. They’ve got a winning mix of pechakucha intros, workshops led by both senior and PhD researchers, keynotes, speed-mentoring, great local food, scenic trips, and lots of time for all those random conversations that go unexpected places!
I was delighted to be invited to give Monday evening’s informal keynote (less a regular talk, more an opportunity to reflect on how you’re developing as a researcher). I shared some of my work-in-progress thinking, as I try to make sense of why the intersecting crises in our newsfeeds barely penetrate the ed-tech academic bubble, and whether the poly/perma/meta-crisis should shape our priorities. As the ridiculous title indicates, the challenges are almost too huge to frame coherently, but if you’re curious, here are the slides (abstract below), where I hope you’ll find at least one interesting thinker to chase down.
It’s always hard to know how such a provocation will go down, so I was delighted with the appetite to wrestle with these questions in many follow-up chats. I loved being immersed in such a cultural melting pot for a week, and given the topic, an added edge was meeting students from countries including Syria, Ukraine and Israel, who have lived/are living the daily hell the rest of us watch on screens.
Kudos to the lead team who orchestrated so effectively, everyone who created such a vibrant atmosphere, and sincere thanks for welcoming me into the special JTELSS community!
Can TEL save the planet?
Abstract. I don’t think it’s overstating matters to say that humanity finds itself at an inflection point. The interlocking crises can feel overwhelming (ecological; political; financial; technological; medical; spiritual…). And I don’t know about you, but I’m finding it increasingly surreal attending conferences where these are not mentioned, and seem to have zero impact on our work. Or is this just ridiculous ranting? Why indeed would irreversible ecosystem collapse (for example) change how we think about TEL, pedagogy, analytics or AI? Sure, it’s really sad, but does it make sense to ask how this impacts our research? So, while it’s an exhilarating time to be working on TEL given all the AI advances, the societal challenges are daunting, and I find myself reflecting increasingly on whether this brings a responsibility to those of us who invent the future of TEL. How do we go about wrestling with this? How do we stay hopeful? I invite you to hear my thoughts-in-progress, and disagree with anything I say! We have a whole week to discuss and sort this out…
In 2018, working with Rebecca Ferguson and Roberto Martínez-Maldonado, I co-edited a special section of the Journal of Learning Analytics introducing the thematic priority of Human-Centred Learning Analytics (HCLA), published in 2019. We called for LA to engage with the rich diversity of HCI theories, design processes and empirical methods, with explicit attention to meaningful engagement with educational stakeholders, to design LA tools that augment teaching practices and learner behaviour, and through this, illuminating the sociotechnical factors influencing the successful adoption (or rejection) of LA tools.
Since then, we have seen the HCLA community grow through annual international workshops linked to major conferences, and several literature reviews have now been published. So 5 years on, it seemed timely to bring together a new collection of work as a snapshot of the current state of the art, and we were delighted that the British Journal of Educational Technology chose from its call for special sections. In our editorial we reflect on the papers, how the field has developed, and what the next 5 years might hold. Here’s the abstract which is now published for early online access, with the special section formally published in May:
Human-Centred Learning Analytics (HCLA) has emerged in the last 5 years as an active sub-topic within Learning Analytics, drawing primarily on the theories and methods of Human-Computer Interaction (HCI). HCLA researchers and practitioners are adopting and adapting HCI theories/methods to meet the challenge of meaningfully engaging educational stakeholders in the LA design process, evaluating systems in use, and researching the sociotechnical factors influencing LA successes and failures. This editorial introduces the contributions of the papers in this special section, reflects more broadly on the field’s emergence over the last five years, considers known gaps, and indicates new opportunities that may open in the next five years.
I hope you find this a provocative collection that inspires you to bring the voices of stakeholders more strongly and meaningfully into the design process.
Campos, F., Nguyen, H., Ahn, J., & Jackson, K. (2023). Leveraging cultural forms in human-centred learning analytics design
In this article, we offer theory-grounded narratives of a 4-year participatory design process of a Learning Analytics tool with K-12 educators. We describe how we design-in-partnership by leveraging educators’ routines, values and cultural representations into the designs of digital dashboards. We make our long-term reasoning visible by reflecting upon how design decisions were made, discussing key tensions and analysing to what extent the developed tools were taken up in practice. Through thick design narratives, we reflect upon how cultural forms—recognizable cultural constructs that might cue and facilitate specific activities—were identified among educators and informed the design of a dashboard. We then examined the extent to which the designed tool supported coaches and teachers to engage in Generative Uncertainty, an interpretive stance in which educators manifest productive inquiries towards data. Our analysis highlights that attuning to cultural forms is a valuable first step but not enough towards designing LA tools for systems in ways that fit institutionalized practices, challenge instrumental uses and spur productive inquiry. We conclude by offering two key criteria for making culturally-grounded design decisions in the context of long-term partnerships.
Hilliger, I., Miranda, C., Celis, S., & Pérez-Sanagustín, M. (2023). Curriculum analytics adoption in higher education: A multiple case study engaging stakeholders in different phases of design
Several studies have indicated that stakeholder engagement could ensure the successful adoption of learning analytics (LA). Considering that researchers and tech developers may not be aware of how LA tools can derive meaningful and actionable information for everyday use, these studies suggest that participatory approaches based on human-centred design can provide stakeholders with the opportunity to influence decision-making during tool development. So far, there is a growing consensus about the importance of identifying stakeholders’ needs and expectations in early stages, so researchers and developers can design systems that resonate with their users. However, human-centred LA is a growing sub-field, so further empirical work is needed to understand how stakeholders can contribute effectively to the design process and the adoption strategy of analytical tools. To illustrate mechanisms to engage various stakeholders throughout different phases of a design process, this paper presents a multiple case study conducted in different Latin American universities. A series of studies inform the development of an analytical tool to support continuous curriculum improvement, aiming to improve student learning and programme quality. Yet, these studies differ in scope and design stage, so they use different mechanisms to engage students, course instructors and institutional administrators. By cross analysing the findings of these three cases, three conclusions emerged for each design phase of a CA tool, presenting mechanisms to ensure stakeholder adoption after tool development. Further implications of this multiple case study are discussed from a theoretical and methodological perspective.
Hutchins, N. M., & Biswas, G. (2023). Co-designing teacher support technology for problem-based learning in middle school science
This paper provides an experience report on a co-design approach with teachers to co-create learning analytics-based technology to support problem-based learning in middle school science classrooms. We have mapped out a workflow for such applications and developed design narratives to investigate the implementation, modifications and temporal roles of the participants in the design process. Our results provide precedent knowledge on co-designing with experienced and novice teachers and co-constructing actionable insight that can help teachers engage more effectively with their students’ learning and problem-solving processes during classroom PBL implementations.
Lawrence, L., Echeverria, V., Yang, K., Aleven, V., & Rummel, N. (2023). How teachers conceptualise shared control with an AI co-orchestration tool: A multiyear teacher-centred design process
Artificial intelligence (AI) can enhance teachers’ capabilities by sharing control over different parts of learning activities. This is especially true for complex learning activities, such as dynamic learning transitions where students move between individual and collaborative learning in un-planned ways, as the need arises. Yet, few initiatives have emerged considering how shared responsibility between teachers and AI can support learning and how teachers’ voices might be included to inform design decisions. The goal of our article is twofold. First, we describe a secondary analysis of our co-design process comprising six design methods to understand how teachers conceptualise sharing control with an AI co-orchestration tool, called Pair-Up. We worked with 76 middle school math teachers, each taking part in one to three methods, to create a co-orchestration tool that supports dynamic combinations of individual and collaborative learning using two AI-based tutoring systems. We leveraged qualitative content analysis to examine teachers’ views about sharing control with Pair-Up, and we describe high-level insights about the human-AI interaction, including control, trust, responsibility, efficiency, and accuracy. Secondly, we use our results as an example showcasing how human-centred learning analytics can be applied to the design of human-AI technologies and share reflections for human-AI technology designers regarding the methods that might be fruitful to elicit teacher feedback and ideas. Our findings illustrate the design of a novel co-orchestration tool to facilitate the transitions between individual and collaborative learning and highlight considerations and reflections for designers of similar systems.
Wiley, K., Dimitriadis, Y., & Linn, M. (2023). A human-centred learning analytics approach for developing contextually scalable K-12 teacher dashboards
This paper describes a Human-Centred Learning Analytics (HCLA) design approach for developing learning analytics (LA) dashboards for K-12 classrooms that maintain both contextual relevance and scalability—two goals that are often in competition. Using mixed methods, we collected observational and interview data from teacher partners and assessment data from their students’ engagement with the lesson materials. This DBR-based, human-centred design process resulted in a dashboard that supported teachers in addressing their students’ learning needs. To develop the dashboard features that could support teachers, we found that a design refinement process that drew on the insights of teachers with varying teaching experience, philosophies and teaching contexts strengthened the resulting outcome. The versatile nature of the approach, in terms of student learning outcomes, makes it useful for HCLA design efforts across diverse K-12 educational contexts.
TLDR: A new paper maps the human-centred design space around AI writing tools, plus an interactive tool to explore the literature behind the design space:
Mina Lee, et al. (2024). A Design Space for Intelligent and Interactive Writing Assistants. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24), May 11–16, 2024, Honolulu. ACM, New York, NY, USA, 34 pages. Open Access Preprint: https://arxiv.org/abs/2403.14117 [Interactive tool]
Writing is thinking. My career-long fascination has been with how computers can support thinking, spanning digital tools for writing and diagramming ideas/arguments. We clarify our thinking by seeing if we can externalise our thinking coherently. When we can’t, it’s a signal to raise our game, or switch tack.
The GenAI writing invasion. Everywhere we look in the world of digital writing, AI is wriggling its way into the apps, from the longstanding, fully featured tools like Microsoft Word and Overleaf, to the niche products like Grammarly, to the myriad new kids on the block targeting specific commercial sectors with the promise of “writing productivity” from GenAI [one of many review listings]. Here at UTS of course, we’ve been deploying and refining our own AcaWriter web app since 2015, building our understanding and research-informed evidence around what works with both students and teaching teams.
Educational implications. From an educational point of view, we are now grappling with the profound questions that Generative AI raises around how we teach and assess writing. The fact that the documents that until now served as plausible proxies for intellectual work, may have been co-authored with/ghost-written by a machine, forces us to ask how and in what ways our students need to demonstrate their writing competency. I’ve reflected on this in The Writing Synth Hypothesis and other GenAI posts. Assessment reform for the age of AI is the challenge.
Design Spaces. A software design space is a framework that clarifies what the key options are that designers can choose from, around key elements of the digital artifact. How many ways are there to provide the user with critical functionality? My PhD 1988-92 was working with Rank Xerox EuroPARC on Design Space Analysis, a form of design rationale capture (Graphical Argumentation and Design Cognition) — so it’s been fun to return to this now.
The AI writing design space. So — what is the shape and size of the design space for AI writing tools? In a new paper we map that space, and will present this in May at CHI24, the leading international conference on human-centred computing. Kudos to Mina Lee and the others in the lead team who coordinated the team of 36 authors who mapped this space, reviewing 115 papers from HCI and NLP, covering the different levels of such tools.
Figure 1: Our design space for intelligent and interactive writing assistants consists of five key aspects—task, user, technology, interaction, and ecosystem—that are interconnected and interdependent. Within each aspect, we define dimensions (bold texts) that represent fundamental components of the aspect and codes (examples associated with bold texts) that represent possible options for each dimension. When necessary, we group semantically relevant dimensions together within each aspect and use a prefix to denote the group name; the interaction dimensions are grouped by user, user interface (UI), and technology; likewise, the technology dimensions are grouped by data, model, learning, and evaluation.
An interactive tool helps explore the literature behind the design space:
Last week I had the pleasure of being welcomed into the Team-Based Learning community, at their Asia Pacific Community Symposium. While everyone is doing “team-based learning” of some sort, I was not familiar with TBL as a set of specific learning design patterns which have been refined across multiple contexts, with a dedicated and passionate community advocating and researching it, and dedicated platforms scaffolding the process.
They invited me to share how UTS has responded to GenAI, to which (given the community) I added a few pointers to work from my other work on Collective Intelligence and Teamwork Analytics. Slides below and as PDF.
I’m looking forward to LAK24 in Kyoto next month. In response to the GenAI workshop call for practical examples and experiences of GenAI, this short paper shares some examples from the last tumultuous year, with some brief reflections…
Abstract: Generative artificial intelligence (GenAI) is now capable of performing tasks that we have considered intellectually demanding. There are justified concerns that this will undermine the agency of both educators and students, if tools are poorly designed, poorly used, or imposed — with consequences for education and the future of work. This short paper contributes practical examples pointing the potential for GenAI to promote critical analysis as part of intellectually demanding tasks, by both students and educators. However, this depends on appropriate usage. The paper then briefly discusses how we may balance the benefits and risks of human cognitive offloading to AI, as a perspective on human agency.
2024 here we come… The current frenzy around artificial intelligence in education was triggered just over a year ago by the explosive arrival of ChatGPT, which made the power of the most mature large language model ever developed, freely available to the masses via an engaging conversational user interface. Every level of the educational sector then spent 2023 grappling with the implications, and I’ve shared my small pieces of that puzzle in other blog posts. (For those interested, R&D in “AIED” is not new, dating back ~40 years depending on how you count.*)
The tech is advancing at a dizzying pace which can leave us disoriented, and few anticipate that 2024 will be any different. But a consistent challenge faced by every school, college and university, is to build and sustain trust that AI will be used responsibly. Easier said than done:
Local ethics. There are endless lists of AI ethics principles that would seem on first inspection to make sense everywhere (“fairness”, “accountability”, “transparency”, etc…) — but translation work is needed. There will be local sensitivities around how these are implemented. What do qualities like “trust” and “responsible” mean to teachers, students, parents, leaders, educational authorities? There will be commonalities for sure that translate across contexts, but building trust means taking your people on the journey, so that they can internalise what these ideas mean, bring abstract principles to life in their own language and metaphors, and tell user stories they can inhabit.
High quality deliberation. Moreover, the issues are complex. How do we convene informed, respectful dialogue between diverse stakeholders? Calling a ‘town hall’ for all interested risks being superficial (there’s no time to grapple with the complexities; contributions are misinformed), tokenistic (those in power have already made the decisions), or attracting only the most confident or strident voices. A brainstorming workshop provides more space to go deep, but often doesn’t involve any learning, participants may not represent the true diversity of the community, and while hugely generative of ideas, may fail to converge on tangible outcomes that actually make a difference.
Agreeing on what WE consider to be acceptable practice in OUR context can provide a sense of orientation and safety amid the turbulence — if they are then implemented of course.
In late 2021, here at UTS we set out to grapple with this, and designed the EdTech Ethics forum for the university community with these concerns in mind. We put out an initial report documenting the process and preliminary feedback in the immediate aftermath, but then did the key work of interviewing participants, analysing their feedback, followed by contributing to the university’s governance processes as it developed and published its AI Operations Policy and Procedures.
So I’m delighted to share a forthcoming journal paper documenting how we ran this, what the participants thought, and the tangible outcomes. The paper acknowledges the many people who made this possible, but special thanks to my co-authors Teresa Swist and Kal Gulson at Sydney University Education Futures Studio, who joined the project as external participants to UTS, and conducted the interviews. This work on Deliberative Democracy intersects with our collaboration around Technical Democracy. Chad Foulkes from Liminal by Design was an awesome workshop session designer and facilitator, under the tricky lockdown conditions. And to Chris Riedy and Nivek Thompson (UTS Institute for Sustainable Futures) whose guidance and teaching on Leading Deliberative Democracy and Doing Deliberative Democracy started me down this road (highly recommended online micro credentials!).
Swist, T., Buckingham Shum, S. & Gulson, K. N. (2024). Co-producing AIED Ethics Under Lockdown: An Empirical Study of Deliberative Democracy in Action. International Journal of Artificial Intelligence in Education Published online: 27 Feb. 2024. https://doi.org/10.1007/s40593-023-00380-z
Abstract: It is widely documented that higher education institutional responses to the COVID-19 pandemic accelerated not only the adoption of educational technologies, but also associated socio-technical controversies. Critically, while these cloud-based platforms are capturing huge datasets, and generating new kinds of learning analytics, there are few strongly theorised, empirically validated processes for institutions to consult their communities about the ethics of this data-intensive, increasingly algorithmically-powered infrastructure. Conceptual and empirical contributions to this challenge are made in this paper, as we focus on the under-theorised and under-investigated phase required for ethics implementation, namely, joint agreement on ethical principles. We foreground the potential of ethical co-production through Deliberative Democracy (DD), which emerged in response to the crisis in confidence in how typical democratic systems engage citizens in decision making. This is tested empirically in the context of a university-wide DD consultation, conducted under pandemic lockdown conditions, co-producing a set of ethical principles to govern Analytics/AI-enabled Educational Technology (AAI-EdTech). Evaluation of this process takes the form of interviews conducted with students, educators, and leaders. Findings highlight that this methodology facilitated a unique and structured co-production process, enabling a range of higher education stakeholders to integrate their situated knowledge through dialogue. The DD process and product cultivated commitment and trust among the participants, informing a new university AI governance policy. The concluding discussion reflects on DD as an exemplar of ethical co-production, identifying new research avenues to advance this work. To our knowledge, this is the first application of DD for AI ethics, as is its use as an organisational sensemaking process in education.
Doroudi, S. (2023). The Intertwined Histories of Artificial Intelligence and Education. International Journal of Artificial Intelligence in Education, 33(4), 885-928. https://doi.org/10.1007/s40593-022-00313-2
Pham, S. T. H., & Sampson, P. M. (2022). The development of artificial intelligence in education: A review in context. Journal of Computer Assisted Learning, 38(5), 1408–1421. https://doi.org/10.1111/jcal.12687
Woolf, B. P. (2015). AI and education: Celebrating 30 years of marriage. In C. Conati, N. Heffernan, A. Mitrovic, & M. F. Verdejo (Eds.), Artificial intelligence in education: 17th international conference, AIED 2015, Madrid, Spain, June 22–26, 2015. Proceedings (pp. 38–47). Springer International Publishing. https://doi.org/10.1007/978-3-319-19773-9
This provocation was posed by one participant in a recent expert forum. Over-dramatic? Not if universities lose the capacity to assure learning.
Assessment Reform for the Age of Artificial Intelligence is a consultation report from the Tertiary Education Quality and Standards Agency (TEQSA), Australia’s independent national quality assurance and regulatory agency for higher education. I had the privilege of being invited to join a TEQSA-convened group for 2 days in August, which we hosted here at UTS, to consider how the assessment landscape has been redrawn by the emergence of widely available generative AI.
“The emergence of generative artificial intelligence (AI), while creating new possibilities for learning and teaching, has exacerbated existing assessment challenges within higher education. However, there is considerable expertise, based on evidence, theory and practice, about how to design assessment for a digital world, which includes artificial intelligence. AI is not new, after all. This document, constructed through expert collaboration, draws on this body of knowledge and outlines directions for the future of assessment. It seeks to provide guidance for the sector on ways assessment practices can take advantage of the opportunities, and manage the risks, of AI, specifically generative AI.”
As the report explains in setting the scene, the point of departure is a 2010 report entitled Assessment 2020: Seven Propositions for Assessment Reform in Higher Education, on which this new report is modelled[website/report]…
“We take our starting point for this document from the propositions for assessment outlined in Assessment 2020 (Boud and Associates, 2010). This work outlines how assessment acts as a powerful intervention in student learning and highlights the educational purposes of assessment in parallel with the process of assuring learning outcomes. Good assessment design that allows for ‘rich portrayals’ of student learning is critical. Thus, we take as given that assessment should engage students in learning, provide a partnership between teachers and students, and promote student participation in feedback. These key elements of assessment can then guide how best to consider the role of AI in assessment design.”
The question is — how does AI change things? We propose two propositions and five principles:
The lead team presented the report in this TEQSA webinar:
(This was the latest in a GenAI series co-hosted with Deakin University’s Centre for Research in Assessment and Digital Learning, to which I’ve had the privilege of contributing.)
Feedback is now in from the consultation, and this report will be presented and workshopped next month at the TEQSA conference, where we look forward to hearing more from participants. The question then, of course, is how to implement such changes at scale, in a sustainable way. As ever, Dave Boud has insights to offer…
It was fantastic working with such outstanding colleagues, and special thanks to the team who designed and facilitated the expert forum: Jason M. Lodge, The University of Queensland Sarah Howard, University of Wollongong Margaret Bearman, Phillip Dawson, Deakin University
With Shirley Agostinho, University of Wollongong, Simon Buckingham Shum, University of Technology Sydney, Chris Deneen, University of South Australia, Cath Ellis, The University of Sydney, Tim Fawns, Monash University, Helen Gniel, TEQSA, Rowena Harper, Edith Cowan University, Michael Henderson, Monash University, Danny Liu, The University of Sydney, Lina Markauskaite, The University of Sydney, Jan McLean, University of Technology Sydney, Carlo Perrotta, The University of Melbourne, Lambert Schuwirth, Flinders University, Christine Slade, The University of Queensland
While my day-job is immersed in analytics/AI-enabled ed-tech in higher ed — the co-design of tools, practices and policy — I’m increasingly compelled to step back and survey the bigger picture: as a species, we face overwhelming, interlocking crises — and we seem to be paralysed. I’m asking whether, and if so how, this should more strongly frame and shape my work and that of the communities I’m in. I’m drawing much inspiration from an exciting neuropsychological account of how we attend to/construct the world (Iain McGilchrist’s The Matter With Things), and the increasingly urgent call for education to equip students to create a more equitable society (Henry Giroux’s work on critical pedagogy).
I was honoured to receive invitations to speak at two recent events focused in different but connected ways on the future of education, in the context of current debates about university futures in the age of AI, and the social context for platforms enabling learning at scale. These gave me opportunities to share and get feedback on how this preliminary thinking helps frame these pressing issues. Here are my Universitas 21 and ACM Learning@Scale keynotes — your feedback most welcome.
Universitas 21
Universitas 21 is an international network of research-intensive universities, committed to sharing insights. In 2014 they invited me to share my thoughts on the toddler field that was Learning Analytics, as part of their focus on personalised learning (an interesting flashback to watch that talk!). I had barely set foot in Australia, but had lots of ideas about what would be possible in my new job at UTS. So in June, it was a pleasure to reconnect, and reflect on that journey. They invited me to their Educational Innovation Symposium:
“U21’s Educational Innovation Symposium, titled ‘Scoping the Future in Higher Education: Transition or Transformation?’ brought together delegates from across the network to tackle some of the big questions currently facing university educators. The symposium, held at McMaster University, explored issues arising from swiftly advancing technologies such as Artificial Intelligence, which affects many areas of educational practice. This includes curriculum development, the way in which teaching and learning are delivered, assessment practices, digital ethics and, significantly, how students can be part of the conversation.”
Transition or transformation? In my abstract, I propose that what we have learnt on our journey at UTS running CIC provides some assurance that universities can transition into the effective, ethical use of AI, since we’ve been inventing, piloting, evaluating and scaling analytics/AI-powered ed-tech since 2015. Conversations with diverse stakeholders are at the heart of this process: Boardroom, Staff room, Server room, Classroom. The talk summarises my take on what we’re seeing in the GenAI-for-Education frenzy, examples from my own work (Bing Chat for argument analysis), and unpacks how we have been responding at UTS in the last 6 months since the GenAI rollercoaster launched, to support faculty academics and students. Human-centred design and Deliberative Democracy are important pieces of this jigsaw puzzle.
However, flipping the order in the abstract, before diving into that detail, in the talk I decided to engage with the bigger picture — the transformation question posed to the symposium. This is where the work of Giroux and McGilchrist has important contributions to make, as introduced below.
Buckingham Shum, S. (2023). Learning, Analytics, AI, Trust (and the future of universities). Keynote address, Universitas 21 Educational Innovation Symposium, (29 June, 2023, McMaster University, Hamilton, Canada). [abstract/replay/slides/reflection]
Thanks to U21 for engaging the talented Emma Richard who created this artful graphic recording (click to zoom)
Learning@Scale
Last month I presented the opening keynote to the 10th ACM Conference on Learning@Scale in Copenhagen. For those not familiar with the L@S community, the conference first emerged amidst the excitement (and data deluge) triggered by Massive Open Online Courses. As an ACM conference L@S started with a strong computational flavour, and while maintaining data science, educational data mining and AI, there is also qualitative attention to the critical human dimensions in all forms of large scale learning. The focus for this year:
“The theme of this year’s conference is the learning futures that the L@S community aims to develop and support in the coming decades. Of special interest this year are contributions that examine the design and the deployment of large-scale systems for the future of learning at scale. We are especially welcoming works targeting not only learners but also educators, educational institutions and other stakeholders involved in the design, use and evaluation of large-scale learning systems. Moreover, we welcome qualitative and mixed-methods contributions, as well as studies that are not at scale themselves but about scaled learning phenomena/environments. Finally, we welcome submissions focusing on the role of culture and cultural values in the implementation and evaluation of large-scale systems.”
Given the intersecting crises now confronting us, I took these opportunities to share some of my current thinking on a question that has increasingly troubled me: What difference, if any, should the climate crisis should make to ed-tech research, especially involving analytics/AI? This is of course just one of the interlocking dilemmas we now face, in what some have termed the “meta-crisis”, but this one comes with an hourglass running down all too fast.
Buckingham Shum, S. (2023). Trust, Sustainability and Learning@Scale. In Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S ’23). Association for Computing Machinery, New York, NY, USA, pp. 1–2. https://doi.org/10.1145/3573051.3593375. [abstract/replay/slides]
Diagnosing our collective paralysis
In the talks, I propose that a plausible diagnosis of our current paralysis — whether or not it proves terminal — is failure to learn. We are simply not learning fast enough and deeply enough. No doubt that is a partial diagnosis, but as people passionate about education and lifelong learning, we can hardly wash our hands of any responsibility when we survey the blasted landscape that is our planet, and the dysfunctional state of civic discourse in so many democracies.
I might have added failure to remember: urgently, we need to re-engage with First Nations people’s knowledge systems. This comes up in the talk later, inspired by Iain McGilchrist, and I also point briefly to the work of Angie Abdilla (Indigenous AI protocols) and Tyson Yunkaporta (Sand Talk). I need and want to go much deeper into this in future.
So, at L@S I asked — intentionally rhetorically — given this massive failure to learn@scale, how should the learning@scale community respond? And to U21, is there anything new to say about the kinds of graduates universities should be cultivating?
Dispositions: how we attend to the world
Knowledge and skills are important, and an ever-changing landscape given cognitive automation. I focus instead on dispositions — ways of attending to the world that are short in supply, and seem particularly salient in these times. I draw on two diagnoses of our collective paralysis — Iain McGilchrist’s neuropsychology work on how we attend to the world (notably his acclaimed new book, The Matter With Things), and Henry Giroux’s work on critical pedagogy, continuing the work of Paulo Freire (Giroux is at McMaster University, and we had a spirited and enjoyable hour in his office!). There is much to read and watch online, but to get a flavour of their work, try Giroux’s keynote to this year’s International Society for the Learning Sciences, and McGilchrist’s keynote to the AI World Summit.
I see McGilchrist and Giroux converging in their calls to resist dehumanising, decontextualizing, extremist ways of representing issues, people and nature. Both challenge us to use technology to help nurture citizens who can think differently, and not merely fuel the mindset that has brought us to the precipice. Both call us to engage with the world in a way that honours relationships, context and justice. Both call for defiant, educated hope as a form of resistance in dark times.
In case this slide is misunderstood, the argument is not that “right-wing politics has a neuroscience basis”. It is that extremism of any sort, of any political persuasion, is black and white thinking, erasing nuance, humility, context, empathy, dehumanising, objectifying, and seeking to manipulate. That has all the hallmarks of how the left hemisphere attends to the world so carefully documented by McGilchrist, when not under the balancing disposition of the right hemisphere’s mode of attention. The polarisation we see now in the culture wars is extremist mindsets of all flavours. But since I’m drawing on Giroux, we’re concerned in this case with right-wing extremism as it threatens educational freedom, the marketisation of universities more broadly, and hence threats to democracy when universities are not playing their role in developing graduates with critical consciousness to fight for a more just society.
Worked example: Belonging Analytics
I don’t think this translates into direct implications for all ed-tech research, but I suggest they pose important provocations for any educator to reflect on, especially those of us immersed in educational data, analytics and AI. Descending from high altitude to practices on the ground, I describe how at UTS we build trust in our automated feedback platforms by democratizing the design and governance processes. And in the L@S talk, I take as a worked example an approach that we’ve termed “Belonging Analytics”, to show how data-informed platforms can be aligned with some of the values championed by Giroux and McGilchrist.
What do you think?
I had encouraging feedback at both conferences, helpful ideas on how I might craft a stronger narrative, and some critical questioning of the arguments. There is so much more to learn, better ways to make the case — and the clock is ticking. I’m looking for intellectual soul mates, and welcome your honest feedback.
Connecting the fields of student belonging and learning analytics: “Belonging Analytics”
As with every purposeful human endeavour, motivation for learning and becoming a professional within a discipline is enhanced when individuals feel a sense of belonging. In the context of education, belonging refers to students’ subjective feeling of being a valued member of the learning community, that comes from a sense of connection with others as well as to the course of study. This affective dimension of engagement has notable links with many positive learning outcomes, including transition, retention, success and well-being. The importance of belonging has been underscored by the recent COVID-19 pandemic and subsequent increase in online, remote learning, and more students found themselves learning in isolation. The issue is further compounded for students from equity or disadvantaged groups, who already feel a lower sense of belonging.
Notwithstanding the pandemic, students’ experiences of belonging is dynamic and contextual, which presents challenges for tracking and supporting students in a timely manner. Traditional research methodologies such as surveys and interviews, may be useful sources of data for understanding student belonging, however these are difficult to scale and repeat over multiple episodes.
In response to the urgency of student belonging, CIC researchers Lisa-Angelique Lim and Simon Buckingham Shum are collaborating with belonging experts Peter Felten and Jennifer Uno (Elon University, USA), to conceptualise a scalable approach to this important issue.
In a new paper, we explore the possibility of harnessing learning analytics to monitor and support belonging in timely and personalised ways. Drawing on examples of where learning analytics has been used for personalising feedback to students, we propose a framework for “belonging analytics”, based on the dimensions of agents, data, and feedback mechanisms.
Overall, our framework suggests approaches that leverage a range of quantitative and qualitative data to monitor and support student belonging over time and at scale. Clearly, as with other substantial learning concepts, much care is needed to ensure that any approach drawing on learning data to inform belonging must be firmly grounded in theory, and that analytical approaches do not foster inequity. We conclude the paper with further questions to explore in this new field.
Watch: We recently presented these ideas in a webinar at the recent Indiana University Learning Analytics Summit.
Engage: Finally, just as belonging is inherently relational, we invite educators and researchers with a shared interest in this topic, to be part of a new Belonging Analytics community on LinkedIn. We look forward to building this community together with you.
Summary: a new paper forges a bridge between data-driven, open automated feedback platforms, and teacher feedback literacy competences:
Buckingham Shum, S., Lim, L.-A., Boud, D., Bearman, M. & Dawson, P. (2023). A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy. International Journal of Educational Technology in Higher Education, 20:40 (12 July 2023). https://doi.org/10.1186/s41239-023-00410-9
The mass availability of generative AI continues to reshape thinking about the future of work and learning. Conversational apps can now give instant feedback to learners about their work — but the educational question is how effective this interaction is. A new design space has opened up for tuning generative AI to give high quality feedback to learners about their work. We are not in uncharted waters here: there is a growing body of knowledge on what “effective feedback” means in higher education, and how to create the conditions for this. It goes far beyond comments accompanying an assignment, with a shift towards “feedback rich ecosystems” in which both teachers and students exercise far greater agency and sensemaking competencies.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
In 2020, I conceived a symposium to bring the editors and authors to UTS to spend 2 days in dialogue with CIC and other researchers developing automated-feedback tools using Learning Analytics/AI. We called for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI. The pandemic shifted this online, but the goals remained the same, and moving online enabled us to more easily bring in additional participants, resulting in DAFFI 2020: Designing Automated Feedback for Impact whose presentations I commend to you.
I’m now delighted to share one of the fruit from this, a collaboration between CIC (Lisa Lim and myself) and our colleagues at Deakin University’s Centre for Research in Assessment and Digital Learning (CRADLE). The focus of the paper is not on generative, conversational AI (which did not exist when we started this work), but on technically less complicated, but correspondingly far more transparent platforms that use simple rules authored by teachers themselves.
“In contrast to closed AF tools, we define “open” AF tools as enabling the educator to specify some or all of the following key parameters in the tool’s behaviour:
the student activity data that the system analyses;
the algorithms that analyse that data;
the feedback information the teacher wishes the software to compile for students;
the modalities via which feedback information is communicated by teachers;
the student-driven feedback processes that are afforded.”
What does it mean to do this skillfully? We demonstrate that Boud & Dawson’s teacher feedback literacy competency framework can be applied very usefully to analysing teaching practices with data-driven, automated feedback platforms. A next step will be to think through what this means for tuning large language models for educational contexts.
A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy
a University of Technology Sydney, AUS b Deakin University, AUS c Middlesex University, UK
Effective learning depends on effective feedback, which in turn requires a set of skills, dispositions and practices on the part of both students and teachers which have been termed feedback literacy. A previously published teacher feedback literacy competency framework has identified what is needed by teachers to implement feedback well. While this framework refers in broad terms to the potential uses of educational technologies, it does not examine in detail the new possibilities of automated feedback (AF) tools, especially those that are open by offering varying degrees of transparency and control to teachers. Using analytics and artificial intelligence, open AF tools permit automated processing and feedback with a speed, precision and scale that exceeds that of humans. This raises important questions about how human and machine feedback can be combined optimally and what is now required of teachers to use such tools skillfully. The paper addresses two research questions: Which teacher feedback competencies are necessary for the skilled use of open AF tools? and What does the skilled use of open AF tools add to our conceptions of teacher feedback competencies? We conduct an analysis of published evidence concerning teachers’ use of open AF tools through the lens of teacher feedback literacy, which produces summary matrices revealing relative strengths and weaknesses in the literature, and the relevance of the feedback literacy framework. We conclude firstly, that when used effectively, open AF tools exercise a range of teacher feedback competencies. The paper thus offers a detailed account of the nature of teachers’ feedback literacy practices within this context. Secondly, this analysis reveals gaps in the literature, signalling opportunities for future work. Thirdly, we propose several examples of automated feedback literacy, that is, distinctive teacher competencies linked to the skilled use of open AF tools.
ChatGPT: What have we learnt?
What do we need to learn next?
I was honoured to join a TEQSA/CRADLE panel yesterday, the 3rd in a series on the implications of ChatGPT (or GenAI more broadly) for higher education. Nearly 3000 people registered, with >1200 joining live, reflecting either the gravity of the situation now facing us — or the consequences of AI and assessment becoming mainstream media fodder! It’s both in fact.
In the 2nd panel in March, in my 8min slot I flagged the absence (at that early stage) of any evidence about whether students have the capacity to engage criticallywith ChatGPT. So many people were proposing to do interesting, creative things with students — but we didn’t know how it would turn out.
But 3 months on, we now have:
myriad demos of GPT’s capabilities given the right prompts
a few systematic evaluations of that capability
myriad proposals for how this can enable engaging student learning
and a small but growing stream of educators’ stories from the field
with peer reviewed research about to hit the streets.
Educators can now articulate the range of critical engagement that their students are displaying, and I share what we’re learning at UTS from some of our leading educators who have been introducing assessments integrating ChatGPT. We now need to track how well these, and other interesting proposals, for AI-informed learning and assessment translate across diverse contexts.
I also urge us to harness the diverse brilliance of our student community in navigating this system shock, sharing what we’re learning from our Student Partnership in AI.
Here are my slides, and the full replay below (jumps to my 12min talk, but watch the whole panel!)
So, all that’s to say that making arguments visible so that you and others can — in a very real sense — “see what you’re saying” has been a career-long passion. A key challenge in this long field of research has been that rigorous thinking is hard work. Bad luck, welcome to university! Argument Mapping and its related techniques use the affordances of visual trees/networks as an extended, external memory to augment personal and collective intelligence. Making one’s ideas visible as coherent diagrams is also hard work — but it’s good pain — the cognitive and discursive effort this entails is designed to clarify one’s thinking by revealing visually where the weaknesses are, in ways that writing and reading chunks of prose cannot tell you at a glance.
Enter NLP and rhetorical parsing
In 2012, we were now in the Web 2.0 era, and an exciting collaboration with NLP and linguistics expert Ágnes Sándor (Xerox) led to a new conception of Contested Collective Intelligence. For the first time in my work, machines could identify argumentative moves in sentences, complementing the argumentative moves that our web annotation tools enabled for people — who unlike machines, can of course can ‘read between the lines’ and see connections between ideas that may not even be in the texts.
This was extremely exciting, and the ideas and open source code carried through to our current Academic Writing Analytics project and web apps. I reflected on the impact of encountering NLP colleagues, in the context of The Future of Text book.
Conversational generative AI
And so we arrive at generative AI based on large language models, which advances the state of the art in language processing and generation in so many ways. Moreover, the conversational paradigm, when a chat application is overlaid, opens so many interesting human-computer/personal-collective intelligence possibilities. I’ve been intrigued to play with GPT-4 to see what its argument analysis capabilities are.
Previously, I’ve shared some early experiments on ChatGPT-3.5’s ability to identify implicit premises in prose arguments, and critique a flawed argument by analogy. I’ve now had the chance to experiment a little with the version of GPT-4 that is Bing Chat, accessed via Microsoft Edge browser. I was dying to see how far I could get in generating an Argument Map from a written argument.
The task is a typical analysis workflow, as prep for teaching:
search for relevant sources
select one for analysis
extract key elements of the argument and their relationships (described using a structured markdown notation called ArgDown)
diagram them to show their key relationships (in the ArgDown web app)
discuss (with the AI)
start thinking about student activities to help them learn
I don’t mind admitting that watching a machine do this for the first time was startling! I tell the story here…
Let’s take a closer look at what Bing Chat did, because it wasn’t perfect.
The gold stars signal what in my view are good summaries of what the authors said, correctly linked.
The blue info circles are “commentary” from Bing Chat about the arguments
The red crosses signal that the authors did not say this, it is a false reconstruction by Bing Chat.
The red underline signals classification of a premise using incorrect, or indeed made-up argument schemes. There is to my knowledge no such argument type as Argument from responsibility, or Argument from precaution. Argument from omission seems to be a jumbling of Fallacy of omission and Argument from ignorance.
If we take this node for example, it reads well as a summary:
However, the authors do not talk about researchers at all, they say:
As an amusing sidenote, Bing Chat was curiously resistant to recognising this, insisting that it was correct, first “quoting” a fabricated passage from the article to me, and then saying that this implied that the authors meant researchers. I thought that this sort of stubbornness had been ironed out after Bing Chat’s earlier escapades! More seriously, this points to the value of dialogic learning, with a partner who can be conversed with 24/7 — but who must still be treated with some caution, certainly at this stage of maturity.
To summarise:
Bing Chat showed intriguing capability, for a machine, to analyse an argumentative article:
extracting the key claim and underlying premises, summarising them in own words
(and without being asked to) attempting to classify some nodes using Walton’s Argumentation Schemes.
However it also introduced fallacious nodes (incorrect summaries of the authors, and incorrect commentary nodes), incorrect links, and argument classifications (inventing argument types, and/or misclassifying nodes).
This is an exploratory example, and more systematic evaluations are required, of the sort we see in the growing Argument Mining literature.
Reflections
It does feel to me that we’ve turned a corner in the long, wintry history of AI. Perhaps this is a passing summer, which will fade like the others. But in my own career, punctuated by eureka moments such as seeing my first Apple Mac, my first web page load, and an iPhone — this is up there.
University is to teach you to think. Argument analysis is serious intellectual work, of the sort that we would hope to see from our students. Nor is there always “one map to rule them all’ — a correct map, since like in spatial cartography, design decisions are made about scale and purpose. The point about knowledge cartography is that it provokes productive reflection and discourse. So even if the AI gets the map wrong (and it will), the conversation this should provoke should be useful. With colleagues Kirsty Kitty and Andrew Gibson, I’ve argued that embracing imperfection in tech can be productive if it promotes deeper critical thinking in learners, e.g., learning by correcting the automated output, or reflecting on questions it asks, or why it seems wrong. Students must, however, be scaffolded to engage in such activity.
Informal learning? This is feasible in formal education, but may be less attractive in other informal learning contexts where we want to promote critical deliberation, e.g. citizens engaged in a policy deliberation, many of whom lack the internal or external motivation to think that hard. But assuming future tools give more accurate argument maps/outlines, that require less debugging, perhaps we can see use-cases including:
assisting facilitators/educators to prepare learning resources for civic deliberations
assisting very engaged citizens to dissect complex arguments, and perhaps lowering the entry threshold for others who might otherwise not engage with such structured, critical deliberation
an article is very different to a multi-author conversation, but we can envisage summarising online discussions (NB: Teams is starting to summarise topics and actions in meeting transcripts)
Did we just supplant student cognition? From a learning sciences perspective, an overriding concern with generative AI is that it does too much cognitive work for the learner. Editing an AI-generated draft is not the same as wrestling with the blank page yourself. Ditto for reviewing an AI-generated argument map.
I have just done what many professionals have enjoyed doing in recent months: putting GPT through its paces to test its technical capability. But learners are not professionals: they don’t know what they don’t know. As I argue elsewhere, they may lack the knowledge, skills and dispositions to engage critically with AI output. They will require suitable scaffolding from mentors and teachers to learn what we mean by critical thinking and argument analysis, in order then to be equipped to use a power tool such as an argument mapping tool. Much empirical research awaits to test the affordances of generative AI like this, to establish when they are most useful to use developmentally, with a given age/stage of learner.
But we do know that argument mapping has struggled to gain traction (in formal education and among professionals) because it’s hard intellectual work. It could be that by generating full or intentionally incomplete argument maps, AI provides a step up for many learners to quickly get feedback on their work, or see examples of arguments about topics they are knowledgeable about — and thus better equipped to critique — compared to examples chosen by the teacher or textbook. Generative AI may open new possibilities because it can generate examples tuned to the interests of each learner, activating their curiosity to go deeper.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
