Tag: assessment

Understanding skilled use of open automated feedback tools as teacher feedback literacy

Summary: a new paper forges a bridge between data-driven, open automated feedback platforms, and teacher feedback literacy competences: 

Buckingham Shum, S., Lim, L.-A., Boud, D., Bearman, M. & Dawson, P. (2023). A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy. International Journal of Educational Technology in Higher Education, 20:40 (12 July 2023). https://doi.org/10.1186/s41239-023-00410-9 

The mass availability of generative AI continues to reshape thinking about the future of work and learning. Conversational apps can now give instant feedback to learners about their work — but the educational question is how effective this interaction is. A new design space has opened up for tuning generative AI to give high quality feedback to learners about their work. We are not in uncharted waters here: there is a growing body of knowledge on what “effective feedback” means in higher education, and how to create the conditions for this. It goes far beyond comments accompanying an assignment, with a shift towards “feedback rich ecosystems” in which both teachers and students exercise far greater agency and sensemaking competencies.

In 2019, an exciting book came out: The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners (Eds. Henderson, Ajjawi, Boud & Molloy):

“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”

In 2020, I conceived a symposium to bring the editors and authors to UTS to spend 2 days in dialogue with CIC and other researchers developing automated-feedback tools using Learning Analytics/AI. We called for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI. The pandemic shifted this online, but the goals remained the same, and moving online enabled us to more easily bring in additional participants, resulting in DAFFI 2020: Designing Automated Feedback for Impact whose presentations I commend to you.

I’m now delighted to share one of the fruit from this, a collaboration between CIC (Lisa Lim and myself) and our colleagues at Deakin University’s Centre for Research in Assessment and Digital Learning (CRADLE). The focus of the paper is not on generative, conversational AI (which did not exist when we started this work), but on technically less complicated, but correspondingly far more transparent platforms that use simple rules authored by teachers themselves.

“In contrast to closed AF tools, we define “open” AF tools as enabling the educator to specify some or all of the following key parameters in the tool’s behaviour:

  1. the student activity data that the system analyses;

  2. the algorithms that analyse that data;

  3. the feedback information the teacher wishes the software to compile for students;

  4. the modalities via which feedback information is communicated by teachers;

  5. the student-driven feedback processes that are afforded.”

What does it mean to do this skillfully? We demonstrate that Boud & Dawson’s  teacher feedback literacy competency framework can be applied very usefully to analysing teaching practices with data-driven, automated feedback platforms. A next step will be to think through what this means for tuning large language models for educational contexts.

A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy

Simon Buckingham Shuma, Lisa-Angelique Lima, David Bouda,b,c, Margaret Bearmanb, Phillip Dawsonb

a University of Technology Sydney, AUS
b Deakin University, AUS
c Middlesex University, UK

Effective learning depends on effective feedback, which in turn requires a set of skills, dispositions and practices on the part of both students and teachers which have been termed feedback literacy. A previously published teacher feedback literacy competency framework has identified what is needed by teachers to implement feedback well. While this framework refers in broad terms to the potential uses of educational technologies, it does not examine in detail the new possibilities of automated feedback (AF) tools, especially those that are open by offering varying degrees of transparency and control to teachers. Using analytics and artificial intelligence, open AF tools permit automated processing and feedback with a speed, precision and scale that exceeds that of humans. This raises important questions about how human and machine feedback can be combined optimally and what is now required of teachers to use such tools skillfully. The paper addresses two research questions: Which teacher feedback competencies are necessary for the skilled use of open AF tools? and What does the skilled use of open AF tools add to our conceptions of teacher feedback competencies? We conduct an analysis of published evidence concerning teachers’ use of open AF tools through the lens of teacher feedback literacy, which produces summary matrices revealing relative strengths and weaknesses in the literature, and the relevance of the feedback literacy framework.  We conclude firstly, that when used effectively, open AF tools exercise a range of teacher feedback competencies. The paper thus offers a detailed account of the nature of teachers’ feedback literacy practices within this context. Secondly, this analysis reveals gaps in the literature, signalling opportunities for future work. Thirdly, we propose several examples of automated feedback literacy, that is, distinctive teacher competencies linked to the skilled use of open AF tools.

Your comments most welcome…

ChatGPT: What have we learnt, what do we need to learn next?

ChatGPT: What have we learnt?
What do we need to learn next?

I was honoured to join a TEQSA/CRADLE panel yesterday, the 3rd in a series on the implications of ChatGPT (or GenAI more broadly) for higher education. Nearly 3000 people registered, with >1200 joining live, reflecting either the gravity of the situation now facing us — or the consequences of AI and assessment becoming mainstream media fodder! It’s both in fact.

In the 2nd panel in March, in my 8min slot I flagged the absence (at that early stage) of any evidence about whether students have the capacity to engage critically with ChatGPT. So many people were proposing to do interesting, creative things with students — but we didn’t know how it would turn out.

But 3 months on, we now have:

  • myriad demos of GPT’s capabilities given the right prompts
  • a few systematic evaluations of that capability
  • myriad proposals for how this can enable engaging student learning
  • and a small but growing stream of educators’ stories from the field
  • with peer reviewed research about to hit the streets.

Educators can now articulate the range of critical engagement that their students are displaying, and I share what we’re learning at UTS from some of our leading educators who have been introducing assessments integrating ChatGPT. We now need to track how well these, and other interesting proposals, for AI-informed learning and assessment translate across diverse contexts.

I also urge us to harness the diverse brilliance of our student community in navigating this system shock, sharing what we’re learning from our Student Partnership in AI.

Here are my slides, and the full replay below (jumps to my 12min talk, but watch the whole panel!)

Human/AI exam proctoring with integrity?

Enjoyed convening this SoLAR Panel with some very knowledgeable colleagues…

The emergence of online exam proctoring (aka remote invigilation) in higher education may be seen as a function of multiple interacting drivers, including:

  • the rise of online learning
  • emergency exam measures required by the pandemic
  • cloud computing and the increasing availability of data for training machine learning classifiers
  • university assessment regimes
  • rising concerns around student cheating
  • accountability pressures from accrediting bodies

Commercial proctoring services claiming to automate the detection of potential cheating are among the most complicated forms of AI deployed at scale in higher education, requiring various combinations of image, video and keystroke analysis, depending on the services. Moreover, due to the pandemic, they were introduced in great haste in many institutions in order to permit students to graduate, with far less time for informed deliberation than would have been expected. Consequently, there was significant controversy around this form of automation, with protests at some universities seeing withdrawal of the services, and research beginning to clarify the ethical issues, and produce new empirical evidence.

However, numerous institutions are satisfied that the services they procured met the emergency need, and are continuing with them, which would make this one of the ‘new normal’ legacies of the pandemic. Critics ask, however, whether this should become ‘business as usual’. Regardless of one’s views, the rapid introduction of such complex automation merits ongoing critical reflection.

SoLAR was delighted to host this panel, which brought together expertise from multiple quarters to explore a range of questions, arguments, and what the evidence is telling us, such as…

  • This is just exams and invigilation in new clothes, right? They’re not perfect, but universities aren’t about to drop them anytime soon, so let’s all get on with it…
  • Are there quite distinct approaches to the delivery of such services that we can now articulate, to help people understand the choices they need to make?
  • What ethical issues do we now recognise that were perhaps poorly understood 2 years ago — or simply couldn’t afford to engage with in the emergency, but which we must address now?
  • What evidence is there about the effectiveness of remote proctoring — automated, or human-powered — at reducing rates of cheating?
  • What answers are there to the question, “Should we trust the AI?” Are we now over (yet another) AI hype curve, and ready for a reality check on what “human-AI teaming” looks like for online proctoring to function sustainably and ethically?
  • What (new?) alternatives to exams are there for universities to deliver trustworthy verification of student ability, and what are the tradeoffs?
  • Who might be better or worse off as a result of the introduction of proctoring?

This panel brought rich experience on the frontline of practice, business and academia:

Phillip Dawson is a Professor and the Associate Director of the Centre for Research in Assessment and Digital Learning, Deakin University. Phill researches assessment in higher education, focusing on feedback and cheating, predominantly in digital learning contexts. His 2021 book “Defending Assessment Security in a Digital World” explores how cheating is changing and what educators can do about it.

Jarrod Morgan is an inspiring entrepreneur, award-winning business leader, keynote speaker, and chief strategist for the world’s leading online testing company. Jarrod founded ProctorU in 2008, and in 2020 led the company through its merger and evolution into Meazure Learning. In his role as chief strategy officer, he is a frequent speaker for the Online Learning Consortium (OLC), the Association of Test Publishers (ATP), Educause, and many others. He has appeared on PBS and the Today Show, and has been covered by the Wall Street Journal, The New York Times, and is a columnist with Fast Company through their Executive Board program.

Jeannie Paterson is Professor of Law and Co-Director of the Centre for AI and Digital Ethics, University of Melbourne. She teaches and researches in the fields of consumer protection law, consumer credit and banking law, and AI and the law. Jeannie’s research covers three interrelated themes: The relationship between moral norms, ethical standards and law; Protection for consumers experiencing vulnerability; Regulatory design for emerging technologies that are fair, safe, reliable and accountable. She recently co-authored “Good Proctor or “Big Brother”? Ethics of Online Exam Supervision Technologies”.

Lesley Sefcik is a Senior Lecturer and Academic Integrity Advisor at Curtin University. She provides university-wide teaching, advice, and academic research within the field of academic integrity. She is a Homeward Bound Fellow and a Senior Fellow of the Higher Education Academy. Dr. Sefcik’s professional background is situated in Assessment and Quality Learning within the domain of Learning and Teaching. Current projects include the development, implementation and management of remote invigilation for online assessment, and academic integrity related programs for students and staff at Curtin. She co-authored “An examination of student user experience (UX) and perceptions of remote invigilation during online assessment”.

(Chair) Simon Buckingham Shum is Professor of Learning Informatics and Director of the Connected Intelligence Centre, University of Technology Sydney, where his team researches, deploys and evaluates Learning Analytics/AI-enabled ed-tech tools. He has helped to develop Learning Analytics as an academic field for the last decade, and has served two terms as SoLAR Vice-President. His background in ergonomics and human-computer interaction always draws his attention to how the human and technical must be co-designed to work together to create sustainable work practices. He recently coordinated the UTS “EdTech Ethics” Deliberative Democracy Consultation in which online exam proctoring was an example examined by students and staff.

Further resources shared during the webinar:

Collaborative Learning Analytics (CSCL Handbook chapter)

Delighted to share this preprint of a chapter that I’ve been working on with Alyssa Wise and Simon Knight. It’s due out in the exciting new CSCL Handbook this year!

Wise, A., Knight, S., Buckingham Shum, S. (In Press) Collaborative Learning Analytics. In: Cress, U., Rosé C., Wise, A. & Oshima, J. (Eds.), International Handbook of Computer-Supported Collaborative Learning (Springer). Preprint: PDF

Abstract

The use of data from computer-based learning environments has been a longstanding feature of CSCL. Learning analytics can enrich this established work in CSCL. This chapter outlines synergies and tensions between the two fields. Drawing on examples, we discuss established work to use learning analytics as a research tool (analytics of collaborative learning – ACL). Beyond this potential though, we discuss the use of analytics as a mediational tool in CSCL – Collaborative Learning Analytics (CLA). This shift raises important challenges regarding the role of the computer – and analytics –in supporting and developing human agency and learning. LA offers a new tool for CSCL research. CSCL offers important contemporary perspectives on learning for a knowledge society, and as such is an important site of action for learning analytics research that both builds our understanding of collaborative learning, and support that learning.

Evidence: Fair & Robust AI-based Assessment

The All Party Parliamentary Group on AI is a multi-year initiative to help anticipate the widespread impacts that AI could have on society. They convene Evidence Sessions on different themes, in which Lords and MPs have the opportunity to hear from, and question, diverse experts.

As noted in the introduction to one of their recent reports,

“The evidence APPG AI has been gathering since 2017 shows education at the heart of both the opportunities and the risks in the narratives forming around AI.”

“[…] In 2019, APPG AI launched the Education Pillar to tackle some of these multi-faceted questions over the next two years. We will focus on:

  • how AI can be used as a tool to improve learning,
  • what skills we need to prioritise as a society,
  • how school curriculums need to transform,
  • and what the role of ethics in education should be.”

Reports (access requires a free signup) on the education theme to date have collated evidence on:

The latest meeting focused on Designing Fair & Robust AI-based Assessment Systems. This brought together a very interesting set of people, to which I was honoured to be invited.

The meeting switched from the House of Lords to Zoom (so a distinct loss of oak panelling and leather upholstery there!) but it meant many others could tune in live.

The guiding questions they set were:

  1. What are the benefits and challenges of different types of AI-based assessment systems in education?
  2. How can it be guaranteed that they will deliver reliable and fair results?
  3. How might AI-based assessment systems change the teacher-student relationship?
  4. How will these technologies affect students’ motivation and trust in a fair evaluation of their performance?
  5. How to prepare students, teachers, and parents before implementing AI-based assessment technologies in education?
  6. How do AI and human understandings of assessment differ?

With just 5 minutes/speaker, it was an interesting challenge to decide what and how to present. Here’s the video and the underpinning written statement which includes the sources I mention (with thanks to several colleagues for their input).

To see all the contributions, here’s the full meeting replay and final Parliamentary Brief.