UTS awards CIC’s multi-year, multidisciplinary faculty collaboration, using the OnTask tool to scale personalised feedback and build student belonging.
Since 2017, CIC has been supporting academics to embed OnTask messaging in learning design, offering workshops and consultations. When Lisa-Angelique Lim joined CIC in 2021, she brought deep expertise from her PhD at UniSA, and took this work to a new level, leading a learning community to implement OnTask across a wider range of subjects, thereby enabling personalised feedback and support for students in multiple faculties.
So it’s a true mark of recognition to see that CIC, in partnership with UTS academics from multiple faculties, received an award in recognition for our sustained efforts in the Student Experience category, for fostering students’ belonging in multiple disciplines through personalised feedback. This accolade was part of the Vice-Chancellor’s 2024 Learning and Teaching Awards and Citations, and was celebrated at the 2025 Learning and Teaching Awards Ceremony at UTS on 4 April 2025.
Our large, multidisciplinary team was led by Dr Lisa-Angelique Lim (CIC), with Associate Professor Amanda White (Business), Dr Amara Atif (FEIT), Chris Croese (Law), Associate Professor James Wakefield (Business), Dr Keith Heggart (FASS), Associate Professor Nicole Sutton (Business), Ram Ramanathan (CIC PhD student), Dr Rina Dhillon (Business), Dr Simone Faulkner (Business), and Professor Simon Buckingham Shum (CIC).
Addressing the challenges of student belonging with personalised feedback using OnTask
Belonging is a cornerstone of the student experience, critically impacting engagement, retention, and overall success. At UTS, the Student Experience Framework places a strong emphasis on belonging. However, the challenge of nurturing a sense of belonging at the classroom level is significant, given the diverse and large student population, particularly in first-year core subjects. Personalised feedback has emerged as a potent tool to address this challenge, serving as a form of ‘relational pedagogy’ that builds self-efficacy and fosters greater engagement and thriving at university.
To tackle the challenge of belonging, our team leveraged OnTask — a tool designed to support students through personalised communication and feedback across multiple disciplines, based on their data. OnTask uses learning analytics to provide tailored messages that help students stay on track with their studies, understand their progress, and feel supported throughout their academic journey. Check out this short animation on how OnTask works.
Quick teaser video of our academics’ perspectives of their implementation of personalised feedback in their context:
What this looks like in practice
Here, we highlight a few examples of the work by our team.
The UTS Business First and Further Year Experience (FFYE) team, led by A/Prof James Wakefield and Dr Simone Faulkner, used OnTask to personalize orientation communications for both undergraduate and postgraduate commencing students, leading to a significant increase in orientation registrations.
A/Prof Amanda White used OnTask in the Accounting for Business Decision A (ABDA) subject to tailor emails based on students’ progress, encouraging higher engagement and leading to improved exam grades.
Dr Keith Heggart created personalised video messages tailored to students’ confidence levels in the fully online Graduate Certificate in Learning Design, significantly enhancing student engagement and perceptions of support.
The positive impacts of OnTask are well-documented. For instance, in the large first-year subject, 22208 Accounting, Business and Society (ABS), led by Dr Rina Dhillon and A/Prof Nicole Sutton, personalised messages based on weekly quiz results led to improved pass rates and enhanced feelings of being valued among at-risk students.
Similarly, personalised feedback messages sent by Chris Croese to his students in large Law subjects kept students on track and motivated, correlating with better final grades. More stories of how academics at UTS have used OnTask, with research papers, can be found on CIC’s OnTask page.
Towards future partnerships to enhance student belonging
With this award, we celebrate the collaborative efforts and innovative approaches taken by our team to foster a sense of belonging among students through personalised feedback. Our journey over these past three years demonstrates the sustainability of this practice and its potential to influence and enhance teaching and learning widely.
We are honoured to receive this recognition, and look forward to continuing strong partnerships, to make a positive impact on the student experience at UTS.
Summary: a new paper forges a bridge between data-driven, open automated feedback platforms, and teacher feedback literacy competences:
Buckingham Shum, S., Lim, L.-A., Boud, D., Bearman, M. & Dawson, P. (2023). A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy. International Journal of Educational Technology in Higher Education, 20:40 (12 July 2023). https://doi.org/10.1186/s41239-023-00410-9
The mass availability of generative AI continues to reshape thinking about the future of work and learning. Conversational apps can now give instant feedback to learners about their work — but the educational question is how effective this interaction is. A new design space has opened up for tuning generative AI to give high quality feedback to learners about their work. We are not in uncharted waters here: there is a growing body of knowledge on what “effective feedback” means in higher education, and how to create the conditions for this. It goes far beyond comments accompanying an assignment, with a shift towards “feedback rich ecosystems” in which both teachers and students exercise far greater agency and sensemaking competencies.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
In 2020, I conceived a symposium to bring the editors and authors to UTS to spend 2 days in dialogue with CIC and other researchers developing automated-feedback tools using Learning Analytics/AI. We called for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI. The pandemic shifted this online, but the goals remained the same, and moving online enabled us to more easily bring in additional participants, resulting in DAFFI 2020: Designing Automated Feedback for Impact whose presentations I commend to you.
I’m now delighted to share one of the fruit from this, a collaboration between CIC (Lisa Lim and myself) and our colleagues at Deakin University’s Centre for Research in Assessment and Digital Learning (CRADLE). The focus of the paper is not on generative, conversational AI (which did not exist when we started this work), but on technically less complicated, but correspondingly far more transparent platforms that use simple rules authored by teachers themselves.
“In contrast to closed AF tools, we define “open” AF tools as enabling the educator to specify some or all of the following key parameters in the tool’s behaviour:
the student activity data that the system analyses;
the algorithms that analyse that data;
the feedback information the teacher wishes the software to compile for students;
the modalities via which feedback information is communicated by teachers;
the student-driven feedback processes that are afforded.”
What does it mean to do this skillfully? We demonstrate that Boud & Dawson’s teacher feedback literacy competency framework can be applied very usefully to analysing teaching practices with data-driven, automated feedback platforms. A next step will be to think through what this means for tuning large language models for educational contexts.
A comparative analysis of the skilled use of automated feedback tools through the lens of teacher feedback literacy
a University of Technology Sydney, AUS b Deakin University, AUS c Middlesex University, UK
Effective learning depends on effective feedback, which in turn requires a set of skills, dispositions and practices on the part of both students and teachers which have been termed feedback literacy. A previously published teacher feedback literacy competency framework has identified what is needed by teachers to implement feedback well. While this framework refers in broad terms to the potential uses of educational technologies, it does not examine in detail the new possibilities of automated feedback (AF) tools, especially those that are open by offering varying degrees of transparency and control to teachers. Using analytics and artificial intelligence, open AF tools permit automated processing and feedback with a speed, precision and scale that exceeds that of humans. This raises important questions about how human and machine feedback can be combined optimally and what is now required of teachers to use such tools skillfully. The paper addresses two research questions: Which teacher feedback competencies are necessary for the skilled use of open AF tools? and What does the skilled use of open AF tools add to our conceptions of teacher feedback competencies? We conduct an analysis of published evidence concerning teachers’ use of open AF tools through the lens of teacher feedback literacy, which produces summary matrices revealing relative strengths and weaknesses in the literature, and the relevance of the feedback literacy framework. We conclude firstly, that when used effectively, open AF tools exercise a range of teacher feedback competencies. The paper thus offers a detailed account of the nature of teachers’ feedback literacy practices within this context. Secondly, this analysis reveals gaps in the literature, signalling opportunities for future work. Thirdly, we propose several examples of automated feedback literacy, that is, distinctive teacher competencies linked to the skilled use of open AF tools.
Here’s the open access preprint of a chapter to appear in a forthcoming book edited by Olga Viberg and Åke Grönlund. I was grateful to be invited to contribute to this, and it was an enjoyable reflective journey figuring out how to tell the story. I hope you enjoy exploring the four rooms!
Buckingham Shum, S. (2022). Embedding Learning Analytics in a University: Boardroom, Staff Room, Server Room, Classroom.In Viberg, O. and Grönlund, Å. (Eds.),Practicable Learning Analytics, SpringerNature.
Abstract: In this chapter, I describe and reflect on the last 8 years at an Australian public university, inventing, piloting and evaluating Learning Analytics tools, specifically focused on data-driven personalised feedback, leading in some cases to integration with the institution’s learning technology ecosystem, and accompanied by staff training and support. I will summarise this as conversations in the Boardroom, the Staff Room, the Server Room and the Classroom, reflecting the different levels of influence, partnership and adaptation required to introduce and sustain novel technologies in the complex system that constitutes a university, or indeed, any educational institution. This chapter is pragmatic, documenting aspects of our work that are typically not the focus in research papers, intending to make a practice contribution.
The role of automated feedback systems in creating feedback-rich environments
Last week I hosted a 2 day dialogue, Designing Automated Feedback for Impact (DAFFI 2020). The original concept was to bring the editors and authors from The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners (Eds. Henderson, Ajjawi, Boud & Molloy) to the UTS Connected Intelligence Centre, to spend 2 days in a workshop. The pandemic shifted this online, but the goals remain the same and moving online enabled us to more easily bring in additional participants.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
We call for a deeper dialogue between researchers in the design of assessment and feedback in higher education, and researchers developing automated-feedback tools using Learning Analytics/AI.
Perhaps we can learn from each other:
designers of automated feedback (for students or educators) are challenged on how they could build more robustly on principles of good feedback design;
researchers and educators working on feedback design are challenged as to whether automated feedback opens up new possibilities not taken into account in prior research;
potentially, new concepts may emerge that provide important language to clarify a changing design space for “feedback-rich environments” (as the book terms them);
new opportunities for learning analytics to tackle obstacles to the uptake of better feedback design practices;
identify topics for future events, and potential next steps.
Over two days, the book’s authors shared examples of this reconceptualisation of designing feedback for impact, and learning analytics researchers showed what is now possible with automated feedback. The extended dialogue was very rich, and we look forward to sharing the fruit from that as we reflect on how to take this forward.
10.30Identifying the Impact of Feedback Over Time and at Scale: Opportunities for Learning Analytics [slides]
Dragan Gaševic (Monash)
This talk explores how learning analytics can help educators design impactful feedback processes and support learners to identify the impact of feedback information, both across time and at scale. In doing so, it offers current examples of how learning analytics could guide policy and educational designs and be usefully employed to support learners to direct their own learning and study habits. This chapter also highlights how learning analytics can help individuals understand and optimise learning, and the environments in which the learning occurs.
30mins: Progress and challenges [Key ref: Book Chapter 12]
30mins: Questions and commentary / General discussion
11.40Break
11.55Automated Feedback on Collocated Teamwork & Classroom Proxemics [slides]
Our work with colleagues in Health focuses on how sensors and multimodal analytics enable automated feedback to nursing teams on embodied, collocated activity. Work with Science has used movement tracking to prototype automated feedback to educators on their use of teaching spaces.
30mins: Questions and commentary / General discussion
12.55Lunch
2.00Role of automated feedback in generating feedback-rich environment [slides]
Michael Henderson (Monash) and Rola Ajjawi (Deakin)
The challenges and opportunities of identifying, influencing and assessing feedback impact.
30mins: Overview – Feedback Research & Practice Challenges [Key ref: Book Chapters 2, 14 and 15] and Rola Ajjawi & David Boud (2018) Examining the nature and effects of feedback dialogue, Assessment & Evaluation in Higher Education, 43:7, 1106-1119, DOI: 10.1080/02602938.2018.1434128
30mins: Questions and commentary / General discussion
3.00Break
3.15Redesigning feedback involves addressing the feedback literacy of students and staff [slides]
David Boud (Deakin)
The challenge of building feedback literacy in students and staff
30mins: Overview Key refs Book Chapter 4 and:
Carless, D. and Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback, Assessment and Evaluation in Higher Education, 43, 8, 1315-1325. DOI: 10.1080/02602938.2018.1463354
Molloy, E., Boud, D. and Henderson, M. (2020) Developing a learner-centred framework for feedback literacy, Assessment and Evaluation in Higher Education, 45, 4, 527-540. DOI: 10.1080/02602938.2019.1667955
30mins: Questions and commentary / General discussion
4.15Reflections on Day 1
4.30Close
Wed 9 Sept (all times AEST)
10.15Fresh Croissants & Reflections for those who want to join early
10.30 Assessment and feedback design at scale [replay][slides]
Jaclyn Broadbent (Deakin)
This is a practice-based discussion of feedback design at scale in a context involving 1500 students. Discussion touches on improving understanding of standards, scaffolded assessment, high-quality audio feedback with feedforward aspects. This practice-based discussion will also mention the use of a tool known as Intelligent Agents which send automated feedback to students based on their digital activity as a way for staff and students to connect.
30mins: Questions and commentary / General discussion
11.30Break
11.45Examining impact and sense-making of personalised feedback messages using OnTask [replay][slides]
Lisa Lim & Abelardo Pardo (UniSA)
An OLT consortium has designed and is now piloting a platform called OnTask which enables an educator to design personalised feedback messages for hundreds of students at a time, based on their digital activity. Evidence is now emerging regarding the student and educator experience of such tools, and how their effectiveness can be judged.
30mins: Examining impact and sense-making of personalised feedback messages using OnTask
Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction. doi:10.1016/j.learninstruc.2019.04.003
Lim, L.-A., Dawson, S., Gašević, D., Joksimović, S., Pardo, A., Fudge, A., & Gentili, S. (2020). Students’ perceptions of, and emotional responses to, personalised LA-based feedback: An exploratory study of four courses. Assessment & Evaluation in Higher Education. doi:10.1080/02602938.2020.1782831
30mins: Questions and commentary / General discussion
12.45Lunch
2.00Where have we got to?
Emerging themes, overlapping interests, next steps…
Significant work has focused on how we co-design automated feedback on writing with educators and students, leading to the refinement and release of an (open source) web tool called AcaWriter (orientation website for staff and students). This uses natural language processing to identify ‘rhetorical moves’ that are hallmarks of different genres of academic writing, in order to generate formative feedback.
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á., Kitto, K., Liu, M., Mogarkar, R. & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12, (1), 141-186. (Published online 12 April 2020). DOI: https://doi.org/10.17239/jowr-2020.12.01.06
Antonette Shibani, Simon Knight and Simon Buckingham Shum (2020). Educator perspectives on learning analytics in classroom practice. The Internet and Higher Education, Volume 46. Available online 20 February 2020. https://doi.org/10.1016/j.iheduc.2020.100730
Automated feedback on online engagement (any platform)
We co-designed and are now piloting a platform called OnTask which enables an educator to design personalised feedback messages or portals for hundreds of students at a time. Other institutions are embedding this or similar platforms (like ECoach and SRES), and evidence is now emerging regarding the student and educator experience of such tools, and how their effectiveness can be judged.
Introductions to OnTask and EClass: see these workshop videos
Lisa-Angelique Lim, Shane Dawson, Dragan Gašević, Srecko Joksimović, Abelardo Pardo, Anthea Fudge & Sheridan Gentili (2020) Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: an exploratory study of four courses, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2020.1782831
Hamideh Iraj, Anthea Fudge, Margaret Faulkner, Abelardo Pardo, and Vitomir Kovanović. 2020. Understanding students’ engagement with personalised feedback messages. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (LAK ’20). Association for Computing Machinery, New York, NY, USA, 438–447. DOI: https://doi.org/10.1145/3375462.3375527
Pardo, A., Bartimote, K., Buckingham Shum, S., Dawson, S., Gao, J., Gašević, D., Leichtweis, S., Liu, D., Martínez-Maldonado, R., Mirriahi, N., Moskal, A. C. M., Schulte, J., Siemens, G. and Vigentini, L. (2018). OnTask: Delivering Data-Informed, Personalized Learning Support Actions. Journal of Learning Analytics, 5(3), 235-249. doi:https://doi.org/10.18608/jla.2018.53.15
Roberto Martinez-Maldonado, Vanessa Echeverria, Gloria Fernandez Nieto, and Simon Buckingham Shum. 2020. From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–15. DOI: https://doi.org/10.1145/3313831.3376148
Critical, human-centred, design of learning analytics
Broader perspectives that could help illuminate how we design Analytics/AI-augmented “feedback rich environments”.
Buckingham Shum, S.J. and Luckin, R. (2019), Learning analytics and AI: Politics, pedagogy and practices. British Journal of Educational Technology, 50, (6), pp.2785-2793. http://dx.doi.org/10.1111/bjet.12880
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-Centred Learning Analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1
Kitto, K., Buckingham Shum, S., & Gibson, A. (2018). Embracing imperfection in learning analytics. Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Association for Computing Machinery, New York, NY, USA, pp.451–460. DOI: https://doi.org/10.1145/3170358.3170413
It is with enormous pleasure that I applaud Vanessa Echeverria Barzola for completing her PhD here at UTS:CIC!
Moving with her family from Ecuador in 2017, Vanessa joined as one of our first doctoral researchers when we launched the Learning Analytics PhD Program, and has wrapped it up within her four years, demonstrating huge personal resilience on the way.
Read her thesis below, related CIC News stories, and browse her website for the extensive publications generated by her work. We wish her every success as she develops her academic career, and can’t wait to see what emerges!
Vanessa Echeverria (2020). Designing Feedback for Collocated Teams using Multimodal Learning Analytics. Doctoral Dissertation, Connected Intelligence Centre, University of Technology Sydney, Australia. http://hdl.handle.net/10453/140936
Abstract: The ability to communicate, be an effective team or group member and collaborate face-to-face are critical skills for employability in the 21st century workplace. Previous research suggests that learning to collaborate effectively requires practice, awareness of group dynamics and reflection upon past activities. However, although having a teacher closely supervising and providing detailed feedback to each group would be ideal, it may be unrealistic in practice. A promising way to approach this challenge could be to capture behavioural traces from group interactions in order to generate comprehensible and actionable feedback to support team reflection. In this sense, Multimodal Learning Analytics (MMLA) is a promising field, offering the potential to track learners’ activity across digital and collocated contexts, using emerging sensing and pervasive computing technologies. Most of the research in MMLA has been conducted in lab conditions, to help researchers validate learning theories or generate more comprehensive learner models. However, one of the most underexplored aspects of MMLA has been the generation of feedback to support teaching and learning, and moreover, in authentic locations and activities.
This thesis reports progress in tackling this challenge by designing and validating computer-based feedback, by means of visual representations and narrative, to support effective, guided reflection using multimodal learning analytics evidence. To achieve this, three contributions are presented. The first contribution is a human-centred design method to translate the informal outputs of co-design sessions with teachers and students, into more meaningful group work constructs with clear MMLA design requirements. The second contribution is a modelling approach to add meaning to low-level multimodal group data based on the characteristics of the context (domain expertise, theory, and the learning design). Finally, the third contribution is an approach for augmenting visual representations with data storytelling elements to facilitate the interpretation of group dynamics insights by educators and students. This thesis is developed in the context of two distinct, collocated group work settings, in the domains of collaborative database design and healthcare simulation. Using a Design-Based Research process, a set of interfaces (i.e. interfaces that communicate insights) was designed and validated with teachers and students. The thesis provides timely and necessary groundwork for researchers and practitioners to design visual representations capable of communicating actionable insights, using multimodal data in complex and authentic collaboration scenarios.
“This book asks how we might conceptualise, design for and evaluate the impact of feedback in higher education. Ultimately, the purpose of feedback is to improve what students can do: therefore, effective feedback must have impact. Students need to be actively engaged in seeking, sense-making and acting upon any information provided to them in order to develop and improve. Feedback can thus be understood as not just the giving of information, but as a complex process integral to teaching and learning in which both teachers and students have an important role to play. The editors challenge us to ask two fundamental questions: when does feedback make a difference, and how can we recognise that impact?”
It is with enormous pleasure that I applaud