UTS awards CIC’s multi-year, multidisciplinary faculty collaboration, using the OnTask tool to scale personalised feedback and build student belonging.
Since 2017, CIC has been supporting academics to embed OnTask messaging in learning design, offering workshops and consultations. When Lisa-Angelique Lim joined CIC in 2021, she brought deep expertise from her PhD at UniSA, and took this work to a new level, leading a learning community to implement OnTask across a wider range of subjects, thereby enabling personalised feedback and support for students in multiple faculties.
So it’s a true mark of recognition to see that CIC, in partnership with UTS academics from multiple faculties, received an award in recognition for our sustained efforts in the Student Experience category, for fostering students’ belonging in multiple disciplines through personalised feedback. This accolade was part of the Vice-Chancellor’s 2024 Learning and Teaching Awards and Citations, and was celebrated at the 2025 Learning and Teaching Awards Ceremony at UTS on 4 April 2025.
Our large, multidisciplinary team was led by Dr Lisa-Angelique Lim (CIC), with Associate Professor Amanda White (Business), Dr Amara Atif (FEIT), Chris Croese (Law), Associate Professor James Wakefield (Business), Dr Keith Heggart (FASS), Associate Professor Nicole Sutton (Business), Ram Ramanathan (CIC PhD student), Dr Rina Dhillon (Business), Dr Simone Faulkner (Business), and Professor Simon Buckingham Shum (CIC).
Addressing the challenges of student belonging with personalised feedback using OnTask
Belonging is a cornerstone of the student experience, critically impacting engagement, retention, and overall success. At UTS, the Student Experience Framework places a strong emphasis on belonging. However, the challenge of nurturing a sense of belonging at the classroom level is significant, given the diverse and large student population, particularly in first-year core subjects. Personalised feedback has emerged as a potent tool to address this challenge, serving as a form of ‘relational pedagogy’ that builds self-efficacy and fosters greater engagement and thriving at university.
To tackle the challenge of belonging, our team leveraged OnTask — a tool designed to support students through personalised communication and feedback across multiple disciplines, based on their data. OnTask uses learning analytics to provide tailored messages that help students stay on track with their studies, understand their progress, and feel supported throughout their academic journey. Check out this short animation on how OnTask works.
Quick teaser video of our academics’ perspectives of their implementation of personalised feedback in their context:
What this looks like in practice
Here, we highlight a few examples of the work by our team.
The UTS Business First and Further Year Experience (FFYE) team, led by A/Prof James Wakefield and Dr Simone Faulkner, used OnTask to personalize orientation communications for both undergraduate and postgraduate commencing students, leading to a significant increase in orientation registrations.
A/Prof Amanda White used OnTask in the Accounting for Business Decision A (ABDA) subject to tailor emails based on students’ progress, encouraging higher engagement and leading to improved exam grades.
Dr Keith Heggart created personalised video messages tailored to students’ confidence levels in the fully online Graduate Certificate in Learning Design, significantly enhancing student engagement and perceptions of support.
The positive impacts of OnTask are well-documented. For instance, in the large first-year subject, 22208 Accounting, Business and Society (ABS), led by Dr Rina Dhillon and A/Prof Nicole Sutton, personalised messages based on weekly quiz results led to improved pass rates and enhanced feelings of being valued among at-risk students.
Similarly, personalised feedback messages sent by Chris Croese to his students in large Law subjects kept students on track and motivated, correlating with better final grades. More stories of how academics at UTS have used OnTask, with research papers, can be found on CIC’s OnTask page.
Towards future partnerships to enhance student belonging
With this award, we celebrate the collaborative efforts and innovative approaches taken by our team to foster a sense of belonging among students through personalised feedback. Our journey over these past three years demonstrates the sustainability of this practice and its potential to influence and enhance teaching and learning widely.
We are honoured to receive this recognition, and look forward to continuing strong partnerships, to make a positive impact on the student experience at UTS.
CONTEXT… Those of you who know my R&D will know that I have a long-standing interest in the role that software can play in human sensemaking around wicked problems, a form of “augmenting human intellect” (Doug Engelbart). A powerful example is structured, visual hypermedia that makes tangible the ways that ideas, data and arguments connect with each other and documents.
This is certainly a story about the evolution of an interactive visual tool for thinking—but far more interestingly, it’s about the co-evolution of software with a set of practices to develop human fluency with the tool. Those practices were (in historical order) Dialogue Mapping, Issue & Argument Mapping, and Knowledge Art.
COMPENDIUM… provides extremely flexible hypermedia linking between conceptual objects (e.g. questions, ideas, arguments), data and documents (local/online), through a visual user interface. As a hypertext system for thinking with, connections can be made in multiple ways: spatial proximity and visual linking within a view, plus tagging and transclusions across views (i.e. embedding a node in multiple views). Views can contain each other non-hierarchically (A can contain B which can contain A). Search can be refined by node types and tags. This 2014 RAE Impact Case distills the research –> impact narrative.
It is an open source, desktop Java application with a full SQL database, with XML and SQL import/export, and HTML publishing. In the course of its development, it was interoperable with (at the time state of the art) Jabber (open source instant messaging) XML, and semantic web RDF.
HYPERTEXT HISTORY… Compendium is descended from the pioneering hypermedia system gIBIS (graphical Issue-Based Information System) at MCC Labs led by Jeff Conklin and Michael Begeman, which led to the commercial corporate memory product CM1, renamed Questmap. Compendium was then developed over the course of around 20 years R&D starting in the early 90s at Bell Atlantic labs (White Plains NY) led by Al Selvin and Maarten Sierhuis, continued at NASA Ames Research Centre by Maarten on the Mobile Agents project providing a human-agent science team tool, in collaboration with my team at the Open University’s Knowledge Media Institute (KMI) from 1995-2014. The gIBIS/CM1/Questmap/Compendium lineage exemplifies an influential strand of hypertext R&D that preceded the invention of the Web, which like Xerox NoteCards, exemplifies what Frank Halasz calledhypertext for idea processing — focusing on visualising and managing the connections between nodes as a form of intellectual work. You can learn a lot more about the intellectual lineage of these ideas in this brief history, another account, on this blog via the compendium tag, and on the memorial archive of Compendium co-inventor, my colleague, PhD student and friend, Al Selvin.
COMPENDIUM INSTITUTE… The Compendium Institute coordinated the international user/developer network, feature requests, code releases, research and training. Software development has been on pause since 2013, and may well not be continued, since much of the world now expects web-based tools (indeed see the great work by DebateGraph, and the KMI team developed quite a few). However, there remains an active user base of people who value the speed and functionality of a Java desktop app which continues to run on current Mac/Win/Linux Java.
I have therefore archived the Compendium Institute website for posterity, since it contains lots of resources:
I can also offer this Mac installer version of CNG which requires no code knowledge to get running
If you’re non-technical on Win/Linux then this version of Compendium comes in an integrated installer:
KMi Open University version 2.0beta with advanced experimental features (like Maps supporting video annotation): Downloads page | QuickStart Guide for Mac or Windows*follow the guide*
CogNexus Institute also offers download links for a slightly older version
COMMUNITY… Since Yahoo closed down their groups end of last year, I’ve created a new Google Group which you’re warmly invited to join if you want to stay connected with fellow users and some of the original team. Collectively, we will hopefully be able to answer any queries about Compendium’s functionality and design rationale — and who knows, possible futures…
In 2015 I blogged about the emergence of what I dubbed “Writing Analytics” as a stream within the wider Learning Analytics field. Five years on, we see regular Writing Analytics workshops at LAK and ALASI (see the Events menu on that link), publications appearing in top tier conferences and journals (e.g. see below), and even the launch of a new conference and journal in the last two years.
This blog is to mark the emergence of Reflective Writing Analytics as a sub-stream. Reflective writing is quite different from the more widely used forms of academic writing (literature review, persuasive essay, research paper; etc.), but is growing in importance as we seek to place learners in authentic learning contexts (e.g. internships; work placements; or simulated teams), specifically so that they experience something of the complexity of real workplaces.
Honest reflection can make the writer vulnerable, as they reflect on their uncertainties, failings, and how they are changing as a learner/professional. They are often deeply personal, connecting to different threads across the many areas of their life. That’s almost the opposite of the other genres of writing that dominate students’ and professionals’ lives, which emphasise rational distance, mastery of the material, and confident rhetoric. Deep reflection can even share transformational moments in someone’s life and learning. Speaking personally, I find some student reflections profound, and even moving — an inspirational reminder of why we do what we do. Yet reflecting is not something that people are always (or even often) given the opportunity to learn how to do well.
As noted in a recent paper:
“Helping people make sense of their thoughts, feelings, reactions and approaches when stretched out of their comfort zones is core business for educators and coaches. Suitably supported, honest reflection makes it safe to question assumptions and consider change, but we also know that this is often difficult to teach, and challenging to learn.
[…] Reflective Writing is a strategy used in education and many professions to help learners, professionals and leaders make sense of challenging experiences, and prepare for the future. It integrates “head and heart”: valuing not only technical/academic knowledge, but how this interplays with experiential/professional ways of knowing, and recognising the fact that learning and working engage our emotions and feelings.
[…] However, while we know there is nothing as valuable as detailed coaching feedback to build this capacity, this is a scarce, costly skillset and labor-intensive. The practical consequence is that most students and leaders do not understand how to reflect deeply, and do not receive good feedback.” (Buckingham Shum & Lucas, 2020)
The desire to provide people with useful and timely feedback on reflection on a widespread basis has led to interest in developing Reflective Writing Analytics — broadly, the use of natural language processing and automated feedback methods to understand and support the process of reflection. This is a nascent area. There are not many people (that we know of) working on the challenge of providing automated analysis of, and feedback on, reflective writing, so it was a delight to convene a post-LAK20 call last week with teams from the USA, UK and AUS, when we spent 2 hours comparing notes on what we’re wrestling with, and how we might collaborate to move the field forward.
Examples of current work are below to help you get up to speed with this emerging field, the different emphases within it, and the scholarly communities who participate. There’s a significant existing body of work outside the field of analytics on the nature of reflection, and how to teach reflective writing (reviewed in the papers). The intriguing challenge is to translate that, with integrity, into the world of text analytics and automated feedback. As we noted during our call, it’s a really exciting nexus of the cognitive, social, affective, pedagogical, ethical, user experience and technical.
Do get in touch if you want to join forces — everyone is most welcome, and we’re sure there must be more people out there doing this we haven’t met!
Cui, Y., Wise, A. F., & Allen, K. L. (2019). Developing Reflection Analytics for Health Professions Education: A Multi-dimensional Framework to Align Critical Concepts with Data Features. Computers in Human Behavior, 100, 305-324. https://doi.org/10.1016/j.chb.2019.02.019
Jung, Y. and Wise, A.F. (2020). How and How Well Do Students Reflect?: Multi-Dimensional Reflection Assessment in Health Professions Education. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK’20). ACM, New York, NY, USA, pp.595-604. https://doi.org/10.1145/3375462.3375528 [Preprint]
Wise, A. F., & Cui, Y. (2019,). Top Concept Networks of Professional Education Reflections. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK’19). ACM, New York, NY, USA pp. 260-264. https://dl.acm.org/doi/pdf/10.1145/3303772.3303840
Wise, A.F., Reza, S. & Han, R. J. (2020). Becoming a Dentist: Tracing Professional Identity Development through Mixed-Methods Data Mining of Student Reflections. Proceedings of ICLS’20: International Conference of the Learning Sciences. Nashville, TN: ISLS. [Preprint]
Queensland University of Technology (Lead: Andrew Gibson)
Willis, J., and Gibson, A (2020). The Emotional Work of Being an Assessor: A Reflective Writing Analytics Inquiry into Digital Self-assessment. In Fox, J., Alexander, C., Aspland,T.(Eds.) Teacher Education in Globalised Times. (Sringer). https://doi.org/10.1007/978-981-15-4124-7 [pre-order]
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing AnalyticsFor ActionableFeedback. Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver, BC, Canada. (ACM Press), pp.153-162. http://dx.doi.org/10.1145/3027385.3027436 [Preprint] [Replay]
The Open University & Warwick University (Lead: Thomas Ullmann)
Ullmann, T. D. (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257. https://doi.org/10.1007/s40593-019-00174-2 [Preprint]
Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing Automatically Detected Reflective Texts with Human Judgements. 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning. CEUR-WS.org.http://ceur-ws.org/Vol-931/paper8.pdf
Try ReflectR – an online tool to classify sentences regarding reflection: http://qone.eu/reflectr
Alrashidi, Huda; Ullmann, Thomas; Ghounaim, Samiah and Joy, Mike (2020). A Framework For Assessing Reflective Writing Produced Within the Context of Computer Science Education. In: Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20, 24/03/2020, Frankfurt, Germany). [Preprint]
University of Technology Sydney (Lead: Simon Buckingham Shum)
Buckingham Shum, S., Á. Sándor, R. Goldsmith, R. Bass and M. McWilliams (2017). Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results. Journal of Learning Analytics, 4, (1), 58–84. https://doi.org/10.18608/jla.2017.41.5 (Open Access)
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing AnalyticsFor ActionableFeedback. Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver, BC, Canada. (ACM Press), pp.153-162. http://dx.doi.org/10.1145/3027385.3027436 [Preprint] [Replay]
Liu, M., Buckingham Shum, S., Mantzourani, E.,Lucas, C. (2019). Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. In: Isotani S., Millán E., Ogan A., Hastings P., McLaren B., Luckin R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science, vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_19 [Preprint]
Buckingham Shum, S. and Lucas, C. (2020). Learning to Reflect on Challenging Experiences: An AI Mirroring Approach. Proceedings of ACM CHI 2020 Workshop on Detection and Design for Cognitive Biases in People and Computing Systems, April 25, 2020 (online). [Preprint]
This project is a collaboration between UTS:CIC and the University of Melbourne SWARM Project. The successful candidate will be based in Sydney, also spending time in Melbourne.
Visit the CIC PhD Scholarships page for full details. Please email us to express interest, ask any questions, and if we can see a potential fit we’ll advise you on writing your proposal.
The Challenge
The real world challenge: improving collaborative EBR
The challenges facing society are so complex that multiple expertises are needed. Consider security, science, law, health, policy-making, finance. The teamis the ubiquitous organisational unit, but the quality of its reasoning, especially under pressure, can vary dramatically. Problems are not provided in neat, well-defined packages: a team must frame problems in creative ways that lead to insights, and resolve uncertainties around possible responses, making the best possible use of evidence, plus their own judgement. Studies of how teams engage in such “sensemaking” highlight the blinkers that can blindside teams, and how the ways that the problem is expressed and visually represented can help or hinder (Weick, 1995). We will term this whole process Evidence-Based Reasoning (EBR). (We note of course that politics and social dynamics are unavoidable whenever people come together, and effective team members learn how to navigate these dynamics effectively.)
Improving collaborative EBR is an interesting scientific and design challenge. A successful support system (i.e. ways of working + enabling tools) must respect the principles of good reasoning, as determined by fields such as logic, argumentation and epistemology, and the domain-specific knowledge (i.e. emergency response, engineering, social work, counter-intelligence, etc.). At the same time, it must accommodate the strengths, weaknesses and vagaries of human reasoners, which is the terrain of cognitive and social psychologists. If part of the support system is interactive software, then it must have a good user interface and a solid underlying architecture. Assessing the resulting performance is a difficult evaluation problem. Building such systems is therefore inherently multidisciplinary.
How do we better equip teams for collaborative EBR? From an educational perspective, teamwork, problem solving and critical thinking skills are now among the most in demand ‘transferable competencies’ (Fiore et al 2018). The challenge of assessing and equipping graduates in these is at the heart of the learning and teaching strategies at UTS and U. Melbourne.
The technology support challenge:
While in some fields, there are specialist tools for modelling and simulation that assist analysts by managing constraints in the problem, but even with machine intelligence, the agency typically rests with the human analysts to decide how much weight to give to the machine’s output. Most other fields, however, do not have such tools: collaborative EBR is typically supported by general-purpose information technologies such as word processors, spreadsheets, databases, and project planners to help with managing information and producing reports. Similarly, generic communication tools dominate, such as email, chat, video conferencing, phone. In most cases, the reasoning itself is typically left wholly to the human reasoners themselves.
There have been remarkably few attempts to provide direct technological support for the processes of inference and judgement that are at the heart of collaborative EBR, and moreover, those attempts have had little impact on the way it is actually conducted in most places (van Gelder, 2012). There are methods and software tools for facilitating group processes and visualising team reasoning, but these require quite an advanced facilitation and software skillset (e.g. Culmsee and Awati, 2013; Okada, et al., 2008; Selvin et al, 2012).
Our interest is in developing computer-support to improve the collaborative EBR of geographically and often temporally distributed teams, that does not require specialist skills to start using beyond using what are now familiar collaboration tools. SWARM is an online platform emerging from an ongoing research project to improve the kind of collaborative EBR undertaken by intelligence analysts making sense of complex sets of qualitative and quantitative information or varying reliability. However, these are the conditions under which most other domains operate, and we hypothesise that it has broader potential, and specifically in this project, for education and training. SWARM is based on three design principles: cultivating user engagement, exploiting natural expertise, and supporting rich collaboration (van Gelder et al, 2018). Central to its approach is the upskilling of team members to equip them with different EBR skills (see in particular the Lens Kit).
Figure: The SWARM workspace
Recent large scale empirical evaluations, in which teams of analysts tackled complex challenges with or without SWARM, indicated that the quality of the reports produced by SWARM teams was significantly better than reports produced by analysts using normal methods (van Gelder et al, In Prep). In a follow-up project, “super-teams” on the platform produced reasoning so good it would plausibly be called “super-reasoning” (van Gelder & de Rozario, 2017) analogous to “super-forecasting” (Tetlock & Gardner, 2015).
This CIC seminar is a great introduction to the work so far:
Learning Analytics for SWARM
The encouraging evidence of SWARM’s effectiveness makes it an attractive candidate platform for use in educational/training contexts. While evaluation of final reports (i.e. the team’s product) is a conventional measure of team performance, and certainly one that educators will be interested in, this is not the only possible indicator of improvement. The emergence of data science, activity-based analytics and visualisation opens new possibilities for tracking the process that teams are following. Learning Analytics connects such techniques to what is known about the teaching and learning of teamwork, and could make the assessment of team performance more rigorous, and more cost effective.
This PhD is therefore focusing on inventing and validating new forms of automated team analytics for collaborative EBR, to provide insights into both process and product. Such analytics might enable not only coaches and researchers to gain insights into a team’s effectiveness, but the teams themselves to monitor their work in real time, or critically review their project on completion. Further, real-time analytics can be used to shape the collaborative environment itself, resulting in better collaboration and better outputs. Some prototype analytics have already been developed to summarise participants’ contributions and interactions. This PhD will build on this work, synthesise the literature, plus insights from the SWARM team and educators, in order to define, design, implement and evaluate automated analytics in different contexts, spanning education and training, research, and potentially more authentic deployments with professional teams.
Figure: Early version of the SWARM group dynamics dashboard. Upper diagrams shows levels of interaction among team members working on a particular problem.
Relevant analytics techniques include, but are not limited to:
Text analysis to identify significant contributions to the team communications and the report they are producing
Social network analysis to identify significant interaction patterns among team members
Process mining to identify significant sequences in the actions that individuals engage in, within or between sessions
Statistical techniques to identify significant differences between teams
Candidates
In addition to the broad skills and dispositions that we are seeking in all candidates (see CIC’s PhD homepage), you should have:
A Masters degree, Honours distinction or equivalent with at least above-average grades in computer science, mathematics, statistics, or equivalent
Analytical, creative and innovative approach to solving problems
Strong interest in designing and conducting quantitative, qualitative or mixed-method studies
Strong programming skills in at least one relevant language (e.g. R, Python)
Experience with web log analysis, statistics and/or data science tools.
It is advantageous if you can evidence:
Design and Implementation of user-centred software, especially data/information visualisations
Skill in working with non-technical clients to involve them in the design and testing of software tools
Knowledge and experience of natural language processing/text analytics
Familiarity with the scholarship in a relevant areas (e.g. high performance teams; collective intelligence; collaborative problem solving)
We will discuss your ideas with you to help sharpen up your proposal, which will be competing with others for a scholarship. Please follow the application procedure for the submission of your proposal.
Fiore, S. M., Graesser, A., & Greiff, S. (2018). Collaborative problem-solving education for the twenty-first-century workforce. Nature Human Behaviour, 2(6), 367–369.
van Gelder, T., & de Rozario, R. (2017). Pursuing Fundamental Advances in Human Reasoning. In T. Everitt, B. Goertzel, & A. Potapov (Eds.), Artificial General Intelligence(Vol. 10414, pp. 259–262). Cham: Springer International Publishing.
Ming Liu and Simon Buckingham Shum (UTS:CIC), Cherie Lucas (UTS:Pharmacy) The supervision team for this PhD is a partnership between CIC and the School of Pharmacy, who together have pioneered reflective writing analytics.
Visit the CIC PhD Scholarships page for full details. Please email us to express interest, ask any questions, and if we can see a potential fit we’ll advise you on writing your proposal.
The Challenge
The societal challenge:
“We do not learn from experience… we learn from reflecting on experience.”
(paraphrasing John Dewey)
The problems now confronting society place an unprecedented urgency on learning from experience. Such is the pace of change that before we can plan for them, citizens and professionals in all sectors find themselves immersed in novel, complex problems. Moreover in education, the learning sciences tell us that crafting authentic experiences is a powerful trigger for learning.
“White water is the new normal”, as they say. But as Dewey noted, critical to this is our capacity to reflect on the experience of shooting those rapids. If we can’t learn how we could do better next time — individually and collectively — we are in deep trouble. From school age students, through higher education, and into professional leadership, we have to make sense of challenging experiences, recognise how we were challenged, how we are changing, and how we can improve.
At the heart of deep learning is our sense of identity. People rarely shift from entrenched positions by force of argument alone. However, when we undergo challenging experiences that force us to question assumptions and worldviews at the heart of our identity, this can indeed be transformational if we are assisted in making sense of this, and can emerge with our identify intact but now under reconstruction. Without such shifts, it’s hard to see how we will move beyond current polarisations around how we relate to each other, and the planet. Given our current political and cultural climate, applied research to help people reflect on how they adjust to threatening transitions is both timely, and of first order importance.
So, we need to get better at deep reflection, and clearly, there’s nothing as valuable as detailed feedback to provoke further reflection. But this is a scarce skillset and very labour-intensive. The practical consequence is that most students and leaders do not understand what good reflective writing is, and do not receive good feedback. For these reasons, there’s interest in educational and professional sectors in the potential of automated techniques to deliver real-time, personalised coaching.
In sum, this PhD is fundamentally about harnessing computational intelligence to deepen human learning in contexts spanning formal education, professional practice, and community transformation.
The writing challenge:
Effective written communication is an essential skill which promotes educational success for university students. However, far too many students have never had the features of good rhetorical moves explained well to them, and most educators are subject matter experts, not skilled writing coaches (Lucas, Gibson & Buckingham Shum, 2018). CIC initiated its Academic Writing Analytics (AWA) project in 2015, as it became clear through consultations across faculties that student writing was a strategically important area for UTS teaching and learning (and indeed, for most other educational institutions). The goal is to more effectively teach the building blocks of good academic writing by providing instant, personalised, actionable feedback to students about their drafts (Knight, Buckingham Shum, Ryan, Sándor, & Wang, 2018).
To deliver on this vision requires integrated expertise including natural language processing, linguistics, academic language pedagogy, learning design, feedback design, user experience, and cloud computing. This is truly a transdisciplinary effort, which has been enormously stimulating. To date, we have worked on critical, argumentative, analytical writing of the sort typically found in literature reviews, persuasive essays and research articles, as well as reflective writing, in which learners make sense of their workplace experiences, try to integrate this with their academic understanding, and share their own uncertainties, emotions and sense of personal challenge/growth (Gibson, Aitken, Sándor, Buckingham Shum, Tsingos-Lucas, & Knight, 2017).
Learning Analytics tools are most effective when co-designed with effective Learning Designs: the features constructed by the analytics align with the assessment criteria, and the tool is coherently embedded in authentic student learning tasks. Our program has demonstrated how this can be accomplished (Knight, Shibani & Buckingham Shum, 2018; Shibani, Knight, Buckingham Shum & Ryan, 2017).
Depending on your interests and skillset, critical advances that this PhD might range across technical and pedagogical contributions to educational technology:
Technical:
Integration between rule-based modelling and machine learning approaches
Accelerated customisation of the parsers to different disciplinary domains and genres of writing
Definition of new computational proxies that can serve as indicators of deep reflection
User experience and machine learning to enable user feedback that teaches the tool when it make errors
Curation of text corpora to advance the field
Pedagogical:
Design and validation of analytics-augmented learning design patterns
Radical improvements in the user experience of automated feedback, e.g. through novel educator/student co-design processes, or user interfaces
Analytics Approaches
We currently implement the underlying concept matching model using a rule-based grammar and human-curated lexicons, which for those not familiar with this kind of work, brings both pros and cons (Buckingham Shum, Sándor, Goldsmith, & McWilliams 2017; Ullmann, 2017). The rules are grounded in scholarly literature on the features of academic research writing, and have been tested on diverse texts by the team through close manual analysis. The lexicons can be edited to tune them to the language used in different disciplines and subjects. This relatively traditional AI approach provides familiar intellectual credentials when introducing the system to educators, and when we’re testing it, the underlying behaviour is easier to explain, and errors can be diagnosed very precisely. However, it brings the limitations associated with any rule-based approach: given the richness of open-ended reflective writing, there are exception cases to debug, and improvements to the system’s performance require manual edits to the rules and lexicon.
We are now beginning work to investigate if a machine learning approach can augment the current infrastructure (Liu, et al. 2019; Ullmann, 2019). Recent years, with the availability of “big data”, such as large question answer banks (Rajpurkar, Zhang, Lopyrev, & Liang, 2016), and effective machine learning algorithms, e.g. deep neural networks (Lecun, Bengio, & Hinton, 2015), data driven approaches based on new data processing architectures have attracted a great attention in natural language processing tasks, such as neural text summarization (Liu & Manning, 2017) and neural machine translation (Bahdanau, Cho, & Bengio, 2014), mainly because these approaches do not require human defined rules and have good generalization power. However, such data driven approaches require a large amount of data, and some statistical learning models such as deep neural networks are not easy to comprehend. Preliminary results are reported
Therefore, we invite your proposals as to which techniques might be best suited to this challenge. What’s more, the creation of a corpus for writing raises ethical challenges, and we invite your thoughts on what these are, and how we might address them.
You will work in close collaboration with one or more academics from other faculties/units in UTS, using co-design methods with academics, potentially external partners, with opportunities for synergy with existing projects and tools as described on the CIC website. For more information about ongoing research in this area, please visit the Academic Writing Analytics homepage and the Writing Analytics blog.
Resources to help you understand the current state of the technology and its educational applications include the references cited, plus:
We’re looking for the broad skills and dispositions that we are seeking in all candidates (see CIC’s PhD homepage). In addition, we envisage that applicants will either come from strong technical backgrounds and are passionate to see these make a difference in education, or from strong educational backgrounds seeking to shape the design of analytics/AI.
Core strengths that we expect from applicants with technical backgrounds:
A Masters degree, Honours distinction or equivalent with above-average grades in computer science, mathematics, statistics, or equivalent
Analytical, creative and innovative approach to solving problems
Strong interest in designing and conducting quantitative, qualitative or mixed-method studies
Knowledge and experience of natural language processing/text analytic
Strong programming skills in at least one relevant language (e.g. C++, .NET, Java, Python)
Experience with statistical and data mining, deep learning, or data science tools (e.g. R, Weka, Tensorflow, ProM, RapidMiner).
Core strengths that we expect from applicants with educational backgrounds:
A Masters degree, Honours distinction or equivalent with above-average grades in teaching, educational theory, instructional design, learning sciences (ideally experience in the pedagogy and scholarship of writing)
Knowledge and experience in Design-Based Research, or a related methodology for authentic design and evaluation of educational technologies
Qualitative and quantitative data collection and analysis skills
It is advantageous if you can evidence:
Skill in working with non-technical clients to involve them in the design and testing of software tools
Peer-reviewed publications
Design and Implementation of user-centred software
Interested candidates should contact the team to open a conversation: Ming.Liu@uts.edu.au; Simon.BuckinghamShum@uts.edu.au; Cherie.Lucas@uts.edu.au We will discuss your ideas to help you sharpen up your proposal, which will be competing with others for a scholarship. Please follow the application procedure for the submission of your proposal.
Buckingham Shum, S., Sándor, Á.,Goldsmith, R.,Bass R.,and McWilliams M.(2017). Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results. Journal of Learning Analytics, 4, (1), 58–84.
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C.,and Knight, S. (2017). Reflective Writing Analytics for Actionable Feedback. In Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge.
The start of the new year is a good moment to distill the key ingredients of the 2 year collaboration around writing analytics (automated feedback to students on their reflective writing), that CIC has built with UTS academic Cherie Lucas from our School of Pharmacy. As with other collaborations (such as with Pip Ryan on her students’ legal writing), it exemplifies the co-design process that we initiate with academics, in which we iteratively seek to design an automated feedback tool that students can use to improve their drafts, prior to submission:
distill key insights from the scholarship into the teaching and learning of good reflective writing, to design a formal, implementable model that – in principle – should be applicable to a wide range of reflective writing contexts (learn more)
understand the academic’s specific educational challenge in context (e.g. students are struggling to produce good reflective writing about their work experience placements)
establish the mapping to the features that our text analytics tool is able to detect
evaluate the performance of the parser (in close partnership with the academic)
In the video, Cherie Lucas describes the nature of the challenge, and what AcaWriter contributes to the learning experience. The interface looks like this, with the text editor frame on the left, and the Reflective Report annotation of the writing on the right, generated after a few seconds on clicking Get Feedback:
Zooming in on the automatically annotated student writing in the Reflective Report:
Not shown in the video is the Feedback Tab providing encouragement when there appear to be good features in the text, and actionable feedback for improvement, e.g.
For us, one of the hallmarks of a successful collaboration is that our academic partners’ own disciplinary community of educators recognise the advance they’ve made. Here’s a brief, very helpful introduction that Cherie wrote for her peers. Her work has excited significant interest with colleagues around the world, who are now initiating their own projects to install our software for piloting with their students.
Learn more about how we design Writing Activities with Writing Analytics through the integration of learning design, analytics, educator and student resources, and evaluation evidence…
Dive deeper…
Gibson A., Aitken A., Sándor Á., Buckingham Shum S., Tsingos-Lucas C. and Knight S. (2017), Reflective writing analytics for actionable feedback. Proceedings of LAK17: 7th International Conference on Learning Analytics and Knowledge, March 13-17, 2017, Vancouver, CA (ACM Press: NY). [Video] (AWARDED BEST PAPER)
Tsingos-Lucas C, Aitken A, Gibson A, Buckingham Shum S. (2017). Utilisation of a Novel Online Educational Toolto Assist Pharmacy Students to Self- Critique Reflective Writing Tasks. Proceedings of the 9th Pharmacy Education Symposium, Prato, Italy, 9-12th July 2017. (AWARDED BEST TEACHING INNOVATION POSTER)
Just kicking off, a project funded by Australian OLT which I’ve long been dreaming about under the heading of Social Learning Analytics [1-2], and now brought to life by Kirsty Kitto and her team at QUT, in collaboration with Mandy Lupton who is developing the concept of teaching+learning in the wild.
“The CLA toolkit helps students and teachers to harvest data about their activities in standard social media environments, and then provide immediate feedback and reports. It is currently in development, but the basic design is sketched out below:
The release of the Experience API (xAPI) makes it possible to capture student behaviour in a highly flexible manner and send it to a Learning Record Store (LRS) for immediate or later analysis. We currently have tools in development that will interface with the API’s of
And if you want to go and check out the source code, then it is available on GitHub.
This project has now been funded by the Australian Government’s Office for Learning and Teaching (OLT). The Proposal that we submitted can be found here if you would like more details.”
[1] Ferguson, R. and Buckingham Shum, S. (2012). Social Learning Analytics: Five Approaches. In: 2nd International Conference on Learning Analytics & Knowledge, 29 Apr – 02 May 2012, Vancouver, British Columbia, Canada, pp. 23–33. Open Access Eprint: http://oro.open.ac.uk/32910
[2] Buckingham Shum, S., & Ferguson, R. (2012). Social Learning Analytics. Educational Technology & Society, 15 (3), 3–26. Open Access Eprint: http://www.ifets.info/journals/15_3/2.pdf
On 31st July I fly to join the University of Technology Sydney, directing the new Connected Intelligence Centre. More on that later!
Meantime, here’s the replay of yesterday’s reflections on nearly 19 years in KMi and the OU (my talk starts 10mins in). My deepest thanks to all my colleagues for the memorable send-off! Slides embedded below, or download as PDF. Read on however, for news of the KMi team who take this work forward in the OU, and for a few ‘Director’s Cut’ additions — the clips that I was dying to show, but just couldn’t squeeze in!
The Contested Collective Intelligence research programme continues!
As well as pursuing this research in Sydney, in KMi my longstanding research fellow Anna De Liddo takes the team’s work forward, with Michelle Bachler continuing to provide the programming that has underpinned our work over the years, joined recently by Brian Plüss (research associate) on the EPSRC Election Debate Visualization Project, and Thomas Ullmann (research assistant/developer) on the EU Catalyst Project. PhD students Duygu Simsek (discourse analytics for academic writing) and Simon Knight (epistiemic commitments and collaborative online search) complete the new look team, of whom I’m very proud 🙂
The EU Catalyst Project represents the current state of the art in our web apps for Contested Collective Intelligence. Check out the demo movies on our YouTube channel to see how LiteMap enables you to highlight clips of interest in source materials, and place them into an emerging summary map of the debate; DebateHub moves us beyond posting an idea to be voted on, to more careful reflection on its pros and cons; and the CI Dashboard enables you to configure views onto deliberation spaces — all based around the ‘DNA’ of reflective dialogue which I introduced: variations of Questions, Ideas, Pros and Cons. Two examples…
The Evidence Hub is a slightly different platform, architected to pool a community’s collective knowledge (as a web of ideas and arguments) about what works (and fails) in tackling pressing issues. Designed to connect researchers, practitioners, policymakers and enterprise, a Hub also geographically maps the people, projects and organisations behind those ideas. It comes with built-in analytics, alerts and report generation:
Knowledge, skills, and dispositions
A clip from the Reinventing University video roundtable on New Metrics, in which Ruth Deakin Crick introduces her work on assessing learning dispositions. You can learn more about dispositional analytics in this replayable workshop, and our conference presentation.
Video from a case study of designing authentic inquiry with explicit assessment of the above learning dispositions, in Bushfield School, Wolverton. I was delighted to be joined on Tuesday by the two Headteachers I’ve worked with closely as Chair of Governors, Andrea Curtis and Steve Springett-McHugh, who embody the kind of vision, courage and compassion needed by school leaders today.
One of the most profound shifts that takes place through education is not simply a grasp of how wonderful and complex our world is, but a growing awareness of the nature of knowledge itself. As we grow from childhood into educated adulthood, we should mature in our ability to hold ideas ‘loosely’, as constructs which can be inspected and critiqued from different perspectives. Within educational research, these are studied under terms such as epistemic beliefs, behaviours and commitments. How sophisticated is a learner’s notion of what counts as a valid source of information, how it is justified, and how stable it is?
Moreover, since this we’re engaged in a learning analytics research programme, seeking to validate digital traces as proxies for deep learning, how might one identify degrees of epistemic commitment from naturalistic online learner behaviour?
Simon Knight is focusing on these questions in his PhD, co-supervised by me with Karen Littleton. The line of attack is to see whether online search and sensemaking behaviours could serve as a lens into the learner’s mind…
Knight, S.J.G. (2013). Learning Analytics for Epistemic Commitments in a Collaborative Information Seeking Environment.Technical Report KMI-13-04 (Dec. 2013), Knowledge Media Institute, The Open University, UK. Available Online: http://kmi.open.ac.uk/publications/techreport/kmi-13-04
Abstract: This report argues that information seeking – the searching, frequently conducted on search engines such as Google, in order to retrieve information for some needs – should be of interest to education. It further suggests that such interest should focus on information commitments, which are implicated in the ways that people find, and process, information. Building on literature researching collaboration in both education and information seeking research, I claim that Collaborative Information Seeking (CIS) is a good lens through which to research information commitments. Indeed, a key component of this report is a preliminary proposal for a new theory of epistemic commitments, which addresses some concerns with prior research on epistemic beliefs, epistemic cognition, and epistemic (or information) commitments. Two novel components of that new theory are a focus on information trace as the core of epistemic activity, and a focus on the ‘dialogic space’ as particularly epistemically relevant. The report goes on to propose a technological solution for the analysis of epistemic commitments in the form of a Computer Supported Collaborative Learning/Work (CSCL/W) environment. This proposal includes an analysis of trace for epistemic commitments, and a discussion of relevant discourse centric learning analytics for the analysis of chat data around epistemic activity. While discourse data has received some analysis in CIS research, the analysis has generally been somewhat shallow in its focus; the proposal made in this report is for a deeper analysis, both as an extension of CIS research, and as of interest to learning analytics (and indeed learning researchers generally) who are interested in the collaborative context of learning. The report is thus proposes a relevant research theory to investigate the ways in which people make commitments in online information seeking environments, and how they might be supported. The report comprises three sections: 1. I start with a literature review, which begins with an overview of some relevant theoretical, and philosophical, literature relating this particularly to learning analytics (section 2). I then introduce the relevant literature on information seeking and epistemic cognition (or, as I propose, commitments) (sections 3-8). Section 9 then introduces some relevant literature on tasks to probe epistemic commitments, while section 10 introduces some core software to do so. 2. Section 11 then discusses my preliminary practical work, with respect to pilot empirical work and some key skills gained 3. Section 12 then introduces my formal research proposal – including research questions, proposed tools, and practical experiments to be conducted.
In a social, data-driven online world, this should of course be answered by what I can evidence. LinkedInLabs have done a very nice job, in my view, of producing a network visualization of my contacts, which really does reflect my thematic networks.
The only thing they haven’t cracked is automatic labelling of the sub-networks. I have to manually assign a pithy summary label, when really they could generate a tag cloud with some ontology-enriched language technology.
One of the most exciting things about releasing Compendium into the wild is that people go off and do amazing things with it that I have no idea about.
Here is a detailed case study from Groupaya, documenting the use of Dialogue Mapping to achieve a breakthrough in collective sensemaking, clearly a hugely collective effort, aided by the expert Dialogue Mapping of two friends and colleagues I’ve known for years, but haven’t seen for a while as I moved into other fields: Jeff Conklin and Eugene Eric Kim.
This report is a very honest account of the sensemaking journey undertaken by a group of people as they wrestled with a classic wicked problem in environmental planning, which had defied previous attempts at bringing people round the table productively.
I am so grateful to these colleagues who are very busy, but deeply reflective practitioners, taking the trouble to document their work in such detail, and so accessibly. Something for us academics to learn there!
