Today’s slides at the Assessment Research Centre, University of Melbourne… [pdf]
Tag: reflection
Framing Professional Learning Analytics as Reframing Oneself
It’s been rewarding working with close colleagues on this, weaving our ideas together over the last year. Here’s the Open Access Preprint (final version has some minor edits). It will appear later this year in what should be a really interesting special issue on “Designing Technologies to Support Professional & Workplace Learning for Situated Practice”.
Buckingham Shum, S., Littlejohn, A., Kitto, K. & Crick, R. (2022). Framing Professional Learning Analytics as Reframing Oneself. IEEE Transactions on Learning Technologies, 15(5), pp.634-649. https://doi.org/10.1109/TLT.2022.3190055
Abstract: Central to imagining the future of technology-enhanced professional learning is the question of how data are gathered, analyzed, and fed back to stakeholders. The field of learning analytics (LA) has emerged over the last decade at the intersection of data science, learning sciences, human-centered and instructional design, and organizational change, and so could in principle inform how data can be gathered and analyzed in ways that support professional learning. However, in contrast to formal education where most research in LA has been conducted, much work-integrated learning is experiential, social, situated, and practice-bound. Supporting such learning exposes a significant weakness in LA research, and to make sense of this gap, this article proposes an adaptation of the Knowledge-Agency Window framework. It draws attention to how different forms of professional learning locate on the dimensions of learner agency and knowledge creation. Specifically, we argue that the concept of “reframing oneself” holds particular relevance for informal, work-integrated learning. To illustrate how this insight translates into LA design for professionals, three examples are provided: first, analyzing personal and team skills profiles (skills analytics); second, making sense of challenging workplace experiences (reflective writing analytics); and third, reflecting on orientation to learning (dispositional analytics). We foreground professional agency as a key requirement for such techniques to be used effectively and ethically.
The emergence of Reflective Writing Analytics
In 2015 I blogged about the emergence of what I dubbed “Writing Analytics” as a stream within the wider Learning Analytics field. Five years on, we see regular Writing Analytics workshops at LAK and ALASI (see the Events menu on that link), publications appearing in top tier conferences and journals (e.g. see below), and even the launch of a new conference and journal in the last two years.
This blog is to mark the emergence of Reflective Writing Analytics as a sub-stream. Reflective writing is quite different from the more widely used forms of academic writing (literature review, persuasive essay, research paper; etc.), but is growing in importance as we seek to place learners in authentic learning contexts (e.g. internships; work placements; or simulated teams), specifically so that they experience something of the complexity of real workplaces.
Honest reflection can make the writer vulnerable, as they reflect on their uncertainties, failings, and how they are changing as a learner/professional. They are often deeply personal, connecting to different threads across the many areas of their life. That’s almost the opposite of the other genres of writing that dominate students’ and professionals’ lives, which emphasise rational distance, mastery of the material, and confident rhetoric. Deep reflection can even share transformational moments in someone’s life and learning. Speaking personally, I find some student reflections profound, and even moving — an inspirational reminder of why we do what we do. Yet reflecting is not something that people are always (or even often) given the opportunity to learn how to do well.
As noted in a recent paper:
“Helping people make sense of their thoughts, feelings, reactions and approaches when stretched out of their comfort zones is core business for educators and coaches. Suitably supported, honest reflection makes it safe to question assumptions and consider change, but we also know that this is often difficult to teach, and challenging to learn.
[…] Reflective Writing is a strategy used in education and many professions to help learners, professionals and leaders make sense of challenging experiences, and prepare for the future. It integrates “head and heart”: valuing not only technical/academic knowledge, but how this interplays with experiential/professional ways of knowing, and recognising the fact that learning and working engage our emotions and feelings.
[…] However, while we know there is nothing as valuable as detailed coaching feedback to build this capacity, this is a scarce, costly skillset and labor-intensive. The practical consequence is that most students and leaders do not understand how to reflect deeply, and do not receive good feedback.” (Buckingham Shum & Lucas, 2020)
The desire to provide people with useful and timely feedback on reflection on a widespread basis has led to interest in developing Reflective Writing Analytics — broadly, the use of natural language processing and automated feedback methods to understand and support the process of reflection. This is a nascent area. There are not many people (that we know of) working on the challenge of providing automated analysis of, and feedback on, reflective writing, so it was a delight to convene a post-LAK20 call last week with teams from the USA, UK and AUS, when we spent 2 hours comparing notes on what we’re wrestling with, and how we might collaborate to move the field forward.
Examples of current work are below to help you get up to speed with this emerging field, the different emphases within it, and the scholarly communities who participate. There’s a significant existing body of work outside the field of analytics on the nature of reflection, and how to teach reflective writing (reviewed in the papers). The intriguing challenge is to translate that, with integrity, into the world of text analytics and automated feedback. As we noted during our call, it’s a really exciting nexus of the cognitive, social, affective, pedagogical, ethical, user experience and technical.
Do get in touch if you want to join forces — everyone is most welcome, and we’re sure there must be more people out there doing this we haven’t met!
Thanks to the kickoff videoconference participants for co-authoring this blog: Alyssa Wise (NYU), Andrew Gibson (QUT), Huda Alrashidi (Warwick), Ming Liu (UTS), Qiujie Li (NYU), Sameen Reza (NYU), Thomas Ullmann (OU), Yeonji Jung (NYU)
New York University (Lead: Alyssa Wise)
Cui, Y., Wise, A. F., & Allen, K. L. (2019). Developing Reflection Analytics for Health Professions Education: A Multi-dimensional Framework to Align Critical Concepts with Data Features. Computers in Human Behavior, 100, 305-324. https://doi.org/10.1016/j.chb.2019.02.019
Jung, Y. and Wise, A.F. (2020). How and How Well Do Students Reflect?: Multi-Dimensional Reflection Assessment in Health Professions Education. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK’20). ACM, New York, NY, USA, pp.595-604. https://doi.org/10.1145/3375462.3375528 [Preprint]
Wise, A. F., & Cui, Y. (2019,). Top Concept Networks of Professional Education Reflections. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK’19). ACM, New York, NY, USA pp. 260-264. https://dl.acm.org/doi/pdf/10.1145/3303772.3303840
Wise, A.F., Reza, S. & Han, R. J. (2020). Becoming a Dentist: Tracing Professional Identity Development through Mixed-Methods Data Mining of Student Reflections. Proceedings of ICLS’20: International Conference of the Learning Sciences. Nashville, TN: ISLS. [Preprint]
Queensland University of Technology (Lead: Andrew Gibson)
Work in progress with RWA and GoingOK: http://goingok.org
Willis, J., and Gibson, A (2020). The Emotional Work of Being an Assessor: A Reflective Writing Analytics Inquiry into Digital Self-assessment. In Fox, J., Alexander, C., Aspland,T.(Eds.) Teacher Education in Globalised Times. (Sringer). https://doi.org/10.1007/978-981-15-4124-7 [pre-order]
Gibson, Andrew P. (2017) Reflective writing analytics and transepistemic abduction. PhD Thesis, Queensland University of Technology. https://doi.org/10.5204/thesis.eprints.106952 [Preprint]
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing AnalyticsFor ActionableFeedback. Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver, BC, Canada. (ACM Press), pp.153-162. http://dx.doi.org/10.1145/3027385.3027436 [Preprint] [Replay]
The Open University & Warwick University (Lead: Thomas Ullmann)
Ullmann, T. D. (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257. https://doi.org/10.1007/s40593-019-00174-2 [Preprint]
Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing Automatically Detected Reflective Texts with Human Judgements. 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning. CEUR-WS.org. http://ceur-ws.org/Vol-931/paper8.pdf
Try ReflectR – an online tool to classify sentences regarding reflection: http://qone.eu/reflectr
Alrashidi, Huda; Ullmann, Thomas; Ghounaim, Samiah and Joy, Mike (2020). A Framework For Assessing Reflective Writing Produced Within the Context of Computer Science Education. In: Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20, 24/03/2020, Frankfurt, Germany). [Preprint]
University of Technology Sydney (Lead: Simon Buckingham Shum)
AcaWriter automated feedback tool: AcaWriter orientation for staff and students • Video intro to the reflective module and embedding in Pharmacy Masters program
A blog post on the challenge of sharing reflective writing datasets
Buckingham Shum, S., Á. Sándor, R. Goldsmith, R. Bass and M. McWilliams (2017). Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results. Journal of Learning Analytics, 4, (1), 58–84. https://doi.org/10.18608/jla.2017.41.5 (Open Access)
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C. and Knight, S. (2017). Reflective Writing AnalyticsFor ActionableFeedback. Proceedings of LAK17: 7th International Conference on Learning Analytics & Knowledge, March 13-17, 2017, Vancouver, BC, Canada. (ACM Press), pp.153-162. http://dx.doi.org/10.1145/3027385.3027436 [Preprint] [Replay]
Liu, M., Buckingham Shum, S., Mantzourani, E.,Lucas, C. (2019). Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. In: Isotani S., Millán E., Ogan A., Hastings P., McLaren B., Luckin R. (eds) Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science, vol 11625. Springer, Cham. https://doi.org/10.1007/978-3-030-23204-7_19 [Preprint]
Buckingham Shum, S. and Lucas, C. (2020). Learning to Reflect on Challenging Experiences: An AI Mirroring Approach. Proceedings of ACM CHI 2020 Workshop on Detection and Design for Cognitive Biases in People and Computing Systems, April 25, 2020 (online). [Preprint]
Tackling cognitive bias with CHI tools
CHI is the premier conference on Computer-Human Interaction, though this year’s isn’t meeting due to COVID-19 (but see the Proceedings if you’re not familiar with its amazing breadth and depth). However, some of the workshops are going ahead online, one of which is this new one on Detection and Design for Cognitive Biases in People & Computing Systems:
“With social computing systems and algorithms having been shown to give rise to unintended consequences, one of the suspected success criteria is their ability to integrate and utilize people’s inherent cognitive biases. Biases can be present in users, systems and their contents. With HCI being at the forefront of designing and developing user-facing computing systems, we bear special responsibility for increasing awareness of potential issues and working on solutions to mitigate problems arising from both intentional and unintentional effects of cognitive biases.
This workshop brings together designers, developers, and thinkers across disciplines to re-define computing systems by focusing on inherent biases in people and systems and work towards a research agenda to mitigate their effects. By focusing on cognitive biases from a content or system as well as from a human perspective, this workshop will sketch out blueprints for systems that contribute to advancing technology and media literacy, building critical thinking skills, and depolarization by design.”
The papers are all open access, so jump in and see what an interesting range of contributions this event attracted. The workshop was an eclectic mix of expertises and interests, and also used a Miro board in a fun way to support activities with sticky note exercises.
I wrote a position paper with Cherie Lucas, contextualising our work on the automated detection of written reflection. Our vision is that such tools could make citizens more self-aware of their biases, making them less reactive, and more open to new perspectives when their assumptions are challenged.
Buckingham Shum, S. and Lucas, C. (2020). Learning to Reflect on Challenging Experiences: An AI Mirroring Approach. Proceedings of the CHI 2020 Workshop on Detection and Design for Cognitive Biases in People and Computing Systems, April 25, 2020. [slides]
Abstract. As citizens are confronted by major societal changes, they find their assumptions being challenged and their identities threatened, bringing the risk that they retreat to like-minded ‘bubbles’ rather than ask whether they might have something to learn. Algorithmically driven media platforms exacerbate this process by amplifying cognitive biases and polarizing debate. This paper argues for a distinctive role that Artificial Intelligence (AI) can play, by holding up a metaphorical ‘mirror’ to online writers, with carefully designed feedback making them more aware of, and reflective about, their reactions and approaches to challenging situations. As an example, we describe a web application that uses Natural Language Processing to annotate written accounts of personal responses to challenging experiences, highlighting where the author appears to be reflecting shallowly or deeply. This open source tool is already in use by students to help them make sense of work placement challenges they encounter, but could find wider application. Our vision is that such tools could make citizens more self-aware of their biases, making them less reactive, and more open to new perspectives when their assumptions are challenged.
Evaluating ML for Pharmacy Student Reflection
Abstract. Reflective writing is widely acknowledged to be one of the most effective learning activities for promoting students’ self-reflection and critical thinking. However, manually assessing and giving feedback on reflective writing is time consuming, and known to be challenging for educators. There is little work investigating the potential of automated analysis of reflective writing, and even less on machine learning approaches which offer potential advantages over rule-based approaches. This study reports progress in developing a machine learning approach for the binary classification of pharmacy students’ reflective statements about their work placements. Four common statistical classifiers were trained on a corpus of 301 statements, using emotional, cognitive and linguistic features from the Linguistic Inquiry and Word Count (LIWC) analysis, in combination with affective and rhetorical features from the Academic Writing Analytics (AWA) platform. The results showed that the Random-forest algorithm performed well (F-score=0.799) and that AWA features, such as emotional and reflective rhetorical moves, improved performance.
WReD alert! Towards a Written Reflection Dataset?
Ming Liu and I just posted on our writing analytics blog some thoughts about a new initiative — the tricky challenge of building a Written Reflection Dataset (WReD) to advance the field.
Please take a look and pitch in…