Future Schools – Simon.BuckinghamShum.net

Evidence: Fair & Robust AI-based Assessment

The All Party Parliamentary Group on AI is a multi-year initiative to help anticipate the widespread impacts that AI could have on society. They convene Evidence Sessions on different themes, in which Lords and MPs have the opportunity to hear from, and question, diverse experts.

As noted in the introduction to one of their recent reports,

“The evidence APPG AI has been gathering since 2017 shows education at the heart of both the opportunities and the risks in the narratives forming around AI.”

“[…] In 2019, APPG AI launched the Education Pillar to tackle some of these multi-faceted questions over the next two years. We will focus on:

how AI can be used as a tool to improve learning,

what skills we need to prioritise as a society,

how school curriculums need to transform,

and what the role of ethics in education should be.”

Reports (access requires a free signup) on the education theme to date have collated evidence on:

The latest meeting focused on Designing Fair & Robust AI-based Assessment Systems. This brought together a very interesting set of people, to which I was honoured to be invited.

The meeting switched from the House of Lords to Zoom (so a distinct loss of oak panelling and leather upholstery there!) but it meant many others could tune in live.

The guiding questions they set were:

What are the benefits and challenges of different types of AI-based assessment systems in education?
How can it be guaranteed that they will deliver reliable and fair results?
How might AI-based assessment systems change the teacher-student relationship?
How will these technologies affect students’ motivation and trust in a fair evaluation of their performance?
How to prepare students, teachers, and parents before implementing AI-based assessment technologies in education?
How do AI and human understandings of assessment differ?

With just 5 minutes/speaker, it was an interesting challenge to decide what and how to present. Here’s the video and the underpinning written statement which includes the sources I mention (with thanks to several colleagues for their input).

To see all the contributions, here’s the full meeting replay and final Parliamentary Brief.

Learning Analytics & AI go to School

Here’s the replay of my keynote to the UTS STEM EdFutures Data in Schools Symposium [#UTSDataInSchools].

A non-technical introduction for school leaders and teachers to the promise and pitfalls around the explosion in data, including different roles for analytics and AI, the role of in-house data scientists, and how we build the bridge between educational constructs and log data [slides].

When children learn to map dialogue and argument

In The rationale of a 5 year old, Paul Culmsee documents coaching his 11 year old daughter in issue and dialogue mapping with our Compendium tool.

This is dead funny, and dead serious! So, in the Christmas gift spirit, here is my story.

Saturday morning and I find my daughter (Year 5 Middle School = 9 years old) doing her Literacy homework:

We have been looking at writing persuasive arguments this week. Look at the following scenario.

Your parents have decided that you watch too much TV. They are going to limit you to just half an hour a day.

In the speech bubbles below, can you list arguments for and against this idea.

She goes to work on it, and completes the template.

Note that two of the arguments Against are in fact not objections to the fascist TV regime being tabled, but attacks on her own For arguments. We discuss this, and she gets what I’m talking about, more or less. The problem of course is that important conceptual connections such as these are invisible in a two-column Pro/Con table.

Then I introduce her to Compendium, starting with a little demo map to see if she understands the correspondence between the simple prose in the highlighted Note, and the map underneath:

No problem. So then I translate the literacy homework into a Debate Mapping template:

I switch to a top-down layout which she seems to prefer over the horizontal. The arrows flow more naturally to the top I believe, than from right-to-left which is what most dialogue mappers do. After verifying that she could drag+drop nodes off the palette, and transcribe her first argument into a node and link it, I went off to the garden and left on her own to complete the map, with 2 visits to check on progress. She seemed to get it without much trouble.

And this is what she created, which made her dad very proud 🙂

And the moral of these little anecdotes?

Time to redefine the Net Generation 😉

TIRED: Young people who amaze certain grown-ups by restarting their browser, and doing their homework plugged into twitter and an mp3 while remixing movies.

WIRED: Young people who understand that ideas are networks, and who can read and write coherent networks, thinking deeply and reflectively, communicating (even disagreeing) with others in an appropriate way!

More systematic evidence around the ability of young people to learn to deliberate and argue critically by making that thinking visible is reported in Evidence-Based Dialogue Mapping for Teenagers, Researching Argumentation in Educational Contexts, Knowledge Cartography, Arguing to Learn, and by the growing network of educators (hey, and parents!) piloting these tools as and when they can.

A Merry Christmas and Peaceful New Year to you!

Working with a top performing school

I’m chair of governors (= school board in US) at Bushfield primary school in Wolverton. This has been one of the most interesting things I’ve undertaken in recent years, very much in tune with work at OpenU since the debates about the future of learning in school and university are pretty much the same. When you read the complaints about the current systems, and scenarios of future, you can more or less swap the nouns school/university and it still all makes sense… but the earlier we start to fix the system, the more profound the impact!

So, at Bushfield we just got a letter from the Minister of State for Schools & Learners, congratulating us on being one of the top performing schools in England. Your critical eye will, of course, be asking so what does top performing actually mean? The letter is quite precise:

“Your results show that you are amongst the top performing schools based on the most sustained year-on-year improvements in Level 4+ aggregate performance from 2005 to 2008.”

Hmm, sounds good. That sounded like they were congratulating us on how much progress children make, which in the glorious world of school league tables is called Contextual Added Value (CVA), a “performance indicator” that was introduced as a more rounded way to assess the quality of education than raw attainment levels in SATS, which take no account of the level at which children enter, your socio-economic context, how many special needs pupils you take, etc.

Well on closer inspection it turns out that no, this was all about our improvements in raw attainment levels. While this has some meaning, and we’re proud of the result, we feel CVA is much more meaningful as an indicator of our philosophy to learning than an absolute league table ranking. And we’re even prouder that we’re amongst the top schools in the city and nationally on this front. No matter where you are in formal academic terms, at Bushfield School you’ll make better than average progress from when you entered to when you leave, than you would in almost any other comparable school. You might score higher if you went to a selective, private institution right at the top of the league tables, but that’s only accessible to a few. For everyone else, this is as good as it gets in the public system. Something for pupils and staff to be very proud of.

However…

Here we are in the 21st Century, gauging learning, and very publicly ranking schools, via a few very selective measures (exam results — scored this very week in May — on literacy, numeracy and science, though science SAT is abolished from next year as an inappropriate way to assess the subject. Hmm…). With the introduction of Contextual Value Added, the indices have grown a little sharper from the initial very blunt instrument of absolute scores, but there is so much more to learning that is left invisible by these categories. To paraphrase Geof Bowker, organisational memory systems are systematic ways of forgetting, since the categorisation systems they employ choose what to erase.

And this is why I love working with the Bushfield staff team, because they’re up for embracing the new conceptions of learning around the skills and habits of mind that the next generation needs to survive, and thrive, in a very fast changing world. In the end, I wrote a summary of my thoughts about our approach to learning, and after some improvements by the Headteacher and other governors, it feels satisfying to paint a much broader, richer picture of how we see learning.

Evidence-Based Dialogue Mapping for Teenagers

KMi’s Compendium tool supports rapid visual mapping of dialogues and debates through a mix of visual language, tagging and hypermedia structuring. With a track record of use in the workplace, we are now beginning to build evidence of its potential for critical thinking in schools.

With >40,000 downloads and an annual workshop, KMi supports an active Compendium user community (and a growing developer group). It has found application primarily, although not exclusively, in the workplace, the tool of choice for many information analysts and facilitators of collective sensemaking. But could we demonstrate that younger users would embrace its visual language and the extra rigour of thinking that argument mapping requires?

In an exciting workshop led by Ale Okada, a Research Fellow in the Hypermedia Discourse research programme, teenagers judged by their schools to be “gifted and talented” attended a science summer school, and were taught how to map and structure their reasoning about the impact of climate change on the UK. The results are summarised in a new article, and represent the first step in our programme to introduce the next generation to visual, network-centric thinking, a literacy that we think will become increasingly important.

Okada, A. and Buckingham Shum, S. (2008). Evidence-Based Dialogue Maps as a Research Tool to Investigate the Quality of School Pupils’ Scientific Argumentation, International Journal of Research and Method in Education, 31(3), pp. 291–315 (Special Issue: Coffin, C. and O’Halloran, K.A, (Eds.) Researching Argumentation in Educational Contexts: New Methods, New Directions). Article PrePrint: http://oro.open.ac.uk/11773