Simon.BuckinghamShum.net

“Classification systems provide both a warrant and a tool for forgetting […] what to forget and how to forget it […] The argument comes down to asking not only what gets coded in but what gets coded out of a given scheme.”

Bowker, G. C. and Star, L. S. (1999). Sorting Things Out: Classification and Its Consequences. MIT Press, Cambridge, MA, pp. 277, 278, 281

Maps are not neutral

Sometimes mapping is held up as a way to avoid formal classification schemes. I think in some people’s minds, it’s something to do with the greater expressiveness that spatial representations have compared to the boring tables or taxonomising trees that they associate with classification schemes. “Mapping the terrain” invokes notions of simply painting what you see without judgement.

That may work to some degree for mapping spatial landscapes, but of course, a map is just another way to systematically distort reality for a particular purpose, and very useful it is too when done well, foregrounding information needed for a particular use-context, and hiding irrelevant details. When it comes to visualizing abstract information onto two or more dimensions, then clearly, many more decisions have been taken by the cartographer, and those who wrote the software. No lens is clean.

Data does not ‘speak for itself’

You are only looking at data because an unknown cloud of people have made a whole load of decisions that have finally delivered that data to your screen. Data requires humans to form it, and humans to interpret it.

danah boyd and Kate Crawford have helpfully unpacked six ‘provocations’ for big data:

boyd, d. and Crawford K. (2011) Six Provocations for Big Data. A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society. Oxford Internet Institute: Oxford, UK.

So when we talk about learning analytics, we should be walking in eyes wide open, bringing all of the above into the conversation. Any notion of analytics that does not do this is at best naive, and at worst unethical. If we take Doug Clow’s diagram of a basic analytics cycle (slide 5), my annotations hopefully make clear that many human judgements must be made at each stage:

Governing algorithms

We see from the huge literature on ethics in ICT, the growing concern about governing algorithms, and emerging forums such as code acts in education, that the algorithms weaving through our lives must be critically deconstructed.

In one scenario (slide 77) I envisaged, I imagined the burden of knowledge that predictive analytics may bring in the future — the system didn’t actually make the decision for you, but did the information make it a harder decision?

However, Naomi Jeffery passed me a chilling example recently showing how  an interview-shortlisting algorithm was carefully validated against human judgements — the only problem was that it faithfully embodied the institutional racism which had crept into faculty decisioning (BMJ pdf article).

Theory considered redundant

The infectious meme behind Chris Andersen’s End of Theory Wired piece was that Google has now rendered ‘theory’ dead, through the power of behavioural statistics and predictive algorithms. Mike Caulfield helpfully critiques this in terms of its limitations for designing learning interventions, arguing that you have to understand why something has happened in order to make an appropriate intervention. Maybe, maybe not. Consider the following — I think we are beginning to see something of Andersen’s vision coming true, and it’s not intrinsically bad.

Theory-free predictive modelling. Some predictive modelling in learning analytics has already entered what I would say is a theory-free zone. John Campbell’s PhD (underpinning Signals) showed that it is indeed possible to predict student failure without a theory of learning, purely by selecting the right variables from student demographics, grade history and usage of the online platform. NO THEORY here. But I’d welcome the Signals team’s comments on the implicit theory, pedagogy or assessment that lies behind the tips given to students. Again, it may be simply codifying educators’ craft knowledge and practice that drives these (not to be devalued in any way) — but not “theory” in any academic sense.

Theory-free student recommendations. Or to take another example, when MOOC students fail but then subsequently pass a test, in principle, machine learning techniques can track those behaviours which are correlated with subsequent success, so that automated interventions/recommendations can be made, again, purely on the basis of behavioural statistics, not a theory of learning. I don’t think that’s happening as a real time service yet anywhere, more offline lab analysis (seeking a pointer here – I recall Daphne Koller talking about this).

Theory-free social network analysis. I recently generated my social graph from LinkedIn, which I found to be an accurate and aesthetically pleasing mirror of my different professional worlds. If this was a visual analytic reflected back to me mid-way through a course, that might well give me pause for thought. What this does not do is give me any clue as to what to do about improving my social capital. If I had a view which wasn’t ego-centric, placing me in the middle, but showed me out on the edge of a cohort, that might well lead me to reflect differently. We’ll return to this shortly.

Do these examples fill you with excitement, and/or make you feel uncomfortable? Do these exemplify the “death of theory”?

They show the value and power of data-driven, algorithmic techniques, which leverage the power of finding patterns in large numbers of users and their activity. They are in no sense neutral, foregrounding certain values and assumptions, but they do not have an explicit ‘theory of learning’. Statisticians are very strong on understanding the assumptions that legitimise the use of a given approach: this is a great ‘middle space‘ area to explore — do the statistical approaches behind analytics make assumptions about learners’ similarity or continuity of behaviour that learning scientists support or challenge?

Next, we might ask: if the above impressive examples are possible with no formal educational theory, how much better can we do once we do bring it in?

Theory-based analytics?

So far, I’ve only talked about how we can’t escape values and assumptions in the socio-technical infrastructure that delivers analytics. What’s the relationship with theory?

Specifically in education, what about epistemology, pedagogy, and assessment regimes? Furious debates rage within education about how one may assess learning with integrity, at scale, quite apart from any digital technology. So in one paper we unpacked how we think these relate to learning analytics technologies. To jump to the end, we concluded:

“We have gone beyond “our learning analytics are our pedagogy” [6], arguing that they embody epistemological assumptions, and they perpetuate assessment regimes. Moreover, as with any tool, it is not only the design of the tool, but the way in which it is wielded in context, that defines its value.”

If we return to the SNA example, whether or not changing my behaviour makes sense for my learning is where SNA  engages with epistemology, pedagogy and assessment regimes. Or in professional workplace learning, less tied to such formal academic concerns, notions of social capital, networked learning or communities of practice is where ‘theory’ may need to enter the conversation. The affordances of visualizations is therefore a key area for critique. The implicit message of a social network view showing me on the edge is that I should try to score higher on certain SNA metrics: I should try to be more densely connected; I should try to be a broker between two sub-communities I care about; I should forge ties with other key people. But what if creative innovators are the ones who resolutely stay on the edge? The ones who do not use the same tags as others? The ones who share ideas between strangely different communities? What if the stats show that non-social learners score at least as well in final assessments on this course?

At the LAK11 conference in Banff, I spoke on the role of theory in analytics, identifying a range of roles it could play — click through to see: