AIF 2.0: human-centred thoughts

The Argument Interchange Format (AIF) is a community effort to agree a formal lingua franca for computer-supported  argumentation. The first version is presented in Knowledge Engineering Review.

I spent 2 days last week in Glenshee, at an EPSRC-sponsored workshop bringing together a diverse group of researchers working on computing and argumentation (conference/journal). Working from 9am-8pm for 2 days solid (!), our mission was to write a report drafting AIF 2.0. This was all “thanks” to Chris Reed who runs the ARG at Dundee U, specifically his Dialectical Argumentation Machines project, who did a great job as convenor and taskmaster 😉

One of the novel aspects of this gathering is that in contrast to AIF 1.0, which was driven by the needs of computer science and AI perspectives on argumentation, they also invited “less formal” people like Tim van Gelder, David Price and me, whose tools are semiformal at most, but are (correspondingly) achieving more rapid uptake, since we do not have to worry so much as our AI colleagues about the hygiene of our representations for formal processing (the price we pay is that it is much harder to automatically evaluate our dialogue/argument maps — we leave that to human analysts scaffolded by the visual language).

As a contribution to the forthcoming report, I wrote a use case (PDF) which sews together a number of the dialogue and argument mapping tools represented at the workshop (Compendium, Cohere, Debategraph, Rationale, Carneades), which centres on an imaginary team of analysts working on the legitimacy of the 2003 Iraq invasion.

Below I’ve also pasted my position statement, and an introductory piece on approaching the design of something like AIF through a human-centred design lens.

Musings (Position Statement) for AIF 2.0 workshop :-)

(this is turning in to a bit of a live blog where I’m recording thoughts)

My work is at the intersection of Human-Computer Interaction, Design Rationale, Hypermedia, Sensemaking and Learning. I’m interested in how we foster argumentation skills from children in school (Okada and Buckingham Shum, 2008), onwards.

Working closely with US colleagues, I lead the Hypermedia Discourse project at the Open University’s Knowledge Media Institute, which has developed two open source mapping/modelling tools, Compendium and Cohere, both of which support the Issue-Based Information System (IBIS) scheme, a simple notation compared to AIF. IBIS and our tools are designed not, in the first instance, for argument modellers, but for analysts and facilitators interested in augmenting human cognition for inquiry into wicked problems through dialogue and debate. They are open-ended visual modelling tools, so can be used in tandem for many other kinds of modelling, and software tool.

Significant work has gone into studying the literacies and practices that emerge around Compendium’s usage, which helps us move it from being a raw technology to a genuine tool with a community of practice sharing experiences about its deployment in authentic contexts, such as Dialogue Mapping, Conversational Modelling, and Knowledge Art (Selvin, Buckingham Shum & Aakhus, In Press).

Technically, Compendium is designed to be as open and extensible as possible to integrate with other tools and software agents (Buckingham Shum, et al 2006; Sierhuis and Buckingham Shum, 2008; Buckingham Shum, et al, submitted). As a web application, Cohere supports many of the common kinds of Web 2.0 interoperabilities we now expect (Buckingham Shum, 2008). Recent work Dialogue Mapping the UK Election TV debates illustrates Compendium (video annotation) and Cohere.

The IBIS constructs of Arguments supporting/challenging Positions, which respond to Issues employs very basic argumentative moves seen in many other tools. As far as I know, IBIS inventor Horst Rittel did not know about any other argumentation schemes: these constructs just make intuitive sense.

We have built on Reed and Walton’s work, to convert the critical questions associated with argumentation schemes, from Araucaria AML formatted XML into visual argumentation scheme templates in IBIS (Buckingham Shum and Okada, 2008). We use the argumentation scheme templates to “explode” and interrogate the implicit substructure sitting behind those simple links. This illustrates the use of IBIS as an intuitive language in which to elicit or express more complex notational schemes.

IBIS has become something of an interlingua for researchers interested in more structured online deliberation platforms (Debategraph; MIT‘s Deliberatorium; overview; demos podcast). Ongoing work with these other IBIS platforms seeks a common format (IBIS, 2010), for precisely the same reasons that AIF was conceived: the IBIS-interoperability challenge is a microcosm of the AIF challenge, and we haven’t even cracked IBIS (!) due to the different flavours and platform-specific nuances. It would make sense for this IBIS effort to align, or be somehow interoperable, with, or expressable in AIF. But the devil is in the detail. If we could express IBIS in AIF, would we use AIF? We are musing at present on whether IBIS is in fact in a different space from AIF, a simple abstraction of AIF, or a subset of AIF. One might use AIF to unpack the implicit nature of the “supports” or “challenges” link in an IBIS structure, as described above.

So, a thorny question to lob in is whether pair-wise mappings between systems is the pragmatic approach, rather than an all-encompassing, context-free ontology. These two orientations are reflected somewhat in the early “Web 2.0 vs Semantic Web” tension: a carefully engineered ontology is powerful, but also brittle if the world being modelled unexpectedly violates assumptions (implicit or explicit). When I want to import your arguments into my system, for a particular purpose, with a given user community, on a specific timescale, in order to provide the following user experience, I might be tempted to just write the convertor tuned to those constraints rather than wrestle with a hugely expressive, but also possibly insufficiently expressive, ontology written by someone else. It’s the old code (non) re-use thing again.

I am not a knowledge representation expert, so see my contribution as being not so much at the detailed level of getting AIF notation right, but in thinking about how it might be visualized, and how usable it would be by different stakeholders who would benefit from it. Clarifying exactly who our user communities are will be important, if we are interested in seeing widespread adoption, rather than “only” defining a unifying conceptual framework (which is of course valuable in its own right for academic purposes — arguably a necessity — but I suspect we have pragmatic ambitions as well in terms of seeing it add value to practical systems).

A closing question I am musing on is whether AIF should somehow be designed to add value to representations of the Social Web. Yes, we want to inject rationality into debates, and raise the level of discourse wherever possible. Argument analysis can also be done in a completely depersonalised manner, by a third party removed from the protagonists (although any pretence to complete objectivity in the normative reconstruction of moves is of course a fiction). But if we’re developing argumentation platforms as a medium in which to engage in discourse, then argumentation is also fundamentally an interpersonal activity. There is now an active research field and industry designing analytics that show online participants’ activity levels, social connectedness, reputation, and so forth. Argumentation platforms (and by extension AIF) should help us define a semantically grounded discourse layer on top of these quantitative and graph-based indices, so that we can profile participants based on the ways in which they engage: can we imagine “challengers” (lots of challenging kinds of moves and exposing of implicit premises), “metaphorical thinkers” (lots of arguments by analogy), etc…? We have begun to ponder this in our group by examining existing social visualization tools, and asking how they would be tuned to argumentation platforms.

References

Buckingham Shum, S. (2008). Cohere: Towards Web 2.0 Argumentation. 2nd International Conference on Computational Models of Argument, 28-30 May 2008, Toulouse. IOS Press: Amsterdam. http://oro.open.ac.uk/10421/

Buckingham Shum, Simon; Selvin, A.M.; Sierhuis, Maarten; Conklin, Jeffrey; Haley, C.B. and Nuseibeh, Bashar (2006). Hypermedia support for argumentation-based rationale: 15 years on from gIBIS and QOC. In: Dutoit, A.; McCall, R.; Mistrik, I. and Paech, B. eds. Rationale Management in Software Engineering. Berlin: Springer-Verlag, pp. 111–132. http://oro.open.ac.uk/3032/

Buckingham Shum, S., Sierhuis, M., Park, J. and Brown, M. (submitted). Software Agents in Support of Human Argument Mapping.

Buckingham Shum, S. and Okada, A. (2008). Knowledge Cartography for Controversies. In: Knowledge Cartography, (Eds.) Okada, A., Buckingham Shum, S. and Sherborne, T. Springer: London. http://oro.open.ac.uk/11768/

IBIS Interchange Specification: http://projects.kmi.open.ac.uk/hyperdiscourse/docs/IBIS-0.1.pdf

Okada, A. and Buckingham Shum, S. (2008). Evidence-Based Dialogue Maps as a research tool to evaluate the quality of school pupils’ scientific argumentation. International Journal of Research and Method in Education, 31(3), pp. 291–315. http://oro.open.ac.uk/11773/

Selvin A. (1999) Supporting Collaborative Analysis and Design with Hypertext Functionality. Journal of Digital Information, 1 (4): http://journals.tdl.org/jodi/article/view/jodi-17/15

Selvin, A., Buckingham Shum, S.J. & Aakhus, M. (2010). The Practice Level in Participatory Design Rationale: Studying Practitioner Moves and Choices. Human Technology Journal. Preprint: http://oro.open.ac.uk/20948/

Sierhuis, M. and Buckingham Shum, S. (2008). Human-Agent Knowledge Cartography for e-Science. In Knowledge Cartography. (Eds.) Okada, A., Buckingham Shum, S. and Sherborne, T. Springer: London. http://oro.open.ac.uk/20301/

User-centred design perspectives

A user-centred perspective to the design of a formalism such as AIF focuses attention on questions such as:

  • What is AIF’s value proposition?
  • Who are the stakeholders?
  • What are the implications of designing for a heterogenous mix of users and platforms?
  • How does AIF fit into the real world activities, tools and intellectual capabilities of the range of envisaged users who are either conducting argument analyses based on material, or engaging in argumentation with each other?

When we consider the socio-technical “ecosystem” of argumentation platforms and users, it is clear that it is heterogeneous: diverse kinds of users and platforms, with different kinds of capability. This has consequences, for instance:

  • The passing of data will not be automatic in all cases. Only “one-way” interchange will be possible from more expressive to less expressive platforms. Data from less expressive platforms will require human intervention to provide missing details before more expressive platforms can add value to the analysis.
  • There is also the possibility that AIF-interoperability between two systems will fail, not because of limitations on AIF’s part, but because the two systems are in principle incompatible. This might be because the recipient platform requires data that the source cannot provide, or because they have incompatible models of argumentation. The introduction of a mediating representation such as AIF will not help in such situations.
  • With “users” ranging from members of the public (e.g. sharing their views in a public debate), to students, to forum moderators, to software developers, to academic argumentation researchers, we are dealing with a vast range of computing expertise, familiarity with argumentation theory, and tasks. To what extent will this shape AIF?

The insights from sociologists of science, and in particular, of how software and data standards are specified (e.g. Bowker and Star, 1999), provide intruiging accounts of the diverse constraints that lead to a given model. These are often compromises: when a diverse community seeks to agree a standard, they are seeking to satisfy many intersecting and sometimes conflicting goals. Ensuring compatibility with one’s own research interests may be in tension with the needs of developers or end-users. The AIF 2.0 standard, like any other, will require not only technical elegance, but a sufficiently aligned mission, and spirit of collaboration that has taken us this far.

Value proposition (VP)

The design of a new “product” must provide a compelling value proposition:

  • What problem is solved?
  • How much does it cost? — in every sense of the word: e.g. money to employ suitably skilled staff; learning curve to use successfully; compatibility with existing technologies and methods…)
  • Who will pay?

What is AIF’s value proposition? To help us learn by analogy, we might challenge ourselves to complete this:

AIF is to < beneficiary community > as

<another model/standard> is to < success story beneficiary community >

We will consider possible answers to this as we consider the possible beneficiaries.

Academic audience

One audience for AIF is the community developing it: researchers in the field that has come to be known as Computational Modelling of Argument (COMMA) are developing it to solve commonly felt problems.

An obvious value proposition question is: Why has AIF 1.0 not taken off and established an active user community already? Consultation with AIF workshop participants identified the following factors:

  1. There was no usable reference implementation, only paper specifications, plus a few small prototypes
  2. It was not expressive enough for COMMA researchers
  3. There was no engagement with communities building argument mapping platforms, who do have real user communities, and data to share. Moreover, it was too complicated for non-COMMA researchers to grasp.
  4. A significant proportion of the COMMA community is working with abstract frameworks such as Dung’s, which has a very different model of argumentation from naturalistic argument, and which does not have significant databases of arguments: cross-platform data exchange was not a pressing problem for these researchers whose focus is on the formal properties of argumentation reasoning engines.

Factor 1 (reference implementation) we hope will be addressed as developers work to the draft specification. Factor 2 is addressed through a modular approach to adding in ontologies of importance to more formal COMMA researchers, and Factor 3 through the involvement in AIF 2.0’s design of argument mappers who have significant user communities (see the use case in the appendix). Factor 4 should begin to be addressed by AIF 2.0, which we hope will make it easier for theoretical researchers to obtain substantive datasets from researchers who have material on their platforms.

Continuing the focus on value proposition, it has been argued that apart from being an interchange format, AIF also provides a theoretical sounding board against which researchers can compare and contrast their approaches. Clearly, researchers already have established ways to do this: the argument is that AIF seeks to add a new kind of value by requiring integration at a much finer granularity and precision. For instance, we can now move beyond agreeing that two approaches are “broadly compatible” or “inspired by the same argumentation model”: AIF invites researchers to prove this very concretely, and to learn from the successes and failures of attempting to do so.

Developers

Software developers building systems to model and/or mediate argumentation, will welcome AIF if it saves them time, or leads to more elegant or efficient systems. They will be looking for clean code, modular architecture, solid APIs, etc. But they in turn will be paid (often) by solution providers who (should) bring to bear the end-user requirements.

Solution providers

From a Human-Computer Interaction/Design orientation, solving real world problems is the key driver.  The mission is to add value to people’s lives, including getting an existing job done more efficiently, getting new kinds of jobs done that weren’t possible before digital tools, and enhancing the aesthetic dimension to life.

Several argument mapping groups have spent a long time designing, trialling, evaluating (and pitching to funders) the benefits of informal/semiformal argument structuring tools. We are teaching pupils, students, researchers and other professionals to engage in the normative reconstruction of naturalistic spoken and written discourse, from which they then gain personal and group cognitive benefit. We face new literacy challenges when it comes to reading and writing dialogues and arguments in more structured ways. Who exactly will use the tools?

User-centred strategies/perspectives

Start with success stories: One strategy for deciding what should be in or out of AIF is to examine whatever we might consider to be our biggest success stories in computer-supported argumentation (broadly defined), and ask what should be learnt, if/how they could have been even better with AIF, or how AIF (as an encapsulation of the community’s collective intelligence) would help propagate key design advances/patterns. We have not yet had the chance to do this.

Design challenges: Another strategy (either in the absence of, or in addition to real success stories) is to compete to engineer systems solutions to challenges which the community agrees are meaningful (cf. Robot Olympics). Solution quality drives this, with a given theoretical/academic perspective forced to justify itself in specific contexts. Understanding those patterns is then extremely important to provide the frameworks preventing others from making the same mistakes.

Argument modelling challenges: Apart from engineering challenges, we might also have argument modelling challenges, such as agreed texts that need to be modelled either manually by analysts, using a given tool, or automatically by approaches using computational linguistics (here TREC is the obvious analogy to draw). An initial experiment along these lines was GlobalArgument.net (GA.net).

Dominant user interaction paradigms: Another approach starts by reviewing what are the most compelling modes of usage in argument mapping tools and online deliberation platforms, and deriving requirements from those that AIF must support. Examples would be:

  • Document annotation (typically Web) is found in Cohere and OVA, Araucaria supports offline annotation, and other Compendium and bCisive support linking to external files/URLs. AIF must support the notions of:
    • restating/paraphrasing/quoting, grounding nodes in the argument network in external sources, and better, in specific locations in those sources
    • distinguishing the authors of those argument nodes from the authors of original sources.
    • Task-specific argumentation: people always conduct argument analysis or engage in argumentation in a context, and domain-specific tools tend to do better (eg. specialising the notation to the language of the domain, integrating with other technologies in that domain’s ecosystem). What are the implications for AIF?

Use cases: Use cases seek to envisage who would use AIF-interoperable tools, and what benefit they would gain, in order to clarify not only the vision of the overall effort, but also identify any requirements on AIF. Examples include the following:

  • Exporting AIF from different platforms, and importing it into an AIF-compatible network visualization package. We assume that these are better than can be provided by passing a generic graph viz package generic graph data. (This might be smoothly delivered via web services etc, should this prove feasible in principle)
  • Exporting AIF enables more powerful statistical NLP than is possible on an unstructured corpus.
  • AIF aggregators, thus enabling searching across argument databases.
  • Use of AIF as structured markup to claim trustworthiness

Leave a Reply

You can use these XHTML tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>