Ethical Design Critique for Learning Analytics Dashboards
The concept of an Ethical Design Critique (EDC) was developed initially to address the pressing need for a UTS process to sign off an analytics dashboard as compliant with formal ethical and other risk factors. EDC was developed by the UTS Connected Intelligence Centre (CIC: a learning analytics innovation centre), to help ensure that discussions about the ethics of data and analytics moved from abstract principles, to being clearly relevant to specific system design and deployment decisions.
An EDC workshop is an intensive 2-3 hour event bringing together key stakeholders to review the proposed user interface to an analytics dashboard. Ideally, this could start with an upstream prototype design ensuring that there is scope for substantial revisions should they be needed, since changes may have profound implications for which data sources are used, how they are aggregated, and the user interface design. However, there is still value further downstream to run an EDC as a pre-deployment review as the basis for formal sign-off.
The participant list gives a sense of how this resembles a “360 degree review” from diverse perspectives:
- CIC: learning analytics, visualisation and usability researchers
- Pro-Vice Chancellor Education: interested in this as a business process
- IT Division & Business Intelligence lead: the dashboard’s developers concerned to get approval for deployment
- Equity & Diversity Unit: to ensure student diversity was suitably visible
- Planning & Quality Unit: data governance
- Director and First Year Experience Coordinator, Institute for Interactive Media in Learning: first year student analytics and knowledge of academics’ practices
- Director Risk Management: verifying that the university was not exposed to any risks by the data displayed
- Director Data Privacy: ensuring compliance with formal regulations
The EDC process is summarised below:
A shared Google Doc template was developed to capture and structure participant input, and participants self-organised into tables focusing on the different dashboard screens. Participants are asked to comment on strengths of the design, but their job is primarily to identify potential problems. Two slides from the workshop are shown below, illustrating the kinds of considerations that participants are invited to consider.
Participants are not there to comment on all aspects of the design (e.g. it is hoped that the user interface designers have already done a reasonable job), but to focus on ethical dimensions, however they construe those from their professional perspective. It is recognised, however, that UI design cannot be divorced from the process, since a misleading visualisation or confusing terminology might lead to misinterpretation.
One example of an issue that surfaced was that it transpired that demographic and grade filters on the dashboard could make it possible to identify a specific student if the sample size was too small, violating the intention that this was a cohort-level analysis tool. Other examples of EDC input from a workshop are shown below (details redacted).
Required changes are agreed, and then made and circulated for comment. Depending on the nature of the critique, this might be a quick edit or a more serious change in what data is sourced, or how it is analysed or rendered. Depending on what stage of the process one is at, and available resources, further user testing might then follow to verify that the changes have not impacted other usability issues.
Prior to deployment, a summary and recommendation are prepared by the EDC Chair (in this case, CIC Director), and submitted to the CIO for approval.
The EDC is at an early stage of development, but all participants in the first workshop agreed that the format was focused enough to achieve the desired goal of formally reviewing a design from the necessary perspectives, in order to deploy the tool with a higher degree of confidence that it was not inadvertently violating ethical principles, or specific legal requirements around confidentiality and privacy.
Leave a Reply