Back to SIG Meetings
Click on topic (5 topics) to move to section:
Welcome and Notices.
e-Assessment: What We Need to Think About - Sharon Perry, CETIS.
e-Assessment: Issues and Experiences - John Kleeman, Questionmark.
Awarding Bodies and
Accessible e-Assessment and the Accessible e-Assessment Checklist Game - Simon Ball,
Work - Andy Heath, Axelrod Access For All.
The Joint Accessibility and Assessment SIG Meeting was held at the
University of Wales Bangor, on 27th June 2006.
The meeting was held in the lovely surroundings of the Oswalds' Conference Centre at the
University of Wales Bangor. Thanks to all the staff at Oswalds for a smooth and pleasant
CAA Conference 2006 - Myles Danson, Director of the annual Computer
Assisted Assessment Conference at the University of Loughborough, reminded participants that
this year's conference would be held on 4th and 5th July. This year will be the biggest to
date, with over 150 delegates indicative of a broad range of stakeholders. Proceedings for
this and previous years are available free online from
http://www.caaconference.co.uk, with all papers
being double blind peer reviewed.
Changes to CETIS - Paul Hollins, CETIS Business Manager, outlined some
forthcoming changes to CETIS. The existing CETIS project was successful in tendering to run
the new JISC Educational Technology and Interoperability Standards and Specifications service
for five years from 1 August 2006, with a review after three years. The new JISC-CETIS
service will be known as the Centre for Educational Technology and Interoperability
Standards, and will cover some of the pragmatic challenges of interoperability such as
retrospective interoperability and closed systems.
Most of the existing SIGs (special interest groups) will continue but their activities will
be reviewed, with more joint events reflecting joining up in standards work.
JISC will be investing a considerable amount in development work over the next few years and
there will be a major role in that for CETIS with closer integration between JISC development
activities and the SIGs, although the SIGs will retain their independence. For example, the
SIGs may be able to award small scale funding to develop ideas in specific areas, report
writing and initial preparations.
A new reader, Dai Griffiths, will be starting with CETIS in Bolton in August, and
recruitment is underway for a Deputy Director, expected to be in place by September. There
has also been some internal reorganisation within CETIS to aid communication and ensure that
community input continues to drive the SIGs.
Back to Top of Page
Sharon introduced some of the issues teachers and developers should be aware of when trying
to make (e)assessments accessible. These include:
- Technical: such as font size, background colour, effectiveness and usability of
assistive technologies and hardware, compliance with appropriate standards and
- Pedagogic: including learning outcomes, test objectives and test validity.
- Strategic: such as departmental or institutional policies, legislation,
interoperability and results handling.
- Practical: such as consultation with students, physical environment, time allowed and
Back to Top of Page
John gave participants an insight into the issues around making software accessible faced by
a major commercial developer. QuestionMark has been in existence for over eighteen years,
developing from a DOS based system to Windows and Macintosh versions and now web based
assessment, and the company has a strong commitment to quality e-assessment. John's
presentation led into a discussion session around the issues raised.
Accessibility for disabilities, rather than location or time, is not often high up
customers' lists of priorities. In commerce, it is often the survival of the fittest, and
those who provide what customers want will survive. However, it is rare that accessibility
features are requested - but why is this? Legislation has had a significant impact,
particularly in the US as a result of Section 508 which requires that software be accessible
from the start. Legislation in the UK also needs to tackle this issue head on as there is
no requirement for developers to adhere to particular standards. There is however a lack of
clarity about what needs to be done, with WCAG v2 being subject to debate about whether it is
'right' or not, combined with varying regional requirements. WCAG is also difficult to
Where an issue is of particular importance to a customer they may 'sponsor' its development.
The ability to override time limits for individual candidates within a cohort (rather than
setting up a separate assessment for them, a significant management issue) is the only
accessibility related example of this, but as a very recent development it may be indicative
that accessibility is becoming and increasing priority.
QuestionMark also offers the ability to construct and print paper-based assessments which
can then be scanned using Principia to produce comma separated files which can then be merged
with results from the same assessment taken online for automatic marking and grading. There
are also means of navigating through versions relevant to individual learners. Paper-based
delivery and scanning is used not only for accessibility reasons but also for practical
reasons, such as when 250 candidates are sitting an exam and there are not 250 computers
available to them.
From the commercial perspective, users need to make the decision to make assessment more
accessible and press developers for this functionality; at the moment users want more 'bells
and whistles'. A good taxonomy of disabilities and how they can be accommodated is also
required. There is excellent information available, such as on the TechDis website, but
people need to access this.
Where an assessment system is embedded in a platform such as WebCT, the software needs to
know how to talk to the underlying architecture to obtain accessibility information.
As accessibility can be seen as an emotive issue, there is no competition between vendors on
the basis of accessibility. This means that it is not very clear to developers what the
issues actually are, and that all vendors are at the same level. Customer surveys never show
accessibility as a high priority, but is this because of poor communication or because they're
using workarounds? Pressure on vendors needs to come from purchasing organisations, yet
purchasers are often very far removed from practitioners who are picking up issues and often
having to spend considerable amounts to meet students' needs. The accessibility of the
authoring software itself is also a low priority for buyers.
There is also a danger of looking at technical elements within a question, such as making a
Flash animation accessible, rather than accessibility within questions themselves. Work flow
and quality assurance processes should be incorporated into the development process, and can
even be automated when using Section 508 which takes a checklist approach to accessibility
evaluation. Constructivist approaches, where students develop learning and assessment
materials for their peers, also tend not to take account of accessibility requirements.
These issues aren't isolated to assessment but are relevant to delivering learning in
general. Validity is a big issue: do accommodations affect validity? The relationships
between pedagogy and technology, and online and offline, are major issues.
Back to Top of Page
Simon delivered his presentation using remote presentation software. After a few initial
teething troubles, the system worked extremely well and provided an excellent example of the
possibility for synchronous remote participation.
TechDis have done considerable work with awarding bodies, which will be fed up to the DfES
for implementation within FE. They have no specific plans at this time for HE, which is
As part of his presentation, Simon asked participants to undertake several exercises to
simulate the experience of individuals with accessibility needs. These vividly illustrated
the challenges faced both by learners and by developers in attempting to increase the
accessibility of resources.
Simon also pointed out a range of accommodations which can make e-assessments more
accessible. These include placing radio buttons or tick boxes for multiple choice or
multiple response questions after their associated distracter rather than before, which
significantly increases accessibility and navigability. This is not possible in some
delivery systems, which is why good, widely adopted guidelines are necessary.
Back to Top of Page
Andy's presentation included a standards update and discussion of blended accessibility
WCAG v2.0 is almost here. It does not cover cognitive disabilities. v1 had many problems
yet represented a significant step forward. The documentation is a mess but that is only to
be expected and it is gradually being improved.
addressed in WCAG v1.0.
The IMS Access For All meta-data standard is being developed into the new ISO Individualised
Adaptability and Accessibility standard and due for release in September 2006.
Digital resource descriptions combined with personal needs profiles can produce matched
content: content located, transformed, augmented or adopted to match personal needs. This
work is based on the concept of distributed resources being brought together to produce
personalised resources, being able to relate media with parts of other media. This is
related to work being undertaken by the IMS Common Cartridge and Rich Media working groups,
together with IMS Content Packaging v1.2 and Digital Rights Management work. Content
Packaging is related because of the solution it has adopted for dealing with varying
resources, solving many use cases not just within accessibility but also for language,
bandwidth, etc. The most appropriate resources are selected at run-time.
The IEEE LTSC RAMlet (resource aggregation model for learning and educational technology)
has produced ontologies which enable automatic transformations.
The IMS Rich Media group may tackle the difficult issue of digital rights management.
Issues relevant to this include screen reader access to .pdfs, and the development of trusted
tools which are able to access the underlying code to allow screen readers to function.
The SC36 working group 7 is looking at blended accessibility solutions in the context of
Access For All.
The emphasis is on a move away from modal thinking towards services architecture.