Accessible e-Assessment

Back to Notes from the Joint Accessibility and Assessment SIG Meeting.

Click on topic (25 topics) to move to section:

1. Agenda.
2. Introduction.
3. Questionmark Perception.
4. John Kleeman.
5. Requirements - Theory.
6. How a Vendor Adds Features.
7. Successful Vendors.
8. Accessibility.
9. Many Different Specs Out There.
10. What Customers Are Currently Asking Us For
11. Solutions.
12. Workflow in Authoring.
13. Workflows Can Be Simple.
14. Workflows Can Include Whatever Steps You Want.
15. Solutions - Management.
16. Override Time Limit.
17. Templating of Online Interface.
18. Print and Scan.
19. Questionmark Printing Options.
20. Demonstration.
21. Questions for Us to Consider.
22. Features vs Accessibility.
23. Confusion Over What Is Needed.
24. Education.
25. Technology Issues.

This is the HTML version of the Accessible e-Assessment (PowerPoint format - 1.65Mb) given by John Kleeman, Chairman, Questionmark.

1. Agenda.
   
   • Introduction.
   • Requirements - What are the requirements to make e-Assessment accessible?
   • Solutions - What does Questionmark to do deal with making assessments accessible?
   • Questions:
     
     • What could we or the community be doing more?
     • How will things change in future?

   Back to Top of Page

2. Introduction.
   
   • Questionmark produces an assessment management system, Perception.
   • Used by more than 2500 organisations worldwide.
   • Lets you:
     
     • Author assessments;
     • Schedule them;
     • Deliver;
     • Run reports.

   Back to Top of Page

3. Questionmark Perception.
   

   
   

   The diagram above shows the functionality in Questionmark Perception. It is capable of the following:

   • Authoring:
     
     • Word templates for SMEs;
     • Windows based Authoring Manager;
     • Browser based Authoring.
     
      
   • Scheduling:
     
     • Manage Participants;
     • Manage Schedules.
     
      
   • Delivery:
     
     • Questionmark Secure
     • Windows CD (no reporting);
     • P2G;
     • PDA;
     • Print and Scan;
     • Standard browsers (IE, Netscape, AOL, etc).
     
      
   • Reporting - Enterprise Reporter.

   Back to Top of Page

4. John Kleeman.
   
   • Founder of Questionmark in 1988.
   • Now Chairman and product manager.
   • Standards work:
     
     • One of original IMS QTI XML v1.0 working group;
     • Convenor of panel that created BS 7988.
   • Accessibility - Only a working knowledge.

   Questionmark vision: The cornerstone of successful education, training and certification is the effective use of assessments. The 21st century offers a real opportunity to use technology to make assessments more widely available and more successful for those involved in the process. In a world where you cannot know everything, assessments will be used to guide people to powerful learning experiences, reduce learning curves, confirm skills, knowledge and attitudes, and motivate by providing a sense of achievement. Questionmark aims to be the world's leading organization in developing, supplying and supporting an assessment platform, software, systems and services for computerizing education and training related assessments. Questionmark strives to bring inspirational meaning to assessment experiences by getting results.

   Back to Top of Page

5. Requirements - Theory.
   
   • E-learning and e-assessment has the potential to make learning much more accessible.
   • Can take assessments:
     
     • Where you want (no need to travel);
     • When you want;
     • With technology such as screen readers to help accessibility.
   • Most vendors have vision and passion to make assessments more available and accessible, that's why we're here.

   Back to Top of Page

6. How A Vendor Adds Features.
   

   
   

   The diagram above shows how a vendor adds features to software. The product is first introduced and then released. After release, the sales team show the software to prospective customers and customers start using it. When customers start using the software, they find bugs and weaknesses, and come up with suggestions/demands which are reviewed. Ideas from within the vendor, features for which customers will pay/sponsor, and features which competitors have which the vendor needs are also reviewed. From this review, a feature list for the next release is drawn up and the next version of the software built and released. The cycle then begins again.

   Back to Top of Page

7. Successful Vendors.
   
   • Survival of the fittest:
     
     • Vendors who survive listen to their customers;
     • Prioritize the features that:
       
       • Customers care about;
       • Make customers more successful;
       • Help sell the product.
     • Vendors who don't listen to their customers don't survive and grow.

   Back to Top of Page

8. Accessibility.
   
   • Must be customer driven:
     
     • Driven by purchase requirements;
     • Driven by user requests for the feature.
   • Questionmark and other vendors are keen to make software more accessible.
   • Education and information by groups like CETIS helps.
   • But customers must drive what needs to be done.

   Back to Top of Page

9. Many Different Specs Out There.
   
   • UK - DDA, SENDA, TechDis, BS 7988.
   • US - Section 508, ADA.
   • International:
     
     • W3C - WCAG V1 and v2.
     • IMS:
       
       • AccessForAll;
       • LIP;
       • IMS Accessibility Guidelines.
   • Which one should we follow?

   Back to Top of Page

10. What Customers Are Currently Asking Us For.
    
    • Section 508 and W3C v1 compliance.
    • Better management features - Override time limit for people who need it.
    • Easy printing of assessments - Many customers have an offline printing option.
    • Separate presentation style from content.

    Back to Top of Page

11. Solutions.
    
    • Workflow in authoring.
    • Management features.
    • Templating of online interface.
    • Print and scan.

    Back to Top of Page

12. Workflow in Authoring.
    
    • Lets questions go through a process for approval.
    • Questions cannot be delivered until process complete.
    • Can use for accessibility review:
      
      • Colour blindness;
      • Review vs standards.

    
    

    The diagram above shows that assessments developed by authors working on individual terminals are sent to a web server and managd by a database server.

    Back to Top of Page

13. Workflows Can Be Simple.
    

    
    

    The diagram above shows a simple authoring workflow. The assessment is authored and if approved, is put into production. If issues are identified, they are then fixed before going through the authoring process again.

    Back to Top of Page

14. Workflows Can Include Whatever Steps You Want.
    

    
    

    The diagram above shows a more complicated authoring workflow. The questions are authored and if the author is happy, they are reviewed by the SME. If they are rejected, they are sent back to the author. If the SME accepts them, they are sent for psychometric review. If they are rejected, then they are sent back to the author. If they are accepted by the psychometic review, they are then sent for an accessibility review. If it is rejected, it is sent back to the author. If it is accepted, then it is included as an experimental assessment. If there is a problem, these are either fixed and tested again or sent back to the author. Once the question statistics have been successfully tested they are sent for a final review before being released and put into production. If it fails the review, it is rejected and sent back for problems to be fixed. If a question becomes out of date, then it is retired.

    Back to Top of Page

15. Solutions - Management.
    
    • Increasingly we are being asked to:
      
      • Identify who has what accessibility needs;
      • Direct them to appropriate methods.
    • One recent improvement (Feb 2006):
      
      • Option to override time limit:
        
        • An assessment has a standard time limit;
        • Senior administrators can override to give people with accessibility needs more time.

    Back to Top of Page

16. Override Time Limit.
    

    
    

    The screenshot above shows the options for the override time limit feature. Features include limiting the number of attempts on an assessment, overriding the time limit by a chosen number of minutes, defining the number of days between retakes of the assessment, and defining when the start and end period of when the assessment can actually be taken.

    Back to Top of Page

17. Templating of Online Interface.
    
    • Questionmark Perception has always separated presentation from content and participant interfaces are controlled by templates and CSS files.
      
      • For instance almost every HTML character produced by our web assessment engine goes through files which are configurable by the user (templates and formats);
      • We meet Section 508 and W3C (v1) guidelines;
      • See www.questionmark.com for more details.

    Back to Top of Page

18. Print and Scan.
    
    • For many real world assessment deliverers, delivery on paper seems a common fallback option for some accessibility needs.
    • Obviously delivery on screen is more ideal.
    • But printing is what a lot of people do.

    Back to Top of Page

19. Questionmark Printing Options.
    
    • Print to Word:
      
      • Introduced June 05;
      • Allows printing of all question types (except rich media ones like Flash) to Word document;
      • Scoring is manual.
    • Print to Scanning:
      
      • In beta June 06;
      • Allows printing of multiple choice / multiple response question types;
      • Scoring is automatic;
      • Online and printed results merged for reporting.

    
    

    The diagram above shows the print workflow. Once the assessment has been authored and reviewed and approved, there is a choice presenting the assessment online or on paper. If the online selection is made, the assessment is scheduled for online delivery, the questions are answered on screen and marked by computer. The results are then stored in a database from which reports can be run. If the paper option is chosen, the assessment is printed, answered on paper, the results scanned in and marked by computer. The results are stored in a database from which reports can be run.

    Back to Top of Page

20. Demonstration.
    
    • Authoring.
    • Printing.
    • Simulated scanning.
    • Reporting.

    Back to Top of Page

21. Questions for Us To Consider.
    
    • Complexity vs accessibility.
    • Confusion over what is needed.
    • Education.
    • Technology changes.

    Back to Top of Page

22. Features vs Accessibility.
    
    • More complex question types (e.g. simulations):
      
      • May allow you to assess more effectively;
      • Involve more effort to make accessible.
    • Trade off:
      
      • Innovative item types;
      • Presentational look and feel;
      • Accessibility.

    Back to Top of Page

23. Confusion Over What Is Needed.
    
    • Too many standards:
      
      • Section 508;
      • W3C v1;
      • W3C v2 (in contention);
      • Possible UK initiatives.
    • Need to agree internationally.
    • Document and communicate clearly.
    • Also a confusion over different kinds of accessibility needs.

    Back to Top of Page

24. Education.
    
    • Communicate accessibility issues:
      
      • To developers;
      • To authors;
      • To other stakeholders.
    • There is good material out there.
    • But how do you get the knowledge inside everyone's head?

    Back to Top of Page

25. Technology Issues.
    
    • How is accessibility impacted by:
      
      • PDAs ;
         
      • Text messaging ;
         
      • Group collaboration/instant messaging ;
         
      • Gadgets/widgets ;
         
      • IPODs .
      
       
    • Rapid change.