RE: [xml-dev] Usability testing of XML vocabularies?

["Michael Kay" <mike@saxonica.com>]

> I just finished reading a book on Web Usability [1].
> 
> The book describes how to test the usability of a Web site: 
> sit down with a user and ask them to perform tasks on the 
> site, ask them if they understand the purpose of the site, 
> ask them if they understand each section on a page, and so 
> forth.  By conducting such usability tests, the tester can 
> get a good idea of where things are not clear, and where 
> users stumble.

I don't know this book, but I was taught that there are essentially three
kinds of usability metric:

* efficiency - how long it takes to perform a given task

* effectiveness - how well the task is performed

* satisfaction - how happy the user is performing the task.

It's not entirely clear to me from the above which of these three you are
trying to measure or to optimize. In particular, asking users questions
about whether they understand the information provided might be asking the
wrong question if the information is not actually necessary to the task.
Also I can't see how to do this without first guiding the user to the
information, which rather misses the point if the problem is that users
can't find the information on their own. We used to do such tests by putting
the user in a closed room with the system, setting them a task, and then
observing their behaviour.

> 
> It occurred to me that when an XML vocabulary is created, or 
> being created, it would be useful to conduct usability 
> testing to see if users - developers - correctly understand 
> the semantics of the elements, the intended purpose of the 
> XML vocabulary, the constraints on and relationships between 
> the elements, and so forth.

For this to count as a usability test, you need a very clearly identified
set of user roles and a clear set of tasks, and you need to define what you
are measuring. "Level of understanding" is not itself a usability metric.
> 
> How do you conduct usability testing on XML vocabularies you develop?
> 
> How do the standards bodies - W3C, OASIS, ISO - conduct 
> usability testing on the XML vocabularies they produce? 

They don't - because the potential set of users and the potential set of
tasks is usually far too open ended. What they do instead is to invite
public comment on the specifications.

Some groups follow the discipline of publishing use cases before the
vocabulary is frozen. This tests whether particular tasks can be
accomplished using the vocabulary, but it does not test the efficiency,
effectiveness, or satisfaction of any particular group of users in
performing the task (except to the extent that the WG, who devise the
solutions to the use cases, regard themselves as typical of the target
users; which is always a dangerous assumption, but better than nothing as a
first approximation.)

One particular user role / task is that of the implementor, and W3C tests
the effectiveness of this task (how well the task of implementation has been
performed) by publishing interoperability tests and collating their results.
This also tends to generate a lot of feedback from implementors about
efficiency and satisfaction, even if this is not formally measured!

Michael Kay
http://www.saxonica.com/