Designing an experiment to gather evidence on approaches todesigning web

["Costello, Roger L." <costello@mitre.org>]

Hi Folks,

Perhaps you have heard this statement:

     Make decisions based on evidence.

That seems pretty obvious right?

But if there is no evidence, how does one make decisions?

Clearly one should seek to gather evidence.

Consider making a decision for which of these approaches to follow in implementing web services:

    (a) Do everything in XML: process all inbound and 
          outbound data using  the XML technologies
          such as XML Schema, XSLT/XPath, XQuery, and XProc.
         Use little, if any, imperative code (Java, C#, C++, etc.), 
         and where it is used, it is used only as glue for the XML
         technologies.

    (b) Do everything in code: immediately extract all inbound 
          data into  imperative code data structures and do all 
          processing in code. For outbound data keep the data in 
          code as long as possible; generate XML only when ready 
          to release the data.

There is no evidence to support one approach or the other. There are anecdotes, but that's not evidence.

Suppose one were to design an experiment with the objective of gathering useful evidence concerning the above two approaches.

What evidence would one seek to gather?

Here are a couple ideas:

1. Source Lines of Code (SLOC): how many lines of code do each approaches require to perform the same task.

2. Performance: what is the response time between inbound data and the resulting outbound data?

What else?

/Roger