Hi
Folks,
A few musings about "scaling" which I thought might be
of interest ...
Note: In this
message I use the word "technology" in its broadest sense.
It can mean a specific tool, or it can mean an approach or
strategy.
A key aspect of
the Web is its ability to scale. Web technologies flourish or die
depending on their ability to scale.
What does it mean
for a technology to scale? What does it mean that a technology does not
scale?
To have a
sensible net-centric vision it is critical to have a deep, intuitive understanding of this
concept of scaling. As a company moves to a net-centric environment it must adopt
technologies that scale well. A technology will
either scale or it will die.
Let's consider some
examples.
A SCALABLE MOVIE
REVIEW TECHNOLOGY
Roger Ebert is a
professional movie critic. He reviews a movie and then provides
information about it:
- a
short description
- a
rating (1 - 4 stars)
[Note that what
Ebert provides is "metadata" about the movie]
Over the years, the
number of movies has multiplied (independent movies, blockbuster movies, etc). It is no longer
possible to Ebert to review all them.
Having a
single professional movie critic review movies doesn't scale. Even a
group of professional movie critics would be soon
overloaded.
As well as not
scaling, there is a problem with "truth". A movie review represents
someone's interpretation, not facts. Everyone has their own opinions about
a movie. Ebert may love the movie, but the rest of the world may hate
it.
An alternative to
using a professional movie critic to review movies is to have each movie
producer generate a movie review. This approach scales well - there
is one movie reviewer for each movie. Of course, the problem with this
approach is that the movie review will be heavily biased.
A third alternative
is to record the comments of the movie-goers. This approach is highly
scalable - there are many reviewers for each movie. Further, since
there are many reviewers there is a diversity of opinions, which give a variety
of perspective to someone who is trying to decide whether to view a
movie.
Thus,
a scalable movie review "technology" is one that can grow as the number of
movies to be reviewed increases.
A SCALABLE BOOK REVIEW
TECHNOLOGY
The New York Times
has a small group of book reviewers. This group reviews books and then
publishes their reviews.
Just as with the
problem of reviewing movies, using professional reviewers does not
scale. The New York times group is unable to review the exponentially
growing number of new and old books.
An alternative
approach is for each book author to elicit a review from a fellow
author. This scales well, but suffers from bias.
A third alternative is to let the users (the
readers) provide a review of the books. This scales well and provides
for a diversity of opinions.
A scalable book review "technology" is one
that can grow as the number of books to be reviewed
increases.
LESSONS
LEARNED
There are some
lessons to be learned from the above examples. There are three approaches to solving a
problem:
1. The high priests
approach: Solve the problem using a small, centralized set of
skilled professionals (the professionals may be geographically separate but
functionally they are centralized)
2. The authors
approach: Solve a problem using the authors.
3. The users
approach: Solve the problem using the large body of distributed
users.
The high priests
approach is not scalable, although it
may produce precise, reliable results.
The author approach
is scalable, but it may produce dubious results.
The user's approach
is scalable and produces multiple results which are then
shared.
A SUCCESSFUL WEB
TECHNOLOGY
A successful web
technology is one that:
-
scales and
-
shares
Technologies that
scale invite participation by the users. When the users become the
professionals then the technology has succeeded:
- When your
users style the XML data to their own liking and share their stylesheets with
others, then your technology for making appealing looking data
has succeeded.
- When your
users markup data using tags that make sense to them and share their tags with
others, then your markup technology has succeeded.
- When your
movie goers write reviews of movies and post them for others to view, then your
movie review technology has succeeded.
- When your
book readers write review of books and post them for others to view, then your
book review technology has succeeded.
A long time ago
(Oct. 28, 2003) John Cowan made the below remark on xml-dev. I now understand
it.
Roger L. Costello
scripsit:
> BTW John, I really liked your paradigm shift on the
telephone operator story.
> Can you think of a similar shift with
programming? /Roger
Yes, in fact. Perhaps ten years ago, I
was sitting next to a stranger
on the New York to Boston shuttle. In
casual conversation, she asked
me what I did. "I'm a computer
programmer", said I. She said, not
unusually, "Oh, I don't know
anything about programming", but then
added in the same breath "except for
some Excel and Basic." I had
an epiphany: the world had
changed.
As always, comments are welcome.
/Roger
