Re: Transactional Web Services (LONG)

["W. E. Perry" <wperry@fiduciary.com>]

I agree with you and Dave Orchard about questions of scalability, and with you
about the problems posed by questions of context and by the statelessness of
http communication. However, the problem goes deeper, as the premises of
two-phase commit are fundamentally incompatible with the exchange of
transactional substance in general XML messages.

The 2PC protocol was developed to address the shortcomings of the previous
monolithic 'mainframe' transaction model. In the monolithic model, all
transactions of a given class were routed serially through a single processor.
Simultaneous or near-simultaneous transaction requests could not cause anomalous
or inconsistent data because the processor would order and process those
requests serially. Nor would resources at that single processor be locked for
longer than the processor itself required to execute each transaction, though
many of those resources were dedicated to the exclusive use of that processor
anyway. Clearly, though, such transaction processing cannot scale beyond the
capabilities of the single processor handling a specific class of transaction.
Also, the implicit system architecture for this transaction model is
mainframe-and-terminals, and it is fundamentally inimical to independent
processors acting in other than a master-slave relationship.

The 2PC protocol introduces different premises:  independent processors
operating two-at-a-time as counterparties to a given transaction, under the
control of a transaction monitor. Intelligently implemented, this protocol is as
scalable--up and down--as the volume itself of the underlying requests for
transactions. It is now clear that the weakness of this protocol is that it
depends upon the basis of the transaction, in the form of a datastructure upon
which both parties (and implicitly the transaction monitor) operate, being
identically defined and identically instantiated by both parties. Indeed that
identical definition is the basis for the fungibility of the datastructure and
therefore of the executability of the transaction itself. Finally, the implicit
system architecture for the 2PC protocol is the homogenous 'enterprise' network
which permits the various processing nodes to identify one another on the basis
of the transactional functions they can execute, based on the premise that the
substance (i.e. the physical datastructure) of each of those transactions is
identically understood by each party.

By serializing Java objects you achieved this requisite uniformity of
transaction 'datastructure' between the nodes across which you implemented your
2PC protocol. You also achieved a de facto homogenous network of the nodes which
you had enabled to deal specifically with that particular object serialization.
If, however, either of those requisites were absent--if you had transactional
communication between nodes not specifically identified in advance as capable of
the specific transaction to be executed, or if you had passed between them
something less than identical in its concrete instantiation at both ends than
serialized Java objects--your 2PC implementation would not have been possible.

In the general case which I believe we should address, we cannot make either of
those assumptions. I assume that the messages which we intend to pass as
transactional requests between processing nodes will be XML documents and that
the topology of those nodes will be an heterogeneous internetwork on which one
node will know little a priori of another besides its address. The problems that
a transaction protocol must address then become, at the originating 'sender'
node, how to address a message to an appropriate processing node--though not
necessarily one whose exact instantiation of data and precise processes are
known to the sender; and, at the 'receiving' node, how to instantiate data which
may be received in various forms from various other nodes.

In the 'contextual' processing case which you present, you have actually
reverted from the premises of 2PC to something very like the monolithic
transaction model, as your use of the concepts client and server indicates. 2PC
is an inherently peer-to-peer operation, but by designating one party as the
server side with additional responsibilities for managing context in a
transaction (which 2PC specifically avoids with its reliance on a transaction
monitor refereeing between equal parties) you are implementing not a
peer-to-peer execution but a request/response for services. In fact, this is how
virtually all apparently transactional services on the Web are now implemented.
Essentially, inside the gateway the homogenous enterprise network is preserved
(along with the 2PC transaction machinery which has long served that network
internally), and at the gateway the incoming message is instantiated as the
specific transactional datastructure expected on the internal network and routed
with the intimate knowledge which the nodes within that network have of one
another's functionality.

In transactional terms this is what I call a 'shop' or retail model. The
transaction requester outside the gateway--whatever the status of its own larger
enterprise vis-a-vis the enterprise it is addressing--is effectively a retail
customer. That requester is obliged, as is any retail customer, to submit to the
procedures of the vendor. Given the transformation of its request which occurs
at the gateway, the transaction process effected here is barely different from
filling out and submitting a web form.

A closer approximation to the 2PC transaction model is achieved where there is a
vertical industry standard data vocabulary (of which more than 2000 have been
promulgated and published in the past three years) or other a priori agreement
between the parties on a specific form for data exchange messages. These
messages may be XML, but in effect their semantics, and not just their syntax,
has been agreed in advance, so there is no question at either end of the
transaction how each node should instantiate the message as the specific data
which its process requires. Such a priori agreements create in effect a cartel
for executing transactions as peers in something very much like the 2PC
environment of a homogenous network. It is however my (admittedly most
unorthodox) belief that this homogeneity is achieved at the cost of contorting
both XML and the internetwork at an architectural level.

I believe that in contrast to both the 'shop' and the 'cartel' models of
transactions, it is possible to achieve what I call a 'souk' or 'bazaar' form,
which is far more consistent with the premises of both XML and the internetwork.
Additionally, this model permits transactions (and presumably profits) from
possibilities which are now overlooked or disregarded because they do not fit an
expected form. In the souk model the various participants may act on either side
of any particular transaction, eliminating the customer/vendor differences
inherent in the shop model. At the same time, each participant acts as a peer to
every other, despite both differences in the scale of dealings which a
participant can undertake (mom and pop versus the conglomerate) and in the
complexity of the data structure around which a transaction might be executed
(the single part versus the subassembly). The souk model necessarily deals with
each transaction on its own terms--instance first--rather than as a realization
of a transactional schema. Neither party may assumes that it understands the
other's resolution of the variables of the transaction, including most
especially the granularity of the data. That resolution is chieved separately by
each party through its separate instantiation of data from specific instance
messages describing the substance of the transaction as understood by the other
party.

In abstract terms, such a transaction is necessarily 'principal' in nature--each
party is acting on its own behalf and own authority to fashion an executable
transaction from what its own processes permit it to do with the transactional
substance--the data--which the other party presents. This is fundamentally
different from the inherently 'agency' nature of 2PC transactions. Each 2PC
transaction is a derived instance of the datastructure schema which defines that
transaction type, and the chief goal of the transaction execution process is
utter fidelity to that schematic. Also, the node specific to executing a
particular class of 2PC transaction acts by delegation on behalf of its entire
enterprise network for transactions of that type. By contrast a souk node may
execute a transaction on one side by instantiating data from one counterparty's
form of message into a form which its own processes can execute and then perform
an offsetting transaction executed by those same processes but against a
counterparty node with a fundamentally different structural understanding of the
data. That is fundamentally different from a 2PC processing node which can
execute transactions on behalf of its enterprise network in only the form that
the datastructures behind those transactions are understood internally on that
network.

The souk model which I propose is therefore 'principal' in transaction nature,
like the monolithic model, but like the 2PC model is inherently peer-to-peer
between the nodes which are parties to a transaction. It does not require the
transaction monitor of the 2PC protocol precisely because the primary task of
each transacting node is to decide, based on its knowledge of its own processes,
whether it can instantiate, from the message offered to it as the basis of a
transaction, data from which those processes can yield a useful result. The
criterion, that is, for whether to consummate a transaction is not, as in 2PC,
that both parties have managed to instantiate precisely the same data, but that
each party has managed to instantiate data which it can use, and will therefore
take principal responsibility for. The transaction monitor is dispensed with
because the criterion is not whether the two parties have obtained identical
results but whether each party has obtained a result that it will commit to.

To implement the souk model requires a method for potential transactional
counterparties to identify one another and to gain an initial understanding of
one another's messages. To work within the premises of this transaction model,
both of these facilities should be implemented instance-first. The heart of this
transactional process is the instantiation of messages as data directly usable
by the specific processes of each node. Therefore the best possible guide to the
transformations which each node is capable of in the course of such
instantiation is the history of the instantiations it has effected. Some portion
of this history may be a trade secret to a particular node, as it would reveal
intimate details of that node's business processes. Yet the necessity to find
transaction counterparties--and to find unexpected opportunities for
transactions (and profits)--in many or most cases will outweigh the need to keep
the history of specific data instantiations secret. Therefore instead of the
vertical market standard data vocabularies of the cartel model, to which a
potential participant must adhere in order to execute the expected transactions,
we have a public history of (some substantial portion of) what forms of messages
have been successfully instantiated for transaction execution. This is in
keeping with the general expectation that Web services must advertise their
availability and the specifics of their use. As a history, it is also in keeping
with the expectation that transactional processes log the details of their
execution, for needs ranging from a verifiable audit trail to the detailed
history required for rollback or offsetting transactions.

Therefore as a necessary part of implementing the souk model, a general purpose
transaction engine should generate an XML description of each data instantiation
it performs. As XML, that description is abstracted from the physical
instantiation of the data:  its element and attribute values are general
representations of the data rather than the physical form in which data was
instantiated in the specific internal format of the processing node. This is
just enough generality for that description to be used schematically. This
mechanism then presents two forms in which a potential transacting party can
implement its interaction with the particular processing node:  either in the
form which that node accepted, and transformed, or in the form which it was
transformed to. In either case, a newly implemented processing node my
instantiate locally whatever data structure is best suited to its processing and
may choose whether to make publicly available the XML representation of the form
into which it transforms the data messages upon which it operates.

This history of instantiated forms fulfills the roles of the three protocols you
cite in the "utopian dream in which business find each other through UDDI, learn
each other languages through WSDL and talk to each other through SOAP".
Furthermore it does so in a manner which, instead of constraining an XML
vocabulary to particular markup and even to particular lexical forms, permits
each processing node in recording its history to make use as it requires of the
full range of lexical possibilities in XML 1.0. And, as you note, UDDI, WDSL and
SOAP have been promulgated without any transactional processing model to back
them up, whereas in the approach I suggest the transactional model is paramount.

Maybe I should apologize for the length of this, but I think it is directly
responsive to your concern about how the transactional model should fit with the
other facilities of Web services and it does address your question of how anyone
else implements such services by describing the (still very incomplete) work
that is my current project.

Respectfully,

Walter Perry


Nicolas LEHUEN wrote:

> I agree that the problem is scalability, but it's the same problem whether
> you're using web service or not (e.g. using CORBA or EJB). The utopian dream
> in which business find each other through UDDI, learn each other languages
> through WSDL and talk to each other through SOAP is all well, but I'm not
> going to even begin to think about business implications if there is no
> transactional capabilities behind.
>
> Speaking or resource locking, we have encoutered interesting problems
> developing contextual web services. In contextual web services, a context is
> created on the server side in which all request from the same client will be
> performed (a context is a session). This is useful for example in our
> Exchange web service interface : we don't want to create a CDO session for
> each request sent by a client, so we create a server-side context containing
> the CDO session on the first request, and reuse it again and again until we
> receive a "close" request.
>
> The problem is that since HTTP is a stateless protocol, we are not notified
> when then client fails, so there are possibilities that some resources are
> never freed. So we implemented support for resource leasing in our SOAP-like
> protocol, just like in RMI or Jini, to work around this problem. This way,
> unused resources are automatically freed avec a timeout period.
>
> I'm not sure that contextual web services are a good idea to generalize, but
> the point is that we needed them for a set of projects. Does anyone else
> implement/use such web services ?