RE: [ebxml-msg] Bundling concerns, continued.

From: Moberg Dale [mailto:] 

Sent: 01 October 2009 17:54
To: Pim van der Eijk; 

Subject: RE: [ebxml-msg] Bundling 
concerns, continued.

The different 
processing in the  bundling case is contrasted with individual message: 
When the message is not bundled,  its  P-mode as determined by its 
user message metadata is used.  In bundling, the processing ignores how the 
message would be processed by its P-mode retrieved by its metadata, but must 
instead apply a different retrieval method for P-mode, and different processing. 
It makes a difference in what is done whether it is in a bundle or not. Even the 
presence of metadata that would govern processing in one case, has to be ignored 
when bundling. That violates Modularity, which I would like to see work. So I am 
not a fan of the RootConvention at all, even in its current lexical ordering 
version. 

 

It is not one Pmode when bundled, and another when 
unbundled.  The current draft says: "A message 
bundle is processed based on the processing mode of the primary message unit for 
the following processing mode sections: Pmode[1].Protocol, 
Pmode[1].ErrorHandling, Pmode[1].Reliability, 
and Pmode[1].Security with the exception of parameters for 
non-repudiation of receipt." The effect is 
that the Pmode of the primary message is used in the security and reliability 
modules, and the Pmode of the user message itself in the ebMS 
module. (Receipts are different in that they are in the security section in 
the PMode, but are not handled in the regular WS-Security module). 
So Service/Action of the user message are set based on that user message's 
Pmode. But WS-Security is configured by the primary unit's security Pmode. 

 

MSHDiscretion means to 
me that bundling is just supposed to be a transmission option, and if it were 
that, then a fallback to an alternative transmission mode would work, just as in 
ebMS 2 when two delivery channels were accepted for a given action. So that 
motivates trying to see how a fallback would work. It doesn’t. So I agree that 
we should not really think of bundling as a transmission option. If it fails, it 
is logged and manual resubmission is probably needed. 

 

I don't get 
this.  The MSH has submit() and deliver() operations 
for user message units.  If you submit a sequence of related 
messages and one fails, it is also safer for the sending MSH to not (re)send 
the units that follow the units in error, just as the receiving MSH 
should not deliver (and the sending MSH should not just resend) units 
in a bundle with a dependency on a failed unit in that bundle.  If 
there are no dependencies among units in a bundle, they can be 
processed as if they were all sent 
separately. 

 

I think it is still a 
very complex proposal. I will study the sections you mention before the next 
meeting to see whether they simplify but I am hoping we can reduce complexity 
and not add features and checks and processing overhead. 

 

The intention was to make the 
spec easy to implement, and unbundling in particular should require little 
changes to an existing ebMS3 product (assuming it has a reasonably 
modular/layered architecture).  The transport, security and 
reliability modules should be reusable with little or no changes. Some 
features may seem complex, but e.g. the ordering and partitioning 
features really just add a few lines of program code/Xpaths for 
sorting/grouping-by. Also, decisions on what to bundle and which 
units impacts which other units in cases of failure is fully driven by 
configuration rather than depending on any intelligence in the 
MSH.   

 

From: Pim van 
der Eijk [mailto:] 
Sent: Thursday, October 01, 2009 8:32 
AM
To: Moberg Dale; 

Subject: RE: [ebxml-msg] Bundling concerns, 
continued.

 

Comments 
inline.

 

Pim

 

From: 
Moberg Dale 
[mailto:] 
Sent: 01 October 2009 01:15
To: 

Subject: [ebxml-msg] Bundling concerns, 
continued.

There are several design principles 
and conventions at work in our current approach to bundling that don’t quite 
hang together cleanly -- or so it seems to me. Let me describe some of the major 
ideas and then discuss some of the tensions in getting them to work 
together.

 

Modularity: Each user 
message should behave in a bundle just as it would by being sent on its 
own.

 

Partly true. For 
routing, transport and security (except receipts) and reliability the secondary 
units are being processed like the primary unit.

 

MSHDiscretion: A MSH 
should be free to be statically or dynamically configured to send messages in a 
bundle or individually. 

 

RootConvention: The P-Mode 
of the SOAP body user message (href attribute omitted in PartInfo) is 
primary.

 

No, the primary message 
is the first eb3:UserMessage in the eb3:Messaging container, which is a SOAP 
header. It doesn't matter where the payloads of that primary message are stored 
(in the SOAP Body, as an attachment).  And perhaps the eb3:UserMessage 
doesn't even have payloads, as the minOccurs of eb3:Payloadinfo is 
0.  

 

SingleSignature: When a 
signed message is indicated by the P-Mode of the Root the signature must cover 
all user messages. 

The PMode of the 
primary unit specifies what (if anything) is signed. If it says: sign the 
eb3:Messaging unit, the SOAP Body, any other SOAP headers per WS-I RSP, and all 
payloads, then the effect is that all user messages are signed. 

 

 

OneSharedReliabilityId (sequence 
number in WS-RX)

 

 

 

Tensions in 
Modularity Root Convention, and Single 
Signature

 

Suppose one non-root body part has a 
P-mode with enveloped encryption. The assembly of parts is to proceed as it 
would for individually sent messages even when in a bundle. But the P-mode of 
the root is signed only. Should the non-root part be encrypted and then signed? 
Or should encryption be omitted and the receiving end indicate a 
Value-Inconsistent error? See section 
3.2.2 of the latest draft.  Whether or not to encrypt payloads is 
determined by the primary unit's PMode, since this is specified in 
the  Pmode[1].Security 
PMode section. This is 
true for both sending and receiving MSH, so there is no Value Inconsistency 
error.  Or when bundled should each non-root 
part be processed by ignoring the defined P-mode for this user 
message?   

 

On the face of it Modularity can 
easily conflict with RootConvention when SingleSignature is operative. In fact 
whenever the root P-mode conflicts with a P-mode of a non-root user-message 
part, then the normal message receiver-side error checks must be disengaged. 
Likewise I think during sender message assembly, the access to the P-mode 
associated with the user message’s AgreementRef will need to be overridden. This 
is a lot of special coding and almost amounts to a new mode of operation.  Bundling is functionality for the 
ebMS module, the security and reliability modules don't need to know much if 
anything about it.  Suppose you have an existing ebMS3 implementation that 
you want to adapt to support bundling.  The only thing is to check that it 
picks the right (=first) UserMessage for PMode setting (agreement) 
validation, since the only new thing is that there may be more of them. The 
security and reliability modules just use a different key to the agreement/pmode 
store than the ebMS module. Please explain why this would be a lot of 
special coding.  

 

So I agree with Sander that it might 
be better to restrict user message bundling to messages with the “same” P-mode, 
assuming we can explain what that means. I doubt that this constraint will be 
sufficient to solve all the tensions, however.

If trading partners 
want to restrict bundling this way, then the parameters in section 3.3 give them 
the configuration parameters they need to do so.  But if partner P1 
exchanges 30 types of messages with a trading partner P2 (hence 30 PModes), it 
may be perfectly acceptable to use a single set of security/reliability 
parameters for all 30 message types and to allow all of them to be bundled 
together, and this will increase the benefit on bundling. Many current 
ebMS user communities use configurations based on a handful of profiles (or CPA 
templates). If they exchange messages regularly, they immediately 
benefit from bundling.  

 

 

Tensions in 
MSHDiscretion and OneSharedReliabilityId

 

Here is a case I was struggling with 
describing today. Supposedly the MSH can “just decide” to bundle messages 
together. And if the bundle does not work for some reason (errors come back 
about message parts) it could decide to “back off” and send messages separately. 
In the case of a signature failure, I think we agree that it might be reasonable 
to see whether each part could be sent individually. Presumably this makes sense 
because no part “should” be delivered, given that we have SingleSignature. 

This is certainly an 
option, but in practice I suspect some system manager will want to inspect this 
message and see what went wrong, instead of having a policy to automatically 
sending the units unbundled.  We could define a PMode parameter to do this, 
or leave it as an implementation choice. 

If the error is due to 
e.g. the use of an untrusted certificate, then the primary unit will also be 
rejected when sent unbundled.  There could be a benefit in sending a 
secondary unit whose PMode does not specify signing (or references another 
-trusted- certificate) unbundled, since they would not suffer from the 
certificate errors of the primary unit's PMode.  But this doesn't help if, 
as you and Sander want, only units that have the same PMode are bundled: 
 then you are certain that they will all fail when sent unbundled, since 
the PMode specifies the certificate and it is the same for all. So then, when a 
bundle fails, all units in it 
fail permanently. 

If there is an error in 
calculating the hash for some cid: MIME attachments and not for other, this 
looks like a product error that would probably also occur with standalone units. 
Resending the units is unlikely to solve anything .

Sending each part individually would at least be 
preferable to sending the same bundle again. This is because the message 
processing failed before the bundle was acked, and so retrying the same bundle 
with its security defect could go into an endless loop of retries as long as 
acks might not be sent.

 

Yes, just as you 
presumably do not resend a failed unbundled message unit.

Now suppose for a different bundle 
that its security checks out, and reliability is processed, and we arrive at the 
separate message parts. Resending the whole bundle would probably just be acked 
as a duplicate, so that will not be an option for a transparent recovery. So if 
we do get error messages, should we trust that the parts for which we did not 
receive an error message were ok and delivered? What happens when you send an unbundled user message unit, 
you get an RM acknowledgement and no error.  Why would you not question 
whether it was delivered in that case, whereas you have doubts about it being 
delivered if it is sent in a bundle?  You can specify that you want an NRR 
receipt.  Those are specified per message unit, so you would get 
separate eb:Receipts for each user message unit that didn't fail. 
What if (some of ) the error messages get lost? What if an error for an unbundled message 
is lost?  With WS-ReliableMessaging, an acknowledgment does not mean that 
no error occurred, even when using InOrder delivery. There is no difference when 
bundling applies, so why is this raised as an issue with bundling?  Not 
sure how to specify it in PModes, but an MSH could send errors back over a 
reliable sequence, to make sure they don't get lost.Or to be 
on the safe side should we send all the parts again, individually, adhering to 
MSHDiscretion and the hope that a more determinate outcome of errors and/or acks 
comes back? If our received error messages were infallible indicators of exactly 
what went wrong, then we might try to send each payload with an error and see 
whether that remedies the error message. Otherwise, we retry all the body parts 
separately. The problem here is that if some of the parts had been ok and had 
been delivered optimistically, then retrying individually will get duplicates 
delivered. Yes, that's why I think 
resending a message unit that you did not get an error for separately, 
if you sent them before in a bundle that you got an acknowledgment for, is a 
questionable idea.  You have to build a second level of duplicate 
elimination at the ebMS level that you otherwise wouldn't need. 
And that violates our modularity assumption because when each part 
has its own reliability id value, duplicates would be eliminated. It could be 
that we just give up and do nothing about these ebMS level errors, just logging 
them. I think the receiving 
MSH should generate errors for the units that have errors and no errors (or 
receipts, if NRR is requested) for the units that are fine at 
ebMS level. We could mandata the errors for units are always bundled, so 
that the sending MSH receives them together, providing strong evidence that some 
units failed but others not (since that bundle would contain explicit errors for 
those if they were).   But that would violate 
modularity too, because in that case, we would not know that any of our parts 
got delivered successfully as we could with an acked, errorless single message 
sent individually.  Not 
true, with WS-ReliableMessaging, an acknowledgment does not mean that no 
error occurred, even when using InOrder delivery, also with unbundled messages. 
Being part of the bundle has made our state alignment a bit 
foggier in the case of some errors at the ebMS header level. Or so it seems to 
me.   Have a look at 3.4 
and 3.9.3.

 

Let me stop with these two issues, 
and ask for simplifications that would diminish the interactions threatening 
Modularity. I think that Sander had a good suggestion to disallow P-modes that 
were “different” but I would like a cleaner way of stating this requirement. I 
personally would not mind dropping the single signature idea because that 
definitely makes sending a message in a bundle different from sending it on its 
own.  I think that if signatures 
are per unit, can use different certificates or algorithms, that some units use 
encryption and others not we are loosing the biggest efficiency benefit for 
bundling.  In that case, I would not use bundling at all.  Or define a 
batching protocol (multiple SOAP envelopes in distinct MIME parts) 
if saving a separate HTTP connection set-up is worth the effort. 
With batching the various SOAP envelopes could also have separate reliabilty 
sequence numbers.  

(One security header can have many 
separate signatures within it, although I need to check WSI BSP to see that they 
have not profiled that away.)  

 

As far as ebMS level errors go, 
either we should state that nothing is to be done in response to an error by the 
sender as part of our specified protocol. Or we might state that each part is 
sent again individually (but I think this is bad because the receiver will have 
to support a second layer of duplicate detection by ebMS message id). Or we 
state that a receiver MAY retry a message with an error received when it was 
sent in a bundle. My preference is now headed toward saying that the specified 
protocol does not say what to do when an ebMS error is received for one part of 
a bundled message. Maybe we even say the same thing about the failed-signature 
error message when received by the sender.  Please review 3.4 and 
3.9.3. 

 

I do not think that going into the 
dependency graph direction is going to make this situation simpler or better. I 
think that we should say that when choosing bundling, that is the mode of 
transmission, and when it fails, each message payload must be resubmitted by 
operations (but with a do-not-bundle flag set! )  Please review 3.4 and 
3.9.3.