On 7/11/12 6:38 PM, "Chad La Joie" <> wrote:
>I think a big drawback is that these sorts of test usually just test
>to see if one can eventually get product A to talk to product B. They
>do not, as far as I can tell, test for spec compliance.
It really varies by feature/test. Some of the tests are fairly specific
and demonstrate that, and some don't. Usually the problem is more that
they only test for compliance in a minimal sense, and don't check some of
the options well.
>Both in my professional consulting and in my work with Shibboleth I have
>encountered a number of impls (especially commercial ones) that simply
>are not standards compliant. This invariablly leads to a significant
>amount of consternation and comments of how "this SAML stuff" is crap.
This is very true of almost all the ill-advised one-offs that a lot of
service providers use, a bit less true in my experience where the actual
full implementations are concerned, with the obvious exception of
metadata. And a lot of that was a simple lack of emphasis.
The testing that Liberty did was only on actual implementations of the
more complete sort, and there were simply no tests for a huge set of
things that are critical to scaled deployment.
-- Scott
