Re: [security-services] SAML Testing Referral Program Proposalfor SSTC Review & 30 Day Feedback - Response to SSTC Feedback

I talked to some colleagues here at Novell and we have mixed thoughts about the conflict of interest questions.  IMHO, If there existed 3 or four different referral providers, we (SSTC) wouldn't care whether one is also a vendor.  Let implementors worry about conflict of interest and NDA's.  In any case this should be an OASIS question and not a TC question.
    

    
 
    

    
I think our (SSTC) emphasis should be on specifying what the tests should be.  That way if my implementation passes Ping's tests, I should also pass IEEE-Liberty tests.  Prateek called for some 'champions' to come forward and help create these tests: http://lists.oasis-open.org/archives/security-services/200503/msg00055.html.  
    

    
 

    
OK, ping just stepped up and gave us some tests - I think. Is ping letting us use these tests, or did they just want feedback?  My concern is that no on has commented on the tests (me included - because I thought they just wanted feedback).  Can we approve a referral provider without specifying what they should test?  
    

    
 

    
That said, we should actively seek additional independent third parties to also do conformance testing.  I'm thinking of IEEE-Liberty, GSA, and the Open-Group.  I mention the Open-group because they hosted a SAML plugfest along with their LDAP certification event last year: http://lists.oasis-open.org/archives/security-services/200408/msg00198.html
    

    
 

    
Then once they are on board, lets have ping/opengroup/gsa/ieee-liberty hammer out the exact conformance tests details.  
    

    
 

    
- Cameron Morris
    

    
 

    

>>>Greg Whitehead <> 04/24/05 3:58 am >>>
Andy,

I haven't seen an answer to Tony's question about the process going 
forward. If you've replied privately, would you please post the answer 
to the list?

Trustgenix has been a strong supporter of SAML interoperability testing 
in both Liberty and Oasis and of independent certification programs, 
such as the ones run by IEEE for Liberty and by the GSA for the US 
Government. However, we continue to see a fundamental problem with a 
vendor of SAML products running a certification program for other 
vendors of SAML products (their competitors). I don't know of any other 
industry that operates this way.

I finally got a chance to read through your response last night and 
here are some initial comments (by number from your response):

1) You say that Oasis defines the test suite and that changes can't be 
made without a vote, but in the general background info on PingDeploy
it is made clear that it exists independently of Oasis and is owned and 
managed by Ping. I don't understand how both can be true. How does 
Oasis know that PingDeploy implements the test suite specified by 
Oasis, or that it does not favor some implementations over others?

2) The complexity of the attached "Privacy Directive" just reinforces 
the fact that all parties acknowledge a fundamental conflict of 
interest in having a vendor of SAML products run the SAML certification 
program. It raises many more questions than it answers. How can we be 
sure that the Privacy Directive is sufficient or can even be 
implemented successfully.

3) It's that Ping, a vendor of SAML products, would be selected to run 
an Oasis branded SAML certification program that is the problem. As 
noted in (2), the "Privacy Directive" raises more questions that it 
answers.

6) I don't understand this. If this is not an Oasis program, why is 
Oasis involved at all?

7) This seems like something that should be corrected in the CURRENT 
program, not left to future programs.

-Greg