Improving conformance testing modularity

[Jim Marino <jim.marino@gmail.com>]

Hi Mike et al,

 From last week's email thread, I promised to take a look at the  
conformance tests and provide feedback as to how they can potentially  
be refactored to accommodate specification "modularity," i.e. the  
ability for vendors to configure the test harness to be run against  
proprietary implementation languages. Here's a start of that feedback.  
I'm happy to supply code-level examples based on responses to some of  
my questions and proposals below.

I'll start with an initial set of questions and comments and then move  
on to some proposals for how modularity might be structured.

Here are the initial questions:

1. How are test artifacts built? What tools are required (e.g. Ant,  
Maven)? From the file layout, it appears Maven is being used. I think  
the build artifacts should be checked into the svn repository.

2. Will contribution artifacts be bundled in zip archives only? I  
assume this has to be the case since SCA requires only support for  
that format.

3. Why are import.java statements in ASM_ contributions? I'm don't  
understanding why they are needed even when Java is used as the  
implementation language. Actually, I think they should be removed to  
verify import.java/export.java work correctly as a contribution using  
a composite in another contribution should not have to import Java  
artifacts referenced by that composite as they are an implementation  
detail.

4. The purpose of General, Contribution1 and Contribution2 is not  
clear to me. I think they are intended to be different contributions  
but I am not sure (it would be helpful to have it explained). If they  
are intended to be separate contributions and Maven is the build  
system, then my recommendation would be to create separate modules for  
them as they zip files will need to be produced. Otherwise, Maven  
won't behave nicely.

5. In the General contribution, the namespace http://docs.oasis-open.org/ns/opencsa/scatests/200903 
  is imported and exported. This type of behavior is defined in OSGi  
but is it defined in Assembly? I don't remember seeing it and if it is  
not defined, it probably should be removed.

6. Java package names should follow JLS recommendations instead of,  
for example, "test"

7. Given the following interface:

public interface RuntimeBridge {

	/**
	 * Sets the Test configuration for a particular test run
	 * @param testConfiguration
	 */
	public void setTestConfiguration(TestConfiguration testConfiguration);

	/**
	 * Start the contribution(s) which represent the application under  
test in the SCA runtime
	 * @param contributionURIs - an array of contribution URIs as strings
	 * @return true if the contributions were loaded and started in the  
SCA runtime, false otherwise
	 */
	boolean startContribution(String[] contributionURIs) throws Exception;

	/**
	 * Stop the contribution(s)
	 */
	void stopContribution();

	String getContributionLocation(Class<?> testClass);
	
} // end interface RuntimeBridge


a. What does getContributionLocation(Class<?> testClass) do? I believe  
it returns a URL for the contribution archive assuming the "testClass"  
is in it. If this is the case, why are the test classes bundled in the  
contribution archive? My assumption is the following:  the contents of  
a contribution archive should not be visible to the test client  
classloader. The test harness should also ensure that the contents of  
the contributions are visible to the classloader used to boot the SCA  
runtime. Otherwise, the the Java import/export mechanisms may not  
function properly and cannot be verified. Regardless of this, and as a  
general design point, the method seems generic and should not be part  
of the runtime-specific SPI as it is not determined by the runtime  
implementation.

b. What is the array contributionURIs in startContribution(String[]  
contributionURIs)? Are they URLs, i.e. intended to be deferenceable?   
If so, I think the type should be URL[] or (better) List<URL>.

c. How is a runtime notified of which deployable composites to deploy  
in a particular contribution? Related to that, I noticed only one  
composite is defined as a deployable (test:TEST_ASM_12001). That seems  
problematic.

8. Given the following interface:

public class ASM_4001_TestCase extends BaseJAXWSTestCase {


     protected TestConfiguration getTestConfiguration() {
     	TestConfiguration config = new TestConfiguration()
     	config.testName 		= this.getClass().getSimpleName().substring(0,  
8);
     	config.input 			= "request";
     	config.output[0] 			= config.testName + " " + config.input + "  
service1 operation1 invoked";
     	config.composite 		= "Test_" + config.testName + ".composite";
     	config.testServiceName 	= "TestClient";
     	config.testClass 		= ASM_0002_Client.class;
     	config.contributionNames	= new String[] { "General", "General" +  
_Lang };
     	config.serviceInterface = TestInvocation.class;
     	return config;
     }

}

a. What is "General"+_Lang  in "config.contributionNames	= new  
String[] { "General", "General" + _Lang };" used for? Can't the URLs  
of language specific  contributions be explicitly supplied by the  
vendor?

----------------------------------------------------------

Here is the start of a proposal for introducing more modularity:

One approach to modularizing the conformance tests would be assume the  
build system is Maven for assembly (an alternative can be chosen which  
impose different constraints on how modularity is achieved and a  
choice would have to be made). Languages other than Java and BPEL may  
choose to have their own build systems.

Assuming Maven, the conformance tests should form several multi-module  
project. Something like:

Assembly Project
	|
	|------------------- Assertions Module:
	|				- Forms a single SCA contribution
	|				- Imports a language-specific namespace
	|				- Contains composites against which test assertions are run.  
These composites define components whose implementations are  
composites in the language-specific namespace
	|------------------- Assembly Test Client Module
	|				- Contains Test clients (JAX-WS) which perform and verify test  
assertions
	|				- Uses the RuntimeBridge SPI to deploy the assertions module  
contribution and language specific contributions

Java Project
	|
	|------------------- Java Implementation Module
	|
	(other modules for testing things like import.java/export.java)

BPEL Project, etc.

One advantage of separating these projects is that they can be  
released independently. For example, the language X can be released  
after the Assembly project. I don't think this is far off from the  
current structure of the SCA composites, although it would involve  
restructuring the conformance test layout.

In terms of the SPI, I was thinking the RuntimeBridge SPI will need to  
be refactored for a number of reasons. Probably the most important is  
there will need to be a mechanism for testing deployment semantics  
(e.g. include-in-domain, remove, etc.). Another reason is the current  
SPI does not really fit some runtimes. For example, Fabric3 has no  
notion of start/stop contribution. It does, however, provide  
facilities for installing/uninstalling contributions in a domain and  
deploying/undeploying composites. Further, Fabric3 conformance tests  
will likely be run against our standard software distribution, which  
is a separate server process that can be managed through JMX  
(customers will want the standard distribution verified for  
conformance).

Based on my previous comments, I was thinking of an SPI similar to the  
following:

public interface RuntimeBridge {

         void setTestConfiguration(TestConfiguration testConfiguration);

	DeploymentResult deployUsingIncludeInDomain(URL contributionUrl,  
Set<QName> deployables);

	DeploymentResult deployUsingAddDomain(URL contributionUrl, Set<QName>  
deployables);

	DeploymentResult removeFromDomain(Set<QName> deployables);   // note  
I'm not quite sure about the parameters to this operation but  
deployables seems reasonable

}

As a final comment, we probably also need to consider allowing the  
test client harness to be something other than Java. I realize this is  
potentially a can of worms, but consider the following situation. A  
vendor/open source project wants to implement an SCA runtime in some  
language other than Java. The management interface for the runtime is  
built using a technology that is not accessible (or easily accessible)  
from Java. I don't have an answer for how to solve this problem yet.  
If wire-level data were tested, I don't know how deployment errors can  
be verified. Maybe the answer is to require a vendor who does not want  
or cannot use Java to supply a port of the test harness. Then the  
issue is verifying the test harness is faithfully reproduced in the  
proprietary language.

Jim