TEAM Engine & CTL Quick Start

 

Note: This instructions apply for TEAM Engine 3.0. TEAM Engine 4.0 Beta have been mavenized and building of the code is different. More details at Sourceforge.

 

 

The Test, Evaluation, And Measurement (TEAM) Engine is a test script interpreter. It executes test scripts written in Compliance Test Language (CTL).

Installation 

Make sure you have JAVA

TEAM Engine is a Java application. It depends on Java 1.5 or greater. To check the version of Java installed on your system open a command prompt window, and try the java command:

C:\>java -version

This should display the version of Java that is in your path. If the version number is 1.5 or greater, you are good to go. If you get an error message or a number less than 1.5, you may need to install a standard Java Runtime Environment (JRE). You can find it on Oracle's Java page.

 

Download TEAM Engine from SourceForge

Download the TEAM Engine binary distribution archive from SourceForge, and extract the files. The root directory of your hard drive would be a good location to extract them to. You can also get the latest code from SVN as explained here.

Test TEAM Engine

Go to where the extracted files are and navigate to the bin folder:

...test-release/teamengine2.0/bin

 

Make sure you can execute the files under these bin folder (*.sh and *.bat). .sh is used for *nix environment, and .bat is used for the Win32 environment. You may need to change the permissions so you will be able to execute these files. For example, when you are under the bin directory, in unix you can type

chmod 744 *.*

 

To test the installation,run the file test. For example:

./test.sh

 

This should display a short synopsis of the test utility.

No working directory supplied.  Using /var/folders/HI/HI3ayq87Fi...


Test mode:

Use to start a test session.



./test.sh [-mode=test] -source=ctlfile|dir [-source=ctlfile|dir] ...

[-workdir=dir] [-logdir=dir] [-session=session] [-base=baseURI]

[-suite=qname|-test=qname [@param-name=value] ...] [-profile=qname|*] ...



qname=[namespace_uri,|prefix:]local_name]


Explanation of how TEAM Engine works

TEAM Engine executes tests written in CTL, a test scripting language which specification can be found at the OGC Portal. When a test suite is executed, the source CTL files are converted into XSL files, a separate XSL file for each test. TEAM Engine presents a web form for a user to select a test suite to the user. The values the user enters are parameters to the starting test, and they are converted into parameter XML. Then the starting test is executed by starting a Saxon transformation using the parameter XML as input and the XSL file that was generated for the starting test as the stylesheet.

The XSL code makes calls to java extension functions. These include a method for calling a subtest, which works by performing another Saxon transformation using the XSL that was generated for the subtest. The transformations write output to the console and generate log files as they progress.

Writing a simple Hello World Test Script

The following is a simple Hello World CTL test script.

hello.ctl
<ctl:test
xmlns:ctl="http://www.occamlab.com/ctl"
xmlns:my="urn://mynamespace"
name="my:hello"
>
<ctl:assertion>This test displays a greeting.</ctl:assertion>
<ctl:code>
<ctl:message>Hello, world!</ctl:message>
</ctl:code>
</ctl:test>

This is one of the simplest test scripts that can be written. It contains a single test. Tests are identified by a namespace qualified name attribute. In this case, we have made up our own namespace "urn://mynamespace" identified by the prefix "my" which we are using in the test name. A test has an assertion, which is a statement that is true if what is being tested is compliant. It also has code that determines whether the assertion is true or false. This test isn't really testing anything. The code merely displays a greeting on the console, and the assertion just describes what the code does.

To execute the test, use the TEAM Engine test utility.

C:\teamengine2.0>bin\test -source=hello.ctl -test=my:hello 

You will get the following message

No working directory supplied.  Using C:\DOCUME~1\morrich\LOCALS~1\Temp\te_work 
Testing my:hello (s0001)...
Assertion: This test displays a greeting.
Hello, world!
Test my:hello Passed

TEAM Engine starts by processing the source file, converting the CTL source code into XSL code that it will execute. The XSL files it produces are stored in a work directory. We could have told TEAM Engine what directory to use for these files using the -workdir option. Since we didn't, it created its own work directory. This directory and the files in it are not removed. TEAM Engine will reuse the files instead of recreating them if we execute the same source file again, unless the source file has been modified.

TEAM Engine displays the test name and the assertion, and then executes the test code. In this case, the code just displays the message "Hello, world!" Then TEAM Engine tells us whether the test passed or failed. In this case, the test will always pass since we did not provide for any failure conditions in the code.

A Real Test Script

Let's try another script that actually tests something.

W3Schools' XML Tutorial has a simple XML example of a note.

http://www.w3schools.com/xml/note.xml
<?xml version="1.0" ?> 
<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>

Suppose we want to test this note. We want to make sure that it has a heading that contains more than just whitespace characters. We also want to make sure that it has from and to fields that identify known users. Here is a test script.

note.ctl
<ctl:package
xmlns:ctl="http://www.occamlab.com/ctl"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:note="urn://notetest"
>
<ctl:test name="note:main">
<ctl:assertion>The note is valid.</ctl:assertion>
<ctl:code>
<xsl:variable name="response">
<ctl:request>
<ctl:url>http://www.w3schools.com/xml/note.xml</ctl:url>
</ctl:request>
</xsl:variable>
<xsl:choose>
<xsl:when test="$response/note">
<ctl:call-test name="note:check-heading">
<ctl:with-param name="heading" select="$response/note/heading"/>
</ctl:call-test>
<ctl:call-test name="note:check-user">
<ctl:with-param name="user" select="$response/note/to" label="The to user"/>
</ctl:call-test>
<ctl:call-test name="note:check-user">
<ctl:with-param name="user" select="$response/note/from" label="The from user"/>
</ctl:call-test>
</xsl:when>
<xsl:otherwise>
<ctl:message>Failed to retrieve the note.</ctl:message>
<ctl:fail/>
</xsl:otherwise>
</xsl:choose>
</ctl:code>
</ctl:test>

<ctl:test name="note:check-heading">
<ctl:param name="heading"/>
<ctl:assertion>The heading contains more than whitespace.</ctl:assertion>
<ctl:code>
<xsl:if test="normalize-space($heading)=''">
<ctl:fail/>
</xsl:if>
</ctl:code>
</ctl:test>

<ctl:test name="note:check-user">
<ctl:param name="user"/>
<ctl:assertion>{$user} is valid.</ctl:assertion>
<ctl:code>
<xsl:choose>
<xsl:when test="$user='Tove'"/>
<xsl:when test="$user='Jim'"/>
<xsl:when test="$user='Jan'"/>
<xsl:otherwise>
<ctl:fail/>
</xsl:otherwise>
</xsl:choose>
</ctl:code>
</ctl:test>
</ctl:package>

There is still just one starting test called note:main, but it calls several subtests. Because there are several tests in the same file, they are grouped inside a package element.

The request instruction retrieves the note XML. The note contents are assigned to the variable named response, that encloses the request instruction.

If the root node in the response variable is a note element as it should be, then we will call the subtests that check various aspects of the note. Otherwise, we display an error message and use the fail instruction so that this test fails.

Our check-heading test receives the heading as a parameter. It uses the XPath normalize-space function to condense whitespace in the header's text content. If the header contains nothing but whitespace so that it normalizes down to an empty string, we use the fail instruction so the test fails.

The check-user test is used for both the from and the to elements in our note. It receives the user as a parameter. The assertion text contains a parameter name substitution code for the user parameter. When the test is executed and the assertion text is displayed, the value of the label attribute supplied with the user parameter replaces "{$user}". The code checks to see if the user is Tove, Jim, or Jan. If not, it falls through to the fail instruction so the test fails.

Now, let's execute the main test.

C:\teamengine2.0>bin\test -source=note.ctl -test=note:main

No working directory supplied. Using C:\DOCUME~1\morrich\LOCALS~1\Temp\te_work
Testing note:main (s0001)...
Assertion: The note is valid.
Testing note:check-heading (s0001/d1e23_1)...
Assertion: The heading contains more than whitespace.
Test note:check-heading Passed
Testing note:check-user (s0001/d1e28_1)...
Assertion: The to user is valid.
Test note:check-user Passed
Testing note:check-user (s0001/d1e33_1)...
Assertion: The from user is valid.
Test note:check-user Failed
Test note:main Failed (Inherited failure)

The second call to check-user fails, since in our test "Jani" is not one of the names we recognize. Because this test fails, the parent test (main) also fails. When a test fails because a child test fails, it is an inherited failure.

Specifying Parameters on the Command Line

Suppose we want to check our own note rather than always checking the note on the w3schools site. We can create our own note, with a valid "from" user. Then with a slight modification to our code, we can check it by passing the URL for our note to our test as a parameter.

Here is our new note.

note2.xml
<?xml version="1.0" ?> 
<note>
<to>Tove</to>
<from>Jan</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>

 

Copy note.ctl to note2.ctl, and modify the main test to use a url parameter.

main test in note2.ctl
...
<ctl:test name="note:main">
<ctl:param name="url"/>
<ctl:assertion>The note is valid.</ctl:assertion>
<ctl:code>
<xsl:variable name="response">
<ctl:request>
<ctl:url>
<xsl:value-of select="$url"/>
</ctl:url>
</ctl:request>
</xsl:variable>
<xsl:choose>
<xsl:when test="$response/note">
<ctl:call-test name="note:check-heading">
<ctl:with-param name="heading" select="$response/note/heading"/>
</ctl:call-test>
<ctl:call-test name="note:check-user">
<ctl:with-param name="user" select="$response/note/to" label="The to user"/>
</ctl:call-test>
<ctl:call-test name="note:check-user">
<ctl:with-param name="user" select="$response/note/from" label="The from user"/>
</ctl:call-test>
</xsl:when>
<xsl:otherwise>
<ctl:message>Failed to retrieve the note.</ctl:message>
<ctl:fail/>
</xsl:otherwise>
</xsl:choose>
</ctl:code>
</ctl:test>
...

 

To execute this new test, specify the url parameter on the command line.

C:\teamengine2.0>bin\test -source=note2.ctl -test=note:main @url=file:///c:/teamengine2.0/note2.xml

This time, all of the tests should pass.

Supply a Starting Form

Specifying parameters on the command line may be fine for developers, but it is somewhat tricky for end users. To help them along, we can create a formal test suite element that includes a starting form.

suite.ctl
<ctl:suite
xmlns:ctl="http://www.occamlab.com/ctl"
xmlns:xhtml="http://www.w3.org/1999/xhtml"
xmlns:note="urn://notetest"
name="note:suite"
>
<ctl:title>Note test suite</ctl:title>
<ctl:description>Validates notes.</ctl:description>
<ctl:starting-test>note:main</ctl:starting-test>
<ctl:form>
<xhtml:p>Enter the URL of the note that you want to validate:</xhtml:p>
<xhtml:input type="text" name="url" size="100"/>
<xhtml:p/>
<xhtml:input type="submit" value="OK"/>
</ctl:form>
</ctl:suite>

The suite element formalizes our set of tests by giving them a title and description. It identifies the starting test to execute, and provides a form that is used to prompt the user for the values of the parameters of the test. The form contains XHTML elements, including <xhtml:input type="text"/> (text box) elements. There is an input element with a matching name attribute for each parameter in starting-test. The form also has a <xhtml:input type="submit"/> (Submit button) element so the user can submit the form.

We could have put the suite element inside the package element in the same file that contains our tests, but we need to know how to work with multiple source files anyway. We can start the test suite by supplying two -source parameters.

C:\teamengine2.0>bin\test -source=suite.ctl -source=note2.ctl

Alternatively, we could put suite.ctl and note2.ctl into their own directory (we'll call it notedir), and set the -source parameter to the directory name.

C:\teamengine2.0>bin\test -source=notedir

With the second form, TEAM Engine reads all .ctl and .xml files in notedir as source files.

 

In either case, TEAM Engine should find our suite element, display the form, and call the starting test using the input we enter as parameter values. This works because there is only one suite element in our source files. If there were more than one suite, then we would also have to supply a -suite option to identify the one that we want to execute.