Tests are very important part of Tigase server development process.
Each release goes through fully automated testing process. All server functions are considered implemented only when they pass the testing cycle. Tigase test suite is used for all our automatic tests which allows to define different test scenarios.
There is no tweaking on databases for tests. All databases are installed in a standard way and run with default settings. Databases are cleared each time before the test cycle starts.
There are no modifications needed to be made to Tigase’s configuration file as well. All tests are performed on a default configuration generated by the configuration wizards.
The server is tested in all supported environments:
Basic checking to see if all the functions work at correctly. These tests are performed every time the code is sent to source repository.
Version | XMLDB | MySQL | PGSQL | Distributed |
3.3.2-b889 | none | |||
3.3.2-b880 | None | |||
3.0.2-b700 | ||||
2.9.5-b606 | ||||
2.9.3-b548 | ||||
2.9.1-b528 | ||||
2.8.6-b434 | ||||
2.8.5-b422 | ||||
2.8.3-b409 | ||||
2.7.2-b378 | ||||
2.6.4-b300 | ||||
2.6.4-b295 | ||||
2.6.0-b287 | ||||
2.5.0-b279 | ||||
2.4.0-b263 | ||||
2.3.4-b226 | None | None | None |
Checking to see whether the function performs well enough.
Version | XMLDB | MySQL | PGSQL | Distributed |
3.3.2-b889 | none | |||
3.3.2-b880 | None | |||
3.0.2-b700 | ||||
2.9.5-b606 | ||||
2.9.3-b548 | ||||
2.9.1-b528 | ||||
2.8.6-b434 | ||||
2.8.5-b422 | ||||
2.8.3-b409 | ||||
2.7.2-b378 | ||||
2.6.4-b300 | ||||
2.6.4-b295 | ||||
2.6.0-b287 | ||||
2.5.0-b279 | ||||
2.4.0-b263 | ||||
2.3.4-b226 | None | None | None |
Checking to see whether the function behaves well in long term run. It must handle hundreds of requests a second in a several hour server run.
Version | XMLDB | MySQL | PGSQL | Distributed |
2.3.4-b226 | None | None | None |
Tigase Test Suite is an engine which allows you to run tests. Essentially it just executes TestCase implementations. The tests may depend on other tests which means they are executed in specific order. For example authentication test is executed after the stream open test which in turn is executed after network socket connection test.
Each TestCase implementation may have it’s own set of specific parameters. There is a set of common parameters which may be applied to any TestCase. As an example of the common parameter you can take -loop = 10 which specified that the TestCase must be executed 10 times. The test specific parameter might be -user-name = tester which may set the user name for authentication test.
The engine is very generic and allows you to write any kind of tests but for the Tigase projects the current TestCase implementations mimic an XMPP client and are designed to test XMPP servers.
The suite contains a kind of scripting language which allows you to combine test cases into a test scenarios. The test scenario may contain full set of functional tests for example, another test scenario may contain performance tests and so on.
To obtain TTS, you will first need to clone the repository
git clone https://repository.tigase.org/git/tigase-testsuite.git
Once cloning is finished, navigate to the TTS root directory and compile with maven:
mvn clean install
Maven will compile TTS and place jars in the necessary locations. From the same directory, you can begin running TTS using the following command:
./scripts/all-tests-runner.sh
You should see the following, which outlines the possible options to customize your test run
Run selected or all tests for Tigase server ---- Author: Artur Hefczyc <artur_hefczyc@vnu.co.uk> Version: 2.0.0 ---- --help|-h This help message --func [mysql|pgsql|derby|mssql|mongodb] Run all functional tests for a single database configuration --lmem [mysql|pgsql|derby|mssql|mongodb] Run low memory tests for a single database configuration --perf [mysql|pgsql|derby|mssql|mongodb] Run all performance tests for a single database configuration --stab [mysql|pgsql|derby|mssql|mongodb] Run all stability tests for a single database configuration --func-all Run all functional tests for all database configurations --lmem-all Run low memory tests for all database configurations --perf-all Run all performance tests for all database configurations --stab-all Run all stability tests for all database configurations --all-tests Run all functionality and performance tests for database configurations --single test_file.cot --other script_file.xmpt ---- Special parameters only at the beginning of the parameters list --debug|-d Turns on debug mode --skip-db-relad|-no-db Turns off reloading database --skip-server|-no-serv Turns off Tigase server start --small-mem|-sm Run in small memory mode ----------- Other possible parameters are in following order: [server-dir] [server-ip]
You may run the tests from a command line like above, however you may create and edit the /scripts/tests-runner-settings.sh file to fit your Tigase installation and avoid having to have long complex commands as this template shows:
#!/bin/bash func_rep="func-rep.html" perf_rep="perf-rep.html" db_name="tigasetest" db_user="tigase" db_pass="tigase" root_user="root" root_pass="root" TESTS=("derby" "mysql" "pgsql" "mssql") IPS=("127.0.0.1" "127.0.0.1" "127.0.0.1" "127.0.0.1") server_timeout=10 server_dir="/home/tigase/tigase-server" database="derby" #database="mysql" server_ip="127.0.0.1" MS_MEM=100 MX_MEM=1000 SMALL_MS_MEM=10 SMALL_MX_MEM=50
This will allow you to maintain identical settings through multiple runs of TTS. See the next section for learning how the scripting language works and how you can create and run your own custom tests.
The test suite contains scripting language which allows you to combine test cases into a test scenarios. On the lowest level, however the language is designed to allow you to describe the test by setting test parameters, test comments, identification and so on.
Let’s look at the example test description.
Short name@test-id-1;test-id-2: Short description for the test case { -loop = 10 -user-name = Frank # This is a comment which is ignored } >> Long, detailed description of the test case <<
Meaning of all elements:
As for the testing script between open curly brackets { and close one } you can put all the test case parameters you wish. The format for it is:
-parameter-name = value
Parameter names always start with -. Note, some parameters don’t require any value. They can exist on their own without any value assigned:
-debug-on-error
This imitates if you were to put yes or true as the value.
The scripting language includes also support for variables which can be assigned any value and used multiple times later on. You assign a value to the variable the same way as you assign it to the parameter:
$(variable-name) = value
The variable name must be always enclosed with brackets () and start with $.
The value may be enclosed within double quotes "" or double quotes may be omitted. If this is a simple string like a number or character string consisting only of digits, letters, underscore _ and hyphen - then you can omit double quotes otherwise you must enclose the value.
The test case descriptions can be nested inside other test case descriptions. Nested test case descriptions inherit parameters and variables from outer test case description.
You can write tests in a simple text file which is loaded during test suite runtime.
You simply specify what should be send to the server and what response should be expected from the server. No need to write Java code and recompile the whole test suite for new tests. It means new test cases can be now written easily and quickly which hopefully means more detailed tests for the server.
How it works:
Let’s take XEP-0049 Private XML Storage. Looking into the spec we can see the first example:
CLIENT:
<iq type="set" id="1001"> <query xmlns="jabber:iq:private"> <exodus xmlns="exodus:prefs"> <defaultnick>Hamlet</defaultnick> </exodus> </query> </iq>
SERVER:
<iq type="result" id="1001"/>
This is enough for the first simple test. I have to create text file JabberIqPrivate.test
looking like this:
send: { <iq type="set" id="1001"> <query xmlns="jabber:iq:private"> <exodus xmlns="exodus:prefs"> <defaultnick>Hamlet</defaultnick> </exodus> </query> </iq> } expect: { <iq type="result" id="1001"/> }
And now I can execute the test:
testsuite $ ./scripts/all-tests-runner.sh --single JabberIqPrivate.test Tigase server home directory: ../server Version: 2.8.5-b422 Database: xmldb Server IP: 127.0.0.1 Extra parameters: JabberIqPrivate.test Starting Tigase: Tigase running pid=6751 Running: 2.8.5-b422-xmldb test, IP 127.0.0.1... Script name: scripts/single-xmpp-test.xmpt Common test: Common test ... failure! FAILURE, (Received result doesnt match expected result., Expected one of: [<iq id="1001" type="result"/>], received: [<iq id="1001" type="error"> <query xmlns="jabber:iq:private"> <exodus xmlns="exodus:prefs"> <defaultnick>Hamlet</defaultnick> </exodus> </query> <error type="cancel"> <feature-not-implemented xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/> <text xml:lang="en" xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"> Feature not supported yet.</text> </error> </iq>]), Total: 100ms Test time: 00:00:02 Shutting down Tigase: 6751
If I just started working on this XEP and there is no code on the server side, the result is perfectly expected although maybe this is not what we want. After a while of working on the server code I can execute the test once again:
testsuite $ ./scripts/all-tests-runner.sh --single JabberIqPrivate.test Tigase server home directory: ../server Version: 2.8.5-b422 Database: xmldb Server IP: 127.0.0.1 Extra parameters: JabberIqPrivate.test Starting Tigase: Tigase running pid=6984 Running: 2.8.5-b422-xmldb test, IP 127.0.0.1... Script name: scripts/single-xmpp-test.xmpt Common test: Common test ... success, Total: 40ms Test time: 00:00:01 Shutting down Tigase: 6984
This is it. The result we want in a simple and efficient way. We can repeat it as many times we want which is especially important in longer term trials. Every time we change the server code we can re-run tests to make sure we get correct responses from the server.
You can have a look in the current build, with more complete test cases, file for JabberIqPrivate.
Now my server tests are no longer outdated. Of course not all cases are so simple. Some XEPs require calculations to be done before stanza is sent or to compare received results. A good example for this case is user authentication like SASL and even NON-SASL. But still, there are many cases which can be covered by simple tests: roster management, privacy lists management, vCard, private data storage and so on.
There is long list of parameters which can be applied to any test case. Here is the description of all possible parameters which can be used to build test scenarios.
There are test report parameters which must be set in the main script file in order to generate HTML report from the test. These parameters have no effect is they are set inside the test case description.
<body/>
element. And is used to embed test result inside other HTML content.-output-cols = (5 | 7) Only valid values are:
5: "Test name", "Result", "Test time", "Description" [, "History" ] 7: "Test name", "Result", "Total time", "OK", "Average", "Description" [, "History" ]
<title/>
element as well as in the first page header.These parameters can be set on per-test case basis but usually they are set in the main script file to apply them to all test cases.
Test parameters which are normally set on per-test case basis and apply only to the test they are set for and all inherited tests. Some of the parameters though are applied only to inherited test cases. Please look in the description below to find more details.
<message/>
packet. You can set the destination address for the packet. Mind, normally every test expects some response for the data sent so make sure the destination end-point will send back the data expected by the test case.