13. Tests

13.1. Tests

Tests are very important part of Tigase server development process.

Each release goes through fully automated testing process. All server functions are considered implemented only when they pass the testing cycle. Tigase test suite is used for all our automatic tests which allows to define different test scenarios.

There is no tweaking on databases for tests. All databases are installed in a standard way and run with default settings. Databases are cleared each time before the test cycle starts.

There are no modifications needed to be made to Tigase’s configuration file as well. All tests are performed on a default configuration generated by the configuration wizards.

The server is tested in all supported environments:

1. XMLDB - tests with built-in simple XML database. This is a simple and efficient solution for small installations. It is recommended for services with up to 100 user accounts although it has been successfully tested with 10,000 user accounts.

2. MySQL - tests with a MySQL database. Much slower than XMLDB but may handle many more user accounts.

3. PostgreSQL - tests with a PostgreSQL database. Again it is much slower than XMLDB but may handle much more user accounts. This is basically exactly the same code as for MySQL database (SQL Connector) but tests are executed to make sure the code is compatible with all supported SQL databases and to compare performance.

4. Distributed - is a test for distributed installation where c2s and s2s components run on separated machine which connects using external an component protocol (XEP-0114) to another machine with SessionManager running.

13.1.1. Functional Tests

Basic checking to see if all the functions work at correctly. These tests are performed every time the code is sent to source repository.

Version	XMLDB	MySQL	PGSQL	Distributed
3.3.2-b889	00:00:12	00:00:17	00:00:17	none
3.3.2-b880	00:00:13	00:00:15	00:00:15	None
3.0.2-b700	00:00:22	00:00:24	00:00:25	00:00:25
2.9.5-b606	00:00:22	00:00:24	00:00:24	00:00:24
2.9.3-b548	00:00:22	00:00:23	00:00:25	00:00:25
2.9.1-b528	00:00:21	00:00:23	00:00:24	00:00:25
2.8.6-b434	00:00:21	00:00:24	00:00:24	00:00:25
2.8.5-b422	00:00:21	00:00:24	00:00:24	00:00:26
2.8.3-b409	00:00:27	00:00:29	00:00:29	00:00:32
2.7.2-b378	00:00:30	00:00:34	00:00:33	00:00:35
2.6.4-b300	00:00:30	00:00:32	00:00:35	00:00:39
2.6.4-b295	00:00:29	00:00:32	00:00:45	00:00:36
2.6.0-b287	00:00:31	00:00:34	00:00:47	00:00:43
2.5.0-b279	00:00:30	00:00:34	00:00:45	00:00:43
2.4.0-b263	00:00:29	00:00:33	00:00:45	00:00:44
2.3.4-b226	None	00:00:48	None	None

13.1.2. Performance Tests

Checking to see whether the function performs well enough.

Version	XMLDB	MySQL	PGSQL	Distributed
3.3.2-b889	00:12:17	00:13:42	00:17:10	none
3.3.2-b880	00:13:39	00:14:09	00:17:39	None
3.0.2-b700	00:10:26	00:11:00	00:12:08	00:24:05
2.9.5-b606	00:09:54	00:11:18	00:37:08	00:16:20
2.9.3-b548	00:10:00	00:11:29	00:36:43	00:16:47
2.9.1-b528	00:09:46	00:11:15	00:36:12	00:16:36
2.8.6-b434	00:10:02	00:11:45	00:36:36	00:17:36
2.8.5-b422	00:12:37	00:14:40	00:38:59	00:21:40
2.8.3-b409	00:12:32	00:14:26	00:37:57	00:21:26
2.7.2-b378	00:12:28	00:14:57	00:37:09	00:22:20
2.6.4-b300	00:12:46	00:14:59	00:36:56	00:35:00
2.6.4-b295	00:12:23	00:14:59	00:42:24	00:30:18
2.6.0-b287	00:13:50	00:16:53	00:48:17	00:49:06
2.5.0-b279	00:13:29	00:16:58	00:47:15	00:41:52
2.4.0-b263	00:13:20	00:16:21	00:43:56	00:42:08
2.3.4-b226	None	01:23:30	None	None

13.1.3. Stability Tests

Checking to see whether the function behaves well in long term run. It must handle hundreds of requests a second in a several hour server run.

Version	XMLDB	MySQL	PGSQL	Distributed
2.3.4-b226	None	16:06:31	None	None

13.2. Tigase Test Suite

Tigase Test Suite is an engine which allows you to run tests. Essentially it just executes TestCase implementations. The tests may depend on other tests which means they are executed in specific order. For example authentication test is executed after the stream open test which in turn is executed after network socket connection test.

Each TestCase implementation may have it’s own set of specific parameters. There is a set of common parameters which may be applied to any TestCase. As an example of the common parameter you can take -loop = 10 which specified that the TestCase must be executed 10 times. The test specific parameter might be -user-name = tester which may set the user name for authentication test.

The engine is very generic and allows you to write any kind of tests but for the Tigase projects the current TestCase implementations mimic an XMPP client and are designed to test XMPP servers.

The suite contains a kind of scripting language which allows you to combine test cases into a test scenarios. The test scenario may contain full set of functional tests for example, another test scenario may contain performance tests and so on.

13.2.1. Running Tigase Test Suite (TTS)

To obtain TTS, you will first need to clone the repository

git clone https://repository.tigase.org/git/tigase-testsuite.git

Once cloning is finished, navigate to the TTS root directory and compile with maven:

mvn clean install

Maven will compile TTS and place jars in the necessary locations. From the same directory, you can begin running TTS using the following command:

./scripts/all-tests-runner.sh

You should see the following, which outlines the possible options to customize your test run

Run selected or all tests for Tigase server
----
Author: Artur Hefczyc <[email protected]>
Version: 2.0.0
----
  --help|-h This help message
  --func [mysql|pgsql|derby|mssql|mongodb]
              Run all functional tests for a single database configuration
  --lmem [mysql|pgsql|derby|mssql|mongodb]
              Run low memory tests for a single database configuration
  --perf [mysql|pgsql|derby|mssql|mongodb]
              Run all performance tests for a single database configuration
  --stab [mysql|pgsql|derby|mssql|mongodb]
              Run all stability tests for a single database
              configuration
  --func-all  Run all functional tests for all database
              configurations
  --lmem-all  Run low memory tests for all database
              configurations
  --perf-all  Run all performance tests for all database
              configurations
  --stab-all  Run all stability tests for all database
              configurations
  --all-tests Run all functionality and performance tests for
              database configurations
  --single test_file.cot
  --other script_file.xmpt
----
  Special parameters only at the beginning of the parameters list
  --debug|-d                 Turns on debug mode
  --skip-db-relad|-no-db     Turns off reloading database
  --skip-server|-no-serv     Turns off Tigase server start
  --small-mem|-sm            Run in small memory mode
-----------
  Other possible parameters are in following order:
  [server-dir] [server-ip]

13.2.2. Customizing Tigase Test Suite

You may run the tests from a command line like above, however you may create and edit the /scripts/tests-runner-settings.sh file to fit your Tigase installation and avoid having to have long complex commands as this template shows:

#!/bin/bash

func_rep="func-rep.html"
perf_rep="perf-rep.html"
db_name="tigasetest"
db_user="tigase"
db_pass="tigase"
root_user="root"
root_pass="root"

TESTS=("derby" "mysql" "pgsql" "mssql")
IPS=("127.0.0.1" "127.0.0.1" "127.0.0.1" "127.0.0.1")

server_timeout=10

server_dir="/home/tigase/tigase-server"
database="derby"
#database="mysql"
server_ip="127.0.0.1"

MS_MEM=100
MX_MEM=1000

SMALL_MS_MEM=10
SMALL_MX_MEM=50

This will allow you to maintain identical settings through multiple runs of TTS. See the next section for learning how the scripting language works and how you can create and run your own custom tests.

13.3. Test Suite Scripting Language

The test suite contains scripting language which allows you to combine test cases into a test scenarios. On the lowest level, however the language is designed to allow you to describe the test by setting test parameters, test comments, identification and so on.

Let’s look at the example test description.

Short name@test-id-1;test-id-2: Short description for the test case
{
 -loop = 10
 -user-name = Frank
 # This is a comment which is ignored
}
>> Long, detailed description of the test case <<

Meaning of all elements:

1. Short name is any descriptive name you want. It doesn’t need to be unique, just something which tells you what this test is about. @ is a separator between the short name and the test ids.

2. test-id-1;test-id-2 is a semicolon separated of the test cases IDs. The tests cases are executed in the listed order. And listing them there means that the test-id-2 depends on test-id-1. Normally you don’t have to list all the dependencies because all mandatory dependencies are included automatically. Which means if you have an authentication test case the suite adds the network socket connection and stream opening tests automatically. Sometimes however, there are dependencies which are optional or multiple mandatory dependencies and you need to select which one has to be executed. As a good example is the authentications test case. There are many authentication tests: PLAIN-AUTH, SASL-DIGESTMD5, SASL-PLAIN, DIGEST-AUTH and they are all mandatory for most of other tests like roster, presence and so on. One of the authentication tests is a default dependency but if you put on the list different authentication it will be used instead of default one.

3. : is a separator between test cases ids list and the short test description.

4. Short test description is placed between : - colon and opening { - curly bracket. This is usually quite brief, single line test description.

5. { } curly brackets contain all the test parameters, like how many times the test has to be executed or run the test in a separate thread, user name, host IP address for the network connection and many others.

6. >> << inside the double greater than and double less than you put a very long, multiple line test description.

As for the testing script between open curly brackets { and close one } you can put all the test case parameters you wish. The format for it is:

-parameter-name = value

Parameter names always start with -. Note, some parameters don’t require any value. They can exist on their own without any value assigned:

-debug-on-error

This imitates if you were to put yes or true as the value.

The scripting language includes also support for variables which can be assigned any value and used multiple times later on. You assign a value to the variable the same way as you assign it to the parameter:

$(variable-name) = value

The variable name must be always enclosed with brackets () and start with $.

The value may be enclosed within double quotes “” or double quotes may be omitted. If this is a simple string like a number or character string consisting only of digits, letters, underscore _ and hyphen - then you can omit double quotes otherwise you must enclose the value.

The test case descriptions can be nested inside other test case descriptions. Nested test case descriptions inherit parameters and variables from outer test case description.

13.4. Writing Tests for Plugins

You can write tests in a simple text file which is loaded during test suite runtime.

You simply specify what should be send to the server and what response should be expected from the server. No need to write Java code and recompile the whole test suite for new tests. It means new test cases can be now written easily and quickly which hopefully means more detailed tests for the server.

How it works:

Let’s take XEP-0049 Private XML Storage. Looking into the spec we can see the first example:

13.4.1. Example: Client Stores Private Data

CLIENT:

<iq type="set" id="1001">
  <query xmlns="jabber:iq:private">
    <exodus xmlns="exodus:prefs">
      <defaultnick>Hamlet</defaultnick>
    </exodus>
  </query>
</iq>

SERVER:

<iq type="result" id="1001"/>

This is enough for the first simple test. I have to create text file JabberIqPrivate.test looking like this:

send: {

<iq type="set" id="1001">
  <query xmlns="jabber:iq:private">
    <exodus xmlns="exodus:prefs">
      <defaultnick>Hamlet</defaultnick>
    </exodus>
  </query>
</iq>
}

expect: {
<iq type="result" id="1001"/>
}

And now I can execute the test:

testsuite $ ./scripts/all-tests-runner.sh --single JabberIqPrivate.test

Tigase server home directory: ../server
Version: 2.8.5-b422
Database:         xmldb
Server IP:        127.0.0.1
Extra parameters: JabberIqPrivate.test
Starting Tigase:
Tigase running pid=6751

Running: 2.8.5-b422-xmldb test, IP 127.0.0.1...
Script name: scripts/single-xmpp-test.xmpt
Common test:  Common test  ...        failure!
FAILURE,  (Received result doesnt match expected result.,
Expected one of: [<iq id="1001" type="result"/>],
received:
[<iq id="1001" type="error">
  <query xmlns="jabber:iq:private">
    <exodus xmlns="exodus:prefs">
      <defaultnick>Hamlet</defaultnick>
    </exodus>
  </query>
  <error type="cancel">
    <feature-not-implemented xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/>
    <text xml:lang="en" xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">
      Feature not supported yet.</text>
  </error>
</iq>]),

Total: 100ms
Test time: 00:00:02
Shutting down Tigase: 6751

If I just started working on this XEP and there is no code on the server side, the result is perfectly expected although maybe this is not what we want. After a while of working on the server code I can execute the test once again:

testsuite $ ./scripts/all-tests-runner.sh --single JabberIqPrivate.test

Tigase server home directory: ../server

Version: 2.8.5-b422

Database:         xmldb

Server IP:        127.0.0.1

Extra parameters: JabberIqPrivate.test

Starting Tigase:

Tigase running pid=6984

Running: 2.8.5-b422-xmldb test, IP 127.0.0.1...

Script name: scripts/single-xmpp-test.xmpt

Common test:  Common test  ... success,  Total: 40ms

Test time: 00:00:01

Shutting down Tigase: 6984

This is it. The result we want in a simple and efficient way. We can repeat it as many times we want which is especially important in longer term trials. Every time we change the server code we can re-run tests to make sure we get correct responses from the server.

You can have a look in the current build, with more complete test cases, file for JabberIqPrivate.

Now my server tests are no longer outdated. Of course not all cases are so simple. Some XEPs require calculations to be done before stanza is sent or to compare received results. A good example for this case is user authentication like SASL and even NON-SASL. But still, there are many cases which can be covered by simple tests: roster management, privacy lists management, vCard, private data storage and so on.

13.5. Test Case Parameters Description

There is long list of parameters which can be applied to any test case. Here is the description of all possible parameters which can be used to build test scenarios.

There is long list of parameters which can be applied to any test case. Here is the description of all possible parameters which can be used to build test scenarios.

13.5.1. Test Report Configuration

There are test report parameters which must be set in the main script file in order to generate HTML report from the test. These parameters have no effect is they are set inside the test case description.

1. -version = 2.0.0 sets the test script version. This is used to easily detect incompatibility issues when the test suite loads a script created for more recent version of the suite and may not work properly for this version.

2. -output-format = (html | html-content) sets the output format for the test report. There is actually only one format possible right now - HTML. The only difference between these 2 options is that the html format creates full HTML page with HTML header and body. The html-content format on the other hand creates only what is inside <body/> element. And is used to embed test result inside other HTML content.

3. -output-file = “report-file.html” sets the file name for the test report.

4. -output-history = (yes | no) sets logging of the all protocol data sent between test suite and the XMPP server. Normally for functional tests it is recommended to set it to yes but for all other tests like performance or load tests it should be set to no.

5. -history-format = separate-file sets protocol data logging to a separate file. Currently this is the only possible option.

6. -output-cols = (5 | 7) Only valid values are:

   5: "Test name", "Result", "Test time", "Description" [, "History" ]
   7: "Test name", "Result", "Total time", "OK", "Average", "Description" [, "History" ]
   

7. -title = “The title of the report page” This parameter sets the test report title which is placed in the HTML page in the <title/> element as well as in the first page header.

13.5.2. Basic Test Parameters

These parameters can be set on per-test case basis but usually they are set in the main script file to apply them to all test cases.

1. -base-ns = “jabber:client” sets the XML name space used for the XML stream in the XMPP connection. Some test cases can be used to test client to server protocol as well as server to server protocol and possibly different protocols added in the future.

2. -debug switches debugging mode on. All the communication between the test suite and the server is printed out to the text console and all other debugging information including java exceptions are displayed as well. It is especially useful when some test fails and you want to find out why.

3. -debug-on-error switches on debugging mode on error detection. Normally debug output generates lots of message which makes the output very hard to read. Especially in the performance tests not only you can read fast scrolling lines of the protocol data but also it slows the test down. This option however turns debugging off if everything is working well and then generates debug output if any test error us detected.

4. -def-auth = (auth-plain | auth-digest | auth-sasl) sets the default authentication method for the user connection.

5. -def-stream = (stream-client | stream-server | stream-component | stream-bosh) sets the connection stream to be tested and the name space for the connection.

6. -host = “host.name” the vhost name the tested server runs for. It may be the real DNS name or just configured for testing purposes hostname. It must match however the server configuration.

7. -keys-file = “certs/keystore” sets the location of the keys store file. No need to touch it.

8. -keys-file-password = keystore sets the password for the keystore file. Normally you don’t have to touch it.

9. -serverip = “127.0.0.1” defines the XMPP server IP address. You may omit this parameter and then the IP address will be determined automatically based on the server DNS address. However if the DNS address can not be correctly resolved or if you run tests on the localhost you can use this parameter to enforce the IP address.

10. -socket-wait = 10000 sets the network socket timeout in milliseconds that is maximum time the test suite will wait for the response from the server. You may want to increase the timeout for some specific tests which require lots of computation or database activity on the server. Normally 10 seconds is enough for most cases.

11. -stop-on-fail = true causes the script to terminate all actions on the first failed test case. It helps diagnosing the server state at the failure point.

12. -trust-file = “certs/client_truststore” sets the file name for the client trust store file. No need to change it.

13. -trust-file-password = truststore sets the password for the trust store file. Normally you don’t have to touch it.

14. -user-name = tester sets the user name used for the XMPP connections between the test suite and the XMPP server. It is usually set globally the same for all tests and for some tests like receiving the server configuration you may want to use a different account (with admin permissions). Then you can set a different user for this specific test case.

15. -user-pass = tester-password sets the password for the user used for the XMPP connection between the test suite and the XMPP server.

16. -user-resr = resource sets the user JID resource part for the XMPP connection between the test suite and the XMPP server.

13.5.3. Test Case Parameters

Test parameters which are normally set on per-test case basis and apply only to the test they are set for and all inherited tests. Some of the parameters though are applied only to inherited test cases. Please look in the description below to find more details.

1. -active-connection is a similar parameter to -on-one-socket option. If set the suite doesn’t close the network socket and if the test is run in loop each loop run re-uses the network connection. Unlike in the -on-one-socket mode the whole test is executed on each run including XMPP stream initialization and user authentication. This option is currently not recommended in a normal use. It is useful only to debug the server behavior in very special use cases.

2. -background executes the test in a separate thread in background and immediately returns control to the test suite program without waiting for the test to complete. Default behavior is to execute all tests sequentially and run next test when previous one has been completed. This parameter however allows to run tests concurrently. This a bit similar option to the -daemon parameter. The daemon test/task however is ignored completely and results from the daemon are not collected where the background test is a normal test which is run concurrently with another one or possibly many other tests.

3. -daemon creates a task running in background in a separate thread. Such a test runs infinitely as a daemon, it is not recorded in the test report and it’s result is not calculated. The purpose of such test/task is to work as a helper for other test cases. A good example of such daemon test is message responder - the test which runs under a different user name and waits for messages and responding to the sender.

4. -delay = 1000 sets the waiting time in milliseconds after the test case is completed. You may use it if you want to introduce short delay between each test cases run in the loop or if you start the helper daemon thread and you have to add the delay to make sure it is ready to work before next real test starts sending requests to the daemon.

5. -expect-type = error sets the type for a packet expected as a response. Some test cases like message sender expects sometimes response with the same type it has sent the packet ( chat ) but in some other cases when it sends a message to a user who has privacy lists set to block messages the response should be with an error. This way we can use the same test cases for testing different responses scenarios.

6. -loop = 10 sets the number of times the test (and all inherited tests) are repeated. You can use a $(loop) pseudo-variable to obtain and use the current loop run number. This is useful if you want to run every loop run for a different user name like registering 10 different user accounts. To do this you stick the $(loop) variable to the user name string: -user-name = “nick_name_$(loop)”.

7. -loop-delay = 10 sets a delay in milliseconds between each individual loop run for the tests which is run multiple times. This is similar parameter to the -delay one but the -delay option introduces a delay after the whole test (or all loop runs) has been completed. The loop delay options adds waiting time between each run of the looped test.

8. -loop-start = 5 sets the loop starting value. It doesn’t affect number of loop runs in a any way. It only affects the value of the $(loop) variable. Let’s say you want to run a load test for the server with 100k concurrent users and you want to run the test from 3 different machines. To make sure each machine uses distinct user accounts you have to set a different -loop-start parameter on each to prevent from overlapping.

9. -messages = 10 sets the number of messages to send to the server. This is another way of looping the test. Instead of repeating the whole test with opening network connection, XMPP stream, authentication and so on it causes only to send the message this many times. This parameters is accepted by some test cases only which send messages. For the messages listeners - test cases which is supposed to respond to the messages the number set here specifies how many times the the response must be sent before the test successfully terminates it’s work.

10. -multi-thread option causes to run the test case and all inherited in all levels test cases in separate threads. Normally the test case where you put the parameter doesn’t have a test ID (what you put between ‘@’ and ‘:’ characters so it doesn’t run a test on it’s own. Instead it contains a series of test cases inside which are then run in a separate thread each. This is a key parameter to run tests for many concurrent users. (Not a load tests though.) For example you can see whether the server behaves correctly when 5 simultaneous modifies their roster. The execution time all inherited tests run in a separate threads is added together and also results from each individual test is calculated and added to the total main test results.

11. -no-record is used for kind of configuration tests (tasks) which are used to prepare the XMPP server or database for later tests. As an example can be creation of the test user account which is later on used for the roster tests. Usually you don’t want to include such tests in the test report and using this parameter you essentially exclude the test from the report. The test and the result however shows in the command line output so you can still track what is really going on.

12. -on-one-socket is a modifier for a looped test case. Normally when we switch looping on using -loop parameter the suite resets the state, closes the network socket and runs the test from the very beginning including opening network socket, XMPP stream, authentication and so on. This parameter however changes this behavior. The network socket is not closed when the test run is completed (successfully) and next run executes only the last part of the test omitting the XMPP stream initialization, authentication and all others but last. This is useful when you want to send many messages to the server (although this effect may be accomplished using -messages parameter as well) or registering many user accounts on the server, unregistering user accounts and any other which might make sense repeating many times.

13. -port = 5223 this parameter is similar to the IP address setting and can be also set globally for all tests. Normally however you set it for a selected tests only to check SSL connection. For all other tests default port number is used. Therefore this parameters has been included in this section instead of “Basic test parameters”.

14. -presence this parameter enables sending initial presence with positive priority after connection and binding the session.

15. -repeat-script = 100 and -repeat-wait = 10 are 2 parameters are specific to the common test cases. (The test cases which reads the test input/output data from the pseudo-xml text file. The first parameter is another variation of test looping. It sets how many times the test has to be repeated. It works very much like the -on-one-socket parameter. The only difference is that the common test can preserve some internal states between runs and therefore it has more control over the data. The second parameter sets the timeout in milliseconds to wait/delay between each individual test run and it is a very similar parameter to the -delay one but it sets a timeout inside the common test instead.

16. -source-file = “dir/path/to/file.cot” is a parameter to set the “common test” script file. The common test is a test cases which depends on the authentication test case and can read data to send and responses to expect from the text file. The “cot” file is a pseudo-xml file with stanzas to send and stanzas to expect. The the test cases compares the received packets with those in the text file and reports the test result. This is usually a more convenient way to write a new test cases than coding them in Java.

17. -time-out-ok is set for a test case when we expect socket timeout as a correct result from the test case. Normally the timeout means that the test failed and there was no response from the server at all or the response was incorrect. For some tests however (like sending a message to the user who is blocking messages through privacy lists) the timeout is the desired correct test result.

18. -to-jid = “user_name@host.name“ sets the destination address for packets sending packets somewhere. As an example is the test case sending <message/> packet. You can set the destination address for the packet. Mind, normally every test expects some response for the data sent so make sure the destination end-point will send back the data expected by the test case.