testing concurrency with SoapUI

Testing concurrency in SoapUI is easy, if you know how – just like everything else in life. 🙂 There are several approaches that you can take.

First, let’s define what “concurrency” is. Concurrency, also called a “race condition”, is when multiple requests arrive at the server at the same time, but they each need to be handled separately. A simple example: let’s say that you have $100 in your bank account and you send several requests to withdraw all the money – obviously only one of the requests should succeed and the others should fail due to insufficient funds. If more than one of these requests succeeds, you found a race condition. The cause of concurrency is often improper database locking.

SoapUI has the ability to send requests in parallel at the project level (parallel test suites) or at the test suite level (parallel test cases). I will discuss several possible approaches based on different situations.

Note that in the below discussion in each case we are talking about only one test! However, when I use the terms “test case” and “test suite” I am talking about the level of hierarchy in a SoapUI project.

Case 1: the straightforward suite

In the simplest case you have a situation where only one of multiple requests should succeed, such as the case of withdrawing all the money from one bank account. This case assumes there is no setup or cleanup of any kind.

Start with creating a new test suite in SoapUI and setting the run mode to parallel.

Next create the first test case to make one transfer. Do not add any assertions that verify success / failure of the transfer itself – you are expecting only one to succeed and you do not know which one that is going to be. Since timing is important in a concurrency test, keep this lean. Normally I have just a single SOAP Response assertion, or a Valid HTTP Status Codes assertion (if dealing with a REST request).

Clone the test case (using F9) multiple times. As will be seen later, it is a good idea to keep the names of the test cases simple; something like: tranfer0, transfer1, transfer2, etc.

Lastly, you need to handle the verification in the test suite TearDown Script. The exact script will depend on what response you are going to get and what you are looking for. Let’s say that a failed transfer will respond with a SOAP fault, then your verification script could look like the following.

import com.eviware.soapui.support.XmlHolder
def goodTransfer = 0
testSuite.testCases.each {
	def response = new XmlHolder(it.getTestStepByName("transfer").getProperty("Response").value)
	if(response.getDomNode("//*:Fault") == null)
		goodTransfer++
}
assert goodTransfer == 1 : "Too many transfers!"

You will have to adjust the script for your particular situation, but the idea is simple. Read the response of each of the steps (within the test cases in the SoapUI hierarchy) and count the number of successes – there had better be only one.

Case 2: login first

What if you first need to login or generate some other setup?

The only difference from case 1 is the setup – the login. Start with creating everything from case 1.

Next add a test case for the login. It does not matter where in the order of tests you place it (everything runs in parallel anyway), but it helps visually if you place it first. Make sure this test runs correctly and transfers any credentials you may need to your other steps.

Next comes the secret sauce. Disable this test case; this test must be run from the test suite Setup Script with:

testSuite.getTestCaseByName("login").run((com.eviware.soapui.support.types.StringToObjectMap)context, false)

The login could get more complicated if your application requires that HTTP session be maintained. The exact solution to your specific situation is left as an exercise for the reader. 🙂

Case 3a: verify afterwards (at project level)

What if, similarly to case 2, you need to run some sort of cleanup after everything has run. For example: deposit money back so that next run of the same test will work again.

You could take the same approach as in case 2: create a test case at the end of your test suite, disable it, and run it from the suite TearDown Script. However, any tests that are run this way will not get considered when determining if the suite failed or not (the final exit status of the test runner) and will not have any logs created!

In order to work around this, you could set up your entire test as a new SoapUI project. In the project, you will have three test suites: setup, transfers, and cleanup. All test suites are run sequentially, nothing gets disabled, nothing gets run from Setup or TearDown. The setup portion will have only one test case similar to the login from case 2. The transfers portion will have multiple test cases such as all the transfers from case 1 (possibly minus the verification), all run in parallel. The cleanup portion will have one test case like the deposit and any cleanup (and possibly the verification).

Practical example of this situation is that instead of withdrawing all the money from your account, you send in multiple deposits and withdrawals for smaller amounts, and then afterwards the transaction history (a separate API request?) has to show the correct running balance.

The exact implementation of all of this should be pretty straightforward if you got through all the discussion so far.

However, if you are running in a continuous integration system, it is undesirable to have a separate SoapUI project for every concurrency test.

Case 3b: verify afterwards (at test suite level)

You can have a test case wait until other test cases have completed. The magic sauce here is the SoapUI monitor.

Start by creating everything you have done in case 2.

Now create your verification test case. Just as before, it does not matter where in the test suite you place it (everything runs in parallel), but visually it makes more sense to place it at the end. This test does not get disabled, since this is where the verification will take place – you need this test to be considered in the final runner exist status and you want all the logs generated from this step.

In the test suite Setup Script initialize the monitor.

def monitor = new com.eviware.soapui.monitor.TestMonitor()
testSuite.testCases.each {
	testCase = it.value
	if(testCase.label.contains("transfer")) {
		log.info("monitoring: ${testCase.label}")
		monitor.monitorTestCase(testCase)
	}
}
runner.runContext.testMonitor = monitor

If you are less careful about how you name your test cases, the if statement above may become slightly more complicated?

	if(testCase.label.contains("withdrawal") || testCase.label.contains("deposit"))

In the last test case, the verification one, create a new Groovy Script step which will wait for everything else to complete. This step has to be first!

def monitor = context["#TestSuiteRunner#"].runContext.testMonitor
if (monitor == null) {
	log.warn("monitor not found in completion check")
} else {
	while(monitor.hasRunningTests()) {
		log.info("waiting for tests to complete ...")
		try {
			Thread.sleep(100)
		} catch(InterruptedException ignored) { }
	}
	log.info("Tests completed: ${!monitor.hasRunningTests()}")
}