Thursday, December 15, 2011

Database integration with SOAP UI - in way over my head...

Today I embarked on a journey to create that real database integration test I always wanted.
Many times in the past I have created tests for applications that use a complex database structure to produce some results. Every time I have resorted to setting up a static data structure and then make my tests agains that collection of data.

The problem with my old approach is that it is not very stable. What if someone tampers with your data? What if you want to move the tests to a UAT environment, or worse yet - production?

The ideal scenario, I think, is to have the test
1. Set all the necessary data up
2. Do the test
3. Clean up the data  

But that sounds a lot simpler than it is.

Setting the data up
Using Soap UI I created a bunch of JDBC test steps to create every tiny bit of data that I needed. I put all of these steps in a disabled test case that I trigger manually from the actual cases.
I divided the data into two categories

  1. Stable data, or "master data"
    This data will most probably exist in all environments, but not necessarily with the same ids.
    Therefore a needed a bunch of "find XXX" test steps. I then transferred all those values into properties I could access later. 
  2. Dynamic data
    This is the data I will perform my actual tests against. Its important that I have full control over this data to create valid tests and to be able to trust the results. For this I created a bunch of JDBC steps that insert data into different tables.
One thing I discovered is that having GUIDs as primary keys, as opposed to having auto-ids, makes testing easier. I created some Groovy test steps that produced new GUIDs  for all new entities. I stored these values as properties for later access.

As you can see on the right - there were a whole bunch of steps to make this happen...

Testing against the data
This is the easy part. Since all values I needed were transferred into properties I could use them in my actual tests. I actually put the setup data test case in a disabled test suite, and that is quite useful. In my tests I don't want to care about where the data comes from or how it is created. I only want to be provided with some values that will make my tests do what I want them to. Properties in Soap UI is your friend here.

Cleaning up
To clean up I created another test case in my disabled suite that took all of the generated Ids and deleted them from the database. No trace whatsoever! But... the infrastructure to do this is quite tiresome. Read on...

Maintaining tests
I wanted to create my data once and then do many tests against it. In fact, I wanted to create some base data that would be available for all tests and then some per-test data. The per-test data could be setup using a test step in my test. It fits nicely under a GIVEN step. (See my post on BDD in Soap UI) But the clean up fits nowhere in the test. Soap UI gives you Setup and TearDown possibilities on many levels so for this I put a TearDown step on my test case.

I created a disabled test case that took care of the cleaning and I made a Groovy script call to execute it.





While this is nice and dandy for the per-test data, the global data was still an issue. I could put a setup and teardown script on the test suite level, but that requires anyone that uses these tests to never run any case in isolation. That would mean that global data would not be available or linger on long after the test was finished. 

So, I put all of the data creation and cleaning on all the test cases. That did nothing good to my performance... What I am considering now is to have some sort of flagging for the setup so that a test case will not set global data up if it has already been setup.

BUT... I think flags like "IsSetup" is kind of a smell. So, here I am - I have stable tests that perform crappy. Do I care? Well, if the trade-off is between performance and stability, I choose stability any day. But I would really like to find a better way of doing this. Maybe it is not supposed to be done at all? Maybe I am grasping for test Utopia?

With those indecisive words, I bid you good night. 

Ps. Any other suggestions on how to do these kind of tests, or motivations on why I shouldn't at all, are appreciated. Ds.

Thursday, December 1, 2011

Going end-to-end with Soap UI

One cool thing we managed to do with Soap UI is create some real end-to-end tests. Many times you might test almost an entire feature. It simply takes to long to test it all, and you settle for some manual testing of those final things.
One example we had was a web service that, when called, would change the status of something in the database and then publish a file on a network share. Testing the database change is simple but the file part was too hard, or at least I thought it was...

My scenario looked something like this:

GIVEN an existing order
WHEN the order status is changed to "Processed"
THEN the order changes status to "Processed"
AND an event for "Processed" is published to an XML file

The Given means to setup something either using a Test Request step or a JDBC Step.
The When is the actual service call, a Test request step.
The Then we did by asking another service to give us the status of the order. But it could have been done by a JDBC step.
The And is the interesting part. That required a bit of Groovy script programming.

Checking that a file exists
This is dead simple in Groovy:
def file = new File(orderStatusFilePath);
assert file.Exists();


Finding the newest file
In my case I did not know the file name. I knew that I file would be published in a folder and that I had to grab the latest one. Maybe this is something bad in our design, but thats the way it is - how do we test it?

First, lets declare our search criteria. I want files that are at most 1 day old:
def today = new Date()
def criteria = {file -> today - new Date(file.lastModified()) < 1}


Then, I want to find the newest file. I always use a setting in my test suite for storing file paths. The actual searching might seem complex, but its really just a matter of listing the files that match our criteria, define how we want them compared, sort them using that comparison and take the last one. 


def orderStatusFilePath = 
        context.expand( '${#TestSuite#OrderStatusFilePath}' )
def xmlFile = new          File(orderStatusFilePath).listFiles().findAll(criteria).sort() {
   a,b -> a.lastModified().compareTo b.lastModified()
 }.last();


Its worth noticing that a File object in Groovy can point to a particular file or a directory. In my first example I located a single file. This last example used a file to locate a directory. I found this solution using Google for maybe half an hour, there is tons of material on Groovy out there!

Parsing XML
Just finding that file made me very happy - but not satisfied. Since it is an XML file, why not look inside and see that the correct status was set. That is the kind of testing you'll often do once or twice and then trust it works forever. By making it a part of the test suite, we'll know.

Groovy has some real good support for XML. First, create a parser based on our file contents.

def parsedXML = new XmlParser().parseText(xmlFile.Text)


Then, within that XML document, find the tag <Status> and get the text within.
def eventStatus = parsedXML.status.text()


And now - for the icing on the cake, assert!
assert 'Processed' == eventStatus

In conclusion
Testing service calls and databases has been part of our test suites in Soap UI in the past. But adding file checking and XML parsing really boosted those suites. They have now become real end-to-end tests. And I thought I saw a small tear in the eye of my test leader...

I have just begun to use Groovy test steps in my test suites but they seem really powerful. The only downside is that its hard for a non-developer to grasp the concepts in detail. But I think that if we name our steps using the Given-When-Then syntax, what lies behind the steps becomes less important.

But what if Soap UI came with this feature built-in? That would be sweet!

Wednesday, November 30, 2011

It's little tricks like these...

Sometimes you stumble upon a hidden feature of software that just makes you smile. Someone thought about something before I knew I wanted it there!

Today I was creating a new Soap UI test project and I needed to put a password somewhere to use in an authentication request. I decided to create a new property in my test case, called "password". And just as I did, i thought "This means I'll have to display the password in plain text. Oh, well - that kind of sucks, but it's ok".
But, what happens?








The password is masked automatically, simply because the property is called "password". I know it's small and silly- but it is these things that make our lives easier!


Tuesday, November 29, 2011

Executable specifications podcast

I recently listened to this episode of Hanselminutes again. This time I think I got it.
Gojko Adzic, Jonas Bandi and Aslak Hellesöy talks to Scott Hanselman about basic BDD concepts, Executable speicifcations and tools. 45 minutes of easy listening.

Download the episode here

Sunday, November 27, 2011

BDD With Soap UI

The tool
Soap UI is a testing tool, primarily aimed at testing web services. This blog post is by no means a commercial for the tool, it is only a recap of my experiences with it. We use it extensively where I work, and it seems a good fit for both developers, testers and analysts.

One might argue that Soap UI is a narrow tool, only suitable for testing web services. That might be true, but breaking your architecture down into services is not limiting at all. It is a common way of enabling composability. Therefore I argue that having a tool that targets services in your BDD tool belt is not at all bad. Since its focus lies in services, much of the wiring is yours for free.

I assume that you know a little bit about Soap UI, at least its core constructs. If not, try the tool. There is a free version located at the Smartbear site. I will however point out that I am using the Pro version, which is not free. If you plan to do serious testing with Soap UI, buy that version.

BDD
Now this is not a post on BDD. Actually, I am quite new to the concept. But I strongly recommend reading the "Introducing BDD" post by Dan North or the "Specification by example" book by Gojko Adzic.  For the scope of this post, knowing the concept of Given-When-Then is quite enough. For the remainder of this post I will recklessly use the term BDD, even though you might as well use specification by example or other similar terms.

Mapping BDD to Soap UI
First we identify three common building blocks used when building software with BDD:
  • Feature
    The actual feature we are building, such as Login, Withdrawal or PublishBlogPost
    In Soap UI, this maps well to Test suite. For this particular feature, we will have a suite of tests running. 
  • Scenario/Example
    Examples of using the feature. Some will be "happy", like Succesful login. Some will be "sad", like FailedLogin or NoConnectionWithBank. 
    This maps well to the TestCase of SoapUI. We will build our suite using several test cases presenting different usages of the feature.
  • Step
    Now this is where we find the "Given-When-Then".
    Soap UI gives us the test steps that can be used for this level, even if we sometimes have to be a bit "clever" to keep it intact. 
An example
Lets assume that we have an authentication service in our architecture. Lets assume that it enables users to login and be authenticated. Yes - I am using the textbook example of login, but what else is business-agnostic like that...?

So, our first feature is Login. Let's create a test suite for it:

The next step is to identify a scenario and what gives more business value than "Succesful login"? We'll create a test case for it:

Finally lets create the test steps for this example:









There we are! Let's examine the steps in more detail.

The Given 
We use this step to setup the "Pam" user in the system. This might be a database operation, a set of service calls or something similar. I always try to create as much of the context from scratch so that I am not relying to much on data still being in the database/system. What I found quite common is to use the step type "Run test case". This gives us the possibility of reusing code between cases. I also use properties of this test case to configure this particular instance to setup a user named "Pam". If the test case gives me data back, such as a user id, I might use that to drive further testing.

The When
In a service-oriented architecture this will most certainly be a service call. We'll call a service using some parameters, such as UserId = "Pam". This was the only way I used Soap UI in the past. 
However, there is a tricky part involved. The normal way of things is to make assertions against the actual web service request, in the request test step. But that would mean that we would assert in our when step, which would break our BDD flow. Read on please...

The Then
This is where we want the assertions. We want the test case to fail in one of the then clauses, or even better - not fail... 
To enable this, we will use a groovy script step to assert against the response of the when step.  This is quite simple to do using the "get data" feature of Soap UI. And you don't need to know Groovy to do it!

Create a new groovy test step. Right-click in the code window and select "get data". Select the "When Pam logs in...." test step and the response.
This will bring up a window to select a property from the response. You will then be prompted to define a variable name. In our case, since we are using the "AuthenticationToken" property, lets call it something simple like authenticationToken.

This will bring in code to the window on the form of 
def authenticationToken = context.expand( '${WHEN Pam logs...

What the code actually says to the right is not that important at this point. You now have the returned token i a variable. Lets now assert that it holds the correct value.

assert authenticationToken != null, 'No authentication token was returned'

The assert statement holds three parts. The assert keyword, the condition (in our case that the token is not null) and a message that will be displayed if the condition is not met. This statement will give us the same type of assertion errors that a normal assert against a web service call gives us. 

I'll admit that this requires extra work and that it might not seem as straight-forward as the normal approach. But since it is the key to make the Given-When-Then syntax happen I am willing to accept the extra work. And its not that hard. More importantly, once you start to use groovy scripts in your tests, a whole new range of possibilities presents itself.

Also, it is not always required. Sometimes your then step will just be another service call that you can assert against in the normal way. The when might be a request to do something to the system and the then another call to make sure stuff happened. Using the groovy script approach is only needed when the evidence of then occurring is only located in the response.

And...
Once we have these constructs in place it is easy to add more steps into the test, using "and". 
We can use this to add extra context in the given, perform multiple actions in the when or assert many things in the then. In my example I added an extra JDBC step to ensure that an audit of the login was stored in the database. 

In conclusion
Making this happen with Soap UI did not only satisfy my technical ego, it really opened my eyes. I am a passionate TDD guy, but my unit tests are not supposed to be understood by the "non-dev" crowd. Since a broader audince where I work are familiar with using tools like Soap UI, I find it very useful to communicate both requirements and the current state of things.

Another key takeaway for me is that the tool you select for implementing BDD or Specification by example is really not that important. You might use SpecFlow, FitNesse or your own custom code. At our company we already had invested time into learning Soap UI and that made it a practical choice. The tool is not important, the process of specifying requirements in the shape of test is.

I know this is not a full-fledged guide or tutorial but I felt really inspired when this easy mapping became apparent to me. I will dig further into both BDD and Soap UI since they are both great additions to my tool-belt. Whatever I find - I'll share.