BDD style reporting in SoapUI Pro - Part 1

Tests as documentation
Since I discovered how well BDD concepts fit into SoapUI, I have been trying to spread those ideas and get more people to write tests in the Given-When-Then syntax. It can be a big step but I believe it is worth it. Tests become so much more than tests, they become documentation and provide an easy way in for someone trying to understand what the system actually does.

Recently I have been doing a lot of exciting work in SoapUI Pro. I have created some custom test steps, guided by Ole Lensmar's excellent blog post, that made SoapUI more usable in more scenarios. I'll go in to that in another post, but for now I'll just say that SoapUI has now become my primary testing tool for many projects.

The point of documenting something about your system is to have someone else understand it. Even if I am definitely in the target audience (how well do you remember what you wrote last week...?) the main point is to make the documentation understandable by other stakeholders. That's where I think my idea of BDD in SoapUI started to crumble. I have sent several screen shots from SoapUI to customers, but that's not really a professional approach. I have also opened up SoapUI projects to explain, but in those cases the tool sort of comes in my way. Many stakeholders don't care about what tool I use. They just want the information, preferably as short and concise as possible.

Enter SoapUI reporting
Recently I discovered the reporting capabilities of SoapUI. The built-in reports are mostly concerned with displaying test results, performance and coverage. They are all good things, but what I was after was a report that could be used as documentation of the system, not a test result report. I am interested in red and green test cases, customers will assume that the system is working.

What I wanted was a specification that only contained my BDD features and scenarios with their given-when-then steps and some additional information. Plain and simple - I thought. I did have to battle both the SoapUI object model, my lacking Java skills and JasperReports, but I think I won. After two nights of battle, I have ended up with something that is not yet perfect, but shows some great promise!

An example
Lets say you have created a test suite for a feature in your system. I will use the "withdraw cash" example, from Dan Norths excellent intro to BDD, again and possibly exhaust it even further beyond its limit...

Basically, this project now contains one feature with two scenarios. One happy flow and one where the customer is rejected. Tests are written with the Given-When-Then syntax. And yes, they're all fake.

To be even more expressive, I have added a description to the test suite "Customer withdraws cash".

Now, I could give this screen-shot to the customer, but it would not be very...cool. It contains a lot of noise and may lead to discussions things like "why is there no Security Tests?" or "What do the symbols before test step names mean, why are some starred and some not...?". So, instead I'll open the project by double-clicking the "ReportingDemo" node, click the "Test suites" tab and then the "Create report" button:

Now, a list of available reports appears. Among them, my own "BDDProjectReport". How I got it there? Be patient...

If I select my report, and click ok - This is what happens:

Magic! Instead of a screen-shot, I can now hand this document to my customer. Away with all the noise, in with a clear and concise specification! The report actually iterates over all the test suites, test cases and test steps in my project. It adds some custom formatting, such as the bold given-when-then. It also finds the description of each suite and adds it to the report. It is actually dead simple. But that's what I think is so brilliant. This is exactly what I see in those SoapUI test suites, but for some stakeholders it is not that visible.
They don't need to know how I implemented the tests, just that I did and that they define the system that we agree on. And the best part is that this is not an external document, it's the actual tests - only formatted for the intended audience.

I think this is pretty great. How I did it?

Given that this blog post is already too long
When I consider adding the actual "how I did it" section
Then I'll put that in a blog post of its own
And put some pressure on myself to actually write it
And come off looking quite geeky writing this in Gherkin

Until then!

soapUI 4.5 Beta 2

The 4.5 Beta 2 of soapUI just got released and it certainly has some new cool features. This post is in no way a complete review, but two of the new features came surprisingly close to what I had on the top of my wish-list.

1. Environment handling

I had a disussion with my co-worker some time ago. We spoke about how we could port our test projects from one environment to the another. "Why can't we just have a drop-down where we select the current environment?!" - that was our wish. So, since the 4.0.1 version contained no such thing, I decided to work something out for myself, resulting in this blog post. Even if that did the trick, it was messy, manipulative and certainly no drop-down...  

In the 4.5 version we have a new tab on the project level, called "Environments"

This tab will contain all of your defined environments where you will run your tests. Going from an old project, with no envrionment handling, to a new one is dead simple. 

When you click the "+" to create a new environment you can select to "copy endpoints and credentials from the project". This means that all of your current endpoints will be saved in that environment.  

An environment is a set of endpoints, rest services, properties and database connections. Each of these artifacts will have a name which you can then use in your tests. For instance, if you are setting up a JDBC test step you will be able to select from the defined database connections by name. When you switch between environments all the JDBC steps using that name will be automatically targeted against the JDBC connection defined in that environment under that name. Awesome! Using environments is now completely transparent.

And the best of all, I got my drop-down:






This dropdown is available on all levels: project, test suite and test case. Switching the environment on any of these levels has a global impact, meaning that all consecutive requests (SOAP requests, JDBC requests) will target the selected environment.

For me this feature is a huge improvement. I have not yet tried the custom properties on the environment level, but I think that feature has some potential as well. It is the whole transparency aspect that appeals to me.

2. Assertion test step
Now this is a feature that many might be excited about for completely different reasons than me. This feature means that we now have a step type that is focused only on asserting. With it, you can make very complex assertions using the interface we are used to with all the guidance for XPath expressions etc.

For me, this just makes my argument for BDD using soapUI even stronger. In this blog post I described how I implement the Given-When-Then syntax into soapUI. The only quirk about it was that I hade to do some Groovy-script-ninja-tricks to get the syntax to be clean. Lets do a short recap:

• The GIVEN steps constitute the background of the tests, the circumstances in which the test is executing. Normally this is a bunch of SOAP and JDBC requests setting up data. Many times we´ll need a bunch of steps to set everything up.
• The WHEN steps is the actual execution of my web service. This is normally only one step, the web service call.
• The THEN steps are the assertions. If I am asserting something from my WHEN request, the plain approach is to put the assertions inside that step. But then, we'll have no THEN step. Previously I solved this by adding a "virtual" assertion step using Groovy, but that caused some frowning among not-so-Groovy-familiar-co-workers...

Now, this last issue with the THEN steps is history. Now we have the assertion test step which is a perfect match for my THEN step. So now there really is no reason not to go BDD with soapUI.


In summary 
These two features really strengthened soapUI's position as my tool of choice for web service testing. The Environments improve maintainability, assertion steps improve readability by enabling BDD. Both features reduce the need for "ninja-stuff" in Groovy. Don't get me wrong - I enjoy a good ninja-coding-spree as much as the next developer but for me testing is all about understanding. It is about making as many stakeholders as possible understand what is happening, what the requirements are and if we are meeting them. I think we are on the right track with this tool.

Executable specifications podcast

I recently listened to this episode of Hanselminutes again. This time I think I got it.
Gojko Adzic, Jonas Bandi and Aslak Hellesöy talks to Scott Hanselman about basic BDD concepts, Executable speicifcations and tools. 45 minutes of easy listening.

Download the episode here

BDD With Soap UI

The tool
Soap UI is a testing tool, primarily aimed at testing web services. This blog post is by no means a commercial for the tool, it is only a recap of my experiences with it. We use it extensively where I work, and it seems a good fit for both developers, testers and analysts.

One might argue that Soap UI is a narrow tool, only suitable for testing web services. That might be true, but breaking your architecture down into services is not limiting at all. It is a common way of enabling composability. Therefore I argue that having a tool that targets services in your BDD tool belt is not at all bad. Since its focus lies in services, much of the wiring is yours for free.

I assume that you know a little bit about Soap UI, at least its core constructs. If not, try the tool. There is a free version located at the Smartbear site. I will however point out that I am using the Pro version, which is not free. If you plan to do serious testing with Soap UI, buy that version.

BDD
Now this is not a post on BDD. Actually, I am quite new to the concept. But I strongly recommend reading the "Introducing BDD" post by Dan North or the "Specification by example" book by Gojko Adzic.  For the scope of this post, knowing the concept of Given-When-Then is quite enough. For the remainder of this post I will recklessly use the term BDD, even though you might as well use specification by example or other similar terms.

Mapping BDD to Soap UI
First we identify three common building blocks used when building software with BDD:

• Feature
  The actual feature we are building, such as Login, Withdrawal or PublishBlogPost
  In Soap UI, this maps well to Test suite. For this particular feature, we will have a suite of tests running. 
• Scenario/Example
  Examples of using the feature. Some will be "happy", like Succesful login. Some will be "sad", like FailedLogin or NoConnectionWithBank. 
  This maps well to the TestCase of SoapUI. We will build our suite using several test cases presenting different usages of the feature.
• Step
  Now this is where we find the "Given-When-Then".
  Soap UI gives us the test steps that can be used for this level, even if we sometimes have to be a bit "clever" to keep it intact. 

An example

Lets assume that we have an authentication service in our architecture. Lets assume that it enables users to login and be authenticated. Yes - I am using the textbook example of login, but what else is business-agnostic like that...?

So, our first feature is Login. Let's create a test suite for it:

The next step is to identify a scenario and what gives more business value than "Succesful login"? We'll create a test case for it:

Finally lets create the test steps for this example:

There we are! Let's examine the steps in more detail.

The Given 

We use this step to setup the "Pam" user in the system. This might be a database operation, a set of service calls or something similar. I always try to create as much of the context from scratch so that I am not relying to much on data still being in the database/system. What I found quite common is to use the step type "Run test case". This gives us the possibility of reusing code between cases. I also use properties of this test case to configure this particular instance to setup a user named "Pam". If the test case gives me data back, such as a user id, I might use that to drive further testing.

The When

In a service-oriented architecture this will most certainly be a service call. We'll call a service using some parameters, such as UserId = "Pam". This was the only way I used Soap UI in the past. 

However, there is a tricky part involved. The normal way of things is to make assertions against the actual web service request, in the request test step. But that would mean that we would assert in our when step, which would break our BDD flow. Read on please...

The Then

This is where we want the assertions. We want the test case to fail in one of the then clauses, or even better - not fail... 

To enable this, we will use a groovy script step to assert against the response of the when step.  This is quite simple to do using the "get data" feature of Soap UI. And you don't need to know Groovy to do it!

Create a new groovy test step. Right-click in the code window and select "get data". Select the "When Pam logs in...." test step and the response.

This will bring up a window to select a property from the response. You will then be prompted to define a variable name. In our case, since we are using the "AuthenticationToken" property, lets call it something simple like authenticationToken.

This will bring in code to the window on the form of 

def authenticationToken = context.expand( '${WHEN Pam logs...

What the code actually says to the right is not that important at this point. You now have the returned token i a variable. Lets now assert that it holds the correct value.

assert authenticationToken != null, 'No authentication token was returned'

The assert statement holds three parts. The assert keyword, the condition (in our case that the token is not null) and a message that will be displayed if the condition is not met. This statement will give us the same type of assertion errors that a normal assert against a web service call gives us. 

I'll admit that this requires extra work and that it might not seem as straight-forward as the normal approach. But since it is the key to make the Given-When-Then syntax happen I am willing to accept the extra work. And its not that hard. More importantly, once you start to use groovy scripts in your tests, a whole new range of possibilities presents itself.

Also, it is not always required. Sometimes your then step will just be another service call that you can assert against in the normal way. The when might be a request to do something to the system and the then another call to make sure stuff happened. Using the groovy script approach is only needed when the evidence of then occurring is only located in the response.

And...

Once we have these constructs in place it is easy to add more steps into the test, using "and". 

We can use this to add extra context in the given, perform multiple actions in the when or assert many things in the then. In my example I added an extra JDBC step to ensure that an audit of the login was stored in the database. 

In conclusion

Making this happen with Soap UI did not only satisfy my technical ego, it really opened my eyes. I am a passionate TDD guy, but my unit tests are not supposed to be understood by the "non-dev" crowd. Since a broader audince where I work are familiar with using tools like Soap UI, I find it very useful to communicate both requirements and the current state of things.

Another key takeaway for me is that the tool you select for implementing BDD or Specification by example is really not that important. You might use SpecFlow, FitNesse or your own custom code. At our company we already had invested time into learning Soap UI and that made it a practical choice. The tool is not important, the process of specifying requirements in the shape of test is.

I know this is not a full-fledged guide or tutorial but I felt really inspired when this easy mapping became apparent to me. I will dig further into both BDD and Soap UI since they are both great additions to my tool-belt. Whatever I find - I'll share.