Automated acceptance testing with Cucumber for .NET and JAVA

One of the biggest problems in software development is communication. Although we have a lot of technology at hand, the more complex a project is, the harder it is to communicate efficiently. In order to improve the interaction between clients, project managers, analysts and developers, a definition of the requirements is needed.
Another important part is testing. With the ever-growing complexity of software, acceptance testing becomes increasingly important. Manual acceptance testing is very time consuming and error-prone though.

Background

Writing specifications and acceptance tests, is usually approached in one of two manners:

  • With the waterfall-approach you’d create plain text documents up front that capture all the clients needs. These documents can be become problematic in that they go out of synch, are not executable, cannot be properly versioned and there’s a long feedback loop between the client, the analyst and the developer. Acceptance tests are based on testing plans. These consist typically of detailed step-by-step instructions with the expected results. Executing these tests is very time consuming, requires a lot of planning, reporting, internal communication and is error-prone.
  • The agile approach has user stories and advocates executable unit tests as a form of specification. These tests fail in the communication towards the client and they don’t provide a global vision of the application and its features. Acceptance testing is often forgotten because the reliability of unit tests is overestimated.

Of course, these are the two extremes, usually there’s a either a mix of both, or the methodology leans more towards one or the other.
In both approaches, however, there’s a gap between the requirements and verifying whether these requirements are met.

Cucumber

Cucumber is a tool that tries to bridge the gap between specifications and acceptance tests by allowing you to write plain text, human readable scenarios which can be executed and verified.

It started in Ruby and it’s a real gem (pun intended). It has been ported to a variety of platforms; .NET has SpecFlow and JAVA the Cucumber-JVM package.

Getting the tools

Regardless of which platform you’re developing for, you will need two items:

  • A plug-in for your IDE so you can write specifications with syntax highlighting and automatic statement completion
  • A library which allows you to link specifications to code

The first one is not really a requirement, but it makes life a lot easier

.NET

The plug-in for Visual Studio is the SpecFlow extension. You can download it from the Visual Studio gallery website.
The library you need to use can be downloaded from nuget: SpecFlow library, alternatively you can get the package with a built-in connection to NUnit by installing the following package:

PM> Install-Package SpecFlow.NUnit

Once you have done this, you can now have a new item type called “.feature” in your “Add new item”-dialog and you can execute features with NUnit.

JAVA

For Java, things are a bit more complicated (but that’s probably more to my lack of knowledge). For Eclipse I have found this plug-in to be very useful: https://github.com/rlogiacco/Natural
A thorough installation guide can be found here: https://github.com/rlogiacco/Natural/wiki/Installation-Guide

As for the library, you can use Cucumber-JVM. The GitHub-page has decent tutorials to get started.

Once these are installed and configured correctly, you can start writing specifications and execute them using jUnit.

The basics

In essence, writing an executable scenario consists of three parts:

  1. Writing the specification steps in plain text through a DSL called Gherkin
  2. Linking the steps to step definitions
  3. Verifying the specified behavior with the application under test

Writing the specification

The gherkin-language is used to define our scenarios. It’s a very simple grammar with a few keywords that allow you to write concise specifications with the given-when-then approach. Specifications are written in .feature-files. The following is an example of a simple specification:

Feature: Transfer money

Scenario: Transfer money with sufficient funds

	Given I have 50€ in my account
	When I transfer 50€
	Then my balance is 0€

A scenario consists of several different steps. Each step starts with one of the following keywords:

  • Given: Put the system into a starting state
  • When: Describe the action you are performing
  • Then: Validate the behavior of the system
  • And, But: If you have several Given, When or Then steps you can link them with And and But.

For the actual outcome of the test, these keywords all behave the same. If I were to change all the keywords in the previous scenario to Given, the outcome would not be influenced. These keywords are merely aliases to define a step and their only objective is making the scenarios more readable.

Linking the steps to step definitions

Next, we need to link these steps to something tangible. The code is in c#, but is very similar to what a JAVA-file would look like:

[Binding]
public class TransferFundsSteps
{
    [Given(@"I have 50€ in my account")]
    public void GivenIHave50InMyAccount()
    {
    }

    [When(@"I transfer 50€")]
    public void WhenITransfer50()
    {
    }

    [Then(@"my balance is 0€")]
    public void ThenMyBalanceIs0()
    {
    }
}

The class TransferFundsSteps defines three methods. By decorating them with attributes, we link the steps with step definitions where we can put executable code.

Verifying the behavior

The last step is to effectively verify the behavior of the application under test. For this example, I will be using a web application, driven by Selenium (you could also use another driver such as Capybara). I have excluded some setup code for brevity’s sake (such as opening a browser and navigating to the page)

[Binding]
public class TransferFundsSteps
{
    private IWebDriver driver;
    public TransferFundsSteps()
    {
        driver = new FirefoxDriver();
    }

    [Given(@"I have 50€ in my account")]
    public void GivenIHave50InMyAccount()
    {
        // setup the DB so 50€ is in the account
    }

    [When(@"I transfer 50€")]
    public void WhenITransfer50()
    {
        driver.FindElement(By.Id("amount")).SendKeys("50");
        driver.FindElement(By.Id("submitBtn")).Click();
    }

    [Then(@"my balance is 0€")]
    public void ThenMyBalanceIs0()
    {
        var currentBalance = driver.FindElement(By.Id("currentBalance")).Text;
        Assert.That(currentBalance, Is.EqualTo("0"));
    }
}

Now, when you run you’re feature-file, these steps will be executed, simulate the user input and assert any incorrect outcome.

Although this is nice, it’s not really all that impressive. Basically we’ve created a selenium script where each step can be accompanied by a description. Let’s see what else we can do.

Step parameters

Suppose we want to add a new scenario: Transferring money when there aren’t enough funds. We could write a scenario as follows:

Scenario: Transfer money with insufficient funds

Given I have 40€ in my account
When I transfer 60€
Then my balance is 40€

Now we could create three new steps with these descriptions and run it. But our steps wouldn’t be DRY because we’re repeating the same logic, just with different values.

Cucumber allows us to parameterize our steps through the use of regular expressions. Let’s take a look at how we can modify the current steps to accommodate for this new scenario:

[Binding]
public class TransferFundsSteps
{
    private IWebDriver driver;
    public TransferFundsSteps()
    {
        driver = new FirefoxDriver();
    }

    [Given(@"I have (.*)€ in my account")]
    public void GivenIHaveInMyAccount(string balance)
    {
        // setup the DB so 50€ is in the account
    }

    [When(@"I transfer (.*)€")]
    public void WhenITransfer50(string amount)
    {
        driver.FindElement(By.Id("amount")).SendKeys(amount);
        driver.FindElement(By.Id("submitBtn")).Click();
    }

    [Then(@"my balance is (.*)€")]
    public void ThenMyBalanceIs(decimal balance)
    {
        var currentBalance = driver.FindElement(By.Id("currentBalance")).Text;
        Assert.That(currentBalance, Is.EqualTo(balance));
    }
}

Cucumber checks whether one of your steps matches with a step definition. If it finds one, it parses out the values and passes them as a parameter to your step definitions. As you can see, we didn’t have to create new steps. Step parameters allow you to run multiple scenarios with different input parameters and check how the application reacts to this. Because it’s all in plain text, stakeholders can easily verify the behavior of the application.

Scenario outlines

Now we want to add one more requirement: an overdraft of 10€ should be allowed. With our step parameters in place we could of course just copy one of the scenarios, change the values and run it. Obviously now the step definitions are DRY, but now our scenarios aren’t DRY anymore since we’re copying them. Let’s see how a scenario outline can help with this:

Scenario Outline: Transfer
	Given I have <startbalance> in my account
	When I transfer <amount>
	Then my balance is <endbalance>

	Examples:
	    | startbalance | amount | endbalance |
	    |  50          |  50    |  0         |
	    |  40          |  60    |  40        |
	    |  50          |  60    |  -10       |

This takes the parameterization one step further: now our scenario has “variables” and they get filled in by the values in each row. To be clear: by defining this, the scenario will run three times, passing in one row at a time. This makes it very easy to define a lot of examples, edge cases and special outcomes.

Backgrounds

For all of the previous scenarios I omitted the initial setup. However, in  a real world situation you would want to log in the user and navigate to the correct page before you can execute the transfer-scenario. You could of course include all of this in several “Given”-steps. The problem is that this is not really part of the scenario we’re describing and thus it obfuscates our scenarios. Let’s see how we can move the setup out of the scenarios:

Feature: Transfer money

Background: 
	Given a user is logged in with username Kenneth and password Cucumber
	And is on the page /Account/TransferFunds	

#scenarios go here

A background definition allows you to create an unnamed scenario that will run before each other scenario in that feature. It allows you to define a context where your scenarios are applicable. Using a background allows for a very clean and readable feature definition. Let’s take a look at what the complete file looks like now:

Feature: Transfer money

Background: 
	Given a user is logged in with username Kenneth and password Cucumber
	And is on the page /Account/TransferFunds			


Scenario: Transfer money with sufficient funds

	Given I have 50€ in my account
	When I transfer 50€
	Then my balance is 0€

Scenario: Transfer money with insufficient funds

	Given I have 40€ in my account
	When I transfer 60€
	Then my balance is 40€

Scenario Outline: Transfer
	Given I have  in my account
	When I transfer 
	Then my balance is 

	Examples:
		| startbalance | amount | endbalance |
		|  50          |  50    |  0         |
		|  40          |  60    |  40        |
		|  50          |  60    |  -10       |
    

As you can see, the feature definition is very concise and to the point. A reader can see very clear what the context is for this feature and which scenarios are applicable.

Other features

Apart from the examples I showed you here, you can do a few more things:

  • Multiline arguments: Instead of writing the parameters inline, you can add a table of data (syntactically the same as the examples in the scenario outline) to a step. You will then receive an extra parameter in the step definition which contains that table.
  • Tags: In order to group features and scenarios independently from the files, you can add tags. (for example front-office / back-office). This can come in handy if you want to integrate it in a continuous integration build and on certain builds want to run certain groups.
  • Multi language: Since improving communication and eliminating ambiguity is the main goal, it’s important that features are communicated as clearly as possible. If you’re not native in English, it can be a good idea to write your features in your native language. This is 100% supported and there currently more than 50 languages available (including Lolcatz: I can haz => when => den). You can see the list of languages here: Cucumber spoken languages
  • Transformations: You can intercept the transformation from the parameters in the step to the parameters in the step definitions. This allows you to write even more expressive scenarios without having to write to much conversion code in your step definitions.
  • Hooks: Hooks allow you to intercept any part of the execution of your scenarios.
  • Reporting: The results can be rendered in a variety of formats: HTML for publishing, JSON to post-process and JUnit (which is understand by most CI-servers). There’s even a possibility to include screenshots for the failed scenarios.

Common pitfalls and tips for avoiding them

Although Cucumber is very powerful, there are a few common mistakes. A few of these are general with acceptance testing, while others are specific to Cucumber. Most of them can be easily solved though. This is a short list of what can go wrong and how to prevent or fix it:

  • Slow execution / lots of scenarios: Since we’re testing the whole application (which can have a lot of scenarios) it’s common that test suites grow a lot. If we have a lot of tests, it can take a long time to execute all of them, thereby making the feedback-loop too long. This can be solved by using a continuous integration server which supports build grids that can execute them in parallel.
  • Leaky scenarios: Because every step basically puts the application in a different state, a common problem is that one scenario influences the outcome of another. You need to make sure that the application gets to a known state before executing a scenario. This can be done with a background or with hooks.
  • UI dependency: When you’re automating a UI, your tests can become brittle when implementation details change (ie: an ID of a field changes, a select-box is replaced with a jQuery component, etc.). There are a few design patterns that can decouple your steps from these implementation details. One of them is using a Page Object. The basic idea is that you model everything you can do on a single page in a class. Your steps can then use this class to drive the UI. That way, if the UI changes, you only have to fix it in one place. Essentially you’re creating an application driver,
  • Imperative vs declarative scenarios: When you’re writing acceptance tests, you should write them from a business perspective, not from a developer perspective. As developers, we are used to writing imperative scenarios (if variable x equals y then do z …). While this is useful when you’re programming, it ties requirements to implementation details. Let’s take a look at the following example:
    Scenario: Create account
    	Given I'm on the page "/account/create"
    	When I enter my name in the name textbox
    	And I enter my last name in the last name textbox
    	And I click the button "Create Account"
    	Then the page "account/created" should be shown
    	And the label should contain my first name and last name

    At first sight, this seems a decent test. It will work well and it will tell you if something goes wrong. However, it contains implementation details such as the fact that it’s a web application, it contains pages, has URL’s and that there are textboxes. If, instead of a web application, you want to create a mobile application, you’d have to create a new acceptance test. Acceptance tests should try to describe requirements, not implementations. Let’s take a look at how this could be written better:

    Scenario: Create account
    	Given I want to create an account
    	When I provide my name
    	And I provide my last name
    	And I create an account
    	Then a message with my name and last name should be shown

    As you can see, this rewritten scenario does not contain any implementation details and could be applied to a web application, a mobile application, a desktop application or even a REST-api.

Conclusion

When Cucumber scenarios are written well, it allows a very tight communication between stakeholder, analysts and developers. Requirements can be tested often and validated by business users. Because they will fail if requirements aren’t met, there is immediate feedback. When properly integrated with a continuous integration server, they can provide additional testing that unit tests can’t deliver. Since they’re written in a human readable format, everyone can get involved in the testing process, not only developers and testers, but also stakeholders, analysts, product owners and clients. They improve the overall communication within a team as well from the team to the client.

Comments are closed.