Automated acceptance tests play a vital role in continuous delivery. Contrary to unit tests though, they’re quite hard to get right. This is not only because end-to-end testing is harder than testing single units, but also because of the way they need to be executed. You need the a fully working version of the application under test and a client that represents a real-world scenario. When you apply this to a browser based app, things can get complicated.If we want to test our website fully, we need to be able to test it on a variety of browsers, devices and screens. Building all this infrastructure is very costly and time consuming. But new companies such as BrowserStack and SauceLabs have a solution for that: they provide VM’s for all kinds of browsers, configurations and emulators. We can make use of their infrastructure by running our acceptance tests on their resources.
Another problem acceptance tests usually pose is that they are slow (in comparison to unit tests). By carefully maintaining your tests, you can get them as fast as they can possibly be, but they are slow by nature so there’s only so much you can do.
In this post I’ll work through an example on how to run a SpecFlow test remotely on multiple BrowserStack VM’s at the same time. These are the goals:
- No need to build/buy infrastructure
- Run tests simultaneously, so that adding configurations doesn’t slow down the process
- Make it easy to add configurations
- Make tests executable from the command-line, by anyone, at any time
1. The tests
As an example I’m going to use a simple test that executes a google search and then verifies whether results were returned:
Feature: Searching google Given I am on the google page When I search the web Then I get search results
The step definitions are quite straightforward (intentionally so):
[Binding]
public class GoogleSteps
{
readonly IWebDriver _driver;
public GoogleSteps()
{
_driver = (IWebDriver)ScenarioContext.Current["driver"];
}
[Given(@"I am on the google page")]
public void GivenIAmOnTheGooglePage() =>
_driver.Navigate().GoToUrl("http://www.google.com");
[When(@"I search the web")]
public void WhenISearchTheWeb()
{
var q = _driver.FindElement(By.Name("q"));
q.SendKeys("Kenneth Truyers");
q.Submit();
}
[Then(@"I get search results")]
public void ThenIGetSearchResults()
{
Assert.That(_driver.FindElement(By.Id("resultStats")).Text, Is.Not.Empty);
}
}
Note that here I’m accessing the UI immediately in my step definitions. This is done for simplicity’s sake. In a real-word scenario, you probably want to use some indirection through the use of Page Objects. For more information on how to create maintainable specflow tests, you can refer to my article about Automated acceptance tests with Cucumber
2. The driver
To drive the browser I will be using Selenium (as seen in the example above). Selenium can drive the browser through an instance called a WebDriver. There are a few different WebDrivers available for IE, Chrome, FireFox, … but there’s also a driver called RemoteWebDriver. This is a driver that can drive a browser on a different machine. This is the one we will be using.
Nevertheless, we don’t want to tie any of our tests to a particular type of WebDriver. Luckily they all implement the IWebDriver interface.
2.1 Instantiating the driver
If we could use constructor injection, we could inject an instance of an IWebDriver into our step definitions. Unfortunately, SpecFlow doesn’t allow this. The next best thing we can do is use a Service Locator pattern. SpecFlow provides a Dictionary-like object called a ScenarioContext, which contains objects available to the current scenario. In the example above you can see that upon instantiation of the GoogleSteps-class, we get the driver from this dictionary. The next bit of code shows how we set up the driver at the beginning of a scenario:
[Binding]
public class Setup
{
IWebDriver driver;
[BeforeScenario]
public void BeforeScenario()
{
driver = new FirefoxDriver();
ScenarioContext.Current["driver"] = driver;
}
[AfterScenario]
public void AfterScenario() {
driver.Dispose();
}
}
Through the use of some SpecFlow hooks, we create a driver before each scenario and tear it down after the scenario finishes. In this example we just created a local FireFox driver. At this point we can execute our test and it will run successfully in a local FireFox instance.
2.2 Configuring the driver to use browserstack
The next step is executing the same test, but on a remote machine managed by BrowserStack. To do this, we need to create a RemoteWebDriver, and configure it accordingly. When you create a RemoteWebDriver, you need to provide two arguments:
- The Url of where the RemoteDriver will accept commands (this is provided by BrowserStack)
- The capabilities: this is a loosely typed dictionary of parameters that will be sent to the RemoteDriver. In this case, BrowserStack dictates which parameters you need.
So first of all, let’s create an instance of the DesiredCapabilities-class and set the properties accordingly:
[BeforeScenario]
public void BeforeScenario()
{
if (Process.GetProcessesByName("BrowserStackLocal").Length == 0)
new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "BrowserStackLocal.exe",
Arguments = ConfigurationManager.AppSettings["browserstack.key"] +
" -forcelocal"
}
}.Start();
var capabilities = new DesiredCapabilities();
capabilities.SetCapability(CapabilityType.Version, "33");
capabilities.SetCapability("os", "windows");
capabilities.SetCapability("os_version", "8");
capabilities.SetCapability("browserName", "firefox");
capabilities.SetCapability("browserstack.user",ConfigurationManager.AppSettings["browserstack.user"]);
capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
capabilities.SetCapability("project", "Google");
capabilities.SetCapability("name", ScenarioContext.Current.ScenarioInfo.Title);
capabilities.SetCapability("browserstack.local", true);
driver = new RemoteWebDriver(new Uri(
ConfigurationManager.AppSettings["browserstack.hub"]),
capabilities);
driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
ScenarioContext.Current["driver"] = driver;
}
I have separated the capabilities in to three different sections:
- The first four capabilities determine what browser and platform you want to run these tests on.
- The next two, indicate your BrowserStack username and key. These you can get from the BrowserStack interface after you create an “Automate”-account (or a trial)
- The last two items are optional and are merely so you can see what project and what test is running from the BrowserStack interface
With these capabilities set up, we can now create an instance of the RemoteWebDriver:
driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
The driver is still of type IWebDriver, so we don’t need to change anything to our steps. Provided we have entered the correct username and key, we can now run our tests remotely on a VM managed by BrowserStack.
2.3 Allowing BrowserStack to access local resources
In the above example, I’m only accessing a public website. Chances are that in a real world scenario you will be testing a QA environment or even a dev environment on your PC. These are usually not publicly accessible. To allow BrowserStack access to these resources, they allow you to set up a tunnel through your PC and then access the resources from your PC. To set this up, you need to:
- Download the appropriate binary from their website (https://www.browserstack.com/local-testing#command-line)
- Run the binary providing your key as a command-line argument (ie: browserstacklocal.exe -<yourkeyhere>)
- Indicate that you need local access in the capabilities
So before we run our tests, we can run the executable from the scenario hook:
if (Process.GetProcessesByName("BrowserStackLocal").Length == 0)
new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "BrowserStackLocal.exe",
Arguments = ConfigurationManager.AppSettings["browserstack.key"] + " -forcelocal"
}
}.Start();
3. Running tests from the command-line
The tests can now be ran from Visual Studio. To run these tests from the command-line, we can use the unit test runner. In this case I have used NUnit as the underlying test framework, so I’ll be using the NUnit runners. To run the tests from the command line you run the following command:
nunit-console.exe /xml:nunit.xml /nologo /config:release <pathtodll>.dll
4. Running tests in parallel
Now that we have the tests running in BrowserStack from the command-line, it’s time to start running them simultaneously on various configurations. In order to run them on various configurations we need a few things:
- Parameterize the capabilities (namely os, os_version, browsername and version)
- Run several test at the same time with different parameters
There’s one problem with this approach: when we run NUnit through the console there’s no way to pass parameters to the tests. The general approach given on various forums is to set environment variables. Since we are running the tests simultaneously, this doesn’t help us. There’s one thing however that lets us parameterize how we run the tests: NUnit allows us to specify the configuration we want to run the tests with. So I have come up with the following approach:
- Use one solution configuration for every configuration we want to test
- Extract the os, os_version, browsername and version to the configuration
- Use config transforms to vary these parameters based on the solution configuration
- Use Powershell to run various instances of the NUnit-console process with different configurations
4.1 Creating different solution configurations
For each configuration we want to test, we will add a different solution configuration. Usually I start by deleting the standard release configuration and renaming the debug-configuration to something more sensible (eg: Win8Firefox33). You can add as many solution configurations as you want. It’s best to copy them from the original Debug-config as this will have all the settings necessary to be able to debug your code.
4.2 Extract the parameters to the configuration
We also need to change the capabilities to fetch these values from the config:
capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);
capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
4.3 Add config transforms
When you have created the different solution configurations, you now need to add a config transform for each configuration. This is an example for my Win8Firefox33 configuration (note that the name is not important, it’s merely a convention)
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<appSettings>
<add key="browser" value="firefox" xdt:Transform="Insert"/>
<add key="os" value="windows" xdt:Transform="Insert"/>
<add key="version" value="33" xdt:Transform="Insert"/>
<add key="os_version" value="8" xdt:Transform="Insert"/>
</appSettings>
</configuration>
If your not sure how to add config transformations to a class-library, you can use a tool such as SlowCheetah to do it for you.
4.4 Run various instances of the NUnit-console process with Powershell
We can now already run a different configuration on BrowserStack by varying the config-parameter of the previous command. Here are two examples of how to run a different configuration:
nunit-console.exe /xml:nunit.xml
/nologo
/config:Win8Firefox33
<pathtodll>.dll
nunit-console.exe /xml:nunit.xml
/nologo
/config:Win7Chrome38
<pathtodll>.dll
The next step is executing all the configurations we have in parallel with powershell. To do this we’ll need to execute the following steps:
- Get all the configurations in the solution
- Compile the project in each configuration
- Run all configurations in parallel
To get all the configurations in the solution, we can use this small function:
function Get-SolutionConfigurations($solution)
{
Get-Content $solution | Where-Object {$_ -match "(?<config>\w+)\|"}
| %{ $($Matches['config'])}
| select -uniq
}
This will open the .sln file, use a regex to find the configurations available, deduplicate them and return them as an array.
Next step is compiling the project against the different configurations. We will be using MsBuild for this:
@(Get-SolutionConfigurations "<path.to.sln>") | foreach {
msbuild <path.to.csproj> /p:Configuration=$_ /nologo /verbosity:quiet
}
Now that the project is compiled in all configurations we can run the tests in parallel. For this, we will be using Powershell jobs:
@(Get-SolutionConfigurations "<path.to.sln>") |
foreach {
Start-Job -ScriptBlock {
param($configuration)
nunit-console.exe /xml:nunit_$configuration.xml /nologo /config:$configuration <path.to.dll.for.this.config>
} -ArgumentList $_
}
Get-Job | Wait-Job
Get-Job | Receive-Job
This snippet retrieves the configurations and it start a job for each one. When all jobs are started it waits for all of them to complete and then receives and writes the output of all of them to the console one by one.
5. Reporting
Once we have run all our tests, we want to report the results. First of all, you can look at your tests performing live via the BrowserStack website. This is great and allows you visually verify any errors if the occur. It also shows you a complete log of everything that has happened in the scenario. The image below shows what you can see while the tests are running:
You can see in real-time which tests are running, how far they are and even follow on-screen what they are doing. That’s pretty awesome for live debugging and monitoring.
Apart from live debugging, we also need to create reports when tests have finished so we can check them afterwards or even archive them for later review. To do this, you can run the specflow tool:
specflow.exe nunitexecutionreport
<path.to.csproj>
/out:specresult.html
/xmlTestResult:nunit.xml
/testOutput:nunit.txt
When we add this to our parallel test runner, the script becomes as follows (we need to parameterize the text-files because we’ll have one for each configuration):
@(Get-SolutionConfigurations "<path.to.sln>") |
foreach {
Start-Job -ScriptBlock {
param($configuration)
try {
nunit-console.exe /labels
/out=nunit_$configuration.txt
/xml:nunit_$configuration.xml
/nologo
/config:$configuration
<path.to.dll.for.this.config>
}
finally {
specflow.exe nunitexecutionreport <path.to.csproj>
/out:specresult_$configuration.html
/xmlTestResult:nunit_$configuration.xml
/testOutput:nunit_$configuration.txt
}
} -ArgumentList $_
}
Get-Job | Wait-Job
Get-Job | Receive-Job
Now every NUnit test run will dump two files: an XML-file with the test results and a TXT-file with more information. The SpecFlow runner will then parse these files and generate an HTML-report.
6. Putting it all together
Now that we have gotten all steps finished, we can put everything together. To execute the tests we need the following:
- A parameterized setup of the driver before each scenario
- A config transformation for each configuration we want to run
- A PowerShell build script that can execute the configurations in parallel and report on them
For completeness, here is the full code for each of these requirements. Alternatively, you can clone the code from GitHub and play with it yourself.
6.1 Parameterized driver
[BeforeScenario]
public void BeforeScenario()
{
if (Process.GetProcessesByName("BrowserStackLocal").Length == 0)
new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "BrowserStackLocal.exe",
Arguments = ConfigurationManager.AppSettings["browserstack.key"] + " -forcelocal"
}
}.Start();
var capabilities = new DesiredCapabilities();
capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);
capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
capabilities.SetCapability("browserstack.local", true);
capabilities.SetCapability("project", "Google");
capabilities.SetCapability("name", ScenarioContext.Current.ScenarioInfo.Title);
driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
ScenarioContext.Current["driver"] = driver;
}
6.2 Config transformations
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<appSettings>
<add key="browser" value="firefox" xdt:Transform="Insert"/>
<add key="os" value="windows" xdt:Transform="Insert"/>
<add key="version" value="33" xdt:Transform="Insert"/>
<add key="os_version" value="8" xdt:Transform="Insert"/>
</appSettings>
</configuration>
6.3 Build script in Powershell
function Get-SolutionConfigurations($solution) {
Get-Content $solution |
Where-Object {$_ -match "(?<config>\w+)\|"} |
%{ $($Matches['config'])} |
select -uniq
}
@(Get-SolutionConfigurations "<path.to.sln>") |
foreach {
iex { msbuild <path.to.csproj>
/p:Configuration=$_
/nologo
/verbosity:quiet
}
}
@(Get-SolutionConfigurations "<path.to.sln>") |
foreach {
Start-Job -ScriptBlock {
param($configuration)
try {
nunit-console.exe /labels
/out=nunit_$configuration.txt
/xml:nunit_$configuration.xml
/nologo /config:$configuration
<path.to.dll.for.this.config>
} finally {
specflow.exe nunitexecutionreport
<path.to.csproj>
/out:specresult_$configuration.html
/xmlTestResult:nunit_$configuration.xml
/testOutput:nunit_$configuration.txt
}
} -ArgumentList $_
}
Get-Job | Wait-Job
Get-Job | Receive-Job
7. Demo project
In the code above, I have omitted a few things for brevity’s sake. In order to be able to run msbuild, the nunit console runner and specflow, you need to ensure that all the paths are set up correctly (ie: added to your environment variables). Therefore, I have created a full working implementation, which you can find on GitHub.
To run it you need to do the following:
- Clone the repository to your PC
- Create a trial account on BrowserStack
- Go to Account => Automate and copy the UserName and Value into the app.config
- Open a command prompt and run “powershell –file build.ps1”