Modern Web Development Setup – Acceptance Testing


In this article we’ll continue setting up the CI/CD system. We’ll setup Kontena-based acceptance testing environments into UpCloud and configure them into our Shippable CI/CD pipeline.

Acceptance Test Environment Setup

New Kontena Environment Creation

To start with, we’ll create acceptance testing CI (atci) and acceptance testing environments. The atci environment is meant for running automated acceptance tests and the acceptance testing environment is meant for manual acceptance testing:

We’ll follow the setup according to the test environment setup in previous articles. We’ll set up the acceptance testing environments under at. and atci. subdomains.

Before integrating the environments with the CI/CD it’s good to make sure everything is working as expected by making a manual deployment after everything is set. So at this point we’ll write the needed variables into the Vault using the acceptance-testing-ci and acceptance-testing grids. We’ll also configure the DNS records, setup Let’s Encrypt certificates and make sure the services work well after manual deployments.

Acceptance Tests

Tech Selection and Setup

There are several viable tech choices for implementing automated acceptance tests for JavaScript applications. Selenium based solution are quite popular, but there are also interesting alternatives like Nightmare.js. Nightmare.js uses Electron under the hood, supports headless mode and runs tests very fast compared to many other solutions. It’s also quite independent package and simple to setup for the environments we’re using. It has some limitations – it’s not possible to run the tests with different browsers out-of-the-box, for example –  but in this project those are not major limitations.

Some frameworks support generic languages like Gherkin language to define the tests while some use their own DSL. I really like the idea of writing the tests using an easily understandable language and the Cucumber.js + Gherkin language is a very compelling alternative. With Cucumber it’s also possible to follow BDD practices.

I ended up selecting Cucumber and Nightmare.js for this setup to make it possible to use BDD. Therefore we’ll define the acceptance test scenarios using Gherkin language, use Nightmare.js to implement the tests scenarios and use Cucumber to run the tests. If BDD wasn’t a requirement, I’d have selected Jest and Nightmare.js to keep things more simple and straightforward.

Acceptance Test Project Setup

For acceptance tests we’ll create a new acceptance-tests repository in GitLab. We’ll install the needed npm modules with the following command:

To keep the test code in good shape we’ll use the same eslint airbnb configs as with the code repositories. The Cucumber-junit module is used for converting the test results into Shippable compatible junit format.

To run the tests, we’ll use the following test script. The script writes results to results.xml in junit format and also displays the results in console:

Test Definitions

Cucumber finds and reads the features and scenarios from .feature files and calls related step definitions from the step_definitions folder:

The implemented application is just an example and therefore the related scenario and feature descriptions are also somewhat imaginary:

For the moment the acceptance tests wouldn’t pass, since we haven’t written the application components yet. To make it possible to verify the pipeline functionality, we’ll write the feature step definitions in a way that they always pass. In the upcoming articles we’ll finalise the application components and implement the proper step definitions for the acceptance tests.

Cucumber implementation for the Given I visit the service website step (see the acceptance-tests repo for more):

And related Nightmare.js test implementation:

Before continuing to the system integration we’ll verify the acceptance tests functionality locally:

Acceptance Test Integration into the Build System

Shippable Machine Image

First of all, we’ll take the 5.5.1 Shippable Machine image into use via Subscription -> Settings -> Options. Latest versions provide Ruby and xvfb-run out-of-the-box which is very convenient.

PEM Keys to Integrations

To get rid of the Kontena SSL connection warnings and making the connection between CI system and Kontena master slightly more secure, we’ll use the .pem file instead of ignoring the SSL warnings with SSL_IGNORE_ERRORS=true definition.

We’ll add the needed integration via Account settings -> Integrations -> Add Integration -> Create Integration (PEM key) and set the Kontena key from ~/.kontena/certs/<kontena_master_ip>.pem. While adding it, we’ll also link it into the article-project subscription and enable the integration for all the projects belonging to the subscription via article-project subscription integration settings. With the integration, the SSL key is now available in the /tmp/ssh/ folder for our runCI and runSH jobs.

Acceptance Tests Project

We’ll enable acceptance-tests project for the Shippable CI. It would’ve been possible to run the acceptance tests CI as runSH job, but runSH jobs don’t display test results in the GUI for the moment. The runCI job is also more flexible and faster to configure since environment variables can be defined via shippable.yml without using external resources, for instance.

Following shippable.yml defines the acceptance tests runCI job:

Overall Pipeline

Until these configurations, only the CI build jobs have been shown in the subscription specific Shippable Pipeline view. We’ll use shippable.resources.yml and files to configure the pipeline and in this example the files are located in the deployment-scripts repository.

To enable the pipeline, GitLab integration has to be enabled for the article-projects subscription via Subscription -> Settings -> Integrations -> Add Integration. After enabling the integration, the deployment-scripts repository can be set as syncRepo via Single Pane Of Glass (SPOG) view’s Add resource button. After the pipeline is read from the repository, the pipeline is automatically updated from the repository.

In the deployment-scripts Git-repository the latest Kontena descriptors are put into the descriptors folder and some shippable configuration scripts are refactored into script files under the scripts folder. To enable access to the Git-repository, the Shippable SSH key (Subscription -> Settings -> Options -> Deployment Key) was given access to the GitLab repository (Settings -> Repository -> Deploy Keys).

We’ll skip quite a few details, but here is the overall pipeline:

Shippable Pipeline Overview

In general, the pipeline is now triggered by Front, App and Auth service Git-repository changes. Changes made to the deployment-scripts repo will also trigger a job to automatically check if the pipeline related definitions have changed.

An acceptance test runCI job is automatically triggered when either Front, App or Auth service CI builds are passing and a new image is created. For CI deployments we created a new descriptor (kontena-ci-areas.yml) which is similar to other test environment descriptors except for some details regarding database persistence. CI jobs should be always run from a clean slate and therefore we don’t use stateful database services. Commits and pull-request web hooks were disabled for acceptance-tests project via project settings, since there isn’t any need to automatically trigger the acceptance tests CI build when the acceptance tests Git-repository is updated.

The testing environment deployment works as before, so that all components passing their unit and API tests are automatically deployed to the testing server; whether they pass the atci or not. The testing environment deployment functionality was removed from the Front, App and Auth runCI jobs and refactored into a new runSH job (testing-deploy in

An acceptance testing deployment job (acceptance-testing-deploy in is defined in such a way that the job isn’t triggered automatically when the related resources are changed. The job must be triggered manually from the web UI and the deployment is done using the latest component images which have passed the acceptance tests CI.

All the pipeline related definitions are available in the deployment-scripts repository.


Overall the pipeline fulfils the requirements we set earlier in the testing and deployment overview article. The final part of the CI/CD pipeline – the production environment – will be added later after we have implemented the system components.