When execution time of your tests is longer than a coffee break, it is a good reason to think about making your tests faster. If you have already tried to run them on SSD drive, or to use PhantomJS instead of Selenium, and the execution time still upsets you, it might be a good idea to run your tests in parallel.
Codeception does not provide a command like run-parallel. There is no common solution that can play well for everyone. Here are the questions you will need to answer:
There are two approaches to achieve parallelization. We can use Docker and run each process inside isolated containers, and have those containers executed simultaneously.
Docker works really well for isolating testing environments. By the time of writing this chapter, we didn’t have an awesome tool like it. This chapter demonstrates how to manage parallel execution manually. As you will see we spend too much effort trying to isolate tests which Docker does for free. Today we recommend using Docker for parallel testing.
Please make sure you have docker or Docker Toolbox installed. Docker experience is required as well.
Run official Codeception image from DockerHub:
docker run codeception/codeception
Running tests from a project, by mounting the current path as a host-volume into the container. The default working directory in the container is /project.
docker run -v ${PWD}:/project codeception/codeception run
To prepare application and tests to be executed inside containers you will need to use Docker Compose to run multiple containers and connect them together.
Define all required services in docker-compose.yml file. Make sure to follow Docker philisophy: 1 service = 1 container. So each process should be defined as its own service. Those services can use official Docker images pulled from DockerHub. Directories with code and tests should be mounted using volume directive. And exposed ports should be explicitly set using ports directive.
We prepared a sample config with codeception, web server, database, and selenium with firefox to be executed together.
version: '3'
services:
codecept:
image: codeception/codeception
depends_on:
- chrome
- web
volumes:
- .:/project
web:
image: php:7-apache
depends_on:
- db
volumes:
- .:/var/www/html
db:
image: percona:5.6
chrome:
image: selenium/standalone-chrome Codeception service will execute command codecept run but only after all services are started. This is defined using depends_on parameter.
It is easy to add more custom services. For instance to use Redis you just simple add this lines:
redis:
image: redis:3 By default the image has codecept as its entrypoint, to run the tests simply supply the run command
docker-compose run --rm codecept help
Run suite
docker-compose run --rm codecept run acceptance
docker-compose run --rm codecept run acceptance LoginCest
Development bash
docker-compose run --rm --entrypoint bash codecept
And finally to execute testing in parallel you should define how you split your tests and run parallel processes for docker-compose. Here we split tests by suites, but you can use different groups to split your tests. In section below you will learn how to do that with Robo.
docker-compose --project-name test-web run -d --rm codecept run --html report-web.html web & \ docker-compose --project-name test-unit run -d --rm codecept run --html report-unit.html unit & \ docker-compose --project-name test-functional run -d --rm codecept run --html report-functional.html functional
At the end, it is worth specifying that Docker setup can be complicated and please make sure you understand Docker and Docker Compose before proceed. We prepared some links that might help you:
If you want to automate splitting tests by parallel processes, and executing them using PHP script you should use Robo task runner to do that.
Parallel Test Execution consists of 3 steps:
We propose to perform those steps using a task runner. In this guide we will use Robo task runner. It is a modern PHP task runner that is very easy to use. It uses Symfony Process to spawn background and parallel processes. Just what we need for the step 2! What about steps 1 and 3? We have created robo tasks for splitting tests into groups and merging resulting JUnit XML reports.
To conclude, we need:
Execute this command in an empty folder to install Robo and Robo-paracept :
$ composer require codeception/robo-paracept:dev-master
You need to install Codeception after, if codeception is already installed it will not work.
$ composer require codeception/codeception
Initializes basic RoboFile in the root of your project
$ robo init
Open RoboFile.php to edit it
<?php
class RoboFile extends \Robo\Tasks
{
// define public methods as commands
} Each public method in robofile can be executed as a command from console. Let’s define commands for 3 steps and include autoload.
<?php
require_once 'vendor/autoload.php';
class Robofile extends \Robo\Tasks
{
use \Codeception\Task\MergeReports;
use \Codeception\Task\SplitTestsByGroups;
public function parallelSplitTests()
{
}
public function parallelRun()
{
}
public function parallelMergeResults()
{
}
} If you run robo, you can see the respective commands:
$ robo
Robo version 0.6.0
Usage:
command [options] [arguments]
Options:
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
Available commands:
help Displays help for a command
list Lists commands
parallel
parallel:merge-results
parallel:run
parallel:split-tests Codeception can organize tests into groups. Starting from 2.0 it can load information about a group from a files. Sample text file with a list of file names can be treated as a dynamically configured group. Take a look into sample group file:
tests/functional/LoginCept.php tests/functional/AdminCest.php:createUser tests/functional/AdminCest.php:deleteUser
Tasks from \Codeception\Task\SplitTestsByGroups will generate non-intersecting group files. You can either split your tests by files or by single tests:
<?php
function parallelSplitTests()
{
// Split your tests by files
$this->taskSplitTestFilesByGroups(5)
->projectRoot('.')
->testsFrom('tests/acceptance')
->groupsTo('tests/_data/paracept_')
->run();
/*
// Split your tests by single tests (alternatively)
$this->taskSplitTestsByGroups(5)
->projectRoot('.')
->testsFrom('tests/acceptance')
->groupsTo('tests/_data/paracept_')
->run();
*/
} Let’s prepare group files:
$ robo parallel:split-tests [Codeception\Task\SplitTestFilesByGroupsTask] Processing 33 files [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_1 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_2 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_3 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_4 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/_data/paracept_5
Now we have group files. We should update codeception.yml to load generated group files. In our case we have groups: paracept_1, paracept_2, paracept_3, paracept_4, paracept_5.
groups:
paracept_*: tests/_data/paracept_* Let’s try to execute tests from the second group:
$ codecept run acceptance -g paracept_2
Robo has ParallelExec task to spawn background processes.
If you are using Docker containers you can launch multiple Codeception containers for different groups:
public function parallelRun()
{
$parallel = $this->taskParallelExec();
for ($i = 1; $i <= 5; $i++) {
$parallel->process(
$this->taskExec('docker-compose run --rm codecept run')
->option('group', "paracept_$i") // run for groups paracept_*
->option('xml', "tests/_log/result_$i.xml") // provide xml report
);
}
return $parallel->run();
} If you want to run tests locally just use preinstalled taskCodecept task of Robo to define Codeception commands and put them inside parallelExec.
<?php
public function parallelRun()
{
$parallel = $this->taskParallelExec();
for ($i = 1; $i <= 5; $i++) {
$parallel->process(
$this->taskCodecept() // use built-in Codecept task
->suite('acceptance') // run acceptance tests
->group("paracept_$i") // for all paracept_* groups
->xml("tests/_log/result_$i.xml") // save XML results
);
}
return $parallel->run();
} In case you don’t use containers you can isolate processes by starting different web servers and databases per each test process.
We can define different databases for different processes. This can be done using Environments. Let’s define 5 new environments in acceptance.suite.yml:
actor: AcceptanceTester
modules:
enabled:
- Db:
dsn: 'mysql:dbname=testdb;host=127.0.0.1'
user: 'root'
dump: 'tests/_data/dump.sql'
populate: true
cleanup: true
- WebDriver:
url: 'http://localhost/'
env:
env1:
modules:
config:
Db:
dsn: 'mysql:dbname=testdb_1;host=127.0.0.1'
WebDriver:
url: 'http://test1.localhost/'
env2:
modules:
config:
Db:
dsn: 'mysql:dbname=testdb_2;host=127.0.0.1'
WebDriver:
url: 'http://test2.localhost/'
env3:
modules:
config:
Db:
dsn: 'mysql:dbname=testdb_3;host=127.0.0.1'
WebDriver:
url: 'http://test3.localhost/'
env4:
modules:
config:
Db:
dsn: 'mysql:dbname=testdb_4;host=127.0.0.1'
WebDriver:
url: 'http://test4.localhost/'
env5:
modules:
config:
Db:
dsn: 'mysql:dbname=testdb_5;host=127.0.0.1'
WebDriver:
url: 'http://test5.localhost/' After the parallelRun method is defined you can execute tests with
$ robo parallel:run
In case of parallelExec task we recommend to save results as JUnit XML, which can be merged and plugged into Continuous Integration server.
<?php
function parallelMergeResults()
{
$merge = $this->taskMergeXmlReports();
for ($i=1; $i<=5; $i++) {
$merge->from("tests/_output/result_paracept_$i.xml");
}
$merge->into("tests/_output/result_paracept.xml")->run();
} Now, we can execute :
$ robo parallel:merge-results
result_paracept.xml file will be generated. It can be processed and analyzed.
To create one command to rule them all we can define new public method parallelAll and execute all commands. We will save the result of parallelRun and use it for our final exit code:
<?php
function parallelAll()
{
$this->parallelSplitTests();
$result = $this->parallelRun();
$this->parallelMergeResults();
return $result;
} Codeception does not provide tools for parallel test execution. This is a complex task and solutions may vary depending on a project. We use Robo task runner as an external tool to perform all required steps. To prepare our tests to be executed in parallel we use Codeception features of dynamic groups and environments. To do even more we can create Extensions and Group classes to perform dynamic configuration depending on a test process.
© 2011 Michael Bodnarchuk and contributors
Licensed under the MIT License.
https://codeception.com/docs/12-ParallelExecution