Achieving Software Testing Nirvana in a CI/CD environment with a sprinkling of Agile – Part III

Part III – What is CI/CD?

Continuous Integration is achieved when developers continually commit their deemed production ready code to a version control system in which changes checked into this repository are picked up by a Continuous Integration server and validated against a set of build steps for things like static code analysis, cyclomatic complexity and various levels of software testing. Most importantly it is a check on how work done by a developer integrates with work done by other developers and the integrations with other systems.

Continuous Delivery is the automated act of delivering various aspects of software into a production environment with minimal manual intervention. Continuous Integration and deployment are aids which help the overall continuous delivery process. CI/CD is important when trying to establish quick feedback, practice and improvement. It’s a continuous cycle along a pipeline whereby various parts of the software system makeup are tested and verified against a set of automated tests and the deployment process known as promotion to each environment is a test of the release process to production.

CD Pipeline

What to do when a test fails in the pipeline?

When a test fails, the remaining suite of tests in that environment should continue to run; as we can at least clarify that nothing else is failing and focuses on fixing the test that flagged an issue during the test run. The exception to the rule, are the acceptance tests and capacity tests, if something fails in these, then due to the general length of testing in these environments, it’s best to stop the test run and focus on fixing the flagging issue.

When a test fails during the commit stage, the team must all stop what they are doing and help get to the bottom of the problem, every member of the team must come together and help locate the issue, every commit to source control is a potential release candidate, and continuing to work on a developers code without helping the developer who checked in a failing test can cause a backlog of failing tests and create a much more difficult scenario of locating the fix for the problem, when more tests could begin to fail. A solution to this is called Pre-tested commit, more on that later.


What should the pipeline model?

The continuous deployment pipeline should model the release process. It is the practiced promotion to production like environments that will give the business the confidence that on release day, a simple flick of a switch and we are live with nothing to worry about, as it has been rehearsed numerous times through the automated continuous process.

Creating Value Stream Maps

Value stream maps come from the lean management methodology; it is essentially a map of the process of a user story from inception to production and the various stages a user story will follow in its lifetime. We take the value of a story and the time it takes to get it into production and create a map around that, constantly evolving the process to increase the time to market and creating a map to visualise this. Value stream maps can and possibly should be extended to the wider software team, not only the development team but management as well.

Value Stream Map

Automation and Orchestration

Automation as previously mentioned makes up a large part of the continuous delivery process. When a developer checks in some production code, the next stage, known in some circles as the commit stage is an automated process whereby a developers code is checked against things like static code analysis and unit and component level testing. If all goes well, this code is promoted to the acceptance test environment, where the code is executed against a suite of acceptance tests derived from the acceptance criteria of various stories, including the story the code was committed against. The next stage is capacity testing, which can run the tests from the acceptance stage in a capacity and security testing environment that matches production to gain confidence that code does not breach CPU or memory usage amongst others. Afterwards comes the staging environment, which like the capacity environment must match production, this can be used for exploratory testing. After all this, a release to production can be created. The large part of this pipeline can be completely automated, with the exception of the exploratory testing environment. A developer should only have to worry about the commit to the CI server, the continuous delivery pipeline will then take their code all the way to the staging environment, where a team of QAs can verify certain tests that can’t be automated, and then use a push button deployment system to get the code into production.

Orchestration is the process whereby environments are completely automated and can be part of the continuous delivery process. Once code has passed a stage, such as the commit stage, an acceptance test environment can be spun up that matches production based on configuration files that represent all entities of that system, from hardware requirements to web applications. This information can all be stored in configuration files and checked into source control and tested along the CD process, tools like CFEngine, Chef and Puppet are but a selection of tools that can help with this process.


Smoke Testing

The term smoke testing comes from plumbing, whereby artificial smoke was forced through pipes to see if there were any leakages of smoke coming through. The same idea applies to software testing, this process can be automated and manual. It is a sanity check that the deployment process was successful, smoke testing your deployment can be achieved by pinging various software components used in the architecture of the system. Or checking a certain number of features are what were expected and nothing has changed, though acceptance tests should cover this aspect.

Smoke Testing

The CI/CD process needs a little pain

Getting the right level of confidence in the CI/CD process requires a little pain and is not an easy process to get right first time. That’s why the aspect of continuous improvement, feedback and learning are vital; it is a staged process of improvement that will take a software delivery process to a continuous delivery dream-line. It is the little scars we receive during this learning process that help us remember and improve it next time.

Dilbert - Embrace Failures



Thanks for reading and keep building awesome.

Until next time!

Ian B

P.S. You can also find me on Twitter: @ianaldo21

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s