Sometimes existing DevOps practices aren’t enough

I recently worked with a large financial organization that had already adopted agile and selected DevOps working practices. Although this was a good start, it was not enough. The company needed to come up with further improvements to enhance the speed and efficiency of their development work.

The company had several feature teams working on different locations. All of the teams had their own repositories and Continuous Integration pipelines for features, and also common ones that everybody shared. This setup did not deliver the results the company was expecting.

My team and I were invited in to improve the situation. Let’s look at the two major findings and recommendations that we came up with in terms of pipeline harmonisation and quality management.

Harmonising pipeline management without interfering ongoing development

We acknowledged that the various pipelines per feature area were only slightly different from each other. Hence, we proposed using a new common pipeline to achieve better control, traceability and automatic validation. 

We harmonised pipeline management and took a new common pipeline into use in parallel with the old pipelines. Transfer to the new pipeline was handled one repository at a time, which allowed us to replace old pipelines completely, step by step, without interfering the ongoing development work.

Addressing fluctuating quality

The fluctuating quality of the master branch was another major challenge. The company had feature teams that developed and validated their changes and merged the changes to the master. Frequently, these changes did not work as expected with the rest of the system.

For the other teams, this meant that something which worked earlier suddenly did not work anymore. Hunting for the reasons and getting the faults corrected meant lost time and effort for everyone. 

We addressed the problem by focusing on certain core activities before merging code to the master. The new pipeline was equipped with three pre-merge gating controls to ensure code quality: 

1. Static code analysis with SonarQube: This guaranteed test coverage of the new code and prevented adding new code with faults.
   
   
2. Peer review of new code with Gerrit UI: At least two persons needed to be familiar with each update. This promoted working together and improved team responsibility. The practice was extended to also cover test cases and test automation code, which allowed a common way of working in the project according to the paradigm everything-as-code. Team members were encouraged to work iteratively with small incremental changes, making self-contained commits that can be more easily cherry-picked and reverted if needed. The visibility and traceability of code were improved before and after the merge.
   
   
3. Automated functions: For example, building code in build servers and always running unit tests after every build.
   
   

A solid foundation for DevOps success

Put together, these simple steps – harmonising pipeline management and addressing fluctuating quality with pre-merge gating controls – enabled creating a common pipeline without any disruption to the ongoing development work, and ensured that no obvious faults were present in the code and all changes were easily identified and traced. The result? A great foundation for faster and far more efficient development work.