Wednesday, 06 May 2015 15:02

Is There a Place for QA in DevOps? Featured

Rate this item
(1 Vote)

DevOps has been around for several years, but there is still a lot of mystery around the term.  There are as many definitions of DevOps as there are tech blogs and articles out there.  

So what is DevOps?  Is it a buzz word, a process, a tool, an answer to all our questions?   Or is it like the Cloud - Everyone knows it’s the next best thing, but no one fully understands why. 

Like anything new, there are as many definitions of DevOps or at least opinions of what DevOps really is.   Similar to the Cloud, nobody fully understands it.  Surprisingly, unlike the Cloud, if you look up DevOps in Webster’s dictionary, you will not find any results.  But let’s see what other reputable sources have to say. 

According to Wikipedia[1], DevOps is a software development method which helps to rapidly produce software and bring Dev and Ops closer together. Since there is no mention of QA in this definition, we can assume that it is included into Dev.   Based on definition from “DevOps for Dummies” book by IBM[2]: DevOps is an approach based on lean and agile principles.  It mentions all stakeholders – including Dev, Business, Ops and QA.  

Another definition I heard at a DevOps training, is that DevOps is Lean for IT.  This is also supported by Gene Kim who wrote “The Phoenix Project”.  His quote is that “IT is the factory floor of this century”.   Lean principles are derived from Japanese Toyota manufacturing industry.  It considers any resources that do not create value for end customers to be wasteful and target for elimination. To become “lean” one needs to shed “fat” that is causing inefficiency and reduces productivity. cannot write about the “Phoenix Project” and DevOps, without mentioning the 3 Principles defined by Gene Kim.  First way is about understanding of the entire system as opposed to silos of work.  Second way is about shortening and amplifying the feedback loop for external and internal customers (for example, QA -> Dev).   Third way is about continual experimentation, taking risks and learning from failure.  A good example of that is Chaos Monkey, which is a tool created by Netflix that randomly simulates AWS system failures by shutting down the VMs.  

 One of my favorite definitions of DevOps was created by John Willis and is knows as CAMS.   “C” stands for Culture which includes People and Processes, which have to be defined and adapted first, before technology.  “A” stands for Automation which includes Tools for release management, provisioning, configuration management, testing, monitoring and orchestration.  “M” stands for Measurement, which is an essential part of continuous improvement via Performance, Process, and People and Quality metrics.  And “S” stands for Sharing that promotes sharing of ideas and knowledge via conferences, Meetups and Blogs. 

DevOps Bottlenecks

No matter how DevOps is defined, the real purpose of DevOps is to identify and resolve bottlenecks in the existing SDLC.    Since DevOps promotes Continuous Integration, Continuous Testing and Continuous Deployment, anything that breaks this Continuity is a potential bottleneck.  In many companies,  QA becomes that bottleneck for one or all of the following reasons: Extended Test Cycle,  Unstable test environments,  Brittle Test Framework , Manual Processes.   According to the "Theory of Constraints", a Chain is not stronger than its weakest link.  This is reinforced by Gene Kim in "The Phoenix Project", who said that "Any improvements made anywhere besides the bottleneck are an illusion".   I would like to explore each one of these weakest links, and discuss how they can be addressed so that QA becomes an enabler between Dev and Ops, instead of a bottleneck. 

Environment Bottleneck

If you ever heard the developer say “but it works on my computer”, you will agree that Environment is one of the major DevOps bottlenecks.  You cannot have a continuous build-test-deploy process, if developers and testers need to raise service requests with the Ops team to setup and provision build and test environments.    And whenever there is a manual process involved in the environment setup, there is a big chance that no two environments will be the same, which results in unstable and unreliable testing.   

In order to address this issue, there are great tools, a lot of them Open Source, that can be used to automated the Environment setup and provisioning processes.  Configuration management tools like Chef, Puppet and Ansible allow to document the infrastructure in the form of configuration scripts that can be stored in a Version Management system and treated as code.   Vagrant can be a great tool for automated environment provisioning and can be used in conjunction with VMWare, VirtualBox and Docker containers. 

Other techniques to ensure Environment availability are Environment Readiness Smoke Test and Service Virtualization. 

Test Cycle and Manual Processes Bottlenecks

If you work in an Agile environment, Test Cycle is often considered a bottleneck by Agile development teams.  Even if majority of testing is automated, automation execution takes time and results are often not reliable and take time to analyze.   Applying the Testing Pyramid to Automated testing, majority of automation should be done on the Unit Test Level, followed by API testing and only small portion of automation testing should be done on the GUI level.   Other techniques that can be used to reduce the Test Cycle are parallel execution, Selective testing and Automated Test Results analysis.  It’s also very important to automate any manual processes in the lifecycle.  Builds can be automated using tools like Gradle and Maven.  Test data management needs to be automated including gathering, generation and cleanup. 


Now that we explored all the definitions of DevOps, we all fully understand what DevOps is and are ready to go back to work and implement it.  Right?  Probably not.

There was a great article published recently in InformationWeek called DevOps: Stop the Existential Angst[3].  If you’re familiar with existentialism philosophy - The meaning of life, of the universe – is defined by what we do, not by what we say. So how about rather than asking what DevOps is, let’s ask what it does for us and our organization.  By answering what DevOps will do, we get a better understanding of what it is.

How does DevOps Improve Time to Market?   By automating testing, environment provisioning and data management.

How does DevOps Decrease Risks?  By promoting the practice of infrastructure as code and reducing the number of keystrokes thru automation.  This reduces human error which in turn reduces the risk.

How does DevOps Promote collaborative environments?  In order to fully understand the system and the process flows, all teams need to collaborate with each other. That means business, QA, security and development must cooperate to define the entire package. 

So let’s take the next steps – identify the bottlenecks, explore available frameworks and tools and Transform inefficient processes.  It’s up to IT practitioners like us to drive process improvements in our organization, get the product to the customers quicker and improve product quality.  



[1] Wikipedia:   DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and Information Technology(IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.”

[2]  DevOps for Dummies: DevOps is an approach based on lean and agile principles in which business owners and the development, operations, and quality assurance departments collaborate to deliver software in a continuous manner that enables the business to more quickly seize market opportunities and reduce the time to include customer feedback. “  



Read 357 times Last modified on Wednesday, 06 May 2015 16:10