11 comments on “Organisation Antipattern: Release Testing”

  1. Excellent article, Steve.
    Release Testing is one of the ‘attractive’, alluring technical practices which seems to make sense initially to managers from a risk viewpoint, but which if anything *increases* risk by making quality and testing “someone else’s problem”.
    Release Testing does not scale: as the software produced by an organisation becomes both larger and more fundamental to that organisation’s strategy (revenue, communication, etc.), with Release Testing in place it becomes more and more difficult to release software, and more and more frustrating for developers, testers, and budget-holders, as the ‘downward spiral’ of ever-larger releases leads to more (not fewer) bugs and more disgruntled customers.
    Release Testing is in effect a sticking plaster over an open wound – it’s the wound which needs healing, and the sticking plaster only makes the problem worse in the end.

    • I’ve noticed that as an organization expands to have multiple teams working on a product, release testing is introduced. The organization continues to add teams, and it becomes more difficult to “remove” or diminish release testing even though release testing is slowing down the organization’s ability to deliver value. It’s an interesting cycle.

      • Hi Alison

        Are you suggesting organisations introduce Release Testing to solve scalability concerns? If so… good grief. That will only achieve the opposite of what you are after.

        Release Testing is very hard to improve, as it requires a level of transparency and courage within an organisation that is often lacking. I find it particularly frustrating when people acknowledge it is a problem, but then do nothing about it.

        Thanks for commenting


    • Hi Matthew

      Thanks for commenting. I’m not sure I agree Release Testing is attractive or alluring. I see it more as a) the status quo in large organisations – many people still believe in the segregation of Development and Testing, or b) an artifact of an Agile transformation that failed to permeate throughout an organisation.

      You are entirely right than Release Testing indirectly increases risk due to increased feedback loops/rework, but I believe the bigger issue is that it *cannot* reduce the risk of production defects without incurring an increased transaction cost. That really is risk management theatre.

      Scalability of Release Testing is something I omitted, but as development teams increase in number it’s certainly possible for a Release Testing team to become an ever-more visible bottleneck – which sadly is often “solved” by adding more people to the Release Testing Team, which means Brook’s Law comes into play.



      • Ah, I meant that Release Testing is initially attractive or alluring to managers looking to ‘manage risk’ as the software release process gets bigger/more complicated, and this initial attractiveness as a possible ‘silver bullet’ to kill quality woes is one reason why it takes hold in organisations. Another reason is a lack of understanding of how systems work (systems thinking), but that’s another story 🙂
        Release Testing is certainly *not* attractive or alluring, but too many orgs seem to realise this either too late or never at all.

  2. Great article Steve. I think release testing is harmful for all the reasons you list plus the fact that it produces large numbers of bug reports which don’t obviously belong to a development team. Bugs being raised by people outside the development teams are also unlikely to take in to account the prioritisation process required in Agile. Getting hung up on trying to produce perfect software is the fastest way to create project delays.

    As a tester my role is to find and share information about the product. Successful CD results in small batch sizes and frequent releases. Automated testing and cross-functional teams will usually give you enough confidence to be able to ship a decent product but you certainly won’t have done enough testing to have a full picture of the quality. I believe that to carry out decent testing you will need to be performing some sort of integration testing on the whole product. Depending on team size and schedule this may be something that can happen within the cross-functional teams but I don’t think it is particularly damaging to have it as a separate test team provided everyone understands that they are a specialist test team with a very clear objective. Security testing for example would fit this model without necessarily needing to prevent CD, depending on the risk to the company it might be enough to know that something has been broken rather than to prevent something from breaking.

    Every project is different and I think the problem is that many people adopt team set ups and practices without really considering why. Understanding how other people work is vital for emulating success but at the end of the day there are no best practices.

    • Hi Amy

      Thanks for replying. You’re right about the extraneous bug reports raised by a Release Testing team – they are a smell that a team is under pressure to find some defects, somewhere – but in my experience they tend to be very minor defects. Serious flaws in a product tend to be uncovered during unit/acceptance testing or exploratory testing (none of which can occur within a Release Testing team due to critical path constraints).

      I dislike the term “integration testing”, but I agree with your sentiment that there needs to be a holistic assessment of product quality off the critical path. I am still *very* wary of creating dedicated security, performance teams etc. as I believe it again breaks authority and responsibility. From the start of LMAX it was decreed there would be no dedicated performance team, and that performance was the responsibility of everyone – that culture can really work, but it is hard to retrofit it later on. Perhaps security specialists could be attached to the product team for a period of time.

      Thanks very much


  3. What seems to be a little bit missing here is HOW does a company that WAS using waterfall, and has (kinda) switched to some sort of water-Ag-fall hybrid that still has release testing, switch over (magically, in mid-stream?) to –

    a comprehensive, automated, unit-based suite of tests (would need to be multiple suites actually as we have many APIs) that:

    1. Runs nightly at a minimum
    2. On every platform (with both x86 and x64 user applications executing at the same time on 64-bit platforms)
    3. Is constantly added to during development by developers, in the language they use to add features and in the same codebase
    4. Has all tests evaluated for completeness by developers and testers (ideally other stakeholders as well)
    5. Has the authority to enforce a task/change is incomplete until doneness is satisfied
    6. Is supplemented with automated end-to-end tests using real world customer use cases and tools
    7. Reports back to stakeholders in a meaningful and non-ignorable way

    while carrying on business, not disappointing customers (who, in turn, compete in their industries using our products), and not letting any killer bugs out the door where lives and livelihoods (and trillions of dollars of transactions, and state secrets, and…) all depend on the security and robustness of the released product?

    From here, it looks a wee bit like crossing a chasm in two leaps.

    • Hi Kevin

      Thanks for that.

      It’s a fair criticism that I don’t go into a lot of detail on the antidote to Release Testing, or how to build up an extensive suite of automated tests.

      When I’m writing, I try to practice what I preach about batch size reduction and only ever write about one thing. In this particular case, my focus was upon the problems introduced by Release Testing and a potential solution. A lot of stuff was cut out, for example on the history of quality inspection.

      For more details on how to create a really effective suite of automated tests that follow the Testing Pyramid, check out the Continuous Delivery book by Dave Farley and Jez Humble if you haven’t already – there’s a whole chapter on it.

      Thanks again


  4. Often the challenge is how to wean your organization off the Release Testing model toward continuous delivery. I believe smaller development batch sizes, relentless focus on automation, cross-functional teams, and real-time communication and reporting associated with each revision candidate as it travels through the deployment pipeline, can be the best first steps.

    -Dennis Ehle, CEO http://www.cloudsidekick.com

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>