AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Have you read our latest blog post?

Print

Continuous Deployment

You probably have heard of continuous integration that is the practice of checking code into the source code repository early and often.  The goal of which is to ease the often very painful process of integrating multiple developer’s code after weeks of independent work. If you have never had the pleasure of experiencing this pain, let me give you another example that we have experienced recently. In the process of writing The Art of Scalability, we have seven editors including an acquisition editor, a development editor, and five technical editors who all provide feedback on each chapter. Our job is to take all of this separate input and merge it back into a single document, which at times can be challenging when editors have different opinions for the direction of certain parts of the chapter. The upside of this process is that it does make the manuscript much better for having gone through the process. Luckily software engineering has developed the process of continuous integration designed to reduce wasted engineering effort. In order to make this process the most effective the automation of builds and smoke tests are highly recommended. For more information on continuous integration there are a lot of resources such as books and articles.

The topic of this post is taking continuous integration to an extreme and performing continuous deployment. And it is exactly what it sounds like, all code that is written for an application is immediately deployed into production. If you haven’t heard of this before you’re first thought is probably that this is the ultimate in Cowboy Coding but it is in use by some household technology names like Flickr and IMVU. If you don’t believe this check out code.flickr.com and look at the bottom of the page, last time I checked it said:

Flickr was last deployed 20 hours ago, including 1 change by 1 person.

In the last week there were 34 deploys of 385 changes by 17 people.

Eric Ries, co-founder and former CTO of IMVU, is a huge proponent of continuous deployment as a method of improving software quality due to the  discipline, automation, and rigorous standards that are required in order to accomplish continuous deployment. Other folks at IMVU also seem to be fans of the continuous deployment methodology as well from the post by Timothy Fitz. Eric suggest a 5 step approach for moving to a continuous deployment environment.

The topic of this post is taking continuous integration to an extreme and performing continuous deployment. And it is exactly what it sounds like, all code that is written for an application is immediately deployed into production. If you haven’t heard of this before you’re first thought is probably that this is the ultimate in ‘Cowboy Coding’ but it is in use by some household technology names like Flickr and IMVU. If you don’t believe this check out code.flickr.com and look at the bottom of the page, last time I checked it said:
Flickr was last deployed 20 hours ago, including 1 change by 1 person.
In the last week there were 34 deploys of 385 changes by 17 people.
Eric Ries, CTO of IMVU, is a huge proponent of continuous deployment as a method of improving software quality due to the  discipline, automation, and rigorous standards that are required in order to accomplish continuous deployment. Eric suggest a 5 step approach for moving to a continous deployment environment.
  1. Continuous Integration – Obviously before moving beyond integration into full deployment, this is a prerequisite that must be in place.
  2. Source Code Commit Checks – This feature which is available in almost all modern source code control systems,  allows the process of checking in code to halt if one of the tests fail.
  3. Simple Deployment Script – Deployment must be automated and have the ability to rollback, which we wholeheartedly agree with here and here.
  4. Real-time altering – Bugs will slip through so you must be monitoring for problems and have the processes in place to react quickly
  5. Root Cause Analysis – Eric recommends the Five Why’s approach to find root cause, whatever the approach, finding and fixing the root cause of problems is critical to stop repeating them.

Admittedly, this concept of developers pushing code straight to production scares me quite a bit, since I’ve seen the types of horrific bugs that can make their way into pre-production environments. However, I think Eric and the other continuous deployment proponents are onto something that perhaps the reason so many bugs are found by weeks of testing is a self-fulfilling prophecy. If engineers know their code is moving straight into production upon check in they might be a lot more vigilant about their code, I know I would be. How about you, what do you think about this development model?

Author:

Comments RSS TrackBack 6 comments

  • Julian Simpson

    in June 23rd, 2009 @ 05:19

    I think it all comes down to the industry, and (as usual) the amount of money and lives at stake. I’m all for making it clear to developers that quality is a process that starts before they write code and ends once the code works in prod, but I’m not sure I’d suggest it for a medical records system, for example :)

  • Mr. Hericus

    in June 23rd, 2009 @ 08:10

    First, Continuous Integration is not just committing code to the repository early and often. It includes that, but it goes beyond that to actually checking out the new code, running the compiles, and performing the integration tests on it to ensure that the newly committed code does in fact integrate well with the existing code.

    I agree with Julian’s comments that continuous deployment probably depends highly on the specific industry in which it is applied. However, even in highly non-fault-tolerant industries (medicine, nuclear, NASA, etc.) continuous deployment can be implemented, as long as it is done in stages.

    You can continuously deploy to development servers, and even QA servers and realize a lot of the benefit of the practice. There just has to be even more that goes into the production release.

    Thanks for the post!

    Mr. Hericus

  • Fish

    in June 23rd, 2009 @ 08:48

    Mr. Hericus, thanks for pointing out that Continuous Integration goes beyond committing code. Your idea of Continuous Deployment to dev and QA environments makes a lot of sense to gain some of the benefits while mitigating some of the risk.

    Thanks for the comments!

  • Wabb

    in June 23rd, 2009 @ 15:06

    We certainly do appreciate the feedback and comments – thanks Mr. Hericus and Julian! I agree that much of this topic hinges on risk, including both the probability of a problem happening and the impact or size of that problem once it happens. Of course both of these are one step removed from the question of “What creates the most, or best optimizes shareholder value?” – or where do we create value and how much does that value creation cost us?

    While not universally true, smaller releases have proven in many companies to result in smaller more manageable issues upon release (as compared to the larger issues that tend to be happen within very large releases). More frequent releases, however, tend to create more frequent (albeit) smaller issues. Once again – these points don’t hold true everywhere but as a generalization they are consistent with our past experiences while running tech organizations and companies and advising our current clients.

    More frequent releases also offer the benefit of faster time to market. If a company releases once a quarter and a feature is ready one week after the last release date, it might wait another 12 weeks to release as compared to a company that releases weekly in which the average delay for a release will be 3.5 days (assuming an even distribution of probability of completion across the 7 day week).

    The question then should be what is the value of the process as compared to the cost and risk? We’ve always pushed our clients to more frequent releases to reduce the magnitude of a potential issue and decrease time to market. But continuous releases take this to an extreme! I suspect that the comments are correct and that this process is great in some areas (assuming that you have appropriate QA automation and risk mitigation) and potentially not appropriate for others.

    Questions I would look to have answered in determining release frequency/velocity:
    1) Quality cost of the frequency/velocity? Do we need more or less QA? What’s our cost of quality assurance?
    2) Relative quality – does the actual AND customer perceived quality go up or down? Note – these are 2 separate issues in many cases. Sometimes customers don’t like things to change underneath their feet daily and while your absolute quality may go up – customers may perceive something different. Customers may also like the rapid change as the magnitude of any change is imperceptible (v. a release of Office 2007 for instance..>)
    3) Cost of development – including and excluding cost of quality – what happens to your overall development costs? Do your engineers have more available hours for development or less? Do they get more done or less? Do they create more value or less?
    4) Value creation – similar to (3) above – but are engineers getting more features out the door and are they generating more revenue overall?
    5) Risk and magnitude of impact – Are you causing more problems or less? Are the problems bigger or smaller?
    6) Time to market – are you faster, the same or slower with the changes? From ideation to launch – are you getting things out the door faster or more slowly?

    Some of these obviously overlap – and I’m sure there are more. Any more ideas on the right questions by which to evaluate this or any other release process?

  • uberVU - social comments

    in January 26th, 2010 @ 08:11

    Social comments and analytics for this post…

    This post was mentioned on Twitter by mfisher5kavika: New blog post: Continuous Deployment http://bit.ly/QRN2U…

  • Burton Haynes

    in June 20th, 2010 @ 08:37

    Interesting thoughts on this process