We’ve written about risk before but wanted to revisit the topic. Generally people and organizations approach risk management or mitigation from one perspective, reducing the probability of failure. We typically do this with systems by testing extensively. While this is useful there is only so much that can be found by testing in a simulated environment or with simulated users. In addition to testing, the company should consider the duration of the failure and the percentage of customers impacted, as show in the figure below.

Let’s go through each one of the items and identify what specifically you can do to accomplish these.

• Payload Size – The smaller the change, the less risk. This is the concept behind continous deployment where every code commit is released to production, assuming it passes the automated build and test processes. While continuous deployment isn’t right for every organization the concept of releasing smaller more frequent releases to reduce risk is applicable to everyone.
• Testing – As we stated before, you cannot test quality into a system and it is mathematically impossible to test all possibilities within complex systems to guarantee the correctness of a platform or feature. Responsibility for the quality of a feature resides with the engineer and begins with unit tests. Test driven development is the process of writing a failing automated test and then writing or modifying code in order to pass that test. There are mixed opinions about the pros and cons of TDD but anything that makes an engineer write more unit tests is likely to improve quality.
• Monitoring – The key to monitoring is to select a few key business metrics to monitor. Per our earlier article first you need to determine if there IS a problem and then use all the other monitoring that we are used to like Nagios, New Relic, Cacti, etc to determine WHERE and WHAT is the problem.
• Rollback – As much as we’ve written on the importance of being able to rollback code I’m not sure more that I can add except having lived through two major outages without the ability to rollback I will never push code again that can’t be rolled back.
• Architecture – By splitting your applications and database along Y and Z axes will allow parts of the service to continue functioning should one part fail. This swim lane or fault isolation approach will provide greater availability for your overall service.

Having discussed why organizations arranged along functional boundaries (e.g. production or engineering, operations, infrastructure, QA, etc) and multi-disciplinary Agile teams are often at odds and counterproductive, we now propose a method of organizing product teams more aligned with the Agile process and Agile methods.

The potential solution is really very simple.  The design goals behind the solution are: maximization of individual capabilities of any given team to quickly and efficiently create solutions and solve problems; creation of ownership of individual products (or portions of products) and the business success metrics associated with that product in order to maximize innovation; maximization of morale through a sense of ownership and delegation of decision making; and reduction in time to market through the elimination of non-value added conflict.

The solution is clear given these goals – eliminate functional organizations and align the organization to architecture specific services.   In the ideal case, each of the senior leaders of these organizations are capable of owning and leading one more complete agile teams.   The number of teams that an executive has is likely going to depend on the size of the company, the size of the engineering and product teams and the number of discrete services in the company’s product architecture.

Here we come to a very important point – it is critically important that the architecture, the process and the organization structure be aligned to reap the benefits of such a change.  If the product architecture continues to be monolithic, nothing is solved.  The solution described above will get you no further than an “agile overlay across functional organizations”.  Each division of agile teams needs to own their own services so that disputes, problems and opportunities can be resolved or capitalized within the team.  This rapid resolution starts to slow down when outside resources are necessary to resolve a problem or capitalize on an opportunity.

We readily admit that this new approach to eliminating functional organizations in favor of agile teams isn’t for everyone.  Some companies don’t have the need to segment their architectures into services as they aren’t experiencing rapid growth.  As such, they shouldn’t pay the price of re-architecting their product.  Some companies deliver to very clearly defined specifications and as a result can’t really benefit from the product discovery aspects inherent to Agile or the questionable “final dates”.  These companies are probably better off continuing to develop in a waterfall fashion.  Many companies won’t have the right skill sets at the top to have former functional executives own product sections.

This last issue sets up a topic for another post.  The question we aim to answer is “What are the qualities and attributes of the Agile Executive?”

What do you call a set of processes or systems for coordination between development and operations teams? Give up? Try “DevOps”. While not a new concept, we’ve been living and suggesting ARB and JAD as cornerstones of this coordination for years, but it has recently grown into a discipline of its own. Wikipedia states that DevOps “relates to the emerging understanding of the interdependence of development and operations in meeting a business’ goal to producing timely software products and services.” Tracking down the history of the DevOps Wikipedia page, shows that this topic is a recent entry.

There are a lot of other resources on the web that many not have been using this exact term but have certainly been dealing with the development and operations coordination challenge for years.  Dev2Ops.org is one such group and posted earlier this year their definition of DevOps “an umbrella concept that refers to anything that smoothes out the interaction between development and operations.”  They continue in their post highlighting that concept of DevOps is in response to the growing awareness of a disconnect between development and operations. While I think that is correct I think it’s only partially the reason for the recent interest in defining DevOps.

With ideas such as continuous deployment and Amazon’s two-pizza rule for highly autonomous dev/ops teams there is a blurring of roles between development and operations. Another driver of this movement is cloud computing. Developers can procure, deploy, and support virtual instances much easier than ever before with the advent of GUI or API based cloud control interfaces. What used to be clearly defined career paths and sets of responsibilities are now being blended to create a new, more efficient and highly sought after technologist. A developer who understands operations support or a system administrator who understands programming are utility players that are very valuable.

While perhaps DevOps is a new term to an old problem, it is promising to realize that organizations are taking interest in the challenges of coordination between development and operations. It is even more important that organizations pay attention to this topic given the blurring of roles.

It seems that everyone is moving to an Agile approach in their product (or software) development lifecycle.  Some move with great success, some with great fanfare and for some it’s one of the last moves their company and engineering organizations make before failing miserably and shutting doors permanently.  As often as not, companies just move because they believe it will cure all of their problems.  As we’ve written before, no new PDLC will cure all of your problems.  Agile may in fact be best for you, but there are always tradeoffs.

We’ve compiled a top 5 misconceptions about Agile from our experience working with companies to solve problems.  Be careful of these pitfalls as they can cause you to fail miserably in your efforts.

1)      Agile is NOT a reason to NOT manage engineers or programmers

Engineering organizations measure themselves.  In fact, many Agile methods include a number of metrics such as burn down charts to measure progress and velocity for measuring the rate at which teams deliver business value.  As with any other engineering organization, you should seek to find meaningful metrics to understand and improve developer and organizational quality and productivity.  Don’t let your team or your managers tell you not to do your job!

2)      Agile is NOT a reason to have engineering alone make product decisions

You still have to run your business which means nesting the product vision to the vision of the company.   More than likely you are paying business or product people to define what it is you need to do to fight and win in your market.  Someone still has to set the broad strategic vision to which products will be developed.  Engineers can and should contribute to this vision, and that’s where Agile methods can help.

3)      Agile alone is NOT a cure for all of your product delivery problems

As we’ve blogged before, there simply are no silver bullets.  With the right management and oversight, you can make any development lifecycle work.  There are just cases where Agile methods work better.   But don’t look to have a PDLC fix your business, people or management issues.

4)      Agile is NOT an excuse to NOT put in the appropriate processes

There is nothing in the Agile manifesto that states that process is evil.  In fact, it recognizes processes are important by stating that there is value within them.  Agile simply believes that individuals and interactions are more important – a concept with which we completely agree.   Don’t argue that all process is evil and cite Agile as the proof as so many organizations seem to do.  Certain processes (e.g. code reviews) are necessary for hyper growth environments to be successful.

5)      Agile is NOT an excuse not to create SOME documentation

Another often misinterpreted point is that Agile eliminates documentation.  Not only is this not true, it is ridiculous.  The Agile manifesto again recognizes the need for documentation and simply prefers working software over COMPREHENSIVE documentation.  Go ahead and try to figure out how to use or troubleshoot something for the very first time without documentation and see how easy it is…  Programming before designing coupled with not creating  useful documentation makes one a hack instead of an engineer.  Document – just don’t go overboard in doing it.

As a frequent technology writer I often find myself referring to the method or process that teams use to produce software. The two terms that are usually given for this are software development life cycle (SDLC) and product development life cycle (PDLC). The question that I have is are these really interchangeable? I don’t think so and here’s why.

Wikipedia, our collective intelligence, doesn’t have an entry for PDLC, but explains that the product life cycle has to do with the life of a product in the market and involves many professional disciplines. According to this definition the stages include market introduction, growth, mature, and saturation. This really isn’t the PDLC that I’m interested in. Under new product development (NDP) we find a defintion more akin to PDLC that includes the complete process of bringing a new product to market and includes the following steps: idea generation, idea screening, concept development, business analysis, beta testing, technical implementation, commercialization, and pricing.

Under SDLC, Wikipedia doesn’t let us down and explains it as a structure imposed on the development of software products. In the article are references to multiple different models including the classic waterfall as well as agile, RAD, and Scrum and others.

In my mind the PDLC is the overarching process of product development that includes the business units. The SDLC is the specific steps within the PDLC that are completed by the technical organization (product managers included). An image on HBSC’s site that doesn’t seem to have any accompanying explanation depicts this very well graphically.

Another way to explain the way I think of them is to me all professional software projects are products but not all product development includes software development.  See the Venn diagram below. The upfront (bus analysis, competitive analysis, etc) and back end work (infrastructure, support, depreciation, etc) are part of the PDLC and are essential to get the software project created in the SDLC out the door successfully.  There are non-software related products that still require a PDLC to develop.

Do you use them interchangeably?  What do you think the differences are?

You probably have heard of continuous integration that is the practice of checking code into the source code repository early and often.  The goal of which is to ease the often very painful process of integrating multiple developer’s code after weeks of independent work. If you have never had the pleasure of experiencing this pain, let me give you another example that we have experienced recently. In the process of writing The Art of Scalability, we have seven editors including an acquisition editor, a development editor, and five technical editors who all provide feedback on each chapter. Our job is to take all of this separate input and merge it back into a single document, which at times can be challenging when editors have different opinions for the direction of certain parts of the chapter. The upside of this process is that it does make the manuscript much better for having gone through the process. Luckily software engineering has developed the process of continuous integration designed to reduce wasted engineering effort. In order to make this process the most effective the automation of builds and smoke tests are highly recommended. For more information on continuous integration there are a lot of resources such as books and articles.

The topic of this post is taking continuous integration to an extreme and performing continuous deployment. And it is exactly what it sounds like, all code that is written for an application is immediately deployed into production. If you haven’t heard of this before you’re first thought is probably that this is the ultimate in Cowboy Coding but it is in use by some household technology names like Flickr and IMVU. If you don’t believe this check out code.flickr.com and look at the bottom of the page, last time I checked it said:

Flickr was last deployed 20 hours ago, including 1 change by 1 person.

In the last week there were 34 deploys of 385 changes by 17 people.

Eric Ries, co-founder and former CTO of IMVU, is a huge proponent of continuous deployment as a method of improving software quality due to the  discipline, automation, and rigorous standards that are required in order to accomplish continuous deployment. Other folks at IMVU also seem to be fans of the continuous deployment methodology as well from the post by Timothy Fitz. Eric suggest a 5 step approach for moving to a continuous deployment environment.

1. Continuous Integration – Obviously before moving beyond integration into full deployment, this is a prerequisite that must be in place.
2. Source Code Commit Checks – This feature which is available in almost all modern source code control systems,  allows the process of checking in code to halt if one of the tests fail.
3. Simple Deployment Script – Deployment must be automated and have the ability to rollback, which we wholeheartedly agree with here and here.
4. Real-time altering – Bugs will slip through so you must be monitoring for problems and have the processes in place to react quickly
5. Root Cause Analysis – Eric recommends the Five Why’s approach to find root cause, whatever the approach, finding and fixing the root cause of problems is critical to stop repeating them.

Admittedly, this concept of developers pushing code straight to production scares me quite a bit, since I’ve seen the types of horrific bugs that can make their way into pre-production environments. However, I think Eric and the other continuous deployment proponents are onto something that perhaps the reason so many bugs are found by weeks of testing is a self-fulfilling prophecy. If engineers know their code is moving straight into production upon check in they might be a lot more vigilant about their code, I know I would be. How about you, what do you think about this development model?