We had the chance to speak with Phil Windley on the Technometria series at IT Conversations.

Here is the link to the podcast from the show.

If a company has already established a disaster recovery plan that involves failover to a cold or passive datacenter it is often hard to convince them to switch to a solution that involves taking traffic at both datacenters. We call this failover type of architecture active-passive and active-active when you accept traffic at both datacenters. If you too believe active-passive architecture is a satisfactory solution for disaster recovery consider this analogy.

Most cars have four tires and a spare tire in the trunk. However, you might have noticed that semi-trucks have pairs of tires on both the tractor and trailer, except for the front wheels that turn. An active-passive architecture is like a car and the active-active solution is like the semi-trucks. Here are three comparisons that might help sell you on an active-active solution.

1) Even with the best of intentions active-passive solutions don’t get tested regularly. How often do you practice changing or even check your spare tire? If you don’t check your spare tire periodically for the correct air pressure it might be flat when you need it most. Passive datacenters are the same way. If you don’t rollout code to it every release and occasionally take actual traffic, it probably won’t work when you need it.

2) Active-active solutions are is much faster to take over traffic when there is a disaster. If your racing down the road which is faster when you get a flat, stop and replace the flat tire with the spare or keep riding on the extra tire? Even if you use a DNS solution like UltraDNS that can failover quickly, you’ll likely need to warm up cache, apply the last round of data logs, etc. before you can take traffic safely in a passive datacenter.

3) Active-active solutions make better use of the investment in equipment than active-passive solutions. The spare tire in your trunk might get used once every year, if you’re unlucky. The second tire on the semi gets used every day helping carry a greater load and reducing the wear on the other tire.

While active-passive is better than not having a disaster recovery plan it’s not the best that you can do. Consider getting to an active-active solution that exercises your DR solution every day and makes use of all that investment.

Here are a few videos that we did to explain Scalability Rules…ignore the scary faces:

You can find more videos by following this link.

Here are a couple reviews of the book:
InfoQ
Code Ranch
LinkedIn Reviews

In this third installment of our “Agile Organization” series we discuss the qualities and attributes necessary for someone to lead a group of cross functional Agile teams in the development of a web-centric product.  For the purposes of this discussion, the Agile Executive is the person responsible for leading a group of agile teams in the development of a product.

In a world with less focus on functional organizations such as the one we’ve described in our Agile Organization articles, it is imperative that the leadership have a good understanding of all domains from the overall business through product management and finally each of the technical domains.  Clearly such an individual can’t be an expert in every one of these areas, but they should be deep in at least one and broad through all of them.  Ideally this would have been the case in the functional world as well, but alas functional organizations exist to support deep rather than broad domain knowledge.  In the Agile world we need deep in at least area and broad in all areas.

Such a deep yet broad individual could come from any of the functional domains.  The former head of product management may be one such candidate assuming that he or she had good engineering and operations understanding.  The head of engineering and operations may be heads of other agile teams, assuming that they have a high business acumen and good product understanding.  In fact, it should not matter whence the individual comes, but rather whether he or she has the business acumen, product savvy and technical chops to lead teams.

In our view of the world, such an individual will likely have a strong education consisting of an undergraduate STEM (science, technology, engineering or math) degree.  This helps give them the fundamentals necessary to effectively interact with engineers and add value to the engineering process.  They will also have likely attended graduate school in a business focused program such as an MBA with a curriculum that covers finance, accounting, product and strategy.  This background helps them understand the language of business.  The person will hopefully have served for at least a short time in one of the engineering disciplines as an individual contributor to help bridge the communication chasm that can sometimes exist between those who “do” and those who “dream”.  As they progress in their career, they will have taken on roles within product or worked closely with product in not only identification of near term product elements, but the strategic evaluation of product needs longer term as well.

From the perspective of philosophy, the ideal candidates are those who understand that innovation is more closely correlated with collaboration through wide networks than it is to the intelligence of one individual or a small group of people.  This understanding helps drive beneficial cognitive conflict and increased contribution to the process of innovation rather than the closed minded approach of affective conflict associated with small groups of contributors.

In summary, it’s not about whence the person comes but rather “who the person is”.  Leading cross disciplinary teams requires cross disciplinary knowledge.  As we can’t possibly experience enough personally to be effective in all areas, we must broaden ourselves through education and exposure and deepen ourselves through specific past experiences.  Most importantly, for a leader to succeed in such an environment he or she must understand that “it’s not about them” – that success is most highly correlated with teams that contribute and not with just being “wickedly smart”.

In case you missed us at Etsy’s Code as Craft Speaker Series, don’t worry you can watch the full video here:

Having discussed why organizations arranged along functional boundaries (e.g. production or engineering, operations, infrastructure, QA, etc) and multi-disciplinary Agile teams are often at odds and counterproductive, we now propose a method of organizing product teams more aligned with the Agile process and Agile methods.

The potential solution is really very simple.  The design goals behind the solution are: maximization of individual capabilities of any given team to quickly and efficiently create solutions and solve problems; creation of ownership of individual products (or portions of products) and the business success metrics associated with that product in order to maximize innovation; maximization of morale through a sense of ownership and delegation of decision making; and reduction in time to market through the elimination of non-value added conflict.

The solution is clear given these goals – eliminate functional organizations and align the organization to architecture specific services.   In the ideal case, each of the senior leaders of these organizations are capable of owning and leading one more complete agile teams.   The number of teams that an executive has is likely going to depend on the size of the company, the size of the engineering and product teams and the number of discrete services in the company’s product architecture.

Here we come to a very important point – it is critically important that the architecture, the process and the organization structure be aligned to reap the benefits of such a change.  If the product architecture continues to be monolithic, nothing is solved.  The solution described above will get you no further than an “agile overlay across functional organizations”.  Each division of agile teams needs to own their own services so that disputes, problems and opportunities can be resolved or capitalized within the team.  This rapid resolution starts to slow down when outside resources are necessary to resolve a problem or capitalize on an opportunity.

We readily admit that this new approach to eliminating functional organizations in favor of agile teams isn’t for everyone.  Some companies don’t have the need to segment their architectures into services as they aren’t experiencing rapid growth.  As such, they shouldn’t pay the price of re-architecting their product.  Some companies deliver to very clearly defined specifications and as a result can’t really benefit from the product discovery aspects inherent to Agile or the questionable “final dates”.  These companies are probably better off continuing to develop in a waterfall fashion.  Many companies won’t have the right skill sets at the top to have former functional executives own product sections.

This last issue sets up a topic for another post.  The question we aim to answer is “What are the qualities and attributes of the Agile Executive?”

We’ve done a lot of work with organizations attempting to become more Agile by implementing Agile development practices.  One common problem we see time and time again is that the “old school” way of defining organization structure starts to lose its value in an Agile world.  Here I am specifically talking about organization structures developed around functional roles such as development (or engineering), QA, Operations, Infrastructure, etc.

This old method of organizing, which resembles a Y axis split within the AKF scale cube served our industry well for a long time.  And, for many organizations, it can continue to work well.  It particularly works well in organizations that follow waterfall models as the organization structure mimics the flow and passage of work through certain gates.  The structure is also comforting in its familiarity as most long tenured managers and individual contributors have worked within similarly structured organizations their entire professional careers.

But in the Agile world, this organization structure doesn’t add as much value as in the Waterfall world.  In fact, I argue that it’s counter-productive in many ways.  The first and perhaps most benign issue is that the actual structure of the organization does not foster work-flow.  Unlike waterfall development where one group hands off a project to another in phases (development to QA, QA to operations, etc), Agile methods seek to develop and deploy seamlessly.  To be successful the Agile team needs representation from multiple stakeholders within functional groups.  As individuals now spend most of their time in cross functional teams, what value does the functional group offer?  In essence, these functional organizations become the analog to the “home room” in school.

The next problem is the inherent conflict created between the Agile team and the functional organization.  To be truly effective, the team must be empowered to some degree.  What power or responsibility does the functional leader then have?  If he or she isn’t responsible for a specific product, are they to be given some sort of veto power?  Such a notion has meaning in the waterfall world but really runs counter to the time to market and discovery objectives of Agile methods.  The resulting affective conflict simply doesn’t add value to the overall product.  In fact, as research shows, it destroys value.  Some proponents of continuing with functional organizations might indicate that the functional groups allow for more effective management and mentoring of individuals within their domain.  Given how little time managers truly spend on mentoring relative to other tasks, I highly doubt this is the case for most organizations.  Our experience is that the functional groups spend more time arguing over ownership of certain decisions (affective conflict) rather than mentoring, training and evaluating individual contributors.

Perhaps the largest problem – larger than the lack of support for work flow and the creation of conflict – is that implementing agile processes across functional organizations sub-optimizes innovation.  Research indicates that innovation happens most frequently and beneficially within groups of individuals with diverse and non-overlapping experience across a number of domains (functional and experiential diversity) and with non-redundant links to individuals outside of their organization.  By engaging in beneficial debate (cognitive conflict) on approaches to a certain problem or opportunity, the perspective and networks brought by each person widens the potential solution set.  Alas, this is the true unheralded value of agile development teams that properly incorporate multiple disciplines within the team (QA, dev, product, infrastructure).  Each of these individuals not only brings new and valuable expertise in how to develop a product, they also have contacts outside the organization unlikely to be matched by each of their peers.  Innovation quality and frequency therefore increases.  But the inherent conflict in multiple competing organizational affiliations will dampen this innovation.  So not only is there conflict and a lack of workflow, the potential major benefit is removed or at the very least diminished.

Having discussed the problems inherent to functional organizations and agile processes, we’ll next discuss how a company might “organize” around agile to be more effective.

Given the debt crises in Italy, Greece, and the US, just to name a few, the idea of debt is often parts of our conversations. As little as five years ago who would have believed that the US could double it’s debt that took 200 years to accumulate. While we all probably think this is outrageous let’s consider the debt that we accumulate within our systems. We’ve all heard and probably used the term technical debt but do we really understand how bad it can affect us and how quickly?

We often tell clients that they need to spend 12 – 25% of their engineering time on maintenance issues to pay down this debt. Failing to do so can result in some horrendous work stoppages. As an example, we had a client that has been around for a little more than a decade and had a great little profitable business. They were bringing in finance to grow their operations outside of the NY-metro area and needed help scaling. Unfortunately they had ignored refactoring, scaling, and maintenance in general for most of the time they’ve been in business. The result was that when a potential client asked to add a second email address on accounts the estimate came back at 1,500 hours…yes, that’s right 190 days or about 3/4 of a year of a developer to simply add an email field. Given that engineers typically are optimistic about how long it will take to accomplish something, imagine how long this really took.

The take away here for both our countries and our organizations is to not let debt pile up. Face the pain and balance the budget because if you don’t it only gets worse.

Whether you love Scalability Rules or you haven’t gotten around to purchasing your copy, check out the new android application. The app has the what, when, why, and how for each of the 50 rules. Follow this link, scan this QR code, or search for “scalability” in the android market place.

How cool is this?! We’ve been asked to speak at Etsy’s Code as Craft Speaker Series. Our presentation will take place on July 28th at 7pm at the Etsy Labs on the 7th floor at 55 Washington Street in beautiful Brooklyn. If you’re in the neighborhood, sign up here and stop by.