In this third installment of our “Agile Organization” series we discuss the qualities and attributes necessary for someone to lead a group of cross functional Agile teams in the development of a web-centric product.  For the purposes of this discussion, the Agile Executive is the person responsible for leading a group of agile teams in the development of a product.

In a world with less focus on functional organizations such as the one we’ve described in our Agile Organization articles, it is imperative that the leadership have a good understanding of all domains from the overall business through product management and finally each of the technical domains.  Clearly such an individual can’t be an expert in every one of these areas, but they should be deep in at least one and broad through all of them.  Ideally this would have been the case in the functional world as well, but alas functional organizations exist to support deep rather than broad domain knowledge.  In the Agile world we need deep in at least area and broad in all areas.

Such a deep yet broad individual could come from any of the functional domains.  The former head of product management may be one such candidate assuming that he or she had good engineering and operations understanding.  The head of engineering and operations may be heads of other agile teams, assuming that they have a high business acumen and good product understanding.  In fact, it should not matter whence the individual comes, but rather whether he or she has the business acumen, product savvy and technical chops to lead teams.

In our view of the world, such an individual will likely have a strong education consisting of an undergraduate STEM (science, technology, engineering or math) degree.  This helps give them the fundamentals necessary to effectively interact with engineers and add value to the engineering process.  They will also have likely attended graduate school in a business focused program such as an MBA with a curriculum that covers finance, accounting, product and strategy.  This background helps them understand the language of business.  The person will hopefully have served for at least a short time in one of the engineering disciplines as an individual contributor to help bridge the communication chasm that can sometimes exist between those who “do” and those who “dream”.  As they progress in their career, they will have taken on roles within product or worked closely with product in not only identification of near term product elements, but the strategic evaluation of product needs longer term as well.

From the perspective of philosophy, the ideal candidates are those who understand that innovation is more closely correlated with collaboration through wide networks than it is to the intelligence of one individual or a small group of people.  This understanding helps drive beneficial cognitive conflict and increased contribution to the process of innovation rather than the closed minded approach of affective conflict associated with small groups of contributors.

In summary, it’s not about whence the person comes but rather “who the person is”.  Leading cross disciplinary teams requires cross disciplinary knowledge.  As we can’t possibly experience enough personally to be effective in all areas, we must broaden ourselves through education and exposure and deepen ourselves through specific past experiences.  Most importantly, for a leader to succeed in such an environment he or she must understand that “it’s not about them” – that success is most highly correlated with teams that contribute and not with just being “wickedly smart”.

It doesn’t matter if you run a commerce site, a services product (such as a SaaS offering) or simply use your homepage to distribute information:  The table stakes for playing online is high availability.  So many companies just take for granted that they will be highly available because they have multiple instances of systems and multiple copies of their data.  This assumption of availability will likely, at the very least, cost you significant pain and in the extreme cost you either significant market share or close your doors as a business.  Customers expect the unachievable – 100% availability.  At the very least you need to give them something close to that.  What will happen to you if you have a data center failure?  How about if a DBA accidentally drops a critical table in your production database?  What will you do when that marketing campaign triggers a near overnight doubling of traffic?  What happens when that new feature has a significant performance bug and gets adopted so quickly that it brings your entire site to its knees?

We often tell our clients that they should treat high availability as a feature.  Unfortunately, it is a somewhat expensive feature that requires constant investment overtime to achieve and maintain. It is a must have feature that will only differentiate your firm if you have competitors who do not make similar investments; when competition exists, customers are more likely to leave a site for a competitor due to availability and performance issues than nearly any other reason.  If you don’t believe us on this topic, just go ask the folks at Friendster.

Treating availability as a feature means measuring availability from a customer perspective rather than a systems or device perspective.  How many times did customer requests not complete?  In this regard, availability now becomes a percentage of failed transactions against an expected number of transactions.   We define an approach to accomplish this in our first book “The Art of Scalability”.  Every executive in the company should “own” the availability metric and understand its implication to the business.    You should track how much you invest in availability over time and significant decreases in engineering or capital should be questioned as it may be an early indicator that you are under investing and a harbinger of hard times to come.

One of the most common failures we see is to assume that disaster recovery is something that only big companies need.  Make no mistake about it, disasters do happen and given enough time they will happen to you.  Data centers catch on fire, have water (sprinkler) discharges that ruin equipment, have complete power equipment failures that take hours to fix and are prone to damages from vehicles, earthquakes, employees and tornados.  In our past lives as executives and current roles as advisors we’ve seen no less than 4 data center fires, 2 data centers incapacitated from earthquakes and tornados and one data center leveled by a truck running into it.  You are never too young to invest in DR.

And DR need not break your bank.  The dials of RTO (recovery time objective) and RPO (recovery point objective) allow you to determine how much you will invest.  Perhaps you simply replicate your databases to a smaller set of databases at a remote datacenter and have a copy of each of your systems there with an additional copy ready “in the cloud”.  While you won’t be able to run production from that data center, you may be able to leverage the cloud to add capacity for relatively low cost by cloning the cloud based systems.  Such a solution has a fast recovery point objective (you lose very little data) and a moderate recovery time objective (several hours) for very low comparative cost.  Of course, you would need to test the solution from time to time to show that it is viable, but it’s a cheap and effective insurance policy for the business.

So remember – availability is your most important feature.  Customers expect it always and will run away from you to competitors if you do not have it.  Create an availability metric and ensure that everyone understands it as a critical KPI.  Evaluate the company spend against availability quarterly or annually as an additional indicator of potential problems.   Assume that disasters happen and have a DR plan regardless of your company size.

This is a continuation post from last week where we covered the first six leadership principles from the perspective of a technology leader. In this post we’ll cover the remaining five principles.

7) Keep your soldiers informed – In the military the headquarter staff makes an overall plan for a mission, presents the plan to subordinate units, and lets them plan the details of how to execute. This planning hierarch can occur across many levels such as starting from the Division HQ, then to the Brigade HQ, then to the Battalion HQ, then to the Company, the Platoon, and finally to the Squad. All this planning takes time but each subordinate plan requires information from the higher plan. You can imagine that with all this planning you could quickly run out of time to actually perform the mission, assuming it is time sensitive, which most things in life and business are. The rule of thumb that we used as a staff in the military was to take 1/3 of the remaining time to put your plan together and leave 2/3 for your subordinates. If we the Division staff had 1 week prior to the mission they would take 2.3 days to develop their plan, leaving 4.7 days remaining. The Brigade would use 1.5 days, leaving 3.2 days. The Battalion would use 1 day, leaving 3.1. The Company would use 0.7 days, leaving 2.1, the Platoon 0.5 day, leaving 1.4, the Squad 0.3 day, leaving 0.9. This same concept should apply to your teams. Don’t sit on ideas, contracts, or problems for 3/4 of the time before it is due and make your team scramble. Give your teams the majority of the time to plan and execute. Reiterating from principle #3, make sound and timely decisions. Taking up all the time so that you get 95% of the information before you make a decision is being a whimp. Gather the most pertinent data, make the best decision and make sure your team gets as much time to react as possible.

8) Develop a sense of responsibility in your subordinates – This principle is a combination of #2 “Seek responsibility and take responsibility for your actions” and #4 “Set the Example”. If you seek responsibility for your actions and set the example for your team, they will follow and behave the same way. For a technology leader, this might manifest itself in a situation where you expect your subordinates to admit when they changed the production environment without approval resulting in an incident. You should expect and demand this behavior from your team but you first must ensure that you are’re taking responsibility for your own actions. Instill in your team a sense of ownership over the site. In order to do so you need to set the example of owning the business. Blaming the business leaders for poor decisions and expecting your team to take responsibility for the technology will never work. Create a culture where people want to own their areas and do so by owning yours.

9) Ensure that the task is understood, supervised and accomplished – There is a big difference between micromanaging and providing adequate guidance on a project. Per principle #2 “Seek responsibility and take responsibility for your actions”, you can’t delegate your responsibility only your authority. Miscommunication is probably the single biggest cause of rework. Clear, concise, articulate criteria are the way to ensure tasks are started properly and accomplished. If you assign an unclear task without a deadline and without guidance on resources you can expect it to come in over budget, after the deadline, and not accurately completed. Communication of the task is your responsibility. In military aircraft we used a three-way positive transfer of control. This meant that if I were flying and I wanted by co-pilot to fly I would say “you have the controls”, they would respond with “I have the controls”, and I would reiterate “you have the controls.” This way there was very little chance of no one actually flying the aircraft…which in tandem seat aircraft you’d be surprised how easy that can occur.

10) Build the team – As a leader you must be constantly building the team. This doesn’t always mean hiring more engineers. Building the team can take the shape of improving their skill sets or replacing underperforming individuals. We’ve written before about seed-feed-weed-succeed. This simple garden analogy means that you need to hire (seed) the team with great players, grow (feed) them through challenging projects, greater responsibility, and training, get rid of (weed) underperformers once they’ve had a chance to improve, and by doing these three things well you and your teams will succeed.

11) Employ your unit in accordance with its capabilities – This last principle requires that you first follow principle #5 “know your soldiers and look out for their welfare.” In order to deploy or ask your team to take on certain projects, you need to first know what their capabilities are as a team and as individuals. If your team is newly formed, you probably don’t want to take on critical projects for major customers. If you don’t have anyone who knows databases you don’t want to architect a DB heavy application. Setting your team up for failure is not what leaders do but you won’t know if you’re doing that unless you know your team and the team members.

What do you call a set of processes or systems for coordination between development and operations teams? Give up? Try “DevOps”. While not a new concept, we’ve been living and suggesting ARB and JAD as cornerstones of this coordination for years, but it has recently grown into a discipline of its own. Wikipedia states that DevOps “relates to the emerging understanding of the interdependence of development and operations in meeting a business’ goal to producing timely software products and services.” Tracking down the history of the DevOps Wikipedia page, shows that this topic is a recent entry.

There are a lot of other resources on the web that many not have been using this exact term but have certainly been dealing with the development and operations coordination challenge for years.  Dev2Ops.org is one such group and posted earlier this year their definition of DevOps “an umbrella concept that refers to anything that smoothes out the interaction between development and operations.”  They continue in their post highlighting that concept of DevOps is in response to the growing awareness of a disconnect between development and operations. While I think that is correct I think it’s only partially the reason for the recent interest in defining DevOps.

With ideas such as continuous deployment and Amazon’s two-pizza rule for highly autonomous dev/ops teams there is a blurring of roles between development and operations. Another driver of this movement is cloud computing. Developers can procure, deploy, and support virtual instances much easier than ever before with the advent of GUI or API based cloud control interfaces. What used to be clearly defined career paths and sets of responsibilities are now being blended to create a new, more efficient and highly sought after technologist. A developer who understands operations support or a system administrator who understands programming are utility players that are very valuable.

While perhaps DevOps is a new term to an old problem, it is promising to realize that organizations are taking interest in the challenges of coordination between development and operations. It is even more important that organizations pay attention to this topic given the blurring of roles.

Just as we discussed in an earlier post about the evolution of roles in technology startups, we’ve seen the same thing in the technology discipline as a whole. Computer science as a discipline started in mathematics with Kurt Gödel’s incompleteness theorem.  From there Alan Turing and Alonzo Church formalized the notion of an algorithm and the concept of a Turing machine. The first computer that could run stored programs, based on the Turing machine model, was built in 1948 and called the Manchester Baby.

In the beginning there were only programmers, then came system operators, and DBA’s, and architects, etc. We now have many different disciplines that one can specialize in for either part or all of their careers. One of the missing disciplines, in my opinion, is the scalability architect or scalability as a discipline.

While understanding the rules, patterns, and principles of scalability are completely achievable by anyone in the technology organization, this does not mean that they are widely known. Scalability architects would be more like evangelist and teachers rather than the gatekeepers of secret knowledge. Unlike DBA’s or network engineers, whose jobs really aren’t to educate any other technology person on how to create an index or open a port, the scalability architect would educate tech people. All other disciplines from software developers to DBA’s could benefit from additional knowledge about scaling.

If you’re serious about scaling is it time that you looked for or anointed a scalability architect?

If you are a technology organization within a Software as a Service (SaaS) company, there is no such thing as an “internal customer”.  We often see this anachronistic IT phrase thrown around in web X.0 companies by executives and engineers who simply have not adopted the new SaaS mindset.  Do you think you’ll hear the left offensive tackle of an NFL team refer to the quarterback as his “internal customer”?  The quarterback consumes services (energy to block opponents) of the left tackle – so why wouldn’t he be a customer?  The answer is simple – because the notion of a customer relationship is different than the notion of a relationship within teammates.

The first reason why your teammate isn’t your customer is because he or she is, well, your TEAMMATE.  Customers are someone for whom you produce a service or product and teammates are someone with whom you work to accomplish a goal.  The difference between working FOR someone and working WITH someone is HUGE.  This difference creates a contextually activated identity that forces you to think about customers in a different light than you would a teammate.  Very often, as we’ve written before, this can result in affective (role based or bad) conflict between teams.  Affective conflict is bad and it destroys shareholder value.  Working as a team is important and customers aren’t part of your team.

The next reason that your teammate isn’t your customer is that the customer is always right.  Your teammate isn’t always right.  You need to debate certain points as a team to come to better solutions.  This isn’t affective conflict, it is cognitive conflict and if handled properly it is good and helps to create shareholder value.

The most important reason there isn’t a customer relationship here is that your teammate isn’t paying!   “Servicing” your teammate (uggh…that’s an ugly term) doesn’t create shareholder value.  Working as  a team to delivery a service or product to your  “real” customer is what creates shareholder value.  One design, one approach, one ruthless drive as a team to get across the goal line is what is necessary to thrive and succeed.

So stop using the ugly “internal” C word in your SaaS company.  It doesn’t have a place there.  Let the old world, internal IT folks continue to provide services to their internal customers.  Start acting like a team, designing and building services rather than software.

The strategic fit camp emphasizes strategic analysis and negotiation prior to the acquisition whereas the organizational fit research emphasizes the integration of day-to-day operations post acquisition. There has even been recent research on the impact of the acquisition process itself, which makes sense given that first impressions are established during this phase. I’ve had the good fortune of seeing acquisitions from all different perspectives, as the acquired, as the acquiree, as a consultant of the acquiree, etc. I think there are two models that result in successful acquisitions.

First off defining a successful acquisition is tough. Whether or not the acquiring company writes off the purchase cost in a few years might be one way but these write offs can include goodwill which is the amount of the purchase price paid above the value of the target’s identifiable net assets. There are of course accounting rules for testing impairment of goodwill that may require it to be written off. Another way of testing this might be by calculating Return On Investment (ROI) for the cost of the acquisition. A third way of determining success would be whether the acquired company’s product offerings continue 3 or 5 years later. If the acquiring company has discontinued the acquisitions products the purchase might be considered a failure. Lastly, I tend to think of a successful acquisition like Justice Potter Stewart tried to explain what is obscene by saying “I shall not today attempt further to define the kinds of material I understand to be embraced … but I know it when I see it.”

Whether there is a clear definition or I know it when I see it, there appear to be two paths to get there. The first is what I term the GE-approach because of all the acquisitions I saw while at General Electric during the 90′s, this appeared to be the dominant strategy, although I’m sure many other companies have similar methods. It’s simply an approach of overwhelming the acquisition’s culture and turning it into a GE culture as fast as possible. They did this by sending current GE employees into key positions at the acquisition and sending the acquisitions employees to lots of GE training. Within the first year the acquisition was on the GE strategic planning calendar, using GE financial systems, and following GE’s HR guidelines…they were a GE company by that point.

The second approach is to leave the acquisition completely alone and let it run autonomously. The only tie to the acquiring company is through financials. The parent company acts like a board of directors, approving annual budgets and determining the reinvestment ratio. The acquiree is left to their own to manage the day-to-day operations, processes, etc.

If you’ve seen other successful acquisitions handled differently let us know your thoughts.

You get paid to deliver a service.  You want to deliver that service to the level of your customers’ expectations, or at least to some internally defined level.  So how often do you meet to discuss your service delivery quality?

In our experience, most companies meet only when there is a problem.  Day in and day out many Software as a Service companies will operate their services throughout the day and simply not take the time to step back and look at the last day’s worth of issues, all open issues that are not yet resolved and diagnose service delivery problems.  How could this be you ask?  Well, we honestly don’t know!

As we’ve written before, if you are a SaaS company your business is predicated first and foremost on SERVICE DELIVERY!  Developing software is important – but what makes you money is the delivery of a service.  Get this straight folks, because it is a major mind shift.

In our view, it is absolutely critical to start the business day with a review of the past day’s service delivery.  We call this the “Morning Operations Meeting” or “Morning Operations Review”.  Every day we ask our clients to review major issues from the previous day, overall service quality (response times, availability, major interruptions or bugs live on the site, etc), and all major open issues identified in past days.  Ideally the notion of an incident (a thing that happens in production and causes customer complaint) and the notion of a problem (a thing that causes an incident) are separated in this meeting.  Both should be discussed – but they are really two separate things.

Ideally this meeting will have representatives from your customer support organization, technical operations and infrastructure teams and software development teams.  Inputs to the meeting are a representation of customer complaints, complaints regarding service within the company, manual identification of issues, automated identification of issues (such as through a monitoring system to include Service Level metrics), predictive identification of future problems (such as might be the case from a capacity management team) and all appropriate service level information.

Open incidents and problems from the issue tracking system are discussed, updated, etc.  Owners are assigned to new incidents and problems (if they haven’t been already) and new issues are updated if any were missed from the previous day’s operations.

Outputs from the meeting are updated service level reports, scheduling of post mortems for large incidents, updated problem reports and data for monthly or quarterly look backs or reviews (more on this later).

If done well, the morning meeting helps inform architectural changes that are necessary in the scalability summits or in other product development and architecture meetings.  Recurring problems should be easily identified within the issue management as a result of heightened oversight and analysis of the system.

I’ve noticed lately that more companies are putting up signs in hallways and cube farms requesting that people avoid having conversations in these areas. While having a nice quiet work environment makes sense to me as a developer, doesn’t this completely void having people work beside each other? The ad hoc/hallway/water cooler/coffe machine conversations or ones overheard when cube mates are chatting about a new feature are one of the primary benefits of having people work in small open environments.

I haven’t done any sort of scientific study but it seems that these sort of “please be quiet” signs are more prevalent at larger companies. These are the same ones that are trying to mimic the small startup with agile development processes or open work spaces to compete in a fast moving SaaS marketplace. Imitating the actions without understanding the purpose or allowing old school corporate policies to overrule are surefire ways to tank the initiative.

A parallel to the “please be quiet” sign is allowing corporate IT to dictate the architecture of the SaaS offering based on a corporate standard that works for the ERP system. Running Oracle ERP on a 16-way system might be the vendor recommended, preferred approach but for scaling a SaaS offering this is a quick way to run up the costs and ensure lower availability. We often use the analogy of goldfish and thoroughbreds for comparing small, cheap 1U servers with large, expensive multi-processor boxes with lots of memory. The goldfish (small, cheap servers) are inexpensive to purchase and replaceable while the thoroughbred (large servers) are expensive to purchase/maintain and cause big impacts when they go down.

The take away to all of this is that if your part of a corporate initiative to run an internal startup or deliver a Software as a Service from inside a larger organization, don’t allow corporate policies to prevent your success. The differences in approaches, architectures, organizations, and offices have a purpose and should not be discounted as non-critical to the success of your initiative.

Since writing our view on outsourcing, we’ve received a number of questions.  While most people indicate that the article was useful, the most common question is “Why can’t I just outsource everything?”  Here’s your answer:

You absolutely CAN outsource (or even purchase) everything.  The consideration of whether or not to outsource, as we’ve indicated earlier, is roughly the same as whether or not you should buy something.  The two differences are cost and the ease with which someone else can do the same thing you do.  “Buying” something (I mean off the shelf, packaged software, software as a service, etc) means that it’s available to just about everyone today – potentially with some small modifications to fit their business needs.  As such it usually costs less than outsourcing but is also more easily accessible and implementable by your potential competition.  “Outsourcing” something means that you are going to have someone else implement (code) your idea or run your servers (in a hosted rather than a SaaS model), which usually implies higher cost and a bit more difficulty in transferring technology.

In either scenario, you must be willing to say that you are willing to be like “everyone else”.  In other words, you are willing to give up the competitive differentiation that a homegrown solution might offer such as creating a higher barrier to entry, lower barriers to exit, switching costs, etc.  If an outsourcer can develop your code they will take that experience and apply it to someone else.  They may not use the actual code they write for you, but they simply can’t help but use the past experience.  This means that the job to copy you just got a little easier, which in turn means that you lowered the barrier to entry for competition.  And of course if you purchase a solution, then you are also making a decision that you will not differentiate yourself in that particular area.

None of this is bad.  In fact, there are many cases where you SHOULD outsource or purchase software or services.  Most companies and organizations tend towards isomorphism, which means that over time they all look (or should look) to leverage the best known practices to increase efficiencies and reduce costs.  It’s hard to imagine that you are going to differentiate yourself in your accounting systems, customer support systems, sales lead systems, etc.  You might add a unique set of routing rules, etc – but these systems are so standard that the best practices are built in to most pieces of software.

From a product perspective, if your business objective is to be a “low price leader” rather than to compete on technology or to simply “run with the pack” and use standard features while maintaining good margins then it also makes sense to buy or outsource.

But what if you want to have the world’s best product, stock, or media recommendation engine?  By definition you can’t “buy” that as everyone else would have the same thing.  If you outsource it, everyone else might not have your code but the firm that develops it for you can’t help but add it to their experience; they might not copy it but it certainly will influence their future activities.

As we’ve described before – don’t outsource or buy those things that you feel should or will differentiate your business.