AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Tag » Management

5-95 Rule

Previously, I wrote about mitigating risk in the face of uncertainty. I suggested that an agile development model was one way successful companies have been able to mitigate risk. In that post I compared the similarities between organizing into “Scrums” to how Army Special Forces organized in order to succeed in uncertain operating conditions. In addition to the way a company is organized, the way a company approaches project planning can mitigate risk. One of the most often overlooked aspects of project management is contingency planning.

Army Green Berets are fond of the saying, “No plan survives contact with the enemy,” a quote attributed to the famous Prussian General Helmut Von Moltke. Von Moltke and his Prussian contemporaries were brilliant strategists and sought to perfect the “Theory of War.” Their ideas still serve as basis for our doctrine today. Moltke understood that battles can be broken down to a near infinite set of complex options, each of them in turn having still more options depending upon the enemy’s response. A plan only goes so far, at which point the enemy ultimately casts his vote. Von Moltke provided his subordinate commanders with his intentions (a method still taught today in military basic leadership courses), and held them responsible for extensively preparing for all plausible contingencies.
von_moltke

As a Special Forces Detachment Commander, once my team received our orders, we immediately began to plan. Along with the mission, we were given a specific deadline by which we needed to brief our commander. Regardless of the deadline, whether it was 6 hours or a week, we spent approximately 5% of the time coming up with a solid, practical, and safe plan. We spent the other 95% of the time “war-gaming” the plan. During the “war-gaming” portion, we would go through every step of the plan from the time we left our barracks to the time we got back. We thought of every possible thing that could go wrong. What if only two helicopters showed up and not four? What if the helicopter couldn’t land where we wanted it? What if someone got injured? What if more bad guys were on the objective than we had originally thought? We would break our mission down into distinct pieces, or milestones, and would work in teams to come up with solutions to every possible contingency. We would collaborate with our partners, the pilots for example, and let the subject matter experts take the lead in their scope of responsibility. Time permitting, we built mock-ups of the buildings we anticipated operating in, and conducted detailed rehearsals. The rehearsals revealed other contingencies we hadn’t planned for, such as equipment that was being carried by the wrong guy or individuals who needed to work together that weren’t located near each other. We did this because we knew the battlefield was going to be fluid and dynamic. Being prepared not only enabled us to be successful, it saved lives.

While the tech industry doesn’t deal in life or death, although it certainly might feel like that at times, Von Moltke’s wisdom still applies, especially in complex project development. What if equipment doesn’t arrive on time during a data center build? A critical engineer or developer gets sick during a launch or gets hired away? AKF Partners is a company heavily influenced by veterans and our collective experience on and off the battlefield. We encourage our clients to practice the AKF “5-95 Rule” in their Agile methodology. Having a solid plan is a great start but understanding the possible variations in the project’s execution will help ensure the projected is delivered on time and within the budget. Remember the AKF “5-95 Rule” and spend 5 percent of your time planning and 95 percent of your time developing contingencies.


1 comment

Driving Change

Driving big changes can be difficult in an organization and it can be especially difficult for new managers. Change is necessary and we should never be happy with status quo as our competitors and the overall market is changing around us. The larger the organization, the harder change can be. We touched on this in an earlier blog titled Five Leadership Lessons From Former Bosses but decided to dive into a little more detail this time. As a former manager of engineers and other talented technologists, I recall in my early days how I simply wanted to identify where we were headed with my team without giving them time to collaborate and provide input. On one occasion, we were in the midst of changing our PDLC to Agile. Agile was fairly new at the time so many didn’t understand the basis of it and in turn were resistant to using it. That problem is easily solved, right? I could just tell the team this is where we are going and this is how we are going to do it, now make sure you execute. And in a couple weeks time, poof, the change would be made and life would be good. After all, that would have been easy, right?

Although I didn’t say the words “get onboard,” I might as well have. I can tell you this doesn’t work well. My team was zapped. At that point they were no longer empowered and didn’t have any ownership of the change and when that happens a team’s performance will be suboptimal. Yeah, I could use the excuse that my senior management dictated what they wanted and I could drum up a bunch of other excuses but the fact of the matter is I should have raised a flag with my leadership and encouraged better collaboration for input from the engineers, QA testers, product managers, and others.

There were two key learning points that I was able to take away from my experience. Technology leaders should ask themselves two simple questions before they start to drive change in their organizations.

1) How do I make sure that my team has input and ownership of the change we are about to make?

Sit down with the team and work with them on coming up with solutions and their recommended timing. Engineers love to problem solve and they are very good at it. Provide the team with the problem that needs solved and any constraints and let them determine the solution. For example, in the situation I described above, we needed to improve our time to market. Many products and features across the organization were being delivered late, payloads for deployments were too big, and the business was frustrated. As on overall company we decided that we would adopt Agile utilizing Scrum practices but there were still many problems that needed to be solved. Those included determining a branching strategy, determining a deployment strategy, identifying how to measure a sprints performance, and many others. By providing the context of why changes need to be made, the team will recognize the significance of their input and by allowing them to come up with solutions they will feel empowered and have ownership in the game. All of this will help you to create a higher performing engineering team.

2) What do I need to do as a leader to establish guardrails in my organization and sustain the change as my company grows?

Once the team has determined how to solve some of the problems, you will want to make sure that you sustain them in your organization. To do this, work with them to establish documented guardrails or principles that be used as the what. For example, the Agile Manifesto essentially identifies principles that you can use. One principle could be “Business people and developers must work together daily throughout the project.” This describes a behavior that’s needed but doesn’t identify how. The team can decide the specifics on how they will achieve the principle. We at AKF believe in establishing architectural and scaling principles that are essentially guardrails as well. (Visit our earlier blog on architectural principles for more details.) For example, Rule Number 39 – Ensure You Can Wire on and Off Functions in our book Scalability Rules identifies the need to have a framework where you can quickly disable and enable features to minimize impact to customers when a failure occurs. You can solve this many ways and your engineers can determine what the solution should be. Once your principles are established you should monitor them to make sure they are being upheld and reward results.

If you ask yourself these two simple questions before diving into a big change your road to success will be much easier. Now that you have empowered your team to determine how to solve the problem and you have documented your principles and guardrails, the team should be able to operate within them and make decisions on their own. In turn, you will find that they feel empowered and you will have helped to create and maintain a higher performing team.


1 comment

The Agile Executive

In this third installment of our “Agile Organization” series we discuss the qualities and attributes necessary for someone to lead a group of cross functional Agile teams in the development of a web-centric product.  For the purposes of this discussion, the Agile Executive is the person responsible for leading a group of agile teams in the development of a product.

In a world with less focus on functional organizations such as the one we’ve described in our Agile Organization articles, it is imperative that the leadership have a good understanding of all domains from the overall business through product management and finally each of the technical domains.  Clearly such an individual can’t be an expert in every one of these areas, but they should be deep in at least one and broad through all of them.  Ideally this would have been the case in the functional world as well, but alas functional organizations exist to support deep rather than broad domain knowledge.  In the Agile world we need deep in at least area and broad in all areas.

Such a deep yet broad individual could come from any of the functional domains.  The former head of product management may be one such candidate assuming that he or she had good engineering and operations understanding.  The head of engineering and operations may be heads of other agile teams, assuming that they have a high business acumen and good product understanding.  In fact, it should not matter whence the individual comes, but rather whether he or she has the business acumen, product savvy and technical chops to lead teams.

In our view of the world, such an individual will likely have a strong education consisting of an undergraduate STEM (science, technology, engineering or math) degree.  This helps give them the fundamentals necessary to effectively interact with engineers and add value to the engineering process.  They will also have likely attended graduate school in a business focused program such as an MBA with a curriculum that covers finance, accounting, product and strategy.  This background helps them understand the language of business.  The person will hopefully have served for at least a short time in one of the engineering disciplines as an individual contributor to help bridge the communication chasm that can sometimes exist between those who “do” and those who “dream”.  As they progress in their career, they will have taken on roles within product or worked closely with product in not only identification of near term product elements, but the strategic evaluation of product needs longer term as well.

From the perspective of philosophy, the ideal candidates are those who understand that innovation is more closely correlated with collaboration through wide networks than it is to the intelligence of one individual or a small group of people.  This understanding helps drive beneficial cognitive conflict and increased contribution to the process of innovation rather than the closed minded approach of affective conflict associated with small groups of contributors.

In summary, it’s not about whence the person comes but rather “who the person is”.  Leading cross disciplinary teams requires cross disciplinary knowledge.  As we can’t possibly experience enough personally to be effective in all areas, we must broaden ourselves through education and exposure and deepen ourselves through specific past experiences.  Most importantly, for a leader to succeed in such an environment he or she must understand that “it’s not about them” – that success is most highly correlated with teams that contribute and not with just being “wickedly smart”.


Comments Off on The Agile Executive

Availability as a Feature

It doesn’t matter if you run a commerce site, a services product (such as a SaaS offering) or simply use your homepage to distribute information:  The table stakes for playing online is high availability.  So many companies just take for granted that they will be highly available because they have multiple instances of systems and multiple copies of their data.  This assumption of availability will likely, at the very least, cost you significant pain and in the extreme cost you either significant market share or close your doors as a business.  Customers expect the unachievable – 100% availability.  At the very least you need to give them something close to that.  What will happen to you if you have a data center failure?  How about if a DBA accidentally drops a critical table in your production database?  What will you do when that marketing campaign triggers a near overnight doubling of traffic?  What happens when that new feature has a significant performance bug and gets adopted so quickly that it brings your entire site to its knees?

We often tell our clients that they should treat high availability as a feature.  Unfortunately, it is a somewhat expensive feature that requires constant investment overtime to achieve and maintain. It is a must have feature that will only differentiate your firm if you have competitors who do not make similar investments; when competition exists, customers are more likely to leave a site for a competitor due to availability and performance issues than nearly any other reason.  If you don’t believe us on this topic, just go ask the folks at Friendster.

Treating availability as a feature means measuring availability from a customer perspective rather than a systems or device perspective.  How many times did customer requests not complete?  In this regard, availability now becomes a percentage of failed transactions against an expected number of transactions.   We define an approach to accomplish this in our first book “The Art of Scalability”.  Every executive in the company should “own” the availability metric and understand its implication to the business.    You should track how much you invest in availability over time and significant decreases in engineering or capital should be questioned as it may be an early indicator that you are under investing and a harbinger of hard times to come.

One of the most common failures we see is to assume that disaster recovery is something that only big companies need.  Make no mistake about it, disasters do happen and given enough time they will happen to you.  Data centers catch on fire, have water (sprinkler) discharges that ruin equipment, have complete power equipment failures that take hours to fix and are prone to damages from vehicles, earthquakes, employees and tornados.  In our past lives as executives and current roles as advisors we’ve seen no less than 4 data center fires, 2 data centers incapacitated from earthquakes and tornados and one data center leveled by a truck running into it.  You are never too young to invest in DR.

And DR need not break your bank.  The dials of RTO (recovery time objective) and RPO (recovery point objective) allow you to determine how much you will invest.  Perhaps you simply replicate your databases to a smaller set of databases at a remote datacenter and have a copy of each of your systems there with an additional copy ready “in the cloud”.  While you won’t be able to run production from that data center, you may be able to leverage the cloud to add capacity for relatively low cost by cloning the cloud based systems.  Such a solution has a fast recovery point objective (you lose very little data) and a moderate recovery time objective (several hours) for very low comparative cost.  Of course, you would need to test the solution from time to time to show that it is viable, but it’s a cheap and effective insurance policy for the business.

So remember – availability is your most important feature.  Customers expect it always and will run away from you to competitors if you do not have it.  Create an availability metric and ensure that everyone understands it as a critical KPI.  Evaluate the company spend against availability quarterly or annually as an additional indicator of potential problems.   Assume that disasters happen and have a DR plan regardless of your company size.

 


Comments Off on Availability as a Feature

11 Leadership Principles – Part II

This is a continuation post from last week where we covered the first six leadership principles from the perspective of a technology leader. In this post we’ll cover the remaining five principles.

7) Keep your soldiers informed – In the military the headquarter staff makes an overall plan for a mission, presents the plan to subordinate units, and lets them plan the details of how to execute. This planning hierarch can occur across many levels such as starting from the Division HQ, then to the Brigade HQ, then to the Battalion HQ, then to the Company, the Platoon, and finally to the Squad. All this planning takes time but each subordinate plan requires information from the higher plan. You can imagine that with all this planning you could quickly run out of time to actually perform the mission, assuming it is time sensitive, which most things in life and business are. The rule of thumb that we used as a staff in the military was to take 1/3 of the remaining time to put your plan together and leave 2/3 for your subordinates. If we the Division staff had 1 week prior to the mission they would take 2.3 days to develop their plan, leaving 4.7 days remaining. The Brigade would use 1.5 days, leaving 3.2 days. The Battalion would use 1 day, leaving 3.1. The Company would use 0.7 days, leaving 2.1, the Platoon 0.5 day, leaving 1.4, the Squad 0.3 day, leaving 0.9. This same concept should apply to your teams. Don’t sit on ideas, contracts, or problems for 3/4 of the time before it is due and make your team scramble. Give your teams the majority of the time to plan and execute. Reiterating from principle #3, make sound and timely decisions. Taking up all the time so that you get 95% of the information before you make a decision is being a whimp. Gather the most pertinent data, make the best decision and make sure your team gets as much time to react as possible.

8) Develop a sense of responsibility in your subordinates – This principle is a combination of #2 “Seek responsibility and take responsibility for your actions” and #4 “Set the Example”. If you seek responsibility for your actions and set the example for your team, they will follow and behave the same way. For a technology leader, this might manifest itself in a situation where you expect your subordinates to admit when they changed the production environment without approval resulting in an incident. You should expect and demand this behavior from your team but you first must ensure that you are’re taking responsibility for your own actions. Instill in your team a sense of ownership over the site. In order to do so you need to set the example of owning the business. Blaming the business leaders for poor decisions and expecting your team to take responsibility for the technology will never work. Create a culture where people want to own their areas and do so by owning yours.

9) Ensure that the task is understood, supervised and accomplished – There is a big difference between micromanaging and providing adequate guidance on a project. Per principle #2 “Seek responsibility and take responsibility for your actions”, you can’t delegate your responsibility only your authority. Miscommunication is probably the single biggest cause of rework. Clear, concise, articulate criteria are the way to ensure tasks are started properly and accomplished. If you assign an unclear task without a deadline and without guidance on resources you can expect it to come in over budget, after the deadline, and not accurately completed. Communication of the task is your responsibility. In military aircraft we used a three-way positive transfer of control. This meant that if I were flying and I wanted by co-pilot to fly I would say “you have the controls”, they would respond with “I have the controls”, and I would reiterate “you have the controls.” This way there was very little chance of no one actually flying the aircraft…which in tandem seat aircraft you’d be surprised how easy that can occur.

10) Build the team – As a leader you must be constantly building the team. This doesn’t always mean hiring more engineers. Building the team can take the shape of improving their skill sets or replacing underperforming individuals. We’ve written before about seed-feed-weed-succeed. This simple garden analogy means that you need to hire (seed) the team with great players, grow (feed) them through challenging projects, greater responsibility, and training, get rid of (weed) underperformers once they’ve had a chance to improve, and by doing these three things well you and your teams will succeed.

11) Employ your unit in accordance with its capabilities – This last principle requires that you first follow principle #5 “know your soldiers and look out for their welfare.” In order to deploy or ask your team to take on certain projects, you need to first know what their capabilities are as a team and as individuals. If your team is newly formed, you probably don’t want to take on critical projects for major customers. If you don’t have anyone who knows databases you don’t want to architect a DB heavy application. Setting your team up for failure is not what leaders do but you won’t know if you’re doing that unless you know your team and the team members.


Comments Off on 11 Leadership Principles – Part II

DevOps

What do you call a set of processes or systems for coordination between development and operations teams? Give up? Try “DevOps”. While not a new concept, we’ve been living and suggesting ARB and JAD as cornerstones of this coordination for years, but it has recently grown into a discipline of its own. Wikipedia states that DevOps “relates to the emerging understanding of the interdependence of development and operations in meeting a business’ goal to producing timely software products and services.” Tracking down the history of the DevOps Wikipedia page, shows that this topic is a recent entry.

There are a lot of other resources on the web that many not have been using this exact term but have certainly been dealing with the development and operations coordination challenge for years.  Dev2Ops.org is one such group and posted earlier this year their definition of DevOps “an umbrella concept that refers to anything that smoothes out the interaction between development and operations.”  They continue in their post highlighting that concept of DevOps is in response to the growing awareness of a disconnect between development and operations. While I think that is correct I think it’s only partially the reason for the recent interest in defining DevOps.

With ideas such as continuous deployment and Amazon’s two-pizza rule for highly autonomous dev/ops teams there is a blurring of roles between development and operations. Another driver of this movement is cloud computing. Developers can procure, deploy, and support virtual instances much easier than ever before with the advent of GUI or API based cloud control interfaces. What used to be clearly defined career paths and sets of responsibilities are now being blended to create a new, more efficient and highly sought after technologist. A developer who understands operations support or a system administrator who understands programming are utility players that are very valuable.

While perhaps DevOps is a new term to an old problem, it is promising to realize that organizations are taking interest in the challenges of coordination between development and operations. It is even more important that organizations pay attention to this topic given the blurring of roles.


1 comment

Scalability as a Discipline

Just as we discussed in an earlier post about the evolution of roles in technology startups, we’ve seen the same thing in the technology discipline as a whole. Computer science as a discipline started in mathematics with Kurt Gödel’s incompleteness theorem.  From there Alan Turing and Alonzo Church formalized the notion of an algorithm and the concept of a Turing machine. The first computer that could run stored programs, based on the Turing machine model, was built in 1948 and called the Manchester Baby.

In the beginning there were only programmers, then came system operators, and DBA’s, and architects, etc. We now have many different disciplines that one can specialize in for either part or all of their careers. One of the missing disciplines, in my opinion, is the scalability architect or scalability as a discipline.

While understanding the rules, patterns, and principles of scalability are completely achievable by anyone in the technology organization, this does not mean that they are widely known. Scalability architects would be more like evangelist and teachers rather than the gatekeepers of secret knowledge. Unlike DBA’s or network engineers, whose jobs really aren’t to educate any other technology person on how to create an index or open a port, the scalability architect would educate tech people. All other disciplines from software developers to DBA’s could benefit from additional knowledge about scaling.

If you’re serious about scaling is it time that you looked for or anointed a scalability architect?


5 comments

“Internal Customer”: The “C” Word of SaaS Companies

If you are a technology organization within a Software as a Service (SaaS) company, there is no such thing as an “internal customer”.

If you are a technology organization within a Software as a Service (SaaS) company, there is no such thing as an “internal customer”.  We often see this anachronistic IT phrase thrown around in web X.0 companies by executives and engineers who simply have not adopted the new SaaS mindset.  Do you think you’ll hear the left offensive tackle of an NFL team refer to the quarterback as his “internal customer”?  The quarterback consumes services (energy to block opponents) of the left tackle – so why wouldn’t he be a customer?  The answer is simple – because the notion of a customer relationship is different than the notion of a relationship within teammates.

The first reason why your teammate isn’t your customer is because he or she is, well, your TEAMMATE.  Customers are someone for whom you produce a service or product and teammates are someone with whom you work to accomplish a goal.  The difference between working FOR someone and working WITH someone is HUGE.  This difference creates a contextually activated identity that forces you to think about customers in a different light than you would a teammate.  Very often, as we’ve written before, this can result in affective (role based or bad) conflict between teams.  Affective conflict is bad and it destroys shareholder value.  Working as a team is important and customers aren’t part of your team.

The next reason that your teammate isn’t your customer is that the customer is always right.  Your teammate isn’t always right.  You need to debate certain points as a team to come to better solutions.  This isn’t affective conflict, it is cognitive conflict and if handled properly it is good and helps to create shareholder value.

The most important reason there isn’t a customer relationship here is that your teammate isn’t paying!   “Servicing” your teammate (uggh…that’s an ugly term) doesn’t create shareholder value.  Working as  a team to delivery a service or product to your  “real” customer is what creates shareholder value.  One design, one approach, one ruthless drive as a team to get across the goal line is what is necessary to thrive and succeed.

So stop using the ugly “internal” C word in your SaaS company.  It doesn’t have a place there.  Let the old world, internal IT folks continue to provide services to their internal customers.  Start acting like a team, designing and building services rather than software.


2 comments

Successful Acquisitions

How do you successfully integrate an acquisition? Most academic research on this subject seems to suggest a rational choice perspective which focuses on either strategic fit or organizational fit.

The strategic fit camp emphasizes strategic analysis and negotiation prior to the acquisition whereas the organizational fit research emphasizes the integration of day-to-day operations post acquisition. There has even been recent research on the impact of the acquisition process itself, which makes sense given that first impressions are established during this phase. I’ve had the good fortune of seeing acquisitions from all different perspectives, as the acquired, as the acquiree, as a consultant of the acquiree, etc. I think there are two models that result in successful acquisitions.

First off defining a successful acquisition is tough. Whether or not the acquiring company writes off the purchase cost in a few years might be one way but these write offs can include goodwill which is the amount of the purchase price paid above the value of the target’s identifiable net assets. There are of course accounting rules for testing impairment of goodwill that may require it to be written off. Another way of testing this might be by calculating Return On Investment (ROI) for the cost of the acquisition. A third way of determining success would be whether the acquired company’s product offerings continue 3 or 5 years later. If the acquiring company has discontinued the acquisitions products the purchase might be considered a failure. Lastly, I tend to think of a successful acquisition like Justice Potter Stewart tried to explain what is obscene by saying “I shall not today attempt further to define the kinds of material I understand to be embraced … but I know it when I see it.”

Whether there is a clear definition or I know it when I see it, there appear to be two paths to get there. The first is what I term the GE-approach because of all the acquisitions I saw while at General Electric during the 90’s, this appeared to be the dominant strategy, although I’m sure many other companies have similar methods. It’s simply an approach of overwhelming the acquisition’s culture and turning it into a GE culture as fast as possible. They did this by sending current GE employees into key positions at the acquisition and sending the acquisitions employees to lots of GE training. Within the first year the acquisition was on the GE strategic planning calendar, using GE financial systems, and following GE’s HR guidelines…they were a GE company by that point.

The second approach is to leave the acquisition completely alone and let it run autonomously. The only tie to the acquiring company is through financials. The parent company acts like a board of directors, approving annual budgets and determining the reinvestment ratio. The acquiree is left to their own to manage the day-to-day operations, processes, etc.

If you’ve seen other successful acquisitions handled differently let us know your thoughts.


Comments Off on Successful Acquisitions

Morning Operations Meeting

If you deliver a service through software, you need to discuss your service delivery quality every day! Here's how:

You get paid to deliver a service.  You want to deliver that service to the level of your customers’ expectations, or at least to some internally defined level.  So how often do you meet to discuss your service delivery quality?

In our experience, most companies meet only when there is a problem.  Day in and day out many Software as a Service companies will operate their services throughout the day and simply not take the time to step back and look at the last day’s worth of issues, all open issues that are not yet resolved and diagnose service delivery problems.  How could this be you ask?  Well, we honestly don’t know!

As we’ve written before, if you are a SaaS company your business is predicated first and foremost on SERVICE DELIVERY!  Developing software is important – but what makes you money is the delivery of a service.  Get this straight folks, because it is a major mind shift.

In our view, it is absolutely critical to start the business day with a review of the past day’s service delivery.  We call this the “Morning Operations Meeting” or “Morning Operations Review”.  Every day we ask our clients to review major issues from the previous day, overall service quality (response times, availability, major interruptions or bugs live on the site, etc), and all major open issues identified in past days.  Ideally the notion of an incident (a thing that happens in production and causes customer complaint) and the notion of a problem (a thing that causes an incident) are separated in this meeting.  Both should be discussed – but they are really two separate things.

Ideally this meeting will have representatives from your customer support organization, technical operations and infrastructure teams and software development teams.  Inputs to the meeting are a representation of customer complaints, complaints regarding service within the company, manual identification of issues, automated identification of issues (such as through a monitoring system to include Service Level metrics), predictive identification of future problems (such as might be the case from a capacity management team) and all appropriate service level information.

Open incidents and problems from the issue tracking system are discussed, updated, etc.  Owners are assigned to new incidents and problems (if they haven’t been already) and new issues are updated if any were missed from the previous day’s operations.

Outputs from the meeting are updated service level reports, scheduling of post mortems for large incidents, updated problem reports and data for monthly or quarterly look backs or reviews (more on this later).

If done well, the morning meeting helps inform architectural changes that are necessary in the scalability summits or in other product development and architecture meetings.  Recurring problems should be easily identified within the issue management as a result of heightened oversight and analysis of the system.


2 comments