AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

A-Teams

In his book, The Lean Start-Up, Eric Ries defines a start-up as a “human institution designed to deliver a new product or service under conditions of extreme uncertainty.” Operating in uncertainty is inherently risky. What makes many start-ups successful, among other things, is their ability to mitigate risk. One way a company can remain adaptive and responsive during uncertainty, and therefore reduce risk, is employing an agile development model. A good example of such an organization is Spotify, which continues to embrace, practice, and improve on agile fundamentals even after growing to over 30 development teams in three different locations.

There are striking similarities between the way Spotify organizes their teams and the way U.S. Army Special Forces organizes their Operational Detachments-Alpha, aka “A-Teams.” It makes sense that the two would be similar because the A-Team was in fact an innovation designed to deliver a “service” under conditions of extreme uncertainty. In this case, the “service” was military assistance to the struggling Republic of Vietnam, and later to guerrilla forces and resistance movements waging unconventional warfare against the North Vietnamese. The U.S. Army had never undertaken a mission similar in objective and scope. They did not know the conditions, the support, or the communication capabilities that their soldiers would find. The U.S. Army needed a self-sustaining team with all the skills necessary to complete a mission with a broad and evolving scope. Moreover, the team would have to operate autonomously and would have to be trusted to make decisions normally left to upper echelons.

HALO

The U.S. Army assessed and selected twelve of their most seasoned veterans, and then trained them each in one of six specialties: medicine, engineering, communications, weapons and tactics, or intelligence. Having redundancy in each position was desired in case one soldier got injured, and it also gave the team the ability to pair a new member with a more senior member of the same specialty. Redundancy also facilitated the ascension of the team’s most senior soldier to the role of Operations Sergeant. The A-Team Operations Sergeant is comparable to the role of Spotify’s Agile Coach, or sometimes called a “Scrum Master” in other organizations. He no longer fills a specialty function on the A-team. He is the team’s mentor and draws on his experience to coach the team. He wants his team to be efficient and safe, which equates to a quality product in their realm. It is the also the Operations Sergeant’s job to advise and informally train the A-team’s commander on the team’s competences and potential with respect to their assigned mission. He ensures that the commander does not over commit the A-team to the stakeholders.

The A-Team was designed to operate deep behind enemy lines. They would not maintain frequent contact to their command. Because the complexity of the mission at hand, the A-Team worked not only for their command, but on the on behalf of many other stakeholders, such as the U.S. State Department, CIA, and other government agencies. In the absence of guidance from their command, and other key stakeholders, it is crucial that the commander, with input from the team, manage the team’s priorities with respect to the mission’s goals. Just like Spotify’s Product Owner, it is the A-team’s commander who ensures the team is on track to accomplish the stakeholders’ desired end state by prioritizing projects and associated tasks.

The way that Spotify tweaked the idea of a Matrix Org to facilitate product-centered teams while managing professional development and adherence to standards across functional areas resembles the method used by A-teams. Spotify organizes their development teams or “Squads,” into “Tribes” of 100 contributors of varying disciplines such as software engineering, hardware engineering, DBAs, etc. Each Tribe has a “Chapter” in which contributors with the same competencies are grouped. Chapters are led by “Chapter Leads” who are responsible for the quality of work produced by his or her chapter. Special Forces A-Teams are organized into “Companies” of six multi-disciplinary teams (around 70 people). A-Team “Companies” have a headquarters unit that is responsible for the professional development and standards of each specialty on the team. As an example, the medics on each team practice medicine under the license of a medical doctor assigned to the headquarters unit. The doctor’s function is similar to that of a Chapter Lead because he is responsible for each medic’s continued professional growth, refresher training, and the overall standard of care that is provided by his chapter.

The A-Team model’s structure has not significantly changed since their inception. During the Vietnam War, and most recently in Iraq and Afghanistan, Special Forces A-Teams have produced tangible results where larger, conventional military units have not. Their self-sufficient and decentralized approach allows the team to focus on mission accomplishment. The conventional Army’s centralized and hierarchical structure doesn’t allow for the flexibility needed to mitigate risk in uncertain environments. Results often become secondary to strict adherence to protocol for protocol’s sake, and the approval of long reporting chains. Similarly, in the technology industry, companies like Spotify who adhere to agile fundamentals are able to scale and grow successfully amid the uncertainty, while larger, less “agile” organizations become bogged down in bureaucracy, and can not navigate the evolving and uncertain market.

Send to Kindle

Comments Off

Recent Interviews

Below is a list of recent interviews by AKF Partners, Mike Fisher & Marty Abbott on a variety of topics including scalability, healthcare.gov, and customer misbehavior:

Make sure you follow Marty and Mike  and AKF’s Facebook Pages to keep up to date with the latest.

@martyabbott

@MikeFisher_AKF

Power of Customer Misbehavior

AKF Partners

 

Send to Kindle

Comments Off

Seizing the Initiative

As a young Army Lieutenant leading a Platoon in Iraq in 2006, I was faced with a crisis. Our platoon was continually getting ambushed during our patrols in sector. The uncertainty of waiting to be attacked, the physical, mental and emotional strain on the men, not to mention the damage to our vehicles and equipment, started to take its toll. The ensuing battles would generally last for hours. We would stay out in sector for upwards of 16 hours. This would only leave 8 hours to come back to the forward operating base, refit and conduct maintenance on our vehicles. This type of operation couldn’t be sustained. Our morale was starting to fall, tensions were starting to rise between my platoon and the maintenance crews servicing our vehicles. We’d bring in leaking, limping, bullet ridden trucks and expect to jump back in them and roll out in a matter of hours. In military terms, we had “lost the initiative.” That is to say, the enemy was dictating where and when and how the battle would go. We were only reacting. We were compelled to react to the enemy’s strengths and capabilities. We were treading water trying to stay afloat. In this crisis, my men depended on me to come up with a plan to take back control, or “seize the initiative.” The importance of having a plan is emphasized in Sun Tzu’s famous quote, “ Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.” My platoon and I were at war, and trying to win.

sun_tzu_general

The crisis I experienced in Iraq is similar to the challenges many executives face with regard to scaling. Granted, no one is shooting at them, but the enemy can represent any or all of the variables and uncertainty in the industry; the customer, the competition, the software, the hardware, the market, etc. Some executives find they are compelled to react to daily incidents, i.e. “lets just make it another 24 hrs without an outage,” and it’s a struggle to maintain, let alone scale effectively. Just like with my platoon, this type of operation takes a toll on an organization. Aside from being bad for business, it sows the seeds of despair, brings down morale, and possibly creates tension between the engineers and the business folks similar to that between my troops and our maintenance crews. The leader needs to take a step back, understand what is going on and figure out a plan to get his or her team out of it.

For me, the solution was counter-intuitive at first. I knew we needed to take time to regroup, but I hesitated to not keep pressing the enemy. I feared that we’d lose what little ground we had. Then, I remembered something my instructor told me while learning to navigate in the woods with a map and a compass. He warned us, “If you don’t know where you are going, don’t walk as fast. You’ll only get lost in a hurry.” In our eagerness to get out into sector and fight the enemy we were getting lost in a hurry. We had gone to war, and were now trying to win. It is a difficult decision to halt operations, or take a product offline. Aggressive leaders are often very successful because they persevere in the face of adversity. However, there is a fine line between perseverance and stubbornness. I made the call to postpone operations. I needed time to develop an effective sustainable strategy.

“Seizing the initiative” requires leaders to make sound decisions under uncertain conditions and to anticipate events so their organization can recognize and leverage opportunities. The key to anticipating events is to understand the problem. In my case, we really did not understand the enemy. From an institutional standpoint, we had been training to fight the soviets for the last 40 years, and here we were faced against an adaptive and evolving enemy. We scaled our operations back at first, focused on reconnaissance and understanding the enemy. In doing so we were better able to anticipate events, and began setting the conditions of the fight. We were engaging the enemy when and where we wanted. We had seized the initiative. Regardless of your type of organization, if you are struggling just to maintain, if you have lost the initiative, the best solution is to take the necessary pause in operations, understand the problem, and develop a strategy for a sustainable effective approach.

Send to Kindle

Comments Off

The Phoenix Project – Book Review

After several friends recommended that I read the book The Phoenix Project, I at last downloaded it to my kindle and read it on one of my transcontinental flights. I really enjoyed it and thought it was both entertaining and enlightening which is difficult to accomplish. However, I don’t believe it goes far enough in describing how the most modern and efficient technology shops run. More about that later, first a recap.

The story line is that the fictional company, Parts Unlimited, is failing to keep up with competition and at risk of being split up and sold off. The root cause of the company’s issues are IT. We join the story after two IT executives are fired and the hero of our story, Bill, is asked to take over as the VP of IT Operations. He is introduced to his sensei, Erik, a tech / business / process genius, who leads Bill through the path to enlightenment by comparing IT Operations to a manufacturing plant. Through this journey the reader is introduced to concepts such as kanban, work-in-process (WIP), work centers, Theory of Constraints, Lean Startup, and the overarching concept of DevOps. In fact the book ends with most of the main characters at a party where Bill reflects “Maybe everyone attending this party is a form of DevOps, but I suspect it’s something much more than that. It’s Product Management, Development, IT Operations, and even Information Security all working together and supporting one another.”

I’m a fan of DevOps but to me the integration of tech ops and development is one small step for technology driven companies. In order to truly be nimble and effective, teams need to incorporate development, tech ops, product management, quality assurance, and any other function such as marketing or analytics that are needed to make the team autonomous. We’ve written about this concept several times The Agile Org, The Agile Org Solution, and The Agile Exec. We call these teams PODs and most importantly they need to be aligned to swim lanes in the architecture. By doing this, you essentially create mini-startups. These teams can make decisions themselves, deploy independently, and are responsible for their service from idea to production support.

I really enjoyed the book The Phoenix Project and I too highly recommend that you read it, especially if you are not familiar with the concepts mentioned above. As the authors are obviously talented writers and technologist, I would love to see them write another book that takes Parts Unlimited to the next level by creating autonomous teams. The authors hint at this concept when they state

“You’ve helped me see that IT is not merely a department. Instead, it’s pervasive, like electricity. It’s a skill, like being able to read or do math. Here at Parts Unlimited, we don’t have a centralized reading or math department—we expect everyone we hire to have some mastery of it. Understanding what technology can and can’t do has become a core competency that every part of this business must have. If any of my business managers are leading a team or a project without that skill, they will fail.”

Hopefully this is foreshadowing of another book to come.

Send to Kindle

Comments Off

The Power of Customer Misbehavior Video

The Power of Customer Misbehavior is now available for the Kindle. Here is a short video about the book.


 

Hopefully the video enticed you to read the book. Purchase the Kindle version or Pre-order the hardcover at Amazon

 

Send to Kindle

Comments Off

A False Sense Of Security and Complacency = Revenue Loss

Its Monday morning and past Saturday evening issues in one of your datacenters triggered a failover to your second data center for service restoration. In other words, all customer traffic has been routed to a single datacenter. The failover was executed flawlessly and the team went back to bed waiting for Monday morning to permanently fix the issue so traffic could once again run out of both datacenters. On Monday morning, we are expecting a flash sale and will make close to $8000 a minute at peak. All is well and there is nothing to worry about. Right?

Hopefully you cringed at the above scenario. What if the data center you are running out of suffers from a failure? Or what if the only data center and its components that is now live for all of your traffic simply wasn’t sized correctly for acceptable performance during a traffic spike?

If it hasn’t happened yet, it will. If that were the case, your business would stand to lose significant revenue. We see it over and over again with many clients and have also experienced it in practice. Multiple datacenters can serve as a false sense of security and teams can become complacent. Remember, assume everything will fail as a monolith. If you are only running out of a single data center and the other is unable to take traffic, you now have a SPOF and as a whole the DC is a monolith. As a tech ops leader you have to drive the right sense of urgency and lead your team to have the right mindset. Restoring service with a failover is perfectly acceptable. However, the team cannot stop there. They must quickly diagnose the problem and return the site to normal service, which means you are once again running out of two datacenters. Don’t let the false sense of security slip into your ops teams. If you spot it, call it out and explain why.

To help combat complacency from setting in, we recommend considering the following:

  1. Run a Morning Ops meeting with your business and review issues from the past 24 hours. Determine which issues need to undergo a postmortem. See one of our earlier blogs for more information: http://akfpartners.com/techblog/2010/08/29/morning-operations-meeting/
  2. Communicate to your team and your business on the failure and what is being done about it.
  3. Run a postmortem determine multiple causes and actions and owners to address the causes: http://akfpartners.com/techblog/2009/09/03/a-lightweight-post-mortem-process/
  4. Always restore your systems to normal service as quickly as possible. If you have split your architecture along the Y or Z axis and one of the swim lanes fails or an entire datacenter fails, you need to bring it back up as quickly as possible. See one of our past blogs for more details on splitting your architecture: http://akfpartners.com/techblog/2008/05/30/fault-isolative-architectures-or-“swimlaning”/

 

Send to Kindle

Comments Off

Wine-Dark Sea

“There is a land called Crete in the midst of the wine-dark sea, a fair land and a rich, begirt with water, and therein are many men innumerable, and ninety cities.” - Homer (fl. 850 B.C.), Odyssey, Book XIX

And some one shall some day say even of men that are yet to be, as he saileth in his many-benched ship over the wine-dark sea…” – Homer (fl. 850 B.C.), Iliad, Book VII

The phrase “wine-dark sea” appears dozens of times in the Iliad and the Odyssey resulting in much debate at what Homer actually meant by the phrase. One theory is that he was describing an outbreak of red-colored marine algae. Another theory put forward by researchers Wrignt and Cattley, was that the Greeks mixed highly alkaline water with their wine, resulting in blue-wine.

The explanation that sounds most plausible to me is that ancient Greeks did not have a word for “blue”. At the time of Homer’s writing there were only five colors (metallics, black, white, yellow-green, and red). Lacking the appropriate term to describe the world, they used what they knew.

Many of us are not unlike the ancient Greeks. We can’t describe a new architecture or we can’t imagine our business differently because we don’t have words for it. Whether you are just out of school and don’t have a ton of experience or your more senior but haven’t seen highly scalable system architectures, both leave you blind to “seeing” a different architecture or business model. Another way we’re blinded is just by being at a business for a number of years. Companies and institutions have collective beliefs, cultures, and even memories that become our own. This is completely normal. If you don’t adapt to the standards and norms of the company you’re going to have a rough go of it. Just like the body tries to reject transplanted organs because the body doesn’t recognize the foreign object, companies reject employees and even leaders who don’t fit or adapt to fit.

So, what is one to do if you know you suffer from a blind spot? The solution to this is to bring in new people or, occasionally, consultants who have seen this before and can help you learn to describe the new world. As made famous by many twelve-step programs, the first step is to “admit there is a problem”. If you can’t admit that you might not see or be able to accurately describe everything, you’ll eventually get blindsided by either the inability to scale or, even worse, a competitor able to see things differently.

Send to Kindle

Comments Off

It’s Not About the Technology

Perhaps it’s because we’re technologists that we love shiny new technologies. However, for years now AKF has been telling anyone that will listen or read, that “scaling is not about the technology”. Scale comes from system-level or deployment-focused architecture which is the intersection of software and hardware. No doubt, we have some amazingly scalable technologies available to us today like Hadoop and MongoDB but when your entire datacenter goes down (like Amazon and Calgary and GoDaddy and Sears and the list goes on…) these scalable technologies don’t keep your service available. And if you think that your customers care whether it was your fault or your vendor’s fault…you’re wrong. They pay you for your service and expect it available when they need it.

No doubt, the technology decisions are important. Whether you use Backbone or Knockout or whether you choose Memcached or Redis, all of these technology decisions have pros and cons which can effect your team for a long time. But, at the end of the day these decisions are not ones that will affect whether your application and organization can scale with growth. These technology decisions affect the speed and cost factors of your organization and technology. Some technologies your team knows more about or are naturally faster to learn; therefore, these cost you less. Other technologies are very popular (PHP) and thus engineers’ salaries are lower because there is more supply. Yet still other technologies (assembly language) are complex, appeal to a select group of engineers, are very costly to develop in but might cost very little to process transactions because of the efficiency of that technology.

Technology decisions are important but for different reasons than scaling. Relying on a technology or single vendor to scale is risky. To the vendor or open source project, you are one of many customers and the longevity of their business or project doesn’t depend on keeping your service available. However, your business does depend on this. Take control of the future of your business by scaling your service and organization based on systems-level or deployment-focused architecture. Leave the technology decisions outside of the systems architecture.

 

Send to Kindle

Comments Off

AKF Agile Training

It is pretty self-evident that Software as a Service (SaaS) companies have to deliver customer-facing features quickly, at low cost, and high quality. Pay-by-usage models and not having to install software makes switching costs for your customers lower than ever. This makes competition even more fierce, requiring companies to be more nimble than ever in order to stay ahead.

Gartner has indicated that Agile approaches are providing the benefits of fast, accurate delivery of priority application requirements but that the methodology is approaching the trough of disillusionment in its hype cycle. This doesn’t mean that the methodology is flawed, it is simply the normal part of any IT trend that is going mainstream. As Nathan Wilson states, “The early days of any trend are full of promise, followed by a level of hype that the trend is going to be a silver bullet that will solve all problems. Of course no new trend can meet these expectations, and the trough of disillusionment follows when people realize that this is not a silver bullet.”

While no software or product development methodology is a panacea, we believe that for almost all SaaS companies the best approach is an Agile development methodology. We consider Agile as a product development life cycle methodology (PDLC) rather than a software development life cycle methodology (SDLC) because it is a business process that must include business owners on the teams. Note that this is the number one problem we see with companies implementing Agile and often the root cause is a lack of formal training.

Because we believe that any rapidly growing SaaS company must deal with scale issues, we have developed an Agile Training program specifically focused on the critical components of scalability – architecture, organization, and process. If a company misses on any of these three components they are likely to have scale issues at some point.

If you are interested in AKF Agile Training focused on scalability get in contact with us. If you are still undecided about Agile or other methodologies check out this post.

Send to Kindle

2 comments

Newsletter – Summer 2013

Here is a copy of our latest newsletter, if you would like it delivered to your inbox signup here.

The Power of Customer Misbehavior

Our third book, The Power of Customer Misbehavior, has entered the production phase with our publisher, Palgrave Macmillan. This book presents some newly discovered drivers of viral growth – product misuse and self-identity. Below is one of the many stories in the book. In this one we relate a story from the early days at YouTube in which they were able to leverage customer misbehavior to guide their ultimately very successful product development.

As Maryrose Dunton, the Director of Product Development and one of the first product managers at YouTube recalls, “YouTube was initially a technology looking for a business problem to solve.”  The founders experimented with various practical implementations of their technology, which allowed one to upload and share videos in a “flash” player.  They thought the technology and the site might be used to showcase real estate for sale or used as a dating site to give personal testimonials.  “But instead,” said Maryrose, “we found that people were uploading videos of their cats and of them performing skateboarding tricks.  We thought ‘Really?  That’s how they want to use it?’  We took their lead and came up with the moniker ‘Broadcast Yourself’ – something that still exists today.”

This example is a great one that displays not only how customers define their identity online and come up with innovative ways to use and even misuse a product, but also how a company can enable that misuse for its own benefit.  As Maryrose’s incredulous question shows (e.g. “Really?  That’s how they want to use it?”), the team was not expecting people to post funny animal tricks and silly videos of themselves online. They could very well have shut down such usage right then.  But instead, the team enabled the usage and even created a marketing tagline based on it.

This is only one story that we wrote about to bring the concepts to life. You will find many more in the book. Follow the book on Facebook or on Twitter to receive updates and preview chapters.

Associate Partner

We’re very excited to have Mike Paylor, a new Associate Partner, join us at AKF. We’ve known and worked with Mike for almost 15 years and have been continually impressed with his knowledge of technology, management, and building sustainable organizations. Over the last 9 years Mike has been integral to the massive growth of PayPal, most recently as Director of Risk Infrastructure Technology. In that role Mike managed more than 60 software engineers worldwide building industry leading Risk & Data Analytics platforms. Mike earned a B.S. in Management Information Systems from The Pennsylvania State University. If you’d like to contact Mike directly his email is:mike.paylor@akfpartners.com

Agile Training

We have partnered and helped several clients around the globe scale their product architecture, processes, and organizations. Many times we see clients who are handcuffed from old project management methods that slow their TTM or they haven’t been able to realize the full benefits from Agile. We are proud to offer a new service, Agile Training with a focus on scaling the way AKF teaches companies. We will partner with you to ultimately help improve your TTM and Quality making your products and company more competitive in the marketplace. We will do this by providing you with the necessary training, consulting, and coaching to help transition your business and engineering teams to an Agile framework while complementing it with AKF Scaling best practices. The goal is to teach the tools, share the knowledge, and encourage the behavior necessary to apply in your business’s context. If this training would be beneficial to your team, reach out to us info@akfpartners.com.

Send to Kindle

Comments Off