The Power of Customer Misbehavior is now available for the Kindle. Here is a short video about the book.
The Power of Customer Misbehavior is now available for the Kindle. Here is a short video about the book.
Its Monday morning and past Saturday evening issues in one of your datacenters triggered a failover to your second data center for service restoration. In other words, all customer traffic has been routed to a single datacenter. The failover was executed flawlessly and the team went back to bed waiting for Monday morning to permanently fix the issue so traffic could once again run out of both datacenters. On Monday morning, we are expecting a flash sale and will make close to $8000 a minute at peak. All is well and there is nothing to worry about. Right?
Hopefully you cringed at the above scenario. What if the data center you are running out of suffers from a failure? Or what if the only data center and its components that is now live for all of your traffic simply wasn’t sized correctly for acceptable performance during a traffic spike?
If it hasn’t happened yet, it will. If that were the case, your business would stand to lose significant revenue. We see it over and over again with many clients and have also experienced it in practice. Multiple datacenters can serve as a false sense of security and teams can become complacent. Remember, assume everything will fail as a monolith. If you are only running out of a single data center and the other is unable to take traffic, you now have a SPOF and as a whole the DC is a monolith. As a tech ops leader you have to drive the right sense of urgency and lead your team to have the right mindset. Restoring service with a failover is perfectly acceptable. However, the team cannot stop there. They must quickly diagnose the problem and return the site to normal service, which means you are once again running out of two datacenters. Don’t let the false sense of security slip into your ops teams. If you spot it, call it out and explain why.
To help combat complacency from setting in, we recommend considering the following:
“There is a land called Crete in the midst of the wine-dark sea, a fair land and a rich, begirt with water, and therein are many men innumerable, and ninety cities.” - Homer (fl. 850 B.C.), Odyssey, Book XIX
“And some one shall some day say even of men that are yet to be, as he saileth in his many-benched ship over the wine-dark sea…” – Homer (fl. 850 B.C.), Iliad, Book VII
The phrase “wine-dark sea” appears dozens of times in the Iliad and the Odyssey resulting in much debate at what Homer actually meant by the phrase. One theory is that he was describing an outbreak of red-colored marine algae. Another theory put forward by researchers Wrignt and Cattley, was that the Greeks mixed highly alkaline water with their wine, resulting in blue-wine.
The explanation that sounds most plausible to me is that ancient Greeks did not have a word for “blue”. At the time of Homer’s writing there were only five colors (metallics, black, white, yellow-green, and red). Lacking the appropriate term to describe the world, they used what they knew.
Many of us are not unlike the ancient Greeks. We can’t describe a new architecture or we can’t imagine our business differently because we don’t have words for it. Whether you are just out of school and don’t have a ton of experience or your more senior but haven’t seen highly scalable system architectures, both leave you blind to “seeing” a different architecture or business model. Another way we’re blinded is just by being at a business for a number of years. Companies and institutions have collective beliefs, cultures, and even memories that become our own. This is completely normal. If you don’t adapt to the standards and norms of the company you’re going to have a rough go of it. Just like the body tries to reject transplanted organs because the body doesn’t recognize the foreign object, companies reject employees and even leaders who don’t fit or adapt to fit.
So, what is one to do if you know you suffer from a blind spot? The solution to this is to bring in new people or, occasionally, consultants who have seen this before and can help you learn to describe the new world. As made famous by many twelve-step programs, the first step is to “admit there is a problem”. If you can’t admit that you might not see or be able to accurately describe everything, you’ll eventually get blindsided by either the inability to scale or, even worse, a competitor able to see things differently.
Perhaps it’s because we’re technologists that we love shiny new technologies. However, for years now AKF has been telling anyone that will listen or read, that “scaling is not about the technology”. Scale comes from system-level or deployment-focused architecture which is the intersection of software and hardware. No doubt, we have some amazingly scalable technologies available to us today like Hadoop and MongoDB but when your entire datacenter goes down (like Amazon and Calgary and GoDaddy and Sears and the list goes on…) these scalable technologies don’t keep your service available. And if you think that your customers care whether it was your fault or your vendor’s fault…you’re wrong. They pay you for your service and expect it available when they need it.
No doubt, the technology decisions are important. Whether you use Backbone or Knockout or whether you choose Memcached or Redis, all of these technology decisions have pros and cons which can effect your team for a long time. But, at the end of the day these decisions are not ones that will affect whether your application and organization can scale with growth. These technology decisions affect the speed and cost factors of your organization and technology. Some technologies your team knows more about or are naturally faster to learn; therefore, these cost you less. Other technologies are very popular (PHP) and thus engineers’ salaries are lower because there is more supply. Yet still other technologies (assembly language) are complex, appeal to a select group of engineers, are very costly to develop in but might cost very little to process transactions because of the efficiency of that technology.
Technology decisions are important but for different reasons than scaling. Relying on a technology or single vendor to scale is risky. To the vendor or open source project, you are one of many customers and the longevity of their business or project doesn’t depend on keeping your service available. However, your business does depend on this. Take control of the future of your business by scaling your service and organization based on systems-level or deployment-focused architecture. Leave the technology decisions outside of the systems architecture.
It is pretty self-evident that Software as a Service (SaaS) companies have to deliver customer-facing features quickly, at low cost, and high quality. Pay-by-usage models and not having to install software makes switching costs for your customers lower than ever. This makes competition even more fierce, requiring companies to be more nimble than ever in order to stay ahead.
Gartner has indicated that Agile approaches are providing the benefits of fast, accurate delivery of priority application requirements but that the methodology is approaching the trough of disillusionment in its hype cycle. This doesn’t mean that the methodology is flawed, it is simply the normal part of any IT trend that is going mainstream. As Nathan Wilson states, “The early days of any trend are full of promise, followed by a level of hype that the trend is going to be a silver bullet that will solve all problems. Of course no new trend can meet these expectations, and the trough of disillusionment follows when people realize that this is not a silver bullet.”
While no software or product development methodology is a panacea, we believe that for almost all SaaS companies the best approach is an Agile development methodology. We consider Agile as a product development life cycle methodology (PDLC) rather than a software development life cycle methodology (SDLC) because it is a business process that must include business owners on the teams. Note that this is the number one problem we see with companies implementing Agile and often the root cause is a lack of formal training.
Because we believe that any rapidly growing SaaS company must deal with scale issues, we have developed an Agile Training program specifically focused on the critical components of scalability – architecture, organization, and process. If a company misses on any of these three components they are likely to have scale issues at some point.
Here is a copy of our latest newsletter, if you would like it delivered to your inbox signup here.
Our third book, The Power of Customer Misbehavior, has entered the production phase with our publisher, Palgrave Macmillan. This book presents some newly discovered drivers of viral growth – product misuse and self-identity. Below is one of the many stories in the book. In this one we relate a story from the early days at YouTube in which they were able to leverage customer misbehavior to guide their ultimately very successful product development.
As Maryrose Dunton, the Director of Product Development and one of the first product managers at YouTube recalls, “YouTube was initially a technology looking for a business problem to solve.” The founders experimented with various practical implementations of their technology, which allowed one to upload and share videos in a “flash” player. They thought the technology and the site might be used to showcase real estate for sale or used as a dating site to give personal testimonials. “But instead,” said Maryrose, “we found that people were uploading videos of their cats and of them performing skateboarding tricks. We thought ‘Really? That’s how they want to use it?’ We took their lead and came up with the moniker ‘Broadcast Yourself’ – something that still exists today.”
This example is a great one that displays not only how customers define their identity online and come up with innovative ways to use and even misuse a product, but also how a company can enable that misuse for its own benefit. As Maryrose’s incredulous question shows (e.g. “Really? That’s how they want to use it?”), the team was not expecting people to post funny animal tricks and silly videos of themselves online. They could very well have shut down such usage right then. But instead, the team enabled the usage and even created a marketing tagline based on it.
We’re very excited to have Mike Paylor, a new Associate Partner, join us at AKF. We’ve known and worked with Mike for almost 15 years and have been continually impressed with his knowledge of technology, management, and building sustainable organizations. Over the last 9 years Mike has been integral to the massive growth of PayPal, most recently as Director of Risk Infrastructure Technology. In that role Mike managed more than 60 software engineers worldwide building industry leading Risk & Data Analytics platforms. Mike earned a B.S. in Management Information Systems from The Pennsylvania State University. If you’d like to contact Mike directly his email is:firstname.lastname@example.org
We have partnered and helped several clients around the globe scale their product architecture, processes, and organizations. Many times we see clients who are handcuffed from old project management methods that slow their TTM or they haven’t been able to realize the full benefits from Agile. We are proud to offer a new service, Agile Training with a focus on scaling the way AKF teaches companies. We will partner with you to ultimately help improve your TTM and Quality making your products and company more competitive in the marketplace. We will do this by providing you with the necessary training, consulting, and coaching to help transition your business and engineering teams to an Agile framework while complementing it with AKF Scaling best practices. The goal is to teach the tools, share the knowledge, and encourage the behavior necessary to apply in your business’s context. If this training would be beneficial to your team, reach out to us email@example.com.
Our newest book The Power of Customer Misbehavior is due out Dec 14, 2013. This book presents the newly discovered drivers of viral growth – product misuse and self-identity. This information comes from rigorous academic research, has been presented at numerous conferences, and is the subject of various papers. We’ve had a few articles published recently that provide an overview of these topics:
Follow the book on Facebook and get all of our updates, sample chapters, and more The Power Of Customer Misbehavior.
Pre-order the book now on Amazon.
Executives often complain to us about the level of conflict within their teams, or the abilities of their teams to resolve conflict. Often, in the same session, they will wonder as to why they don’t achieve greater levels of innovation in their products. Sometimes we hear things like “My team is too conflict avoidant – they won’t resolve their issues”, or “I simply don’t get why product and engineering won’t get along”, or “My engineers (or finance team, or team XYZ) simply won’t listen to the rest of the business teams”. With respect to innovation, quotes range from “Why aren’t we more innovative” to “It seems like I’m the only one with ideas around here!” Well guess what? It’s YOUR FAULT!
The simple truth is that what we do and just as importantly, do not do as executive teams sets the stage for the creation and escalation of conflict and maximization or minimization of innovation within our teams. Before I get into some of the drivers and results of both conflict and innovation, let me first spend some time trying to define some terms and clear up some fallacies surrounding the term “conflict”.
2 Types of Conflict
Psychologists and sociologists, especially those who study conflict, conflict resolution and innovation, recognize the following:
1) Affective Conflict (sometimes termed Role Based) is “bad conflict”. Affective conflict typically involves “who” should be responsible for doing something or “how” something should be done. Any way you cut it, our jobs as leaders are to try to minimize this type of conflict and set up systems to reduce the probability that it happens. Affective conflict is never good.
2) Cognitive Conflict is “good conflict”. This type of conflict deals with “what” should be done and “why” something should be done. When handled properly, it helps increase strategic possibilities, open up new doors for innovation and drives teams to better answers. When left unhandled by leaders, it will often escalate into affective conflict and be detrimental to overall performance.
I will just call these terms “bad conflict” and “good conflict”.
Outcomes (or consequences) of Conflict
Here we see why affective conflict is “bad” and cognitive conflict is “good”.
1) High levels of bad conflict are highly correlated with lower morale, high rates of employee turnover within teams, slower time to market, lower levels of financial/business performance, and lower levels of perceived performance within teams. Bad conflict minimizes and very often can completely eliminate innovation within a team.
2) High levels of properly handled good conflict are highly correlated with higher morale, higher employee retention, higher levels of innovation, happier teams, faster time to market and higher levels of financial and business performance.
Put simply, you want lots of quickly resolved and properly facilitated good (“what and why”) conflict and very little bad (“who and how”) conflict.
Drivers of Bad Conflict
Research indicates that there are many drivers of “bad” conflict. Here are some of the most frequent drivers:
1) Team Alignment: Without getting into the theoretical specifics, teams find themselves in conflict along team boundaries that are defined by organization structure and the resulting individual identities. If you have an “engineering team” and a “product team” on your organization charts, the employees will identify with one or the other and sometimes conflict along those boundaries.
2) Employee Tenure: Research shows that employees also identify themselves along tenure boundaries. These may be the “old guard” and the “new employees” or the “founding team” and the “new team”, etc. While not codified within an org chart, this stratification of employee identities can lead just as much to conflict creation as an actual organization.
3) Leadership Approach: Engaging in quid-pro-quo management (“Do this for me, and I’ll do that for you”) is highly correlated with conflict creation. Individuals think of themselves, rather than a team.
4) Loosely Defined Responsibilities: Chaos breeds conflict. When people don’t understand the boundaries and expectations, they attempt to expand or define each for themselves which in turn increases strife.
Out with the Bad and In with the Good
Bad conflict is inversely correlated with good conflict. The more bad conflict you have, the less good conflict you have. Good conflict on the other hand correlates directly with innovation. The higher the good conflict, the higher the level of innovation. The fix to ending bad and promoting the good is to address each of the areas above.
1) Team Alignment: Create cross functional teams aligned around product or service initiatives – not functions or experiences. Each team should have all the experience across all domains that it needs to be successful.
2) Employee Tenure: Eliminate old and new employees who cling to tenure based identities. Ensure everyone understands there is one, and only one company identity (though there may be product identities within the company – but each of these has everything they need to be successful).
3) Leadership Approach: End quid pro quo leadership and talk about higher purposes and team accomplishments. Make the job about something other than just the individual. Eliminate people who are “in it for themselves” and won’t contribute to the higher cause.
4) Loosely Defined Responsibilities: A leader has many responsibilities and one of them is to eliminate uncertainty where possible. Help people quickly resolve conflict with roles. Don’t let employees come into conflict or stay in conflict about competing roles and responsibilities.
Over the course of the last year, we have seen several of our clients either start exploring or make plans to move their SaaS products to the “Cloud” or an IaaS provider. We thought we would share some of the misconceptions we sometimes see and our advice.
- I can finally focus on product development and software engineering and not worry about this infrastructure stuff.
The notion that IaaS providers like Amazon have eliminated your worries about infrastructure is only partially true. You may not need to understand everything about designing an infrastructure with bare metal but you need to make sure you understand how your virtual configuration in the Cloud will affect your product. IaaS helps us to quickly deploy infrastructure for our products but it doesn’t eliminate the need for good high availability and fault tolerance design. There are several levers you can pull within an IaaS console and design decisions that will impact your products performance. To ensure good design and configuration, its ultra important that your SaaS product engineering team is made up of talent that has expertise in distributed application architecture design, infrastructure, security, and networking. Having this knowledge will help you design a high performing, fault isolated product for your business.
- Going to the cloud pretty much guarantees me high availability because of auto scaling.
Going to the cloud will provide you with the ability to scale quickly as load increases but it will not provide you with high availability. For example, if you have a monolithic code base that you deployed and you are pushing to production on a regular basis, there is a pretty good chance you will introduce a defect at some point that impacts the availability of your entire service and business. We advise our clients to split their applications appropriately, deploy the services to separate instances, and, assuming you are using Amazon, configure them to run across multiple zones within a region at a minimum (preferably across regions). This allows you to focus dedicated teams to the individual services and reduce the likelihood of introducing a defect that takes down your system.
- The Cloud will be cheaper than a collocation or managed hosting provider.
There are several factors that need to be considered before you can confirm that is cost effective. You should look closely at load on your servers. If your servers are not serving traffic around the clock, it may be better from a cost perspective for you to buy and maintain your own infrastructure in a collocation or in an existing data center you may have. The economics of this decision is changing rapidly as IaaS pricing is declining due to the competition in the industry. A simple spreadsheet exercise will help determine if the move to Cloud would be cost effective for your business.
- The Cloud isn’t secure so we better not use it.
The cloud isn’t necessarily what makes or breaks security around your SaaS product. Many believe that public cloud services like Amazon’s EC2 service isn’t secure. First off, you are far more likely to experience a security breach because of an employee’s actions (either intentionally or unintentionally) than caused by an infrastructure provider. Secondly, Amazon likely has invested much more in security at various layers, including their physical data centers, than most companies we see who have their own data centers. Amazon has designed the infrastructure to isolate customer instances and you can also choose to take advantage of Amazon Virtual Private Cloud that can be configured to create an isolated network. There are various options for encrypting all of your data as well. This only touches the surface for security design options you have and they continue to be enhanced everyday. You can see why it’s important to staff your team with an engineer who has experience in this space.
If you are looking to move to Cloud, don’t rush into the decision. Do your homework and make sure it’s right for your business. Make sure you have the talent that has experience with the technology that will get you there and run your operations. Once you make the leap, you will have to live with it for a while.
One of the most important aspects of managing a successful technology organization is ensuring that you are practicing & instilling the concept of enablement at all levels. This concept applies to both the product/service you are producing and for people. A good example for your organization is enabling decision-making at the lowest levels possible. I have often seen this represented as “delegation” but I believe that enablement of decision-making is a more powerful concept than delegation which is driven from the top-down. I recently had the opportunity to lead a large infrastructure team and one of the first changes we made was breaking apart into reasonable sized PODs with the primary purpose of ensuring that decisions for the product & technology were driven from the bottom-up while guidance was flowing in from various stakeholders. Many teams practice a flavor of Agile but without enabling each POD to make the appropriate decisions you will run into organizational scalability problems.
The allure of IaaS & PaaS is firmly rooted in the concept of enablement. Self-service is an amazing if you are a DBA, developer or even the end user of your product. The cloud may not be suitable for your needs but don’t let that stop your organization from thinking strategically about bringing those processes & technologies “in-house” for scalability reasons. Implemented correctly, the infrastructure and platform you are building should enable the users and not hinder them, as is sometimes the case. Reducing the number of dependencies between technology teams for launching products is not only good for cycle time of product launches but also critical in scaling up.
Consider making enablement part of your technology team’s DNA and you will likely see that employee morale, productivity and other metrics like NPS will rise.