AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Category » Newsletters

Newsletter – Summer 2013

Here is a copy of our latest newsletter, if you would like it delivered to your inbox signup here.

The Power of Customer Misbehavior

Our third book, The Power of Customer Misbehavior, has entered the production phase with our publisher, Palgrave Macmillan. This book presents some newly discovered drivers of viral growth – product misuse and self-identity. Below is one of the many stories in the book. In this one we relate a story from the early days at YouTube in which they were able to leverage customer misbehavior to guide their ultimately very successful product development.

As Maryrose Dunton, the Director of Product Development and one of the first product managers at YouTube recalls, “YouTube was initially a technology looking for a business problem to solve.”  The founders experimented with various practical implementations of their technology, which allowed one to upload and share videos in a “flash” player.  They thought the technology and the site might be used to showcase real estate for sale or used as a dating site to give personal testimonials.  “But instead,” said Maryrose, “we found that people were uploading videos of their cats and of them performing skateboarding tricks.  We thought ‘Really?  That’s how they want to use it?’  We took their lead and came up with the moniker ‘Broadcast Yourself’ – something that still exists today.”

This example is a great one that displays not only how customers define their identity online and come up with innovative ways to use and even misuse a product, but also how a company can enable that misuse for its own benefit.  As Maryrose’s incredulous question shows (e.g. “Really?  That’s how they want to use it?”), the team was not expecting people to post funny animal tricks and silly videos of themselves online. They could very well have shut down such usage right then.  But instead, the team enabled the usage and even created a marketing tagline based on it.

This is only one story that we wrote about to bring the concepts to life. You will find many more in the book. Follow the book on Facebook or on Twitter to receive updates and preview chapters.

Associate Partner

We’re very excited to have Mike Paylor, a new Associate Partner, join us at AKF. We’ve known and worked with Mike for almost 15 years and have been continually impressed with his knowledge of technology, management, and building sustainable organizations. Over the last 9 years Mike has been integral to the massive growth of PayPal, most recently as Director of Risk Infrastructure Technology. In that role Mike managed more than 60 software engineers worldwide building industry leading Risk & Data Analytics platforms. Mike earned a B.S. in Management Information Systems from The Pennsylvania State University. If you’d like to contact Mike directly his email is:mike.paylor@akfpartners.com

Agile Training

We have partnered and helped several clients around the globe scale their product architecture, processes, and organizations. Many times we see clients who are handcuffed from old project management methods that slow their TTM or they haven’t been able to realize the full benefits from Agile. We are proud to offer a new service, Agile Training with a focus on scaling the way AKF teaches companies. We will partner with you to ultimately help improve your TTM and Quality making your products and company more competitive in the marketplace. We will do this by providing you with the necessary training, consulting, and coaching to help transition your business and engineering teams to an Agile framework while complementing it with AKF Scaling best practices. The goal is to teach the tools, share the knowledge, and encourage the behavior necessary to apply in your business’s context. If this training would be beneficial to your team, reach out to us info@akfpartners.com.

Comments Off on Newsletter – Summer 2013

Newsletter – Pod People

Here is a copy of our newsletter, if you would like it delivered to your inbox signup here.


“Pod people” was a nickname for an alien species featured in the 1955 novel The Body Snatchers. They were a group of nomadic, extraterrestrial parasites floating through space and taking over planets. Today we’re seeing a reemergence of “pod people” in our technology organizations but unlike the alien pod people these are productive groups of employees who speed up Time to Market (TTM) and foster greater creativity.

Problems with Hierarchical Organizations
Corporate organizations are primarily modeled after military organizations which are hierarchical. While some corporations have attempted to eliminate managers, most are aligned by division under a C-level executive (finance – CFO, technology – CTO, marketing – CMO, etc). Under these individuals are typically sub-divisions by specific tasks such as software development, QA, product management and technical operations under a CTO. There are three major problems with this hierarchical model – no single owner, conflict, and efficiency.

When delivery of a product requires multiple teams within a hierarchical organization no team or person on the team is actually responsible for delivery. Ownership by all is ownership by none. When the release of the product is late no single person / team is to blame.

Speaking of blame, putting people in teams causes people to start associating their self-identity to that team. In 1968, a teacher named Jane Elliott divided her class into blue-eyed and brown-eyed groups. On the first day, the blue-eyed children were told they were better than the brown-eyed group and on the second day the roles were reversed. Children designated as inferior performed poorly on tests and those designated as superior became mean-spirited. We see similar issues when people are separated into different groups by function – they start thinking other groups are inferior.

So if there are all these problems with hierarchical, functional organizations, why do we organize this way? One reason is that this organization structure is optimized for efficiency of the different functional skills. For example, this isolation by skill is very efficient for software developers to produce code. However, SaaS companies provide their customers with services not code. Certainly code is an important part of the service but it isn’t what the customer is buying. In order to optimize for what the customer is buying (the service) we need to organize around those services.

Swim Lanes
In systems architecture we use the term “swim lanes” or fault isolation zones to describe a service that is completely isolated. In order to achieve this there can be no synchronous calls between services. The reason is that synchronous calls tie the availability and scalability of services together. If you have separate services on your site such as browse, search, and checkout if the third party that provides the checkout service goes down, your users can still browse and search. What if we apply this architecture to an organization?

Org to Architecture Alignment
It turns out that organizations grouped into small cross-functional teams are very efficient in terms of overall service delivery. If we align these teams or pods with the services in our “swim lane” architecture they are able to act even more autonomously which speeds up their TTM. This is what Agile teams are supposed to be, at least according to the Agile Manifesto that states as one of the twelve principles that “the best architectures, requirements, and designs emerge from self-organizing teams.” The teams no longer have to coordinate across teams for changes in code or approval of ideas before implementing.

Individuals on these Pods associate their self-identity with the team rather than with their skillset. Because the team is responsible for delivery of the service – from idea inception to delivery and support – there is less conflict between teams. Team members participate in more cognitive conflict (the good type) where they debate how things should be done and have less affective conflict (the bad type) where they argue about who should do what tasks.

We also have a single team that owns the service rather than this responsibility spread across multiple teams. We have one group of people to praise when things go well and one group to work with when delivery dates are missed. While pods provide a lot of benefits and solve many of the problems with hierarchical teams, there are also some issues to watch out for.

Potential Problems
For maximum efficiency, pods must be aligned to the architecture. They also need to be aligned to business units. Having general managers of different business units pull on a single Pod to help achieve different goals forces the pod to prioritize which business goal is more important. Instead, the GM and pod should jointly decide how important each goal is in achieving and allowing this to determine the priority of work. Having a single GM sponsor of a pod helps ensure that this is done properly.

Another potential issue with pods is when they are not given ground rules to operate under. While we want the teams to be as autonomous as possible, there are some things that should be standardized across teams. For example, teams might find it useful to select whether they use ideal days vs. story points for estimation. This type of “customization” of a process is probably fine. However, customizations that cost more to support, in terms of extra people or training, should be avoided. Having multiple source code control systems (Git, CVS, etc) is probably not necessary and just costs the business more.

Whether you want to be a pod person or stick to a hierarchical organization there is no panacea. Building, delivering, and maintaining great services still require hard work and dedication. However, in the ever present cycle of distributed-centralized we are now seeing the trend towards small decentralized teams or pods. If you’re implementing pods, good luck, and let us know your experiences.


Newsletter – New Associate Partners

Below is part of our recent newsletter. If you haven’t subscribed yet, click here to do so.

New Associate Partners
We are very excited to announce that Geoff Kershner and Dave Berardi are joining AKF Partners.

For those who have worked with AKF in the past, you might recognize Geoff as he was part of our extended team a couple years ago. He most recently spent time at LiveOps running a Program Management Office in efforts including Agile development and PCI-DSS certification. He also spent 6 years at eBay and PayPal.

Dave is a long time friend and colleague as well who recently spent a number of years helping Progressive with incident and problem management processes. He has led various software development teams and spent 8 years at GE working on data warehousing and six sigma projects.

Check out Dave and Geoff’s full profiles.

We’re seeing three interesting areas continue to popup this summer:

  1. Cloud Computing
  2. Becoming a SaaS Company
  3. Agile Organizations

Here are some highlights:

Cloud Computing
One of our latest blog posts discusses the current trends with cloud computing. A common definition of cloud computing is the delivery of computing and storage capacity as a service. A distinct characteristic is that users are charged based on usage instead of a more conventional licensing or upfront purchase. There are three basic types of cloud computing: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Servie (SaaS).The major trend is that we are seeing more and more companies becoming comfortable with cloud offerings. In a 2012 survey with 785 respondents by North Bridge Venture Partners a very low 3% of respondents consider cloud services to be too risky and 50% of the survey respondents now say they have “complete confidence” in the cloud. The demand for SaaS offerings is growing rapidly as Gartner predicts that SaaS will hit $14.5 billion in 2012 continuing through 2015 when it will be $22.1 billion. This means your company should be leveraging the cloud as infrastructure where possible (burst capacity, rapid provisioning, etc) and if you’re not already a SaaS provider you need to be thinking about moving their rapidly. Which brings us to the second trend…

Becoming a SaaS Company
Given the comfort with cloud offerings that is now pervasive in industry if you’re don’t already provide a SaaS offering you need to start thinking about it now. One of the hardest things to do is to move from an on-premise or shrink-wrapped software company and become a Software-as-a-Service company. It is not enough to simply bundle up an application in a hosted fashion and label yourself a “SaaS” company.  If you don’t work aggressively to increase availability and decrease your cost of operations, someone with greater experience will come along and simply put you out of business.  After all, your business is now about SERVICE – not SOFTWARE. We have pulled together some advice such as start thinking about having two different products (on-premise and SaaS) or determine which business you want and kill the other one off. Attempting to satisfy both with a single architecture will likely result in you failing at both.

Our belief is that scalability and ultimately business success require architecture, organization, and process to all be aligned and functioning well. This brings up our last trend, how to organize for success as a SaaS company.

Agile Organizations
We’ve written lots of posts about this issue starting with how“old school” organizations are aligned functionally and then hownew Agile organizations are aligned by delivery of services. The organizations that align this way have remained remarkably nimble and agile despite sometimes having thousands of software developers. As a SaaS provider you need to rethink the traditional organizational structure and start thinking about how to organize to provide the best service possible at the lowest cost.

1 comment

Newsletter – Trends

Below is our most recent newsletter. If you would like to subscribe and have it delivered to your inbox, you can do so here.

We’re privy to meeting and talking with hundreds of high tech, hyper growth companies and therefore we have a unique opportunity to see trends that are taking place in the industry. Here are a few that we thought would be interesting to share with our friends.

NoSQL Skills – Many companies are experimenting with NoSQL solutions including Membase, Hadoop, Cassandra, and many others but are finding that employees with skills in these are hard to come by. Even in areas rich in talent such as the Silicon Valley, Boston, Austin, Seattle and NY the demand for this rather nascent skill set is higher than the current supply. Many companies are relying on on-the-job training for these types of open source solutions. Most importantly, if you’ve properly fault isolated your architecture you can tolerate the risk associated with small segments going down during beta releases while you iron out the kinks.

Oracle’s NoSQL – Oracle recently announced their entry into NoSQL with announcements around Hadoop integration and their own NoSQL key-value solution. The Hadoop integration isn’t much more than a conduit to allow data to and from an Oracle database and a Hadoop cluster. This technology has been around for quite a while in other relational database solutions such as GreenPlum and Asterdata. It was also recently announced by Microsoft that SQL Server would support this Hadoop connection as well.

Oracle’s NoSQL is a distributed, replicated key-value store and is new and exciting even though it has attributes similar to other product offerings including their acquisition, Berkeley DB. Given trend #1 above, it may sound like an interesting alternative to adopting an enterprise supported NoSQL alternative to open source solutions but beware. The marketing materials for Oracle’s NoSQL solution include the BASE acronym (Basically Available, Soft State, Eventaully Consistent) but unlike other NoSQL products such as Dynamo, SimpleDB, or Cassandra, the Oracle NoSQL Database does not support eventual consistency. Oracle’s solution to eventual consistency appears to be by not accepting writes when the primary node for that key is down.

ORM – many companies start off using an object relational mapping solution such as Hibernate or Active Record but we are seeing many of them having difficulty scaling with them. The solution for several companies has been to use ORMs for simple queries but resort to ODBC or their own Data Access Layer for handling more complex queries. Be wary of using a solution to handle your query development as we’ve had a number of clients with incredibly complex and costly queries bring down their platforms for extended periods of time.

Enterprise Monitoring Frameworks – Until very recently, we had not seen a proprietary third party (non open-sourced) monitoring solution at a customer for at least 2 years and that includes our large Fortune 500 clients. Many of our clients have adopted innovative new monitoring solutions from Wilytech and Coradiant (now CA and BMC respectively) that look for and help diagnose patterns “on the wire” – whether it be from the browser to their servers or from app servers to databases. While these are interesting and potentially worth some of your attention, our very best clients design their systems to be monitored from the ground-up – ensuring that their software helps identify performance problems as they happen.

Distributed File Systems – Take your pick of implementations, but many of our clients are eschewing traditional NAS/NFS devices for distributed storage pools the likes of Gluster, MogileFS and Ceph. Nearly every case has resulted in significant savings relative to proprietary systems with few reported impacts to availability or response times. Of course, as with any other architectural change you need to ensure that you are properly managing your risk through pods or swim lanes.


Newsletter – Spring 2011

Below is part of our Fall 2010 Newsletter.  If you haven’t subscribed yet, click here to do so.

In this newsletter:

Scalability Rules

Scalability Rules: 50 Principles For Scaling Websites is available for presale. We are just a few short weeks away from the release date and are very excited about this project. This book is meant to serve as a primer, a refresher, and a lightweight reference manual to help engineers, architects, and managers develop and maintain scalable Internet products. It is laid out in a series of rules, each of them bundled thematically by different topics. Most of the rules are technically focused, while a smaller number of them address some critical mindset or process concern – each of which is absolutely critical to building scalable products.

It is available for preorder from these sites:

You can also help us get the word out about this book by liking and sharing the book’s Facebook page or the book’s official website, where we’ll keep up to date information about reviews and speaking engagements.

With the success of The Art of Scalability, we’ve been asked by a few folks, why write another book on scale? Our answer is that there simply aren’t many good books on scalability on the market yet, and Scalability Rules is unique in its approach in this sparse market.  Also, this is the first book to address the topic of scalability in a rules-oriented fashion. One of our most-commented-on blog posts is on the need for scalability to become a discipline. We and the community of technologists that tackle scalability problems believe that scalability architects are needed in today’s technology organizations. This book will help scalability architects, scalability evangelists, and the like to share their knowledge with others in scaling their systems.  See More…

Our first book The Art of Scalability is still available at these retailers:


Most Popular Posts

We know everyone is busy and often our RSS readers get filled with too many interesting articles to keep up with.  Here are summaries of a few of our posts and some by other authors that we particularly enjoyed.

Why A Technology Leader Should Code
The military teaches that a leader should be “technically and tactically” proficient. Military leaders owe it to their subordinates to understand the equipment that the unit employed and the basic combat tactics that would be followed. This concept is transferable to technology companies; the CTO owes it to their subordinates to understand the technology. They also owe it to the business to understand the economic aspects of the business and be able to straddle these two worlds. Additionally, periodically having to code a feature and deploy it will provide the engineering manager a better understanding and appreciation for what her engineers go through on a daily basis. Read more

What Is That Delay Costing?
Most technologists know that the slower the page the more likely the user will flee the page or the transaction flow and not make the purchase.  Research is teaching us that it may be less important to reduce actual delay rather than create a system where users will be less likely to attribute the delay to the site. An example that we sometimes see is to give the user the option of selecting a low or high graphic site in order to provide the users with the control. Users will likely perceive this as an active effort on the part of the SaaS provider to minimize download time and thus attribute delays to themselves, their computer, their ISP, etc but not the site. Read more

DevOps is an umbrella concept that refers to anything that smoothes out the interaction between development and operations and is a response to the growing awareness of the disconnect between development and operations. There is an emerging understanding of the interdependence of development and operations in meeting a business’ goals. While not a new concept, we’ve been living and suggesting ARB and JAD as cornerstones of this coordination for years, DevOps has recently grown into a discipline of its own. Read more

Google Megastore
Google provided a paper detailing their design and development of “Megastore.” This is a storage system developed to meet the requirements of today’s interactive online services and according to the paper it blends the scalability of a NoSQL datastore with the convenience of a traditional RDBMS in a novel way, providing strong consistency and high availability. The system’s underlying datastore is Google’s Bigtable but it additionally provides for serializable ACID semantics and fine-grained partitions of data. Read more

Scalability at the Cost of Availability
One subtle concept that is sometimes misunderstood is that if not careful an increase in scalability can actually decrease your availability. The reason for this is the multiplicative affect of failure with items in series.  If two pieces of hardware independently have 99.9% uptime, when combined into a single system that relies on both to respond to requests, the availability of the system to go down to 99.9% x 99.9% = 99.8%. Read more

8 Lessons We Can Learn From The MySpace Incident
Robert Scoble wrote a case study, MySpace’s death spiral: insiders say it’s due to bets on Los Angeles and Microsoft, in which he reports that MySpace insiders blame the Microsoft stack on why they lost to Facebook.  Some lessons can be gleaned from this including All computer companies are technology companies first and Enterprise Programming != Web Programming and Intranet != Intranet. Read more

Aztec Empire Strategy: Use Dual Pipes For High Availability
The Aztecs built the great aqueduct 600 years ago but even then thought about uninterrupted supply.  This post states that the purpose of the twin pipes was to keep water flowing during maintenance.  When one pipe got dirty, the water was diverted to the other pipe while the dirty pipe was cleaned. Read more


Research Update and Request for Help
Marty and Mike will both be presenting their research at the 2011 Academy of Management Conference. Marty’s research deals with tenure based conflct and Mike’s research is focused on social contagion (a.k.a. viral growth). You can read the abstracts and full text for both papers here.

We are continuing our research and could use your help. Please consider completing one or both surveys.

If you are an executive team member at a startup, please take this survey and pass it along to your colleagues within your company.

If you participate in any of the following social networks (Facebook, Friendster, LinkedIn, Twitter, MySpace, Ning, Orkut, or Yahoo!360), please take this survey and pass it along to your friends or colleagues.

Thanks for your support!

Comments Off on Newsletter – Spring 2011

Newsletter – Firesheep

Below is part of our Fall 2010 Newsletter.  If you haven’t subscribed yet, click here to do so.

In this newsletter:

Scalability Rules

In between working with some terrific new clients this year we’ve been busy writing the second book, Scalability Rules.  With the help of some terrific technical reviewers we feel the book is taking shape very nicely and the first five chapters are now available on Safari Rough Cuts for those interested in helping review.  Scalability Rulesbrings together 50 rules that we have gathered from our experiences working with over a hundred hyper-growth companies. This format of practical rules of scalability should make it ideal for use as reference manual in formal meetings and informal discussions.

See More…

If you haven’t picked up your copy of The Art of Scalability or have technologist on your holiday gift list here are a couple links for you:

Putting Out Firesheep (Protecting Your Users’ Cookies)

One of our recent blog posts that we found most interesting was submitted by a guest blogger.  Randy Wigginton is a seasoned technologist who after a discussion with us about the security risks, brought to light recently by the Firefox plugin called Firesheep, came up with a solution that we thought should be shared.  Randy’s solution is ideal for companies who want to protect their user session data (login, browsing history, etc) but doesn’t want to be encumbered by the overhead of running their entire site behind SSL.

We also have a simple demo setup for those interesting in testing this solution.  The way it works is when a user logs in, TWO cookies are dropped.  In the demo, one is called “session”, the other is called “authenticate”.  These two cookies are identical except for a single attribute: “authenticate” is a secure cookie.  We authenticate users on non-secure pages by including a reference to a secure javascript at the top of each page.  At the top of pages requiring authentication is this simple line of code:

<script type=”text/javascript” src=”https://verify.akfdemo.com/authenticate.php“>

See More…

Lot18’s Series A

Lot18, a membership-by-invitation marketplace for wine from renowned producers at fantastic values, announced that it has completed a $3 million Series A round of funding led by FirstMark Capital, a New York City-based venture capital firm. Lot18 was founded by Kevin Fortuna and Philip James. Philip was the founder of Snooth.com, the world’s largest wine website, and Kevin was most recently a partner at AKF partners.

Lot18 Screen Shot

See More…

Comments Off on Newsletter – Firesheep

Newsletter – NoSQL

Below is our most recent newsletter. If you would like to subscribe and have it delivered to your inbox, you can do so here.


Over the past few years the NoSQL movement has been gaining in popularity. This is a movement away from the traditional RDBMS such as Oracle or MySQL, that use Structured Query Language (SQL), in favor of high performance data storage systems, that typically use call level interfaces for data storage and retrieval. There are dozens of products, open source and vendor provided, vying for adoption in this space. These products have arisen because of the need for technology teams to find an easier way to scale their data management needs while maintaining high availability, fast performance, and easy administration.

Supporters of the NoSQL movement point out that scaling databases is fraught with difficulties and problems.  Even RDBMS supporters at times promote this view. It is believed that these new technologies have many advantages over databases including that they are much easier to scale across multiple nodes, that they are faster, and that they require much less administration.  There are many companies currently using these data storage systems as either extended memory caches for the application or object caches to offload the database and are doing so with great success. We, in fact, often recommend this as a first step in scaling.

Data storage systems are numerous and there currently is no standard way of classifying all of them. Three often cited categories are key-value stores, extensible record stores, and document stores. My goal is not to classify or review all of the data storage systems available, rather I’d like to focus on some of the key attributes of each of these three classifications and provide some balanced thought about each.

ACID, CAP, BASE, and Cube

Before we begin we need to cover a few topics. The first is ACID, which is an acronym that stands for:

  • Atomicity – all of the operations in the transaction will complete, or none will.
  • Consistency – The database will be in a consistent state when the transaction begins and ends.
  • Isolation – The transaction will behave as if it is the only operation being performed upon the database.
  • Durability – Upon completion of the transaction, the operation will not be reversed.

Most RDBMS guarantee ACID properties. Because of these properties RDBMS can be difficult to scale. When you guarantee consistency of data and you have multiple nodes in your RDBMS cluster, such as with MySQL NDB, synchronous replication is used to guarantee data is written to multiple nodes upon committing the data. With Oracle RAC there is a central database but ownership of areas of the DB are shared among the nodes so write requests have to transfer ownership to that node and reads have to hop from requestor to master to owner and back. No matter the strategy for solving the multiple node plus consistency challenge, the application must wait on this to occur.

According to the Brewer Theorem, developed by Eric Brewer, there are three core requirements that exist when it comes to designing and deploying applications in a distributed environment.  These are expressed in the acronym CAP:

  • Consistency – The client perceives that a set of operations has occurred all at once.
  • Availability – Every operation must terminate in an intended response.
  • Partition Tolerance – Operations will complete, even if individual components are unavailable.

From this we have BASE, an acronym for architectures that solve CAP and stands for: Basically Available, Soft State, Eventually Consistent.  By relaxing the ACID properties of consistency we have greater flexibility in how we scale.

If you are reading this you are probably familiar with the AKF Scale Cube but to quickly refresh your memory the cube consists of X, Y, and Z axes. The X-axis represents read-write splitting of databases such as done through master-slave or standby databases. The Y-axis represents splitting tables onto different database nodes based on which application service or module it pertains to. The Z-axis represents a splitting of the data within tables based on a modulus or lookup.

There are two primary problems that you must face when attempting to scale an RDBMS. As discussed above one of these problems is the requirement of maintaining consistency across the nodes. Eventually you are limited by the number of nodes that data can be synchronously replicated to or by their physical geographical location. The second problem is that data in the RDBMS has a relational structure. The tables of most OLTP systems are normalized to3rd Normal Form, where all records of a table have the same fields, non-key fields cannot be described by only one of the keys in a composite key, and all non-key fields must be described by the key. Within the table each piece of data is very related to others. Between tables there are often relationships, foreign keys. Most applications depend on the database to support and enforce these relationships. Requiring the database to do so makes it difficult to split the database without significant engineering intervention. The simpler the data model the easier it is to scale.

Key-Value Stores

I would include in this group products such as Memcached, Tokyo Tyrant, and Voldemort. These products have a single key-value index for all data that is stored in memory. Some have the ability to write to disk for persistent storage. Across nodes some products use synchronous replication while others are asynchronous. These offer significant scaling and performance by utilizing a very simplistic data store model, the key-value pair. This is obviously somewhat limiting when the application needs to store and relate large amounts of diverse data.  Additionally, the key-value stores that rely on synchronous replication still face the limitations that RDBMS clusters do which are a limit on the number of nodes and their geographical locations. Asynchronous replication provides the same eventually consistent model that an X-axis split in the AKF Scale Cube provides utilizing master-slave or standby databases.

Extensible Record Stores

In this group I would place Google’s proprietary BigTable and Facebook’s, now open source, Cassandra. These products use a row and column data model that can be split across nodes. Rows are sharded on primary keys and columns are broken into groups and placed on different nodes. This method of scaling is similar to the X and Y axes in our Scale Cube, where the X-axis split is read replicas and the Y-axis is separating the tables by services supported. In these products row sharding is done automatically but column splitting requires user definitions, similar to how it is performed in and RDBMS. These products also utilize an asynchronous replication providing eventually consistency. Even though similar scaling can be accomplished with an RDBMS there are still some innovations that are desirable such as Cassandra’s ability to automatically bring nodes in and out of the cluster. By the way, Cassandra in Greek mythology had the ability of prophecy but was cursed that no one would believer her.

Document Stores

I would put CouchDB, Amazon’s SimpleDB, and Yahoo’s PNUTS in this category. The data model used is called a “document” but is more accurately described as a multi-indexed object model. These use domains or collections as combinations of documents and can be queried on multiple attributes. They do not support ACID properties, instead they utilize asynchronous replication providing an eventually consistent model. This is similar to the read replication for x-axis scaling of RDBMS.


In summary, data store systems are very popular and rightly so. They have helped some very large companies scale quite effectively. But, just as cloud computing doesn’t automatically solve your application’s scaling problems, these don’t either. The constraints of 1) numbers and locations of nodes with synchronous replication and 2) the binding of nodes together when relating complex objects to one another, still remain. NoSQL data stores solve these problems by either relaxing the constraint (eventual consistency vs. guaranteed consistency) or restricting the data model (RDBMS table vs. key-value). As Ted Dziuba points out, even Google who owns the most proven highly scalable data storage system, BigTable, still runs AdWords on MySQL. Don’t misconstrue my intent, many of these solutions have places in high scalability architectures but make sure you understand the limitations of each before planning their implementations.

In Other News….

We were recently informed that The Art of Scalability has been selected to be translated into Japanese.  We’re very excited about this and look forward to hopefully many other translations.  If you haven’t picked up a copy here are links to a few online stores AmazonBarnes and NobleBorders, and InformIT.

If you are interested in a preview of the kinds of topics covered in the book check out our latest posts on GigaOM and VentureBeat.  We’ve also been very busy on the blog with posts such as 97 Things – Book ReviewEthical Concerns of China, a leadership example from the Battle of Shilo, and one of the most popular posts 4 Things I Wish I’d Learned As An Undergraduate.


Newsletter – Trends

Below is our most recent newsletter. If you would like to subscribe and have it delivered to your inbox, you can do so here.

It has been several months since our last newsletter and we’ve been very busy working with a lot of new clients as well as getting the opportunity to continue working with some of our existing friends. We’ve also been busy working on our upcoming book, The Art of Scalability. The book will be released January 8th and is available now for preorder at Amazon, Barnes and Noble, Borders, and InformIT.

In our profession, we have a unique opportunity to study a lot of new technology as well as visit with hundreds of technology teams each year. This perspective allows us a vantage point to spot trends that are occurring across the technology spectrum. In this newsletter we are going to cover three of these technology trends that we think you should at least be aware of. Each organization and product offering is different so the applicability of these is going to need to be determined on an individual basis.

Continuous Deployment
The concept of continuous deployment is the natural extension to continuous integration that is the practice of checking code into the source code repository early and often, compiling the code, and performing the integration tests on the new code. The goal of which is to ease the often very painful process of integrating multiple developer’s code after weeks of independent work. In order to make this process the most effective the automation of builds and smoke tests are highly recommended. For more information on continuous integration there are a lot of resources such as these books and articles.

Continuous deployment is when all code that is written for an application is immediately deployed into production. While still a very new concept, there are a growing number of companies that are beginning to adopt this process. Flicker and IMVU are two of the earliest. Eric Ries, CTO of IMVU, believes that this approach can improve software quality due to the discipline, automation, and rigorous standards that are required. In order to be successful Eric suggest a 5 step approach that includes continuous integration, source code commit checks, deployment scripts, alerting, and root cause analysis. You can read more about this process in a recent post.

Before dismissing this idea as not right for your organization consider what one of our clients does that achieves some of the benefits of the approach without so much of the risk. This particular company uses its own software, which many SaaS companies do at least from an administrative perspective, and they deploy each night’s build onto their internal system. They do not deploy each build onto their customer’s production environments but instead wait until the iteration is complete. The concern over disrupting the company’s internal operations are enough to enforce the rigor and achieve higher quality without taking on the risk of disrupting their customers.

Key-Value Stores and Task Specific Databases
In most Software as a Service or eCommerce applications the database is the central part of the entire system. While relational database management systems (RDBMS) are scalable as we have described in many of our posts, we also know that it can be intimidating for some technology orgs.

What we have seen and expect to see more of is the use of in memory key-value stores such as memcached and redis as a shared memory object cache that is the primary data source for the service or application. Writes work their way back to the relational database asynchronously and reads into the object cache work their way forward either scheduled or on demand. An even more cutting edge trend, although the term has been around since at least 1998, is memory-based architectures where the memory storage is the system of record and there is no relational database or persistent storage device.

A similar trend is the utilization of task specific databases such as Apache CouchDB that is a document-oriented database. CouchDB allows objects, that consist of named fields, to be queried and indexed in a MapReduce fashion and also offers incremental replication with bi-directional conflict resolution. This is obviously not a relational database nor is it an object oriented database but rather it is a query-able and index-able, table oriented reporting engine. The advantages are simpler implementation and administration as well as improved performance for a very specific task.
As services become more specialized and service level agreements demand faster response times, in memory data stores and task specific databases are very likely to be involved in more architectures as a layer between the application servers and the persistent storage or in place of them.

Cloud Computing
While not really a hot new trend this cloud computing is still at such a fledgling stage and evolving so rapidly that it will pay to keep a close eye on it. Some of the more recent introductions by Amazon in their cloud portfolio include auto scaling, load balancing, monitoring, and VPN. Auto Scaling allows you to automatically scale Amazon EC2 instances up or down according to predefined conditions. Instead of running your own software load balancer such as HAProxy, Elastic Load Balancing can perform the distribution of incoming traffic across multiple instances. Monitoring is now available through CloudWatch for EC2 instances but it is still pretty featureless other than load and traffic. For companies interested in extending their existing management capabilities such as security services, firewalls, and intrusion detection systems to include their AWS resources, Amazon now offers virtual private cloud. This enables companies to connect their existing infrastructure to a set of isolated AWS compute resources via a Virtual Private Network (VPN).

These are all Amazon specific functionality but the addition of these features are either already offered by other cloud providers or can be expected shortly. We still are weary to recommend a full production deployment in a single provider’s cloud but often see this as a viable solution for many customers for non-production environments, disaster recovery, and surge or flex capacity.

1 comment

Newsletter – The Future of Relational Databases

For those readers who haven’t subscribed to our newsletter here is a copy of what was sent out earlier this week.  If you would like to receive future emails, about once every other month, sign up here.

We were intrigued by a question asked by Tony Bain of Read Write Web, in his recent article “Is the Relational Database Dead?” Since relational databases management systems (RDBMS) are a significant part of most SaaS or Web 2.0 architectures, we spend a lot of time helping people scale their databases. We also often find ourselves part of discussions about the future of relational databases.

Do Relational Databases Scale?
We believe that RDBMS scale very well especially when not on a single server. This is the major principle behind the three axes of the AKF Scale Cube.

The AKF Database Scale Cube consists of an X, Y and Z axes – each addressing a different approach to scale transactions applied to a database. The lowest left point of the cube (coordinates X=0, Y=0 and Z=0) represents the worst case monolithic database – a case where all data is located in a single location and all accesses go to this single database.

The X Axis of the cube represents spreading read requests across multiple instances of the same data set. All writes get executed on a single database that asynchronously propagates to the read replicas. Caching can be implemented to further reduce database load. The Y Axis of the cube represents a split by function or service. The Z Axis represents ways to split transactions by performing a lookup, a modulus or other indiscriminate function (hash for instance).

Key / Value Stores
We don’t believe in the imminent demise of the RDBMS but we do expect more applications and services to use a relational database as a persistent data store and use key/value stores as part of the transactional system. We have seen and expect to see more of the use of key/value stores such as memcached and redis as a shared memory object cache that is the primary data source for the service or application.  Writes work their way back to the relational database asynchronously and reads into the object cache work their way forward either scheduled or on demand.

This trend is possible obviously because of the software being developed to allow this but also the cheaper hardware including virtualization in clouds. As services become more specialized and service level agreements demand faster response times, in memory data stores are very likely to be involved in more architectures as a layer between the application servers and the relational databases.

Specialized Databases
Another trend in RDBMS is the creation of specialized databases such as Hive that is an open source, peta-byte scale date warehousing framework based on Hadoop that was developed by Facebook. Other databases include Amazon’s distributed, web service SimpleDB, Google’s column-oriented BigTable, and the Apache JSON based CouchDB. Each of these with advantages in particular scenarios. It has been shown in research papers like “One Size Fits All? – Part 2: Benchmarking Results” by Michael Stonebraker and Ugur Cetintemel that major RDBMS vendors can be outperformed by 1 – 2 orders of magnitude by specialized database engines.

In “The End of an Architectural Era” they continue their argument  that the current legacy RDBMS code attempting to be a “one size fits all” excels at nothing. In fact they offer data that the completely written from scracth, H-Store database, built in collaboration with MIT, Brown and Yale Universities, can outperform these legacy RDBMS by two orders of magnitude on standard transactional benchmarks.

The demise of the RDBMS has been prophesied almost since it’s inception in E.F. Codd’s 1970 paper “A Relational Model of Data for Large Shared Data Banks“. One of the most promising replacements given the advent of object oriented programming languages was the object-oriented database, which never became mainstream but has resulted in the inclusion of object-relational features in more popular RDBMS.

Even with the improved performance from specialized databases the most popular relational databases including Oracle, MySQL, and PostgreSQL will continue to be an integral part of SaaS and Web 2.0 architectures. Learning how to scale them will continue to be important for many years to come.