AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Tag » Agile software development

Making Agile Predictable

One of the biggest complaints that we hear from businesses implementing agile processes is that they can’t predict when things will get delivered. Many people believe, incorrectly, that waterfall is a much more predictable development methodology. Let’s review a few statistics that demonstrate the “predictability” of the waterfall methodology.

In a study of over $37 billion (USD) worth of US Defense Department projects concluded that: 46% of the systems so egregiously did not meet the real needs (although they met the specifications) that they were never successfully used, and another 20% required extensive rework (Larman, 1995).

In another study of 6,700 projects it was found that four out of five key factors contributing to project failure were associated with or aggravated by the waterfall method, including inability to deal with changing requirements and problems with late integration.

Waterfall is not a bad or failed methodology, it’s just a tool like any other that has its limitations. We as the users of that tool misused it and then blamed the tool. We falsely believed that we could fix the project scope (specifications), cost (team size), and schedule (delivery date). No methodology allows for that. As shown in the diagram below, waterfall is meant to fix scope and cost. When we also try to constrain the schedule we’re destined to fail.

fig1

Agile when used properly fixes the cost (team size) and the schedule (2 week sprints), allowing the scope to vary. This is where most organizations struggle, attempting to predict delivery of features when the scope of stories is allowed to vary. As a side note, if you think you can fix the scope and the schedule and vary the cost (team size) read Brooke’s 1975 book The Mythical Man-Month.

This is where the magical measurement of velocity comes in to play. The velocity of a team is simply the number of units of work completed during the sprint. The primary purpose of this measurement is to provide the team with feedback on their estimates. As you can see in the graph below it usually takes a few sprints to get into a controlled state where the velocity is predictable and then it usually rises slightly over time as the team becomes more experienced.

fig2

Using velocity we can now predict when features will be delivered. We simply project out the best and worst velocities and we can demonstrate with high certainly a best and worst delivery date for a set of stories that make up a feature.

fig3

Velocity helps us answer two types of questions. The first is the fixed scope question “when will we have X feature?” to which the answer is “between A and B dates”. The second question is the fixed time question “what will be delivered by the June releases?” to which the answer is “all of this, some of that, and none of those.” What we can’t answer is fixed time and fixed scope questions.

fig4

It’s important to remember is that agile is not a software development methodology, but rather a business process. This means that all parts of your organization must buy-in to and participate in agile product development. Your sales team must get out of the mindset of committing to product new features with fixed time and scope when talking to existing or potential customers. When implemented correctly, agile provides faster time to market and higher levels of innovation than waterfall, which brings greater value to your customers. The tradeoff from a sales side is to change behavior from making long-term product commitments as they did in the past (but were more often than not missed anyway)!

By utilizing velocity, keeping the team consistent, and phrasing the questions properly, agile can be a very predictable methodology. The key is understanding the constraints of the methodology and working within them instead of ignoring them and blaming the methodology.


1 comment

AKF Agile Training

It is pretty self-evident that Software as a Service (SaaS) companies have to deliver customer-facing features quickly, at low cost, and high quality. Pay-by-usage models and not having to install software makes switching costs for your customers lower than ever. This makes competition even more fierce, requiring companies to be more nimble than ever in order to stay ahead.

Gartner has indicated that Agile approaches are providing the benefits of fast, accurate delivery of priority application requirements but that the methodology is approaching the trough of disillusionment in its hype cycle. This doesn’t mean that the methodology is flawed, it is simply the normal part of any IT trend that is going mainstream. As Nathan Wilson states, “The early days of any trend are full of promise, followed by a level of hype that the trend is going to be a silver bullet that will solve all problems. Of course no new trend can meet these expectations, and the trough of disillusionment follows when people realize that this is not a silver bullet.”

While no software or product development methodology is a panacea, we believe that for almost all SaaS companies the best approach is an Agile development methodology. We consider Agile as a product development life cycle methodology (PDLC) rather than a software development life cycle methodology (SDLC) because it is a business process that must include business owners on the teams. Note that this is the number one problem we see with companies implementing Agile and often the root cause is a lack of formal training.

Because we believe that any rapidly growing SaaS company must deal with scale issues, we have developed an Agile Training program specifically focused on the critical components of scalability – architecture, organization, and process. If a company misses on any of these three components they are likely to have scale issues at some point.

If you are interested in AKF Agile Training focused on scalability get in contact with us. If you are still undecided about Agile or other methodologies check out this post.


2 comments

How to Choose a Development Methodology

One of our most viewed blog entries is PDLC or SDLC, while we don’t know definitively why we suspect that technology leaders are looking for ways to improve their organization’s performance e.g. better quality, faster development. etc. Additionally, we often get asked the question “what is the best software development methodology?” or “should we change PDLC’s?” The simple answer is that there is no “best” and changing methodologies is not likely to fix organizational or process problems. What I’d like to do in this posts is 1) give you a very brief overview of the different methodologies – consider this a primer, a refresher, or feel free to skip and 2) provide a framework for considering which methodology is right for your organization.

The waterfall model is an often used software development processes that occurs in a sequential set of steps. Progress is seen as flowing steadily downwards (like a waterfall) through the phases such as Idea, Analysis, Design, Development, Testing, Implementation and Maintenance. The waterfall development model originated in the hardware manufacturing arena where after-the-fact changes are prohibitively costly. Since no formal software development methodologies existed at the time, this hardware-oriented model was adapted for software development. There are many variations on the waterfall methodology that changes the phases but all have the similar quality of a sequential set of steps. Some waterfall variations include those from Six Sigma (DMAIC, DMADV).

Agile software development is a group of software development methodologies based on iterative and incremental development. The requirements and ultimate products or services that get delivered evolve through collaboration between self-organizing, cross-functional teams. This methodology promotes adaptive planning, evolutionary development and delivery. The Agile Manifesto was introduced in 2001 and introduced a conceptual framework that promotes foreseen interactions throughout the development cycle. The time boxed iterative approach encourages rapid and flexible response to change. There are many variations of Agile including XP, Scrum, FDD, DSDM, and RUP.

With so many different methodologies available how do you decide which is right for your team? Here are a few questions that will help guide you through the decision.

1) Is the business willing to be involved in the entire product development cycle? This involvement takes the form of dedicated resources (no split roles such as running a P&L by day and being a product manager by night), everyone goes through training, and joint ownership of the product / service (no blaming technology for slow delivery or quality problems).
YES – Consider any of the agile methodologies.
NO – Bypass all forms of agile. All of these require commitment and involvement by the business in order to be successful.

2) What is the size of your development teams? Is the entire development team less than 10 people or have you divided the code into small components / services that teams of less than 10 people own and support?
YES – Consider XP or Scrum flavors of the agile methodology.
NO – Consider FDD and DSDM which are capable of scaling up to 100 developers. If you team is even larger consider RUP. Note that with agile, when the development team gets larger so does the amount of documentation and communication and this tends to make the project less agile.

3) Are your teams located in the same location?
YES – Consider any flavor of agile.
NO – While remote teams can and do follow agile methodologies it is much more difficult. If the business owners are not collocated with the developers I would highly recommend sticking with a waterfall methodology.

4) Are you hiring a lot of developers?
YES – Consider the more popular forms of agile or waterfall to minimize the ramp up time of new developers coming on board. If you really want an agile methodology, consider XP which includes paired programming as a concept, and is a good way to bring new developers up to speed quickly.
NO – Any methodology is fine.

A last important idea is that it isn’t important to follow a pure flavor of any methodology. Purist or zealots of process or technology are dangerous because a single tool in your tool box doesn’t provide the flexibility needed in the real world. Feel free to mix or alter concepts of any methodology to make it fit better in your organization or the service being provided.

There are of course counter examples to every one of these questions, in fact I can probably give the examples from our client list. These questions/answers are not definitive but they should provide a starting point or framework for how you can determine your team’s development methodology.


3 comments

The Agile Executive

In this third installment of our “Agile Organization” series we discuss the qualities and attributes necessary for someone to lead a group of cross functional Agile teams in the development of a web-centric product.  For the purposes of this discussion, the Agile Executive is the person responsible for leading a group of agile teams in the development of a product.

In a world with less focus on functional organizations such as the one we’ve described in our Agile Organization articles, it is imperative that the leadership have a good understanding of all domains from the overall business through product management and finally each of the technical domains.  Clearly such an individual can’t be an expert in every one of these areas, but they should be deep in at least one and broad through all of them.  Ideally this would have been the case in the functional world as well, but alas functional organizations exist to support deep rather than broad domain knowledge.  In the Agile world we need deep in at least area and broad in all areas.

Such a deep yet broad individual could come from any of the functional domains.  The former head of product management may be one such candidate assuming that he or she had good engineering and operations understanding.  The head of engineering and operations may be heads of other agile teams, assuming that they have a high business acumen and good product understanding.  In fact, it should not matter whence the individual comes, but rather whether he or she has the business acumen, product savvy and technical chops to lead teams.

In our view of the world, such an individual will likely have a strong education consisting of an undergraduate STEM (science, technology, engineering or math) degree.  This helps give them the fundamentals necessary to effectively interact with engineers and add value to the engineering process.  They will also have likely attended graduate school in a business focused program such as an MBA with a curriculum that covers finance, accounting, product and strategy.  This background helps them understand the language of business.  The person will hopefully have served for at least a short time in one of the engineering disciplines as an individual contributor to help bridge the communication chasm that can sometimes exist between those who “do” and those who “dream”.  As they progress in their career, they will have taken on roles within product or worked closely with product in not only identification of near term product elements, but the strategic evaluation of product needs longer term as well.

From the perspective of philosophy, the ideal candidates are those who understand that innovation is more closely correlated with collaboration through wide networks than it is to the intelligence of one individual or a small group of people.  This understanding helps drive beneficial cognitive conflict and increased contribution to the process of innovation rather than the closed minded approach of affective conflict associated with small groups of contributors.

In summary, it’s not about whence the person comes but rather “who the person is”.  Leading cross disciplinary teams requires cross disciplinary knowledge.  As we can’t possibly experience enough personally to be effective in all areas, we must broaden ourselves through education and exposure and deepen ourselves through specific past experiences.  Most importantly, for a leader to succeed in such an environment he or she must understand that “it’s not about them” – that success is most highly correlated with teams that contribute and not with just being “wickedly smart”.


Comments Off on The Agile Executive

The Agile Organization Solution

Having discussed why organizations arranged along functional boundaries (e.g. production or engineering, operations, infrastructure, QA, etc) and multi-disciplinary Agile teams are often at odds and counterproductive, we now propose a method of organizing product teams more aligned with the Agile process and Agile methods.

The potential solution is really very simple.  The design goals behind the solution are: maximization of individual capabilities of any given team to quickly and efficiently create solutions and solve problems; creation of ownership of individual products (or portions of products) and the business success metrics associated with that product in order to maximize innovation; maximization of morale through a sense of ownership and delegation of decision making; and reduction in time to market through the elimination of non-value added conflict.

The solution is clear given these goals – eliminate functional organizations and align the organization to architecture specific services.   In the ideal case, each of the senior leaders of these organizations are capable of owning and leading one more complete agile teams.   The number of teams that an executive has is likely going to depend on the size of the company, the size of the engineering and product teams and the number of discrete services in the company’s product architecture.

Here we come to a very important point – it is critically important that the architecture, the process and the organization structure be aligned to reap the benefits of such a change.  If the product architecture continues to be monolithic, nothing is solved.  The solution described above will get you no further than an “agile overlay across functional organizations”.  Each division of agile teams needs to own their own services so that disputes, problems and opportunities can be resolved or capitalized within the team.  This rapid resolution starts to slow down when outside resources are necessary to resolve a problem or capitalize on an opportunity.

We readily admit that this new approach to eliminating functional organizations in favor of agile teams isn’t for everyone.  Some companies don’t have the need to segment their architectures into services as they aren’t experiencing rapid growth.  As such, they shouldn’t pay the price of re-architecting their product.  Some companies deliver to very clearly defined specifications and as a result can’t really benefit from the product discovery aspects inherent to Agile or the questionable “final dates”.  These companies are probably better off continuing to develop in a waterfall fashion.  Many companies won’t have the right skill sets at the top to have former functional executives own product sections.

This last issue sets up a topic for another post.  The question we aim to answer is “What are the qualities and attributes of the Agile Executive?”


Comments Off on The Agile Organization Solution

The Agile Organization

We’ve done a lot of work with organizations attempting to become more Agile by implementing Agile development practices.  One common problem we see time and time again is that the “old school” way of defining organization structure starts to lose its value in an Agile world.  Here I am specifically talking about organization structures developed around functional roles such as development (or engineering), QA, Operations, Infrastructure, etc.

This old method of organizing, which resembles a Y axis split within the AKF scale cube served our industry well for a long time.  And, for many organizations, it can continue to work well.  It particularly works well in organizations that follow waterfall models as the organization structure mimics the flow and passage of work through certain gates.  The structure is also comforting in its familiarity as most long tenured managers and individual contributors have worked within similarly structured organizations their entire professional careers.

But in the Agile world, this organization structure doesn’t add as much value as in the Waterfall world.  In fact, I argue that it’s counter-productive in many ways.  The first and perhaps most benign issue is that the actual structure of the organization does not foster work-flow.  Unlike waterfall development where one group hands off a project to another in phases (development to QA, QA to operations, etc), Agile methods seek to develop and deploy seamlessly.  To be successful the Agile team needs representation from multiple stakeholders within functional groups.  As individuals now spend most of their time in cross functional teams, what value does the functional group offer?  In essence, these functional organizations become the analog to the “home room” in school.

The next problem is the inherent conflict created between the Agile team and the functional organization.  To be truly effective, the team must be empowered to some degree.  What power or responsibility does the functional leader then have?  If he or she isn’t responsible for a specific product, are they to be given some sort of veto power?  Such a notion has meaning in the waterfall world but really runs counter to the time to market and discovery objectives of Agile methods.  The resulting affective conflict simply doesn’t add value to the overall product.  In fact, as research shows, it destroys value.  Some proponents of continuing with functional organizations might indicate that the functional groups allow for more effective management and mentoring of individuals within their domain.  Given how little time managers truly spend on mentoring relative to other tasks, I highly doubt this is the case for most organizations.  Our experience is that the functional groups spend more time arguing over ownership of certain decisions (affective conflict) rather than mentoring, training and evaluating individual contributors.

Perhaps the largest problem – larger than the lack of support for work flow and the creation of conflict – is that implementing agile processes across functional organizations sub-optimizes innovation.  Research indicates that innovation happens most frequently and beneficially within groups of individuals with diverse and non-overlapping experience across a number of domains (functional and experiential diversity) and with non-redundant links to individuals outside of their organization.  By engaging in beneficial debate (cognitive conflict) on approaches to a certain problem or opportunity, the perspective and networks brought by each person widens the potential solution set.  Alas, this is the true unheralded value of agile development teams that properly incorporate multiple disciplines within the team (QA, dev, product, infrastructure).  Each of these individuals not only brings new and valuable expertise in how to develop a product, they also have contacts outside the organization unlikely to be matched by each of their peers.  Innovation quality and frequency therefore increases.  But the inherent conflict in multiple competing organizational affiliations will dampen this innovation.  So not only is there conflict and a lack of workflow, the potential major benefit is removed or at the very least diminished.

Having discussed the problems inherent to functional organizations and agile processes, we’ll next discuss how a company might “organize” around agile to be more effective.


1 comment

Book Review: The Lords of Strategy

OK, this isn’t a book review as much as it is a comparison of how the iterative and rapid “productization” of strategy closely parallels Agile methods of software development.  But first, here’s an overview of a particularly good book – Walter Kiechel’s The Lords of Strategy.   I flew through the book – not because I wanted it to end but because I couldn’t put it down.  It’s an incredible history of the people, organizations and ideas that developed the concept of corporate strategy and it’s full of incredible facts and observations.  Take for instance that the notion of strategy consulting as purveyed by the likes of BCG, Bain and McKinsey is only roughly 30 to 35 years old and that perhaps even more interestingly the notion that a company exists to create shareholder wealth is only about 30 years old.  The book does a great job of explaining not only the history of the ideas behind strategy consulting, sometimes told alongside the biography of their inventors, but how those ideas affected the industry for better and for worse.  Ultimately it describes how these ideas “quickened the pace of capitalism” though the reader is left with figuring out whether we are better or worse off for the change.

What struck me as particularly interesting in this book is the parallels one can draw between how corporate strategy (including the product and services surrounding it) developed and how Agile methods of solution development should work.  The germinating idea behind strategy was the identification of the “experience curve” by the Boston Consulting Group.  This curve identified through trend analysis that the longer and more experienced a company became, the lower its cost of producing a certain good or service.  This notion, though flawed (experience alone isn’t what drives cost), came quickly and was brought to market quickly by BCG.  In rapid fashion, the company built upon this to develop the growth-share matrix as its second “product” offering.  Both of these ideas together led to a grouping of offerings that suggested companies take on debt, reduce costs, differentiate themselves on price and expand shareholder value.  The success of BCG led to McKinsey joining the ranks of strategy consultants, Harvard Business School changing its curriculum (via Michael Porter who had now built his own strategy framework – the famous 5 Forces analysis) and created Bain and Company.

Key here is the evolutionary nature of strategy as a product.  In the very early phases, the offerings were quite frankly wrong.  We know now that the notion that companies differentiate themselves on price alone in every industry is flawed.  But the firms and institutions that supported strategy as a product and intellectual endeavor did not try to offer the absolute best solution – they attempted to bring an appropriate solution to the market and then modify it from there.  In effect, for the time, their solution was the minimum viable product.  Did their approach work?  Billions of dollars of consulting revenue and profits and billions in market value would argue it was an effective approach.

While these companies didn’t realize it at the time, they were in fact practicing agile development.  They didn’t know end user requirements – how could they?  The market wasn’t created yet.  They created a quick offering and iterated upon it, simultaneously changing the market demand and adapting to both the shifting demand and their growing understanding of what strategy needed to become.

Where else might agile methods apply?


2 comments

Agile Architects

If you think agile development methods and architecture are at odds, think again. They can not only coexist but can thrive together to build better products and platforms.

We recently posted what agile is not, where we outlined questions that we often hear about agile development. Another question that is often raised is how to combine the seemingly long-term process of architecture with the short-term nature of agile development. We believe that architecture is not at odds with agile development and that the two can not only coexist but complement each other. To ensure your architecture standards are being integrated with each sprint, resulting in a scalable and available architecture, we rely on the Joint Architecture Development (JAD).

We’ve covered JAD before but as a recap, this is the process by which features are designed in a series of meetings where developers, architects, and tech operations come together to create the design. This multi-function representation early in the development process ensures that individuals are aware of standards, there is buy-in from all concerned parties, and that the design is benefitted by the knowledge that exists in different technical fields.

In the IEEE Software article “Architects as Service Providers,” Roland Faber says that “the architect role is to champion system qualities and stability, while the developer role is to champion the system functions and change.” The agile architect interacts frequently and flexibly with the developers, building a trust relationship with them.

The JAD is ideal for flexible interaction that can happen in short bursts of effort that correspond to sprints. The agile architect must understand that, because of the nature of agile development, architecture must be dynamic and not static. Architects must rely on personal interaction with developers not documentation to understand the requirements.

Faber continues in his article describing two phases of the architecture process as preparation and support. During preparation the architect engages in processes such as prepare rules, frameworks, and structures. During support the architect helps resolve conflicts, engages in firefighting, and stimulates architecture communication. He makes a point that if the developers don’t believe the architects will provide support they won’t tell them when they are breaking the rules.


1 comment

5 Things Agile is NOT

Agile Processes can help fix several - but not all - issues. Here are the top 5 misconceptions about the process that we see in our practice.

It seems that everyone is moving to an Agile approach in their product (or software) development lifecycle.  Some move with great success, some with great fanfare and for some it’s one of the last moves their company and engineering organizations make before failing miserably and shutting doors permanently.  As often as not, companies just move because they believe it will cure all of their problems.  As we’ve written before, no new PDLC will cure all of your problems.  Agile may in fact be best for you, but there are always tradeoffs.

We’ve compiled a top 5 misconceptions about Agile from our experience working with companies to solve problems.  Be careful of these pitfalls as they can cause you to fail miserably in your efforts.

1)      Agile is NOT a reason to NOT manage engineers or programmers

Engineering organizations measure themselves.  In fact, many Agile methods include a number of metrics such as burn down charts to measure progress and velocity for measuring the rate at which teams deliver business value.  As with any other engineering organization, you should seek to find meaningful metrics to understand and improve developer and organizational quality and productivity.  Don’t let your team or your managers tell you not to do your job!

2)      Agile is NOT a reason to have engineering alone make product decisions

You still have to run your business which means nesting the product vision to the vision of the company.   More than likely you are paying business or product people to define what it is you need to do to fight and win in your market.  Someone still has to set the broad strategic vision to which products will be developed.  Engineers can and should contribute to this vision, and that’s where Agile methods can help.

3)      Agile alone is NOT a cure for all of your product delivery problems

As we’ve blogged before, there simply are no silver bullets.  With the right management and oversight, you can make any development lifecycle work.  There are just cases where Agile methods work better.   But don’t look to have a PDLC fix your business, people or management issues.

4)      Agile is NOT an excuse to NOT put in the appropriate processes

There is nothing in the Agile manifesto that states that process is evil.  In fact, it recognizes processes are important by stating that there is value within them.  Agile simply believes that individuals and interactions are more important – a concept with which we completely agree.   Don’t argue that all process is evil and cite Agile as the proof as so many organizations seem to do.  Certain processes (e.g. code reviews) are necessary for hyper growth environments to be successful.

5)      Agile is NOT an excuse not to create SOME documentation

Another often misinterpreted point is that Agile eliminates documentation.  Not only is this not true, it is ridiculous.  The Agile manifesto again recognizes the need for documentation and simply prefers working software over COMPREHENSIVE documentation.  Go ahead and try to figure out how to use or troubleshoot something for the very first time without documentation and see how easy it is…  Programming before designing coupled with not creating  useful documentation makes one a hack instead of an engineer.  Document – just don’t go overboard in doing it.


1 comment

Revisiting the 1:10:100 Rule

Has the 1:10:100 rule changed? We think so, though the principles still hold true.

If you have any gray in your hair, you likely remember the 1:10:100 rule.  Put simply, the rule indicates that the cost of defect identification goes up exponentially with each phase of development.  It costs a factor of 1 in requirements, 10 in development, 100 in QA and 1000 in production. The increasing cost recognizes the need to go back through various phases, the lost opportunity associated with other work, the amount of people and systems involved in identifying the problem, and end user (or customer) impact in a production environment. In a 2002 study by the National Institute of Standards and Technology the estimated cost of software bugs was $59.5 billion annually, half the cost borne by the users and the other by the developers.

While there is an argument to be made that Agile development methods reduce this exponential rise in cost, Agile alone simply can’t break the fact that the later you find defects, the more it costs you or your customers.   But I also believe it’s our jobs as managers and leaders to continue to reduce this cost between phases – especially in production environments.  If the impact in the production environment is partially a function of 1) the duration of impact, 2) the degree of functionality impacted, and 3) the number of customers impacted, then reducing any of these should reduce the cost of defect identification in production.  What can we do besides considering Agile methods?

There are at least three approaches that significantly reduce the cost of finding production problems.  These are “swimlaning”, having the ability to roll back code in XaaS environments (our term for anything as a service), and real time monitoring of  business metrics.  These approaches affect the number of customers impacted and the duration of the impact respectively.

Swim Lanes

We think we might have coined the term “swimlaning” as it applies to technology architectures.  Swimlaning, as we’ve written about on this blog as well as in the book, is the extreme application of the “shard” or “pod” concept to create strict fault isolation within architectures.  Each service or customer segment gets its own dedicated set of systems from the point of accepting a request (usually the webserver) to the data storage subsystem tier that contains the data necessary to fulfill that request (a database, file system or other storage system).  No synchronous communication is allowed across the “swimlanes” that exist between these fault isolation zones.  If you swimlane by the Z axis of scale (customers) you can perform phased rollouts to subsets of your customers and minimize the percentage of your customer base that a rollout impacts.  An issue that would otherwise impact 100% of your customers now impacts 1%, 5% or whatever the smallest customer swimlane is.  If swimlaned by functionality, you only lose that functionality and the rest of your site remains functioning.  The 1000x impact might now be 1/10th or 1/100th the previous cost.  Obviously you can’t have less cost than the previous phase, as you still need to perform new work, but the cost must go down.

Rollback

Ensuring that you can always roll back recently released code reduces the duration of customer impact.  While there is absolutely an upfront cost in developing code and schemas to be backwards compatible, you should consider it an insurance policy to help ensure that you never kill your customers.  If asked, most customers will probably tell you they expect that you can always roll back from major issues.   One thing is for certain – if you lose customers you have INCREASED rather than decreased the cost of production issue identification.  If you can isolate issues to minutes or fractions of an hour in many cases it becomes nearly imperceptible.

Monitoring Business Metrics

Monitoring the CPU, memory, and disk space on servers is important but ensuring that you understand how the system is performing from the customer’s perspective is crucial. It’s not uncommon to have a system respond normally to an internal health check but be unresponsive to customers. Network issues can often provide this type of failure. The way to ensure you catch these and other failures quickly is to monitor a business metric such as logins/sec or orders/min. Comparing these week-over-week e.g. Monday at 3pm compared to last Monday at 3pm, will allow you to spot issues quickly and rollback or fix the problem, reducing the impact to customers.


Comments Off on Revisiting the 1:10:100 Rule