We wrote a post back in July 2008 about how in order to get better you must practice. Since that time we’ve seen a lot of interest in this concept of how much must you practice in order to master a skill.  This interest is primarily due to Malcolm Gladwell’s book Outliers in which he showed a number of examples of why you must have about 10 years or 10,000 hours of practice in order to achieve mastery. I’ve come across many variations of this recently and wanted to revisit the topic from a non-programming perspective to show that this is applicable to every aspect of your life and career. If you want to become a better parent, teacher, runner, programmer, leader, or technologist you must practice. If you want to master those skills you must practice them a lot.

One of the more interesting variations is from the 1976 book by William Zinsser, On Writing Well. If you are interested in writing I highly recommend this book and for everyone else you can draw inspiration from his devotion to the craft of writing. In the first chapter William Zinsser talks about an interview he participated in discussing writing as a vocation along with a certain Dr. Brock who was a surgeon that had recently taken up writing. Dr. Brock started by expressing how “The words just flowed.” And when asked what he did when the writing wasn’t going well he said he’d just stop writing. Zinsser countered that “writing is a craft, not an art, and that the man who runs away from his craft because he lacks inspiration is fooling himself.” He continues with “If your job is to write every day, you learn to do it like any other job.” Zinsser concludes the section with a thought that if writing for a surgeon was so easy he’d consider taking up surgery on the side. This was a bit tongue-in-cheek with the point being to get better you must persevere and do so with a critical eye for how to constantly improve.

An interesting theory on how to produce sustained and desirable change is the Boyatzis’ Intentional Change Theory. This is five step process that enables individuals to achieve change and maintain it. If you have ever tried to stay on a diet or start an exercise regime and have found it difficult, you should be able to relate to this. One of the keys to this process is practicing the new behavior in order to build and strengthen neural pathways.  This eventually leads to mastery of the skill and sustained change.

An MIT Computer Science grad student has an interesting list on his blog of posts that cover this subject of dedicated practice to master a skill.  In MJ fashion, he calls this list “Thoughts on living a remarkable life”.  Two of his most interesting from that list are book reviews On the Value of Hard Focus about Murakami’s book on distance running and The Steve Martin Method about the comedian’s life.

Another story comes from Dave Ramsey’s book More Than Enough. Dave tells the story/parable of a professional golfer being approached by a fan saying “I’d do anything to hit like you” and the golfer says “No, you wouldn’t”, to which the fan replies “Oh, yeah, I really would.”  The golfer goes on to explain his secret “Get up every morning and hit 500 golf balls. Hit them until your hands are so blistered they bleed. The next morning, tape over the blisters and do it again.”

A final example comes from Jason at 37Signals on making money.  He postulates that as an entrepreneur you need to practice early and often making money; learning the skills of negotiating, pricing, and selling.  This is why he recommends entrepreneurs not take outside money, in order that they learn to make money quickly. Tim Ferris would probably agree that negotiating can be practiced and Seth Godin would certainly agree that selling takes practice.

So now with all the overwhelming evidence that we need to focus and practice to master a skill, what will you do today to become a master at something?

The partial meltdown of TMI-2 at Three Mile Island in 1979 is one of the best known crisis situations within the US and was the source of several books, and at least one movie.  It also generated two theories relevant to crisis management.

Charles Perrow’s Normal Accident Theory (NAT), described in his book Normal Accidents, states that the complexity inherent to tightly coupled technology systems makes accidents inevitable.  Perrow’s hypothesis is that the tight coupling causes interactions to escalate rapidly and without obstruction.  “Normal” is a nod to the inevitability of such accidents.

Todd LaPorte, who founded the Berkeley school of High Reliability Theory, believes that there are organizational strategies to achieve high reliability even in the face of such tight coupling.  The two theories have been debated for quite some time.  While the authors don’t completely agree as to how they can coexist (LaPorte believes that they are complimentary and Perrow believes that they are useful for the purposes of comparison), we believe there is something to be gained from them.

One paradox from these debates becomes intuitively obvious to our pursuit of high availability and highly scalable systems:  The better we are at building systems that avoid problems and crises, the less practice we have in solving problems and crises.  As the practice of resolving failures are critical to our learning, we become more and more inept at rapidly resolving these failures as their frequency decreases.  Therefore, as we get better at building fault tolerant and scalable systems, we get worse at resolving crisis situations that are almost certain to happen at some point.

Weick and Sutcliffe have a solution to this paradox that we paraphrase as “organizational mindfulness”.  They identify 5 practices for developing this mindfulness:

1)      Preoccupation with failure.  This practice is all about monitoring IT systems and reporting errors in a timely fashion.  Success, they argue, narrows perceptions and breeds overconfidence.   To combat the resulting complacency, organizations need complete transparency into system faults and failures.  Reports should be widely distributed and discussed frequently such as in our oft recommended “operations review” process outlined within the Art of Scalability.

2)      Reluctance to simplify interpretations.  Take nothing for granted and seek input from diverse sources.  Don’t try to box failures into expected behavior and act with a healthy bit of paranoia.

3)      Sensitivity to operations.  Look at detail data at the minute level, such as we’ve suggested in our posts on monitoring.  Include the usage of real time data and make ongoing assessments and continual updates of this data.  We think our book and our post on monitoring strategies have some good suggestions on this topic.

4)      Commitment to resilience.  Build excess capability by rotating positions and training your people in new skills.  Former employees of eBay operations can attest that DBAs, SAs and Network Engineers used to be rotated through the operations center to do just this.  Furthermore, once fixes are made the organization should be quickly returned to a sense of preparedness for the next situation.

5)      Deference to expertise.  During crisis events, shift the leadership role to the person possessing the greatest expertise to deal with the problem.  Our book also suggests creating a competency around crisis management such as a “technical duty officer” in the operations center.

We would add that every operations team should use every failure as a learning opportunity, especially in those environments in which failures are infrequent.  A good way to do this is to leverage the post mortem process.

We’ve had a couple posts on this topic of engineer vs craftsman vs technician but I found myself in the past couple of days discussing this topic in two different settings and thought it would be fun to revisit on the blog. I started the conversation with a post that quoted Tom DeMarco concluding that “software engineering is an idea whose time has come and gone.” Also quoted was Jeff Atwood stating that “If you want to move your project forward, the only reliable way to do that is to cultivate a deep sense of software craftsmanship and professionalism around it.” Marty picked up the conversation in another post stating that:

All of this goes to say that software developers rarely “engineer” anything – at least not anymore and not in the sense defined by other engineering disciplines. Most software developers are closer to the technicians within other engineering disciplines; they have a knowledge that certain approaches work but do not necessarily understand the “physics” behind the approach. In fact, such “physics” (or the equivalent) rarely exist. Many no longer even understand how compilers and interpreters do their jobs (or care to do so).

I think it is possible that we are seeing the evolution of our discipline as it struggles to determine what its final form will take. Computer science, information technology, software engineering, and other related disciplines are all relatively new fields of study. With a new discipline it should be expected that definitions and themes will need to be stretched or reconsidered.

Software engineers, similar to other engineering disciplines who are taught more “true” laws such as Newton’s or Faraday’s, also undergo something of an apprenticeship once their degree is conferred. Mechanical and electrical engineers work beside senior engineers who help them transition from the theoretical to the practical. Software engineers are often apprenticed in the same manner by more senior engineers. If the practical implementation of one discipline is considered engineering because it is based upon laws and principles, I would argue that the principles of architecture for scalability are of a similar nature. This in fact I think is a strong differentiator between technicians and engineers within the software development discipline. A technician can write code, setup a database, or administer a server. An engineer can architect a database or system or pool of servers such that it can scale. We’ve written several posts about the principles and patterns of scalability and a large part of our book is dedicated to these principles. Are these sufficiently established to be called a principle as defined in Wikipedia?

A principle is one of several things: (a) a descriptive comprehensive and fundamental law, doctrine, or assumption; (b) a normative rule or code of conduct, and (c) a law or fact of nature underlying the working of an artificial device

I still like the idea of software developers as craftsmen and -women but as I concluded in the other post, that discussion for me is as much about organizational size and control as it is anything else. The technician vs engineer discussion I think is best held in the light of are they or are they not applying laws or principles. As the American Engineers’ Council for Professional Development defines engineering “The creative application of scientific principles to design or develop…” Have we as a discipline, especially in terms of scalability, advanced enough to call what we use “principles”? Let us know what you think.

Here are a baker’s dozen of items that we feel are Best Practices for Scalability:

1. Asynchronous - Use asynchronous communication when possible. Synchronous calls tie the availability of the two services together. If one has a failure or is slow the other one is affected.
2. Swim Lanes – Create fault isolated “swim lanes” of hardware by customer segmentation. This prevents problems with one customer from causing issues across all customers. This also helps with diagnosis of issues and code roll outs.
3. Cache - Make use of cache at multiple layers including object caches in front of databases (such as memcached), page or item caches for content (such as squid) and edge caches (such as Akamai).
4. Monitoring - Understand your application’s performance from a customer’s perspective. Monitor outside of your network and have tests that simulate a real user’s experience. Also monitor the internal working of the application in terms of query and transaction execution count and timing.
5. Replication - Replicate databases for recovery as well as to off load reads to multiple instances.
6. Sharding - Split the application and databases by service and / or by customer using a modulus. While this requires slightly more complicated logic in the application it allows for massive scaling.
7. Use Few RDBMS Features – Use the OLTP database as a persistent storage device as much as possible. The more you rely on the features offered in most RDBMS for your transactions, the greater load you are putting on the hardest item in your system to scale. Remove all business logic from the database such as stored procedures and move it into the application. When significant scaling is required join in the application and not through the SQL.
8. Slow Roll – Roll out new code versions slowly, to a small subset of your servers without bringing the entire site down. This requires that all code be backwards compatible because you will have two versions of code running in production during the roll out. This method allows you to find problems that your quality and L&P testing missed while having minimal impact on customers.
9. Load & Performance Testing – Test the performance of the application version before it goes into production. This will not catch all the issues, which is why you need the ability to rollback, but it is very worthwhile.
10. Capacity Planning / Scalability Summits – Know how much capacity you have on all tiers and services in your system. Use Scalability Summits to plan for the increased capacity demands.
11. Rollback – Always have the ability to rollback a code release.
12. Root Cause Analysis - Ensure you have a learning culture that is evident by utilizing Root Cause Analysis to find and fix the real cause of issues.
13. Quality From The Beginning – Quality can’t be tested into a product, it must be designed in from the beginning.

Mike Fisher recently blogged about all the recent activity decrying the death of software engineering in his post “Engineering or Craftsmanship”.  The two terms should never have been stuck together in the first place.  Compared to the “true” engineering disciplines, the construct is as ridiculous as the term “sanitation engineer”.

Most other engineering disciplines require school trained engineers with deep technical and scientific knowledge to accomplish their associated tasks.  There probably aren’t many ground breaking airplanes designed by people who do not understand lift and drag, few ground breaking electronic devices designed by people who don’t understand the principles of electromotive force and few skyscrapers designed by people who do not understand the principles of statics and dynamics.  This isn’t to say that such things haven’t happened (e.g. the bicycle manufacturers turned airplane pioneers named the Wright brothers), but rather that these exceptions are examples of contributions by geniuses and savants rather than the norm.

The development of software is simply different than the work performed within true engineering disciplines.  With just a little bit of time and very little training or deep knowledge, one can create a website or product that is disruptive within any given market segment.  You don’t need to learn a great deal about science or technology to begin being successful and you need not be a genius.  The barrier to entry to develop a business changing service on the internet simply isn’t the same as the knowledge necessary to send a man to the moon.  Software, as it turns out, simply isn’t “rocket science”.   To develop it we don’t need a great deal of scientific or technical experience and it’s really not the application of a “real science” (one with immutable laws etc) as is say electrical engineering.

Sure, there are some people who as a result of training are better than other people and there is still incredible value in going to school to learn the classic components of computer science such as asymptotic analysis.  Experience increases one’s ability to create efficient programs that reduce the cost of operations, increase scalability and decrease the cost of development.  But consider this, many people with classical engineering backgrounds simply walk into software development jobs and are incredibly successful.  Seldom is it the case that a software engineer without an appropriate undergraduate engineering background will walk into a chemical, electrical or mechanical engineering position and start kicking ass.

The “laws” that developers refer to (Brooks Law, Moore’s Law, etc) aren’t really laws as much as they are observations of things that have held true for some time.  It’s entirely possible that at some point Moore’s Law won’t even be a “law” anymore.  They just aren’t the same as Faraday’s Law or Bernoulli’s Principle.  It’s a heck of a lot easier to understand an observation than it is to understand, “prove” or derive the equations within the other engineering disciplines.  Reading a Wikipedia page and applying the knowledge to your work is not as difficult as spending months learning calculus so that one can use differential equations.

All of this goes to say that software developers rarely “engineer” anything – at least not anymore and not in the sense defined by other engineering disciplines.  Most software developers are closer to the technicians within other engineering disciplines; they have a knowledge that certain approaches work but do not necessarily understand the “physics” behind the approach.  In fact, such “physics” (or the equivalent) rarely exist.  Many no longer even understand how compilers and interpreters do their jobs (or care to do so).

None of this goes to say that we should give up managing our people or projects.  Many articles decry the end of management in software, claiming that it just runs up costs.  I doubt this is the case as the articles I have read do not indicate the cost of developing software without attempting to manage its quality or cost.  Rather they point to the failures of past measurement and containment strategies as a reason to get rid of them.  To me, it’s a reason to refine them and get better.  Agile methods may be a better way to develop software over time, or it may be the next coming of the “iterative or cyclic method of software development”.  Either way, we owe it to ourselves to run the scientific experiment appropriately and measure it against previous models to determine if there are true benefits in our goal of maximizing shareholder value.

Having gone through a computer science program at a school that required many engineering courses such as mechanical engineering, fluid dynamics, and electrical engineering as part of the core curriculum, I have a good appreciation of the differences between classic engineering work and computer science. One of the other AKF partners attending this same program along with me and we often debate whether our field should be considered an engineering discipline or not.

Jeff Atwood posted recently about how floored he was reading Tom DeMarco’s article in IEEE Software, where Tom stated that he has come to the conclusion that “software engineering is an idea that whose time has come and gone.” Tom DeMarco is one of the leading contributors on software engineering practices and has written such books as Controlling Software Projects: Management, Measurement, and Estimation, in which it’s first line is the famously quoted “You can’t control what you can’t measure.” Tom has come to the conclusion that:

For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business….Software development is and always will be somewhat experimental.

Jeff concludes his post with this statement, ”…control is ultimately illusory on software development projects. If you want to move your project forward, the only reliable way to do that is to cultivate a deep sense of software craftsmanship and professionalism around it.”

All this reminded me of a post that Jeffrey Zeldman made about design management in which he states:

The trick to great projects, I have found, is (a.) landing clients with whom you are sympatico, and who understand language, time, and money the same way you do, and (b.) assembling teams you don’t have to manage, because everyone instinctively knows what to do.

There seems to be a theme among these thought leaders that you cannot manage your way into building great software but rather you must hone software much like a brew-master does for a micro-brew or a furniture-maker does for a piece of furniture. I suspect the real driver behind this notion of software craftsmanship is that it if you don’t want to have to actively manage projects and people you need to be highly selective in who joins the team and limit the size of the team. You must have management and process in larger organizations, no matter how professional and disciplined the team. There is likely some ratio of professionalism and size of team where if you fall below this your projects breakdown without additional process and active management. As in the graph below, if all your engineers are apprentices or journeymen and not master craftsmen they would be lower on the craftsmanship axis and you could have fewer of them on your team before you required some increased process or control.

Continuing with the microbrewery example, you cannot provide the volume and consistency of product for the entire beer drinking population of the US with four brewers in a 1,000 sq ft shop. You need thousands of people with management and process. The same goes for large software projects, you eventually cannot develop and support the application with a small team.

But wait you say, how about large open source projects? Let’s take Wikipedia, perhaps the largest open project that exists. Jimbo Wales, the co-founder of Wikipedia states “…it turns out over 50% of all the edits are done by just .7% of the users – 524 people. And in fact the most active 2%, which is 1400 people, have done 73.4% of all the edits.” Considering there are almost 3 million English articles in Wikipedia, this means each team working on a article is very small, possibly a single person.

Speaking of Wikipedia, one of those 524 people defines software engineer as “a person who applies the principles of software engineering to the design, development, testing, and evaluation of the software and systems that make computers or anything containing software, such as chips, work.” To me this is too formulaic and doesn’t describe accurately the application of style, aesthetics, and pride in one’s work. I for one like the notion that software development is as much craftsmanship as it is engineering and if that acknowledgement requires us to give up the coveted title of engineer so be it. But I can’t let this desire to be seen as a craftsman obscure the concept that the technology organization has to support the business as best as possible. If the business requires 750 engineers then there is no amount of proper selection or craftsmanship that is going to replace control, management, measurement, and process.

Perhaps not much of a prophecy but I predict the continuing divergence among software development organizations and professionals. Some will insist that the only way to make quality software is in small teams that do not require management nor measurement while others will fall squarely in the camp of control, insisting that any sizable project requires too many developers to not also require measurements. The reality is yes to both, microbrews are great but consumers still demand that a Michelob purchased in Wichita taste the same as it does in San Jose. Our technological world will be changed dramatically by a small team of developers but at the same time we will continue to run software on our desktops created by armies of engineers.

What side of the argument do you side with engineer or craftsman?

We often talk to clients about the differences between tuning an application or platform and scaling it.  Often the clients intuitively see them as different things, but oddly still refer to them similarly as in “We scaled the database last year by tuning the top SQL queries and getting 20% better performance”.

Tuning is something that you absolutely must do, but we stress that it’s more of an operational endeavor than an architectural endeavor.  In the ideal world, as a typical part of your operation, you would focus some time monthly or quarterly on identifying the slowest portions of your product or the most costly in terms of hardware, engineering time or clock cycles.  These would then be prioritized based on cost to fix and expected return with changes being made to them as engineering allocations allow.  This process is just good housecleaning; throwing out old clothes, and books or getting rid of the motorcycle and lawn mower that no longer work to make room in a garage bay for other “stuff”.

And that’s just what you’re doing with tuning – throwing out old stuff to make room for new stuff.  Just as with the house analogy, you aren’t really making your house “bigger”, you are making the current house capable of holding different and ideally more valuable “stuff”.   If the occupants of your house aren’t growing (i.e. your business isn’t growing quickly), as with a new addition to a family or several new roommates, then this disposition of old less valuable stuff (old code that is seldom executed but costly, often executed code that is very costly) for newer more valuable stuff (new functionality or more efficient code) may be all you ever need to do.

Scaling, however, is an architectural effort that implies significant modifications to your current structure.  In the house analogy, it might mean adding a second floor or expanding the house.  Structurally the house is different and is capable of holding more “stuff” because the total floor space has increased, as compared to the tuning example where you just freed up existing floor space.  Scaling involves architectural effort in redesign of the underlying fundamentals of your system such as adding replicated read databases or splitting databases along data boundaries while tuning is just “spring cleaning” of your platform.

If you are experiencing rapid or hyper growth, you need to both tune your system for the sake of efficiency and performance and scale it for future growth.  Scaling is what allows you to achieve orders of magnitude of improvement (10x, 100x, etc), while tuning should afford linear growth and more efficient operation over time.

You probably have heard of continuous integration that is the practice of checking code into the source code repository early and often.  The goal of which is to ease the often very painful process of integrating multiple developer’s code after weeks of independent work. If you have never had the pleasure of experiencing this pain, let me give you another example that we have experienced recently. In the process of writing The Art of Scalability, we have seven editors including an acquisition editor, a development editor, and five technical editors who all provide feedback on each chapter. Our job is to take all of this separate input and merge it back into a single document, which at times can be challenging when editors have different opinions for the direction of certain parts of the chapter. The upside of this process is that it does make the manuscript much better for having gone through the process. Luckily software engineering has developed the process of continuous integration designed to reduce wasted engineering effort. In order to make this process the most effective the automation of builds and smoke tests are highly recommended. For more information on continuous integration there are a lot of resources such as books and articles.

The topic of this post is taking continuous integration to an extreme and performing continuous deployment. And it is exactly what it sounds like, all code that is written for an application is immediately deployed into production. If you haven’t heard of this before you’re first thought is probably that this is the ultimate in Cowboy Coding but it is in use by some household technology names like Flickr and IMVU. If you don’t believe this check out code.flickr.com and look at the bottom of the page, last time I checked it said:

Flickr was last deployed 20 hours ago, including 1 change by 1 person.

In the last week there were 34 deploys of 385 changes by 17 people.

Eric Ries, co-founder and former CTO of IMVU, is a huge proponent of continuous deployment as a method of improving software quality due to the  discipline, automation, and rigorous standards that are required in order to accomplish continuous deployment. Other folks at IMVU also seem to be fans of the continuous deployment methodology as well from the post by Timothy Fitz. Eric suggest a 5 step approach for moving to a continuous deployment environment.

1. Continuous Integration – Obviously before moving beyond integration into full deployment, this is a prerequisite that must be in place.
2. Source Code Commit Checks – This feature which is available in almost all modern source code control systems,  allows the process of checking in code to halt if one of the tests fail.
3. Simple Deployment Script – Deployment must be automated and have the ability to rollback, which we wholeheartedly agree with here and here.
4. Real-time altering – Bugs will slip through so you must be monitoring for problems and have the processes in place to react quickly
5. Root Cause Analysis – Eric recommends the Five Why’s approach to find root cause, whatever the approach, finding and fixing the root cause of problems is critical to stop repeating them.

Admittedly, this concept of developers pushing code straight to production scares me quite a bit, since I’ve seen the types of horrific bugs that can make their way into pre-production environments. However, I think Eric and the other continuous deployment proponents are onto something that perhaps the reason so many bugs are found by weeks of testing is a self-fulfilling prophecy. If engineers know their code is moving straight into production upon check in they might be a lot more vigilant about their code, I know I would be. How about you, what do you think about this development model?

What questions do each of your system monitors answer? You probably think they answer questions such as “Is there a problem?” and if so “Where is the problem?” Most likely this is not the case and instead of telling you “Is there a problem?” it really only tells you “Where” or “What” the problem might be. Before we continue this, first a quick detour to discuss metrics, which while different than monitoring are very similar in many ways.

Eric Ries, co-founder and CTO of IMVU, posted an article about the difference between vanity metrics and actionable metrics.  The entire article and accompaning video are worth a read and listen but the take away is that most people are using and looking for metrics that are great soundbites but do not offer any definable actions.  One example is the total number of hits to a website.  Eric ask the questions “Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next?” This makes total sense to me as we  often see teams misusing monitoring in an attempt to determine what actions to take with their systems.

Back to our discussion of what question your monitoring is attempting to answer. We think there are five evolutionary questions that monitoring should answer:

1. Is there a problem?
2. Where is the problem?
3. What is the problem?
4. Why is there a problem?
5. Will there be a problem?

Where most people fail is using a monitoring tool that is designed to answer “Where” or “What” and try to use it to answer “Is”. For example, if you are monitoring all of your servers vitals such as CPU, memory, and I/O what is the appropriate action for your team to take when the CPU utilization goes to 100%? The reason that might be a tough question is that you are missing the vital piece of information “Is this affecting my customers?”.  The “Is there a problem” is intended to be a proxy for customer impact in order to help determine the degree and speed of escalation of the issue.

If you have monitoring services in place now it is worthwhile to determine what question each one answers. If you are missing a monitor for a particular question, the time to remedy it is before you need that question answered.

Want to think big, abstract thoughts? Try sitting in a building that has high ceilings. Have to focus on detailed work? Maybe you should be in an office with lower ceilings. That is according to Joan Meyers-Levy, a professor of marketing at the University of Minnesota. Her concepts are part of an article How Room Design Affect Your Work and Mood in Scientific American. “Ceiling height affects the way you process information,” Meyers-Levy says. “You’re focusing on the specific details in the lower-ceiling condition.”

Another interesting concept covered in this article is the affect that natural surroundings have on people, especially children. The article cites a study by environmental psychologist Nancy Wells that found that children who moved with their family to a home with more views of natural settings, such as a garden, field or forest, made the most gains on a attention test. Another study from the University of Illinois found similar results in children with attention deficit disorder (ADD).

Tim Ferriss, author of The Four Hour Work Week has an indoor garden that his office overlooks.  He uses this space for relaxing, inspiration, and taping episodes of Random with Kevin Rose, founder of Digg. Perhaps the next time you are struggling with a task, whether that’s balancing your checkbook, coding your latest feature, or coming up with a brand new business idea, stop and consider the environment that you are working in and how it might be affecting you.