We recently posted what agile is not, where we outlined questions that we often hear about agile development. Another question that is often raised is how to combine the seemingly long-term process of architecture with the short-term nature of agile development. We believe that architecture is not at odds with agile development and that the two can not only coexist but complement each other. To ensure your architecture standards are being integrated with each sprint, resulting in a scalable and available architecture, we rely on the Joint Architecture Development (JAD).

We’ve covered JAD before but as a recap, this is the process by which features are designed in a series of meetings where developers, architects, and tech operations come together to create the design. This multi-function representation early in the development process ensures that individuals are aware of standards, there is buy-in from all concerned parties, and that the design is benefitted by the knowledge that exists in different technical fields.

The JAD is ideal for flexible interaction that can happen in short bursts of effort that correspond to sprints. The agile architect must understand that, because of the nature of agile development, architecture must be dynamic and not static. Architects must rely on personal interaction with developers not documentation to understand the requirements.

Faber continues in his article describing two phases of the architecture process as preparation and support. During preparation the architect engages in processes such as prepare rules, frameworks, and structures. During support the architect helps resolve conflicts, engages in firefighting, and stimulates architecture communication. He makes a point that if the developers don’t believe the architects will provide support they won’t tell them when they are breaking the rules.

It seems like everyone’s architecture diagrams has a big “Enterprise Service Bus” drawn down the middle of it these days.   It’s as if you get a prize for just including the notion of an ESB and indicating that everything is loosely coupled.   The problem is that an unmanaged, misused or overused service bus can become one big Enterprise Cesspool.

Don’t get me wrong – ESBs are a great addition for many companies if used properly.   If you are publishing a message for which there are many subscribers and if you can lose some messages without causing problems then by all means implement an ESB.  Centralized logging of messages, trickle loading of data warehouses and even updates to prices or inventory in search nodes are all examples of ESB activities.  Just ensure that your application is either resilient to or can recover from intermittent message loss.

Don’t fall into the trap of thinking that message buses are the only solution for communications.  Asynchronous communication is preferable where possible, but there are many options beyond message buses.  You can implement asynchronous point to point communication methods (we often differentiate these from bus architectures and call them queue architectures).  You can also implement async-sync point to point methods that allow a busy application to “fire and forget” into a queue while another application works through the queue and synchronously communicates to another consumer process or to multiple data stores where necessary.  And sometimes, when it absolutely positively has to be there overnight, it is best just to communicate synchronously.  Be a wise craftsman/engineer and choose the right tool for the job.  After all, it’s rare that a carpenter chooses to cut a board with a hammer.

Service buses can easily also become cesspools.  Someone needs to ensure that the bus is built with an appropriate number of independent channels so that communications paths don’t become congested.  Treat buses as you would any other piece of infrastructure and scale them horizontally with multiple channels, routing services, etc.  Monitor them aggressively to include message latency, age of the oldest message, number of undelivered messages, etc.

Okay we really don’t hate stored procedures and have actually used them  extensively on past projects but we so often tell people to remove them  that we’ve been accused of hating them. Here’s the deal, as we mentioned  in a past post, two scalability best practices are 1) put as little  business logic in the database layer as possible and 2) use as few  features of the RDBMS as possible. Both of these are violated by the use  of stored procedures, if they contain any business logic. If the stored  procedures are simply instantiations of SQL data statements then you’re  probably okay.

Expanding on the two reasons for avoiding stored procedures mentioned  above, we’ll first discuss why business logic doesn’t belong in the  database layer. The biggest reason for this is that the database server  is likely the most expensive server in your system and the most  difficult to scale. Adding more computational processing on it makes it  the most expensive layer to perform these and requires you to upgrade  the hardware or split the database sooner than necessary. It is much  cheaper from a cost per computation perspective to have this business  logic processed on a much cheaper application server.

Avoiding the use of RDBMS features is important when it comes to  remaining vendor agnostic and getting the best possible price during  negotiations. Using features specific to certain database vendors locks  you in to that vendor. While you may never want to switch database  vendors it does reduce your negotiating strength. When you are locked in  to a vendor your best alternative to a negotiated agreement, BATNA, is  not good and their sales staff knows this.

While we don’t hate stored procedures we do seem them as an impediment  to scaling. If you already have them in place you may be hesitant to  spend the money moving them into the application logic. One method of  doing so at minimal incremental cost is to start today with the policy  that if any engineer touches a stored procedure they must move it. This  way the cost is small, because you’re already touching the code, and  spread out because it’s not done all at once.

We often consult with clients who have storage costs overruns.  These aren’t just the actual cost of storage, which as we all know have been dropping fairly drastically.  Rather, as we outline in our book, it’s all the other costs of storage:  space for the storage, power for the storage, the cost to traverse huge indices and the resulting response time, etc.  Most often these clients have a single solution for their storage needs.  Regardless of the actual value of the data being stored to the business, it goes into a cookie cutter high speed SAN; easy to access – overall costly to maintain on a relative basis.

Our solution to this is simple:  Not all data is created equal.  No, we aren’t going to tell you to “enslave it” but we are going to violate certain constitutional rights and tell you that you should absolutely “profile” it.  TSA be damned.  We are going to borrow a technique from our marketing friends and apply something known as an RFM cube to the data.  RFM stands for recency, frequency and monetization.  Marketing gurus use this technique to make recommendations to people or send special offers to keep high value customers happy or to “re-activate” those who haven’t been active recently.

“Recency” accounts for how recently the data item in question has been accessed.  This might be a file in your storage system or rows within a database.  Frequency speaks to how frequently the data is accessed.  Monetization is the value that specific piece of data has to your business in general.  The three of them help us calculate overall business value and access speeds.  By matching the type of storage to the value of the data in an RFM-like approach, we might have a cube that has very high cost storage mapped to high value data in the upper right and an approach to delete and/or archive data in the bottom left:

The product of our RFM analysis might yield a score for the value of the data overall.  Maybe it’s as simple as a product or maybe you’ll add some magic of your own.  If we plot this on a cost and value curve, we can start matching storage needs to it.  Low value data goes away or goes on low cost systems with slow access times.  If you need to access it, you can always do it offline and email the report or whatever to the requester.  High value systems might go on very fast but relatively costly SSDs or some storage area network equivalent.  We’ve made some mappings below, though the escalation isn’t meant to represent today’s market prices:

Now go save your company some cash!

It seems that everyone is moving to an Agile approach in their product (or software) development lifecycle.  Some move with great success, some with great fanfare and for some it’s one of the last moves their company and engineering organizations make before failing miserably and shutting doors permanently.  As often as not, companies just move because they believe it will cure all of their problems.  As we’ve written before, no new PDLC will cure all of your problems.  Agile may in fact be best for you, but there are always tradeoffs.

We’ve compiled a top 5 misconceptions about Agile from our experience working with companies to solve problems.  Be careful of these pitfalls as they can cause you to fail miserably in your efforts.

1)      Agile is NOT a reason to NOT manage engineers or programmers

Engineering organizations measure themselves.  In fact, many Agile methods include a number of metrics such as burn down charts to measure progress and velocity for measuring the rate at which teams deliver business value.  As with any other engineering organization, you should seek to find meaningful metrics to understand and improve developer and organizational quality and productivity.  Don’t let your team or your managers tell you not to do your job!

2)      Agile is NOT a reason to have engineering alone make product decisions

You still have to run your business which means nesting the product vision to the vision of the company.   More than likely you are paying business or product people to define what it is you need to do to fight and win in your market.  Someone still has to set the broad strategic vision to which products will be developed.  Engineers can and should contribute to this vision, and that’s where Agile methods can help.

3)      Agile alone is NOT a cure for all of your product delivery problems

As we’ve blogged before, there simply are no silver bullets.  With the right management and oversight, you can make any development lifecycle work.  There are just cases where Agile methods work better.   But don’t look to have a PDLC fix your business, people or management issues.

4)      Agile is NOT an excuse to NOT put in the appropriate processes

There is nothing in the Agile manifesto that states that process is evil.  In fact, it recognizes processes are important by stating that there is value within them.  Agile simply believes that individuals and interactions are more important – a concept with which we completely agree.   Don’t argue that all process is evil and cite Agile as the proof as so many organizations seem to do.  Certain processes (e.g. code reviews) are necessary for hyper growth environments to be successful.

5)      Agile is NOT an excuse not to create SOME documentation

Another often misinterpreted point is that Agile eliminates documentation.  Not only is this not true, it is ridiculous.  The Agile manifesto again recognizes the need for documentation and simply prefers working software over COMPREHENSIVE documentation.  Go ahead and try to figure out how to use or troubleshoot something for the very first time without documentation and see how easy it is…  Programming before designing coupled with not creating  useful documentation makes one a hack instead of an engineer.  Document – just don’t go overboard in doing it.

If you have any gray in your hair, you likely remember the 1:10:100 rule.  Put simply, the rule indicates that the cost of defect identification goes up exponentially with each phase of development.  It costs a factor of 1 in requirements, 10 in development, 100 in QA and 1000 in production. The increasing cost recognizes the need to go back through various phases, the lost opportunity associated with other work, the amount of people and systems involved in identifying the problem, and end user (or customer) impact in a production environment. In a 2002 study by the National Institute of Standards and Technology the estimated cost of software bugs was $59.5 billion annually, half the cost borne by the users and the other by the developers.

While there is an argument to be made that Agile development methods reduce this exponential rise in cost, Agile alone simply can’t break the fact that the later you find defects, the more it costs you or your customers.   But I also believe it’s our jobs as managers and leaders to continue to reduce this cost between phases – especially in production environments.  If the impact in the production environment is partially a function of 1) the duration of impact, 2) the degree of functionality impacted, and 3) the number of customers impacted, then reducing any of these should reduce the cost of defect identification in production.  What can we do besides considering Agile methods?

There are at least three approaches that significantly reduce the cost of finding production problems.  These are “swimlaning”, having the ability to roll back code in XaaS environments (our term for anything as a service), and real time monitoring of  business metrics.  These approaches affect the number of customers impacted and the duration of the impact respectively.

Swim Lanes

We think we might have coined the term “swimlaning” as it applies to technology architectures.  Swimlaning, as we’ve written about on this blog as well as in the book, is the extreme application of the “shard” or “pod” concept to create strict fault isolation within architectures.  Each service or customer segment gets its own dedicated set of systems from the point of accepting a request (usually the webserver) to the data storage subsystem tier that contains the data necessary to fulfill that request (a database, file system or other storage system).  No synchronous communication is allowed across the “swimlanes” that exist between these fault isolation zones.  If you swimlane by the Z axis of scale (customers) you can perform phased rollouts to subsets of your customers and minimize the percentage of your customer base that a rollout impacts.  An issue that would otherwise impact 100% of your customers now impacts 1%, 5% or whatever the smallest customer swimlane is.  If swimlaned by functionality, you only lose that functionality and the rest of your site remains functioning.  The 1000x impact might now be 1/10^th or 1/100^th the previous cost.  Obviously you can’t have less cost than the previous phase, as you still need to perform new work, but the cost must go down.

Rollback

Ensuring that you can always roll back recently released code reduces the duration of customer impact.  While there is absolutely an upfront cost in developing code and schemas to be backwards compatible, you should consider it an insurance policy to help ensure that you never kill your customers.  If asked, most customers will probably tell you they expect that you can always roll back from major issues.   One thing is for certain – if you lose customers you have INCREASED rather than decreased the cost of production issue identification.  If you can isolate issues to minutes or fractions of an hour in many cases it becomes nearly imperceptible.

Monitoring Business Metrics

Monitoring the CPU, memory, and disk space on servers is important but ensuring that you understand how the system is performing from the customer’s perspective is crucial. It’s not uncommon to have a system respond normally to an internal health check but be unresponsive to customers. Network issues can often provide this type of failure. The way to ensure you catch these and other failures quickly is to monitor a business metric such as logins/sec or orders/min. Comparing these week-over-week e.g. Monday at 3pm compared to last Monday at 3pm, will allow you to spot issues quickly and rollback or fix the problem, reducing the impact to customers.

Ways of abstracting the storage of data have been around for a long time.  In data warehouses engineers abstract data into business or domain objects that can be manipulated in reports.  For object oriented programming, engineers can use the active record pattern to create a wrapper class representing a table and methods for inserting, updating, or deleting. Thus the manipulation of the database rows are abstracted into the object oriented parlance of classes and methods. This layer of computing is known as a Data Access Layer (DAL) and hides the complexity of the underlying data store from engineers who do not need to be bothered with those details.

There are many frameworks or object-relational mapping (ORM) tools for creating DAL’s for different programming languages such as Active Record for Ruby and Hibernate for Java.  We often hear from development teams that have adopted an ORM that the pros include: shorter development time, better designed code and reduction in amount of code.

However, a quick search will show you that not everyone is sold on the benefits of a DAL. The reduction in code is debatable when constructing complex queries or considering that Hibernate is over 800 KLOC of Java and XML.  There are also concerns about the ability to scale effectively when using DALs.  While it is possible with an ORM to scale on the X-axis such as with MySQL master-slave replication, the Y and Z axes splits can become much more complicated.

I am a fan of DAL’s for their centralization of data objects and their abstraction of relational data into objects. To me these advantages speed development and testing time and improve quality. Additionally, given the sophistication and open source of ORM’s today, I think it makes sense to consider using one as a framework. However, if you choose to do so, you need to consider ahead of time how you would shard your data along other axes. The time for those considerations is moved up when using an ORM. Think of the D-I-D approach where the cost to make a change during the Design phase is negligible compared to changes made after Deployment.

As a frequent technology writer I often find myself referring to the method or process that teams use to produce software. The two terms that are usually given for this are software development life cycle (SDLC) and product development life cycle (PDLC). The question that I have is are these really interchangeable? I don’t think so and here’s why.

Wikipedia, our collective intelligence, doesn’t have an entry for PDLC, but explains that the product life cycle has to do with the life of a product in the market and involves many professional disciplines. According to this definition the stages include market introduction, growth, mature, and saturation. This really isn’t the PDLC that I’m interested in. Under new product development (NDP) we find a defintion more akin to PDLC that includes the complete process of bringing a new product to market and includes the following steps: idea generation, idea screening, concept development, business analysis, beta testing, technical implementation, commercialization, and pricing.

Under SDLC, Wikipedia doesn’t let us down and explains it as a structure imposed on the development of software products. In the article are references to multiple different models including the classic waterfall as well as agile, RAD, and Scrum and others.

In my mind the PDLC is the overarching process of product development that includes the business units. The SDLC is the specific steps within the PDLC that are completed by the technical organization (product managers included). An image on HBSC’s site that doesn’t seem to have any accompanying explanation depicts this very well graphically.

Another way to explain the way I think of them is to me all professional software projects are products but not all product development includes software development.  See the Venn diagram below. The upfront (bus analysis, competitive analysis, etc) and back end work (infrastructure, support, depreciation, etc) are part of the PDLC and are essential to get the software project created in the SDLC out the door successfully.  There are non-software related products that still require a PDLC to develop.

Do you use them interchangeably?  What do you think the differences are?

There is an interesting article in this months IEEE Computer Magazine on Commitment-Based Service Oriented Architecture.  The authors, Munindar P. Singh, Amit K. Chopra, and Nirmit Desai, put forth that existing implementations of service oriented architecture are low-level abstractions instead of business services. They propose a new Commitment-Based SOA where the components are true business services and the connectors are patterns.

As proof of this claim the authors provide the hypothetical example of a purchasing of an item that combines ordering, paying, and shipping business services. The current approaches such as Business Process Modeling Notation, Business Process Execution Language, and Choreography Description Language all emphasize controll and data flow. In contrast to these the CSOA “gives primacy to the business meanings of service engagements, which are captured through the participants’ commitments to one another.”  Commitments include three agents: debtors, creditors, and context. The autors claim benefits of the CSOA are 1) Services can be changed or modified and judged to be correct as long as the commitments are not violated. Commitments thus support business-level compliance and don’tdictate specific operationalizations. 2) Commitment-based specifications explicitly reflect business requirements. Without this business meaning, there is no basis to establish the validity of reuse or composition.

For more information on this topic check out the authors paper on North Carolina State University’s site or in the Novemeber issue of IEEE’s Computer Magazine.

We wrote a post back in July 2008 about how in order to get better you must practice. Since that time we’ve seen a lot of interest in this concept of how much must you practice in order to master a skill.  This interest is primarily due to Malcolm Gladwell’s book Outliers in which he showed a number of examples of why you must have about 10 years or 10,000 hours of practice in order to achieve mastery. I’ve come across many variations of this recently and wanted to revisit the topic from a non-programming perspective to show that this is applicable to every aspect of your life and career. If you want to become a better parent, teacher, runner, programmer, leader, or technologist you must practice. If you want to master those skills you must practice them a lot.

One of the more interesting variations is from the 1976 book by William Zinsser, On Writing Well. If you are interested in writing I highly recommend this book and for everyone else you can draw inspiration from his devotion to the craft of writing. In the first chapter William Zinsser talks about an interview he participated in discussing writing as a vocation along with a certain Dr. Brock who was a surgeon that had recently taken up writing. Dr. Brock started by expressing how “The words just flowed.” And when asked what he did when the writing wasn’t going well he said he’d just stop writing. Zinsser countered that “writing is a craft, not an art, and that the man who runs away from his craft because he lacks inspiration is fooling himself.” He continues with “If your job is to write every day, you learn to do it like any other job.” Zinsser concludes the section with a thought that if writing for a surgeon was so easy he’d consider taking up surgery on the side. This was a bit tongue-in-cheek with the point being to get better you must persevere and do so with a critical eye for how to constantly improve.

An interesting theory on how to produce sustained and desirable change is the Boyatzis’ Intentional Change Theory. This is five step process that enables individuals to achieve change and maintain it. If you have ever tried to stay on a diet or start an exercise regime and have found it difficult, you should be able to relate to this. One of the keys to this process is practicing the new behavior in order to build and strengthen neural pathways.  This eventually leads to mastery of the skill and sustained change.

An MIT Computer Science grad student has an interesting list on his blog of posts that cover this subject of dedicated practice to master a skill.  In MJ fashion, he calls this list “Thoughts on living a remarkable life”.  Two of his most interesting from that list are book reviews On the Value of Hard Focus about Murakami’s book on distance running and The Steve Martin Method about the comedian’s life.

Another story comes from Dave Ramsey’s book More Than Enough. Dave tells the story/parable of a professional golfer being approached by a fan saying “I’d do anything to hit like you” and the golfer says “No, you wouldn’t”, to which the fan replies “Oh, yeah, I really would.”  The golfer goes on to explain his secret “Get up every morning and hit 500 golf balls. Hit them until your hands are so blistered they bleed. The next morning, tape over the blisters and do it again.”

A final example comes from Jason at 37Signals on making money.  He postulates that as an entrepreneur you need to practice early and often making money; learning the skills of negotiating, pricing, and selling.  This is why he recommends entrepreneurs not take outside money, in order that they learn to make money quickly. Tim Ferris would probably agree that negotiating can be practiced and Seth Godin would certainly agree that selling takes practice.

So now with all the overwhelming evidence that we need to focus and practice to master a skill, what will you do today to become a master at something?