We’ve written about trying to use vendor features to scale but given how often we run across companies that have been convinced by vendors to rely on them, this topic is worth revisiting. To state it as directly as possible, every major SaaS company that has relied on a vendor, software or hardware, to scale them through hyper-growth has failed and had to solve the scale problem themselves.

Since Oracle World took place recently I’ve decided to use Oracle RDBMS as an example of failing to scale with vendor features. We have nothing against using Oracle as an RDBMS, even though there are open source options that can scale just as well, but let’s use one of their scalability features, Real Application Clusters (RAC), as an example. In Oracle’s own words RAC “…enables a single database to run across a cluster of servers, providing unbeatable fault tolerance, performance, and scalability with no application changes necessary.” A nice concept – to scale with “no application changes” – but this isn’t realistic with hyper-growth companies. One large reason is that RAC does not scale across multiple data centers, which is a requirement for hyper-growth companies since everything fails eventually including data centers. Even with the “Extended Distance Clusters” for RAC nodes, they only extend to 25 kilometers using Dark Fiber (DWDM or CWM) technology.

The use of RAC for increased availability is fine but you should review our post on the downside of using vendor features and how to negotiate with vendors. In particular you should be aware that by using this feature you have weakened your position during renewal negotiations. If you think your sales person is being nice by throwing in the RAC feature for a low price, think again. As soon as you start using this feature they have the upper hand in negotiations.

Enough of the RAC rant, especially since this is just one example of many that are out there. Hardware vendors, both servers and storage, are just as guilty of trying to convince SaaS companies to rely on them for scalability. Keep your destiny in your own hands and resist relying on short term solutions to long term problems.

There was a study by Viswanath Venkatesh and Michael G. Morris “Why Don’t Men Ever Stop to Ask for Directions? Gender, Social Influence, and Their Role in Technology Acceptance and Usage Behavior” that looked at 342 workers over five months to observer usage and adoption of technology. The researchers’ results were that men considered perceived usefulness to a greater extent than women in making decisions about the use of new technology. On the other hand, perceived ease of use was more important to women compared to men even after initial training. What’s perhaps even more interesting was that men’s view of ease of use was that it went up after using the system while women’s view remained unchanged. Additionally subject norms were considered much more by women than men. What this suggests is that women are much more balanced in their technology decisions with regards to perceived usefulness, ease of use, and subject norms.

Another study by Fiona Lee “When the Going Gets Tough, Do the Tough Ask for Help? Help Seeking and Power Motivation in Organizations” revealed that individuals do not seek help, even when it is needed and available, because doing so implies incompetence, dependence, and powerlessness.  There is even a whimsical study from insurer Sheilas’ Wheels that claims that the average male drives around lost 276 miles each year costing over $3,000 in fuel.

Our experience with hundreds of clients has been that technologist in general hate to ask for help but would much rather struggle in an attempt to solve the problem themselves. Unfortunately they do so even at the risk of being very inefficient and wasting organizational resources. Whether the reason is genetic or fear of being seen as incompetent the problem is costly to organizations.

Asking for help or advice from someone who has been down the road you’re traveling in my opinion is not only the fastest way to get there but also shows great courage and confidence. You have to be completely comfortable with what you know in order to admit in front of peers or bosses what you don’t know. Some of this confidence comes from experience but some of it also comes from the culture of the organization. If you’ve built or inherited an organizational culture where everyone pretends to know everything and is afraid to ask for help, you’re guaranteed to be very inefficient. If you find yourself faced with this situation be the leader and step forward to show people how to ask for help. Start by calling this issue out at your next all hands meeting or staff meeting. Then show people it’s more important to be unbelievably curious and passionate about your craft than to appear like you know everything.

If you are a technology organization within a Software as a Service (SaaS) company, there is no such thing as an “internal customer”.  We often see this anachronistic IT phrase thrown around in web X.0 companies by executives and engineers who simply have not adopted the new SaaS mindset.  Do you think you’ll hear the left offensive tackle of an NFL team refer to the quarterback as his “internal customer”?  The quarterback consumes services (energy to block opponents) of the left tackle – so why wouldn’t he be a customer?  The answer is simple – because the notion of a customer relationship is different than the notion of a relationship within teammates.

The first reason why your teammate isn’t your customer is because he or she is, well, your TEAMMATE.  Customers are someone for whom you produce a service or product and teammates are someone with whom you work to accomplish a goal.  The difference between working FOR someone and working WITH someone is HUGE.  This difference creates a contextually activated identity that forces you to think about customers in a different light than you would a teammate.  Very often, as we’ve written before, this can result in affective (role based or bad) conflict between teams.  Affective conflict is bad and it destroys shareholder value.  Working as a team is important and customers aren’t part of your team.

The next reason that your teammate isn’t your customer is that the customer is always right.  Your teammate isn’t always right.  You need to debate certain points as a team to come to better solutions.  This isn’t affective conflict, it is cognitive conflict and if handled properly it is good and helps to create shareholder value.

The most important reason there isn’t a customer relationship here is that your teammate isn’t paying!   “Servicing” your teammate (uggh…that’s an ugly term) doesn’t create shareholder value.  Working as  a team to delivery a service or product to your  “real” customer is what creates shareholder value.  One design, one approach, one ruthless drive as a team to get across the goal line is what is necessary to thrive and succeed.

So stop using the ugly “internal” C word in your SaaS company.  It doesn’t have a place there.  Let the old world, internal IT folks continue to provide services to their internal customers.  Start acting like a team, designing and building services rather than software.

When we build products we spend a great deal of time and effort thinking through how our customers will use them. From all this effort we believe that we full understand our customer and system interaction. The reality is often that we don’t know how customers will ultimately decide to interact with our system. An example of this is how social networking sites were originally intended for people to meet and interact. Soon after the launch of Friendster people began setting up accounts for their pets. This came to the attention of Friendster’s management and they began shutting down these “fake” accounts. This, as you can imagine, upset many of these individuals who thought their pets deserved to experience social networking but the point is that customers had decided to utilize the system in a very different manner than the company planned or even approved of.

The academic research that can be used to explain this phenomenon is adaptive structuration theory (AST), adopted by DeSanctis and Poole from Anthony Giddens’ theory of structuration. AST describes structures and agents, where there is a duality of structure  that exhibit an interplay between the structures inherent to advanced technologies and those that emerge in human action with these technologies. Orlikowski also extended Giddens’ work into technology by examining how people enact structures as they interact with a technology that affect their use.

Back to our example of pets having their own accounts on social networking sites. Customers are going to adapt their usage of our technology based on our product’s design as well as their interactions with the product. Our responsibility is not only to focus on the customer during the conceptualization, design, and development phases of our products but also in the maintenance phase. Take note of how the products are really being used by our customers in order to not only support their use but leverage it for further product refinement.

The strategic fit camp emphasizes strategic analysis and negotiation prior to the acquisition whereas the organizational fit research emphasizes the integration of day-to-day operations post acquisition. There has even been recent research on the impact of the acquisition process itself, which makes sense given that first impressions are established during this phase. I’ve had the good fortune of seeing acquisitions from all different perspectives, as the acquired, as the acquiree, as a consultant of the acquiree, etc. I think there are two models that result in successful acquisitions.

First off defining a successful acquisition is tough. Whether or not the acquiring company writes off the purchase cost in a few years might be one way but these write offs can include goodwill which is the amount of the purchase price paid above the value of the target’s identifiable net assets. There are of course accounting rules for testing impairment of goodwill that may require it to be written off. Another way of testing this might be by calculating Return On Investment (ROI) for the cost of the acquisition. A third way of determining success would be whether the acquired company’s product offerings continue 3 or 5 years later. If the acquiring company has discontinued the acquisitions products the purchase might be considered a failure. Lastly, I tend to think of a successful acquisition like Justice Potter Stewart tried to explain what is obscene by saying “I shall not today attempt further to define the kinds of material I understand to be embraced … but I know it when I see it.”

Whether there is a clear definition or I know it when I see it, there appear to be two paths to get there. The first is what I term the GE-approach because of all the acquisitions I saw while at General Electric during the 90′s, this appeared to be the dominant strategy, although I’m sure many other companies have similar methods. It’s simply an approach of overwhelming the acquisition’s culture and turning it into a GE culture as fast as possible. They did this by sending current GE employees into key positions at the acquisition and sending the acquisitions employees to lots of GE training. Within the first year the acquisition was on the GE strategic planning calendar, using GE financial systems, and following GE’s HR guidelines…they were a GE company by that point.

The second approach is to leave the acquisition completely alone and let it run autonomously. The only tie to the acquiring company is through financials. The parent company acts like a board of directors, approving annual budgets and determining the reinvestment ratio. The acquiree is left to their own to manage the day-to-day operations, processes, etc.

If you’ve seen other successful acquisitions handled differently let us know your thoughts.

If I get another phone call from another recruiter about a “Chief Innovation Officer” search, I may just hang myself from my hotel light fixture with my belt.  Don’t get me wrong – I’m not angry at the recruiter (other than for wasting some of my time) – I’m mad at the CEO or person advising the CEO who come up with such a lame title.  Why am I so upset over the name? Pull up a chair and grab a cup of coffee.  You don’t want to miss this rant.

My first gripe is with the name of this “position”.  The name just sucks and flies in the face of everything we know about innovation.  In some companies, the Innovation Officer might just be a rebranded VP or SVP of product.  I’m almost okay with that – except for the fact that in this case it is a complete misrepresentation of what the person is doing.   If the person is something other than the head product person, well, that’s when this position really is nothing more than a shareholder wealth incineration device.

This moronic title seems to imply that there is a person “responsible” for innovation within a company.  Anyone who believes that one person can somehow be responsible for innovation is clueless about whence innovation comes.  You can no more “control” innovation than you can domesticate a rabid dog hyped up on meth.  You can certainly “kill” innovation, just as you can put down that rabid junkie dog – and giving someone a business card with Chief Innovation Officer on it is like putting a 240 grain slug in that dog.  Goodbye junkie dog and goodbye innovation.  The dog will probably die a more humane death however, as the Chief Innovation Officer will kill innovation under the crushing weight of power point presentations and spreadsheets.

Innovation simply can’t be controlled.  It can be “captured”, it might be “harvested” and it certainly can be “found”.   In some cases, it might even be able to be “directed” with the right questions and incentives.   But “controlled”?  Give me a break.  Capturing, harvesting and finding are all by the company culture – not a single individual.  And as we all know, that culture is most often affected by the CEO and to a lesser degree by each level below the CEO.  This is one of those things, then, that the CEO can’t delegate.  It’s not a position – it’s a shared responsibility.  How do you incent people to innovate?  How do you find little pockets of innovation that hide within the organization but never become apparent because your processes require it to be presented within a 40 page Power Point deck?

My partner, Tom Keeven, knows how.  Tom is a man of varied interests.  One interest is running a pizza parlor in Carmel.  His employees come to him with suggestions all the time.  “Let’s start a delivery service”, “Why don’t we make a low carb pizza dough?”, “Let’s advertise in local hotels”.  Tom doesn’t tell them to run off and build a presentation – he listens to them with great interest.  He rewards them for great insights.  He tells them to experiment and try things out.

Google allows engineers to work on interesting problems with great benefit to the innovation process.  Granted, few of these innovations have paid off and the resulting shareholder dilution is very large.  But the process of bottoms -up, “grass roots” innovation produces results in terms of number of innovations brought to market.  Sooner or later one of them will pay off – the question is “at what cost”.

In summary, innovation isn’t an organization, it’s not a title and it’s certainly not something controlled from the top.  There are things you can do to nurture it and maybe even slightly direct it – but there is little you can do to control it.   You can’t hire a person to control it and you are just wasting your time if you expect to try to create an organization to “innovate”.  If you want your team to innovate, work on creating a risk tolerant culture and work to incent your team for their innovation efforts.  Lower the cost of innovation by eliminating ridiculously complex and burdensome innovation hurdles like executive presentations.  Change the ridiculous titles of your innovation officers to something useful and stop wasting shareholder wealth.

You get paid to deliver a service.  You want to deliver that service to the level of your customers’ expectations, or at least to some internally defined level.  So how often do you meet to discuss your service delivery quality?

In our experience, most companies meet only when there is a problem.  Day in and day out many Software as a Service companies will operate their services throughout the day and simply not take the time to step back and look at the last day’s worth of issues, all open issues that are not yet resolved and diagnose service delivery problems.  How could this be you ask?  Well, we honestly don’t know!

As we’ve written before, if you are a SaaS company your business is predicated first and foremost on SERVICE DELIVERY!  Developing software is important – but what makes you money is the delivery of a service.  Get this straight folks, because it is a major mind shift.

In our view, it is absolutely critical to start the business day with a review of the past day’s service delivery.  We call this the “Morning Operations Meeting” or “Morning Operations Review”.  Every day we ask our clients to review major issues from the previous day, overall service quality (response times, availability, major interruptions or bugs live on the site, etc), and all major open issues identified in past days.  Ideally the notion of an incident (a thing that happens in production and causes customer complaint) and the notion of a problem (a thing that causes an incident) are separated in this meeting.  Both should be discussed – but they are really two separate things.

Ideally this meeting will have representatives from your customer support organization, technical operations and infrastructure teams and software development teams.  Inputs to the meeting are a representation of customer complaints, complaints regarding service within the company, manual identification of issues, automated identification of issues (such as through a monitoring system to include Service Level metrics), predictive identification of future problems (such as might be the case from a capacity management team) and all appropriate service level information.

Open incidents and problems from the issue tracking system are discussed, updated, etc.  Owners are assigned to new incidents and problems (if they haven’t been already) and new issues are updated if any were missed from the previous day’s operations.

Outputs from the meeting are updated service level reports, scheduling of post mortems for large incidents, updated problem reports and data for monthly or quarterly look backs or reviews (more on this later).

If done well, the morning meeting helps inform architectural changes that are necessary in the scalability summits or in other product development and architecture meetings.  Recurring problems should be easily identified within the issue management as a result of heightened oversight and analysis of the system.

Recently on the MySQL Performance Blog they had a post that did a great job explaining a problem that we often try to warn our clients about. The crux of the problem is that if you are relying only on a replica for disaster recovery then you are going to lose data when something bad happens.

For minimizing the impact of eventual consistency in our BASE applications, we want our replicas to be very near real time. This unfortunately can be unintended consequences in a disaster. Whether you’re relying on MySQL’s statement-based replication or Oracle’s redo apply replicating at the block-level, both are vulnerable to data corruption.

Any scenario resulting in data corruption on the primary will immediately be replicated to the standby. If a DBA drops a table by the time he stops cursing the drop table has been replicated to the standby. Storage subsystem or HA failover both can corrupt data files which can get propagated to the standby.

The solution to this problem is to create a standby or replica that has a delay on applying the log files. We recommend between 6 – 12 hours delay which gives you plenty of time to catch a logical corruption and stop the replication. You don’t need a large production sized server for this since you’ll never use this database in production but simply recover the database from it. Do this simple thing and it might save your data.

Most of the time the individuals that we interact with during engagements are intelligent, intellectually curious, and open to suggestions for improving their service offering’s scalability and availability.  On rare occasions, however, we run into an individual who either argues just for the sake of arguing or thinks he/she can’t learn anything from anyone. While these people might be brilliant they are likely going to ultimately fail and in doing so negatively impact the company. A rule to live by is that an architecture designed by one person is much poorer than one designed by a diverse group of individuals with different skill sets. This is one of the driving principles behind the JAD and ARB.

This is not to say that all conflict is bad. As Marty mentioned in his post on Team Conflict, cognitive conflict is desired. It is the affective conflict that is not good for the team. An easy way to think about the difference between cognitive (good) and affective (bad) is that arguing over “what” to do is good, arguing over “who” should do it (territorial) or “how” it should be done (micromanagement) can be harmful if not carefully kept in check. Once an someone has been assigned as the “R” (see Chapter 2 in The Art of Scalability) let them own the project.

We’ve posted a lot on our blog about hiring A players, tending your team like a garden, and building high performance teams. Allowing someone who displays an attitude of arrogance and superiority in a leadership position is more than just annoying but harmful to the team. Junior engineers will not push back on this person’s decisions for fear of humiliation, younger leaders are being taught to act this way in order to succeed, and the experiential chasm between this person and other executives is only widening.  No matter how brilliant this person is, they are causing more problems than they are solving. Take steps today to remove them from the organization as quickly as possible.

I’ve noticed lately that more companies are putting up signs in hallways and cube farms requesting that people avoid having conversations in these areas. While having a nice quiet work environment makes sense to me as a developer, doesn’t this completely void having people work beside each other? The ad hoc/hallway/water cooler/coffe machine conversations or ones overheard when cube mates are chatting about a new feature are one of the primary benefits of having people work in small open environments.

I haven’t done any sort of scientific study but it seems that these sort of “please be quiet” signs are more prevalent at larger companies. These are the same ones that are trying to mimic the small startup with agile development processes or open work spaces to compete in a fast moving SaaS marketplace. Imitating the actions without understanding the purpose or allowing old school corporate policies to overrule are surefire ways to tank the initiative.

A parallel to the “please be quiet” sign is allowing corporate IT to dictate the architecture of the SaaS offering based on a corporate standard that works for the ERP system. Running Oracle ERP on a 16-way system might be the vendor recommended, preferred approach but for scaling a SaaS offering this is a quick way to run up the costs and ensure lower availability. We often use the analogy of goldfish and thoroughbreds for comparing small, cheap 1U servers with large, expensive multi-processor boxes with lots of memory. The goldfish (small, cheap servers) are inexpensive to purchase and replaceable while the thoroughbred (large servers) are expensive to purchase/maintain and cause big impacts when they go down.

The take away to all of this is that if your part of a corporate initiative to run an internal startup or deliver a Software as a Service from inside a larger organization, don’t allow corporate policies to prevent your success. The differences in approaches, architectures, organizations, and offices have a purpose and should not be discounted as non-critical to the success of your initiative.