Below is our most recent newsletter. If you would like to subscribe and have it delivered to your inbox, you can do so here.

Trends
We’re privy to meeting and talking with hundreds of high tech, hyper growth companies and therefore we have a unique opportunity to see trends that are taking place in the industry. Here are a few that we thought would be interesting to share with our friends.

NoSQL Skills – Many companies are experimenting with NoSQL solutions including Membase, Hadoop, Cassandra, and many others but are finding that employees with skills in these are hard to come by. Even in areas rich in talent such as the Silicon Valley, Boston, Austin, Seattle and NY the demand for this rather nascent skill set is higher than the current supply. Many companies are relying on on-the-job training for these types of open source solutions. Most importantly, if you’ve properly fault isolated your architecture you can tolerate the risk associated with small segments going down during beta releases while you iron out the kinks.

Oracle’s NoSQL – Oracle recently announced their entry into NoSQL with announcements around Hadoop integration and their own NoSQL key-value solution. The Hadoop integration isn’t much more than a conduit to allow data to and from an Oracle database and a Hadoop cluster. This technology has been around for quite a while in other relational database solutions such as GreenPlum and Asterdata. It was also recently announced by Microsoft that SQL Server would support this Hadoop connection as well.

Oracle’s NoSQL is a distributed, replicated key-value store and is new and exciting even though it has attributes similar to other product offerings including their acquisition, Berkeley DB. Given trend #1 above, it may sound like an interesting alternative to adopting an enterprise supported NoSQL alternative to open source solutions but beware. The marketing materials for Oracle’s NoSQL solution include the BASE acronym (Basically Available, Soft State, Eventaully Consistent) but unlike other NoSQL products such as Dynamo, SimpleDB, or Cassandra, the Oracle NoSQL Database does not support eventual consistency. Oracle’s solution to eventual consistency appears to be by not accepting writes when the primary node for that key is down.

ORM – many companies start off using an object relational mapping solution such as Hibernate or Active Record but we are seeing many of them having difficulty scaling with them. The solution for several companies has been to use ORMs for simple queries but resort to ODBC or their own Data Access Layer for handling more complex queries. Be wary of using a solution to handle your query development as we’ve had a number of clients with incredibly complex and costly queries bring down their platforms for extended periods of time.

Enterprise Monitoring Frameworks – Until very recently, we had not seen a proprietary third party (non open-sourced) monitoring solution at a customer for at least 2 years and that includes our large Fortune 500 clients. Many of our clients have adopted innovative new monitoring solutions from Wilytech and Coradiant (now CA and BMC respectively) that look for and help diagnose patterns “on the wire” – whether it be from the browser to their servers or from app servers to databases. While these are interesting and potentially worth some of your attention, our very best clients design their systems to be monitored from the ground-up – ensuring that their software helps identify performance problems as they happen.

Distributed File Systems – Take your pick of implementations, but many of our clients are eschewing traditional NAS/NFS devices for distributed storage pools the likes of Gluster, MogileFS and Ceph. Nearly every case has resulted in significant savings relative to proprietary systems with few reported impacts to availability or response times. Of course, as with any other architectural change you need to ensure that you are properly managing your risk through pods or swim lanes.

We’ve written about trying to use vendor features to scale but given how often we run across companies that have been convinced by vendors to rely on them, this topic is worth revisiting. To state it as directly as possible, every major SaaS company that has relied on a vendor, software or hardware, to scale them through hyper-growth has failed and had to solve the scale problem themselves.

Since Oracle World took place recently I’ve decided to use Oracle RDBMS as an example of failing to scale with vendor features. We have nothing against using Oracle as an RDBMS, even though there are open source options that can scale just as well, but let’s use one of their scalability features, Real Application Clusters (RAC), as an example. In Oracle’s own words RAC “…enables a single database to run across a cluster of servers, providing unbeatable fault tolerance, performance, and scalability with no application changes necessary.” A nice concept – to scale with “no application changes” – but this isn’t realistic with hyper-growth companies. One large reason is that RAC does not scale across multiple data centers, which is a requirement for hyper-growth companies since everything fails eventually including data centers. Even with the “Extended Distance Clusters” for RAC nodes, they only extend to 25 kilometers using Dark Fiber (DWDM or CWM) technology.

The use of RAC for increased availability is fine but you should review our post on the downside of using vendor features and how to negotiate with vendors. In particular you should be aware that by using this feature you have weakened your position during renewal negotiations. If you think your sales person is being nice by throwing in the RAC feature for a low price, think again. As soon as you start using this feature they have the upper hand in negotiations.

Enough of the RAC rant, especially since this is just one example of many that are out there. Hardware vendors, both servers and storage, are just as guilty of trying to convince SaaS companies to rely on them for scalability. Keep your destiny in your own hands and resist relying on short term solutions to long term problems.

Recently on the MySQL Performance Blog they had a post that did a great job explaining a problem that we often try to warn our clients about. The crux of the problem is that if you are relying only on a replica for disaster recovery then you are going to lose data when something bad happens.

For minimizing the impact of eventual consistency in our BASE applications, we want our replicas to be very near real time. This unfortunately can be unintended consequences in a disaster. Whether you’re relying on MySQL’s statement-based replication or Oracle’s redo apply replicating at the block-level, both are vulnerable to data corruption.

Any scenario resulting in data corruption on the primary will immediately be replicated to the standby. If a DBA drops a table by the time he stops cursing the drop table has been replicated to the standby. Storage subsystem or HA failover both can corrupt data files which can get propagated to the standby.

The solution to this problem is to create a standby or replica that has a delay on applying the log files. We recommend between 6 – 12 hours delay which gives you plenty of time to catch a logical corruption and stop the replication. You don’t need a large production sized server for this since you’ll never use this database in production but simply recover the database from it. Do this simple thing and it might save your data.