There are numerous companies who decided a year or two ago that as part of their product offering to provide storage of user data. Usually this occurred with no foresight or cost calculations and so these companies decided that this was either unlimited in amount, perpetual in duration, or worse, both. Fast forward to the present and these companies are scrambling to figure out ways to lower the storage cost or charge customers for this service. Of course, hindsight is 20/20 but in our opinion this should be taken as a lesson to all companies that product roadmaps without consideration of the revenue versus cost equation is more than likely to result in future problems of features either not being used by customers or the use of the feature not generating enough revenue to cover the cost.
For companies with data storage problems our recommendations are very dependent on their business model, user agreements, customer contracts, etc. So unfortunately there is no panacea or one size fits all solution. In general we usually walk down the follow steps attempting to achieve an acceptable solution:
- Delete what data you can
- Archive to very low cost storage data that is not being accessed
- Establish tiers of storage based on speed, reliability, and availability
Consider situations in which you have a significant amount of archival data such as former employees or customers who are no longer active. The cost of keeping this on your primary storage is not only the space on your fastest and most expensive storage but also the backup and archiving of this data that occurs every day even though it never changes. Incremental backups help this but more than likely you have full backups periodically as well. If this data is in a primary database, you are likely to have one or more standby databases as well as a tape backup. All of that unchanging and rarely accessed data continues to take up storage and bandwidth to move it around.
Possible storage alternatives include the myriad of SAN offerings, NAS devices, open source storage, SATA drive farms, tape, and cloud storage. We recommend that you implement one or more of these in your solution depending upon your particular needs. We also encourage you to consider ahead of time your need for scalability and availability. For a sample architecture of a scalable read or search subsystem check out our previous article.