If you think there are a hundred things you need to worry about as a startup, make it 101; you must not forget about intellectual property infringement. Unfortunately the environment surrounding intellectual property is complex and fraught with dangers for the unwary. Imagine a scenario where you come up with a great idea, spend thousands of hours developing it into a viable product, spend thousands of more hours attracting and retaining customers, only to find out years later that you have infringed someone’s patent and now owe them monetary compensation for the use of the same and potentially punitive damages for willful infringement. Obviously this is not a desirable scenario and therefore it’s worth doing a little bit of homework upfront to try and avoid that outcome.

Our first piece of advice is to seek legal counsel early; pay a little money to an attorney and get some peace of mind. Don’t just read a blog, even ours, and think you can go it alone. Educate yourself but seek experts when needed.

Types of Intellectual Property

A top priority should be to avoid infringing someone else’s ideas and the first step to accomplishing this is to understand the three types of intellectual property: patents, trademarks, and copyrights. These are all governed by federal law.

Copyrights are granted to people for any expressed form of an idea or information that is substantive and discrete. Under the Berne Convention copyright is automatic even without application once it is produced in a “fixed medium”, such as a drawing, document, or electronic file. Copyrights are issued by the U.S. Copyright Office. One of the limitations of the copyright protection is “fair use”, the definition of which is not always clear but allows the use of your work by others for such things as parodies and news.

Trademarks are protect words, names, symbols, sounds, or colors that distinguish goods and services from those manufactured or sold by others. Trademarks are issued by the Trademark Office of the U.S. Patent and Trademark Office (PTO). Unlike patents, trademarks can be renewed forever as long as they are being used in commerce.

Patents grant the receiver exclusive rights to the invention and are granted through the U.S. Patent and Trademark Office (PTO). The length of protection is currently 14 years from date of filing for design patents and 20 years for all other. Design patents protect the unique appearance of items that can be manufactured but not their function, such as furniture. The five primary requirements for an idea to be patentable are:

1. Patentable subject matter – The subject matter must be such that the federal legislature and judicial branches have determined to allow and includes machines, processes/methods, compositions of matter, and improvements of those. Algorithms and laws of nature (e.g. E=mc2) are examples of subject matter than cannot be patented.
2. Utility – The idea must serve a use or purpose.
3. Novelty – The idea must be new and not known by others.
4. Nonobviousness – The idea must not be obvious to someone practiced and working in the field of the matter. For example if the idea involves a method of producing chocolate candy, the idea must not be obvious to an experienced chocolatier.
5. Enablement – Within the patent application the idea must be described in sufficient detail as to enable someone to make use of it.

Research

The next step in protecting your startup is to do some research. Use tools such as the Google’s patent search or the search engine at the PTO site to look for patents and applications that are close to your idea. If there are patents that appear to be close to your idea change your idea, design, and implementation now. If you think there could be any doubt call in an expert, someone very experienced in the field and with the relevant patents, or call a patent attorney to receive a legal opinion.

Patent vs Trade Secret

Once your sure you’re not infringing the next decision is how to protect your own intellectual property. Copyrights and trademarks are usually pretty straightforward decisions. Things get murky when deciding between a patent and a trade secret. Trade secrets are intellectual property that is not widely known outside of the company. This can take the form of formulas, processes, or methodologies. A great example of a trade secret is the formula for Coca-cola. The reason this can be a difficult decision is that as soon as a patent is filed it becomes public record and your competitors will gain insight into your business. If you are eventually awarded the patent you gain the sole right to use your idea. If you keep your idea a trade secret but your competition determines how to reverse engineer your product and begins using your idea, you have no recourse.

Patent Application

Once you decide to proceed with a patent the application process can be very long often taking years to complete. The process of receiving a patent starts with an application that includes a specification, a summary of the invention, all claims being made, and a declaration of invention. This application along with the filing fees initiates the process. A reviewer from the PTO studies the application and investigates prior art. The reviewer can issue a patent or reject some or all the claims. If any or all of the claims are rejected the applicant can resubmit the same or modified application. If rejected a second time there is an appeal process that includes the Board of Patent Appeals and Interferences and eventually the US Federal Circuit Court of Appeals.

Conclusion

If you’re unfamiliar with intellectual property this may all seem a bit overwhelming. The two critical things to take away are 1) spend the time upfront to educate yourself and your team and 2) seek expert help when confused or you arrive at a critical decision. The investment in both will be well worth it in the end.

Now that we’ve completed one of our major writing projects and are on winter break from another writing-intensive project, I’ve spent the past few weeks catching up on some reading. I keep a both a pile of unread books/articles as well as a list of books that I want read. When the pile gets low I go to the list to replinish the pile. We often share reading recommendations amongst ourselves.

All of this reading has made us interested in updating our old recommended reading list with a new reading list. This time we’ve decided to use Amazon’s astore to do this since it makes adding, updating, and categorizing our list quick and easy. You can find the permalink on the righthand side of our blog. We’ll try to keep it updated as much as possible so you know what we’ve read and thought worthy enough to pass along. If you have suggestions of great books let us know in the comments.

NOTE: The purpose of this blog isn’t financial gain but given the new FTC requirements for full disclosure we feel obligated to state that the recommended reading list is an Amazon associate’s page, for which we get some incredibly small amount back in the event you do use our links. We are not really interested whether you use our links or not, we just want an easy way to recommend books to you. For those fellow bloggers that haven’t seen the FTC requirement, here is the relevant couple of sentences:

The revised Guides specify that while decisions will be reached on a case-by-case basis, the post of a blogger who receives cash or in-kind payment to review a product is considered an endorsement. Thus, bloggers who make an endorsement must disclose the material connections they share with the seller of the product or service.

One final quote to leave you with…

Outside of a dog, a book is man’s best friend.  Inside of a dog it’s too dark to read.  ~Groucho Marx

If you’ve had the opportunity to participate in an engagement with us you know that we tend to utilize the Socratic method of asking questions. And, it’s not uncommon for us to hear a company explain how they plan to utilizing a particular vendor’s feature to scale. For databases this is often clustering. We believe relying on a vendor to scale violates several of our architectural and scalability principles. Let us explain a couple of our more significant concerns with this practice of relying on a vendor.

First and foremost you should want the fate of your company, your team, and your career in your own hands. Do not look for vendors to relieve you of this burden. As a CTO if the vendor you vetted and selected fails, causing downtime for your business, your are just as responsible as if you had written every line of code. And if it is significant or frequent enough you should expect to be relieved of your position. All code has bugs, even vendor provided code, and personally I would rather have the source code to fix it rather than have to rely on a vendor to find the problem and provide me with a patch. This statement should not be taken to imply that you should do everything yourself such as writing your own database. Use vendors for things that they can do better than you and that are not part of your core competency. See our post on build vs buy for the four question to answer when making this decision.

With scalability, as with many other things in life, simple is better. The more complex you make your system to provide scalability the more you are likely to suffer from availability issues. More complex systems are more difficult and more costly to maintain. Clustering technologies are much more complex than straightforward log shipping for creating read replicas.

One of our beliefs, and should be one of yours also, is that it is most cost effective to be vendor neutral. Locking yourself into a single vendor, whether hardware or software, gives them the upper hand in negotiations. If you don’t think the sales rep knows this, ask yourself why they are willing to throw in some of these features at such an initially steep discount. The reason is that they can make up for it next year when you have to renegotiate.

If these concerns resonate with you, check out our database scalability cube post for ideas on how to design a scaling strategy that you can own, is relatively simple, and is vendor agnostic.

Almost all technologist get the opportunity at some point in their career to negotiate with vendors and/or manage a vendor relationship. Often this is a critical part of the job for the VP of Operations or the CTO/CIO. Below are some ideas on how to make the negotiation and ultimately the relationship with the vendor successful.

1. Be honest. Don’t lie to vendors about anything. It’s often tempting to stretch the truth about other vendors and their offers, timelines, budgets, approval requirements, etc. It’s better to say nothing, lies will damage the relationship forever.
2. Don’t take things personally. The vendor’s sales reps do this day in and day out, most aren’t going to lose sleep worrying about if you like them or not. Your loyalty and motivation should be to your investors.
3. Consider the relationship. While negotiating is often viewed as a game, be aware that your behavior will follow you into the relationship. See rule #1.
4. Give yourself time. Time is your best friend in negotiations. Many vendors have rearranged their fiscal year because their clients know how desperate many sales reps are to close sales and receive their commission during these times.
5. Give yourself options. Follow our advice on scalability so that you are vendor neutral and can change vendors with little effort or concern.
6. Do your homework. Find out about the vendor, their customers, their solvency, their post sales support, etc.  Use your network, find out what other people pay for similar services. Be prepared for there to be huge differences based on the total size of a purchase a company makes but at least know the ballpark range.
7. Keep negotiators separate from implementers. If possible have the people negotiating the deal different from the people who have to work daily with the vendor. This way in the event either party feels slighted in the deal they don’t have to work with the same people the next day.
8. Don’t discuss your budget. The vendor has no need to know how much you can afford to spend on this purchase. It’s okay to let them know that you will need additional authorization to make the purchase if you think the price is beyond your ability to authorize.
9. If it’s not in writing it never happened. Lots of things get said during negotiations that get put into the contract and therefore never get delivered. If it’s important get it in writing.
10. Ask for anything. Once you’ve exhausted the pricing negotiation ask for other things that aren’t cash related such as additional modules, higher support levels, steeper discounts for future purchases, etc.

Customers often ask us “When should we invest in scalability?”  The answer that we want to give, and that’s financially correct for your shareholders, is to deploy your scalability improvements the day before you need them.  If you could deploy scale improvements the day before the lack of those improvements would cause problem, you would delay investments to be “just in time” and gain the benefits that Dell brought to the world with configure to order systems married with just in time manufacturing.

But most of us have a hard time predicting when we need to deploy that next scalability improvement and an even harder time getting the project to come in just at the right date.  To that end, we developed the “Design-Implement-Deploy” or “D-I-D” approach to thinking about scalability.  Because there are multiple phases to any product enhancement, including the phase in which you start to think about it and design it (even if it’s just in your head), the phase in which you actually “code it” and the phase that you deploy it to production, we matched these phases to your needs for scalability.  Note that these phases don’t argue for a waterfall model.  If you are developing using Agile methods, you still hopefully “think” about how something might look or work before you start coding or developing it.

We start with the notion that discussing and designing something is significantly less expensive than actually implementing that design in code.  As such, we should be willing to spend more time discussing how we might scale something and sketching out a design for that scale well in advance of our need and to an extent significantly greater than our need.  We should, for instance, discuss how we might scale something to at least 20x greater than what we have now and ideally to nearly infinite capacity.  Many times, in SaaS environments, we could facilitate such scale along the Z axis by dedicating systems to individual customers though this would cost us a great deal and would break the leverage achieved by multi-tenancy.   Nevertheless, discussions such as these are beneficial to help us figure out what we need to do in the future and to what degree we need to scale our systems.  The focus then on design of the D-I-D scale model is on scaling to between 20x and infinity.  Intellectual costs are high, but engineering and asset costs are lower because we aren’t writing code and we aren’t deploying systems.  Scalability summits are a good way to identify the areas necessary to scale within the design phase of the D-I-D process.

We should then move to implementing our designs within our software, focusing on ensuring that we can scale to at least 3x our current size and up to 20x.  There might be cases where the cost of scaling 100x (or greater) our current size is not different than the cost of scaling 20x and if this is the case we might as well make those changes once rather than going in and making those changes multiple times.  This might be the case we are going to perform a modulus of our user base to spread across multiple (N) systems and databases.  We might code a variable Cust_MOD that we can configure over time between 1 (today) and 1,000 (5 years from now).  The cost of such changes are high in terms of engineering time, medium in terms of intellectual time (we already discussed the designs earlier in our lifecycle) and low in terms of assets as we don’t need to deploy 100x our systems today if we intend to deploy a modulus of 1 or 2 in our first phase.

The final phase is deployment.  Using our modulus example above, we want to deploy our systems in a just in time fashion; there’s no reason to have idle assets sitting around diluting shareholder value.  Maybe we put 1.5x of our peak capacity in production if we are a moderately high growth company and 5x our peak capacity in production if we are a hyper growth company.   Asset costs are high, and other costs range from low to medium.  Total costs tend to be highest for this category as to deploy 100x of your necessary capacity relative to demand would kill many companies.

The focus of your scale factor (20 to infinity, or 1.5x to 3x moving from design to deploy) varies with your rate of growth.  If you have 10x annual growth, you are going to want to be on the high end of our ranges whereas if you have 10% annual growth you can be on the lower end.  The chart below summarizes the discussion above.

One of the processes that we often recommend to clients is known as a Scalability Summit. The purpose of this summit is to identify which component in your application is most likely to prevent you from scaling. This idea of fixing the next bottlenecks or the next thing that is going to prevent you from scaling is how YouTube.com scaled. You can see a presentation by Cuong Do at a Google Tech Talk. About three minutes into the video Cuong expresses his algorithm for “Handling Rapid Growth” as:

while (true)

{

identify_and_fix_bottlenecks();

drink();

sleep();

notice_new_bottleneck();

}

YouTube’s growth was so rapid that this cycle of identifying the next bottleneck and fixing it was often weeks or even days. For all other “normal” scaling issues performing this bottleneck identification is usually done on a quarterly basis. When done at this interval we refer to them as Scalability Summits. We recommend that a select group of individuals should be invited to participate and discuss what they believe to be the next set of issues the platform will experience. The participants should include people representing architecture, operations, engineering.

When we run Scalability Summits we generally will go through this exercise twice. Once for the expected growth rate of the business and then once again for the expected growth rate multiplied by 10. So if you plan on growing by 200,000 users over the next quarter use that number first then use 2,000,000 users and identify which components would fail at those usage numbers.

Once these potential bottlenecks are identified they are prioritized by a return on investment analysis that takes into account factors such as how expensive it is to fix (in terms of both capital expenditure as well as personnel), the component’s Time To Break (how much growth it can sustain), and the severity in the event it does break.

The most important step comes after the Scalability Summit. A set amount of labor from each team must be set aside to focus on scalability related issues that come out of the summit. If a team spends several hours identifying bottlenecks that get ignored no one is going to participate again. As an organization you must take action on these or 1) you will likely experience these issues that will hamper your growth and 2) participants will lose interest in the process.

One of our earliest post was the Path To CTO/CIO, where we focused on not only the “path” but the path that would make you successful once you arrived in that position. One of the necessary skills that we mentioned you must gather along the way is “great technical experience”. We promised to revisit this topic in a later post so I thought I’d come back to this question of how technical does the CTO/CIO need to be? This is especially relevant for those individuals coming from a non-technical background but I think it is a question often asked by technologist as well. Do you need to have engineering and operations experience? Can you come from QA and become a CTO? Do you have to know how to code?

CTO and CIO jobs come in all shapes and sizes. In some businesses the CTO is the chief architect and not a manager, in others it is the VP or SVP of all technology teams. For the purpose of this discussion I’ll define the CTO/CIO as the role that has the technology organizations (such as engineering, quality assurance, operations, etc) reporting to them.

To be upfront about answering the question in the title of this post, I think a CTO should be very technical. I don’t think there is a prescribed path to the top technology job in a company and you don’t necessarily have to come up the technology ranks. I do, however, believe that possessing certain technical skills and experiences are far more likely to land you in that role than if you do not have them. More importantly, while these skills and experiences won’t guarantee your success in that role, they lack of them almost ensure problems or even failure. The skills and experiences that I mentioned fall into two categories, broad and deep.

Deep experiences and skills are ones that are most likely gained early in your career and should bring you proficiency in a subject area. For some this might be programming in a specific language, automating testing on a specific tool, or administering an operating system. If you believe Malcolm Gladwell in Outliers this process takes about 10,000 hours. Thinking of this in terms of travel or language, these deep experiences and skills are the kind you gain by living in a foreign country and becoming immersed in the culture and able to speak the language fluently. Deep experiences and skills are important because they develop in you a strong knowledge foundation that can be built upon when broadening your experiences. This deep foundation allows you to learn other technologies easier, similar to how proficiency in one foreign language makes the next one easier to learn. These deep experiences also give you a base of confidence that when peered with other experts provides credibility and when faced with uncertainty provides a history of solutions.

Broad experiences and skills are ones that are somewhat superficial but serve to give you a general understanding. Continuing our travel and language analogy, broad experiences and skills are the ones you acquire by spending a few weeks in another country and being able to get by asking for directions and food. The broad experiences that a CTO/CIO should have are working with multiple technology disciplines (engineering, quality assurance, architecture, operations, etc.) as well as business disciplines (marketing, finance, legal, etc). These experiences should serve to give you an understanding of their responsibilities, their day-to-day jobs, and most importantly their perspectives on technology and product development. You don’t have to have a job in each of these departments to gain this experience. Other ways to gain these include, acting as a liaison, serving on joint boards, working together on special projects, or volunteering to stay late to help the other teams accomplish their work.

Perhaps not prerequisites but rather as identifiers that will set you apart and prepare you well for the top technology role, look for establishing first deep skills and experiences. Once that foundation is firmly built then begin to broaden those through interdisciplinary work. Don’t forget that this is focusing of the technical skills and experiences. There are still other skills such as leadership, management, communication, and business that must be developed as well if you not only want the top technology job but want to keep it and do a great job while you are there. It’s not unusual for the most technical CTO’s to be the ones who need the most management and business coaching.

Having thrown down the gauntlet that a CTO must be technical it is only fair to address those who didn’t rise up through technology roles and are currently CTO’s or desire to be CTO’s.  We’ll save this for a future post but in the mean time break out a coding book.

If you are curious about the food that you eat a couple of books that I’d recommend are The Omnivore’s Dilemma and In Defense of Food both by Michael Pollan. One of the key pieces of advice to his readers is “Eat food. Not too much. Mostly plants.” Admittedly, I don’t follow that advice but it sounds reasonable. The other interesting point in his books is how we have arrived at so much highly processed food. As Michael explains it this is because the food chain wants to grow more than the 1% per year population growth and therefore by processing the food more they can charge more and achieve faster growth. The result is not only higher prices but food with much higher caloric values. This “value add” by the members of the food chain reminded me of technology vendors.

A product (hardware or software) perhaps starts out as a unique technology that is innovative and sells because of this. Over time this product becomes less unique and eventually commoditized, everyone sells essentially the same product only differentiated by price. High tech companies generally don’t like selling commodity products because the profit margins continue to get squeezed each year. The way to prevent this is to add features to the product, sort of like making highly processed food products. The more “value” the vendor adds the more they can keep the price high. What we as the consumers are left with is highly processed products that are not what they were originally designed for.

What’s the problem you say? What’s the harm in a piece of network gear that does it all?  It routes, load balances, firewalls, manages sessions, and even can have business logic. The problem is that it was more than likely designed to do one of those things and when the vendor sells it to us as a “do everything” panacea we’re setting ourselves up for problems. This is not to say that it’s never okay to dual use hardware or software or that you can’t use added features of a vendor’s product. What we do say is that the more you rely on these the more you are tied to this vendor and the more likely you are to have problems with that product.

We have a term that we use with our customers to explain this, we call it “purpose driven hardware/software”. The general principle is to use the hardware or software for its original purpose, a load balancer as a load balancer and a database as a database. Use an X, Y, or Z axis split to scale your database, don’t rely on a vendor for scale. Following this principle will 1) allow you to remain vendor agnostic, achieving the best pricing 2) rely on yourself for scaling not a vendor who may or may not release the next version/bug fix when you need to scale and 3) simplify your architecture which leads to less human error when everyone understands how the system works.

Researchers at UC Berkeley have outlined their take on cloud computing in an paper “Above the Clouds: A Berkeley View of Cloud Computing.“ They cover a lot of material in this paper and it’s well worth reading.  Section 7 was particularly interesting to us because it covers the top 10 obstacles that companies must overcome in order to utilize the cloud.  According to them these are:

• Availability of service
• Data lock-in
• Data confidentiality and auditability
• Data transfer bottlenecks
• Performance unpredictability
• Scalable storage
• Bugs in large distributed systems
• Scaling quickly
• Reputation fate sharing
• Software licensing

These look very similar to our top five concerns that we outlined in our article on Venturebeat.com.  Our list was:

• Security
• Non-portability
• Control (availability)
• Limitations (non-persistent storage) 
• Performance

Their article concludes with “Although Cloud Computing providers may run afoul of the obstacles …we believe that over the long run providers will successfully navigate these challenges…” They continue saying “Hence, developers would be wise to design their next generation of systems to be deployed into Cloud Computing.”  

We agree and reiterate our conclusion from “The Cloud Isn’t For Everyone”

“Of course, most importantly, we should all keep an eye on how cloud computing evolves over the coming months and years. This technology has the potential to change the fundamental cost and organization structures of most SaaS companies. And as cloud providers mature, we’re sure they’ll address our top five concerns, becoming more viable companies in their own right.”

In the spirit of the New Year, we thought we would share our list of top things that you should consider putting on your technology’s roadmap for 2009.

1. Develop the ability to rollback: If you can make only one change to your product and process in 2009 and you don’t currently have the ability to rollback, this should be at the top of your list.  Being able to push code changes and then pull them back from production in the event of a problem will save you more customers and more effort than any other single item.
2. Break changes into smaller pieces: There is almost never a need to redesign the entire site or service at once.  Break it into parts and take it one piece at a time.  This will be lower risk and give you an opportunity to learn along the way.
3. Remove SPOFs:  Commit to removing all single points of failure in your architecture.  Single servers, firewalls, load balancers, power supplies, etc should all be listed and tackled one at a time until they are all eliminated
4. Remove synchronous calls:  Having one service call another service in a synchronous manner causes a multiplicative effect of failure.  Five synchronous calls on servers with five 9’s availability (99.999% uptime) leads to a maximum of 99.995% for the system.  Eliminate synchronous calls wherever possible and create fault-isolative architectures to help you identify problems quickly.
5. Incent a culture of excellence:  Hire the right people and hold them to high standards. Set aggressive yet achievable goals and motivate them with your vision. Be a leader.
6. Develop a disaster recovery plan: Disasters happen, no one expects an entire data center to be down but things like that happen.  Plan on it and start making changes today to keep your services up and running in the event of a disaster.
7. Develop quality into the product from the start:  Don’t expect QA to ensure quality is built into the product.  We’ll post more about this in the New Year but quality starts much, much earlier in the product development life cycle.
8. Split your application or database:  Start this year thinking about how to split your application and database.   We recommend our cube model for both because working on all three axes gives you unlimited scalability in both your app and database.
9. Start Logging:  As we discussed in a recent post, start logging your application data but follow our three key guidelines – 1) logging must not impede the performance of the application 2) use a common framework and 3) look at the data
10. Celebrate your success:  Take time now and throughout the year to look back and congratulate your team and yourself.  If you want to foster creativity on your team, celebrating victories is a great way to keep the energy up on your team.  If you are getting ready to present your 2009 goals to your team, we recommend that you start by focusing on the amazing accomplishments that you have had in 2008.

Wishing you and your teams a great New Year!

-The AKF Team