Log Every Change
In well run technology organizations, any event that has the potential of impacting customers will trigger an alert that brings a cross disciplinary team together in person or on the phone to start troubleshooting the potential (or actual) problem. Ideally the person responsible for running the incident management and problem resolution process will ask what most recently changed and then listen (or read) as the operations team reads (or displays) the change log. We often joke that you only need to wait for someone to say “Yeah, but that change couldn’t possibly have caused this issue” to find the root cause and fix the problem.
In our experience, changes are one of the most common cause o f customer and revenue impacting issues. Sometimes these changes are feature enhancements or functionality additions, and sometimes they are infrastructure or architectural changes. Very often, they are simple configuration changes like an addition of a range of IP Addresses to an access control list, or the modification of DNS. In some companies, these changes (identified as any modification to a production environment other than that made by the actual software or system itself) happen at a rate of several thousand per day. It is virtually impossible to track them unless a change logging system is put in place. Very often, it is the change that is undocumented and therefore difficult to isolate and roll back that costs the company the greatest downtime or revenue.
Too many companies allow too many changes to go undocumented. The most commonly cited reason for a lack of change logging is that it simply takes too long to log each and every change. But change logging doesn’t have to be cumbersome and it need not always include the notion of risk management inherent to a change management system. Just the logging of a change for later identification can save between hundreds and millions of dollars of revenue and hundreds or thousands of customers – especially in a SaaS environment. Something as simple as always logging the time, date, reason for a change, person making the change and the system being modified can make a world of difference. Many web enabled tools offered by companies like Service-Now make such logging very simple. Most tools offer smtp interfaces that allow people to make a change and email it to the system. For a minute or two of time per change, hours can be saved in customer impact.
Log your changes – every change, every time.