AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Tag » Risk

5-95 Rule

Previously, I wrote about mitigating risk in the face of uncertainty. I suggested that an agile development model was one way successful companies have been able to mitigate risk. In that post I compared the similarities between organizing into “Scrums” to how Army Special Forces organized in order to succeed in uncertain operating conditions. In addition to the way a company is organized, the way a company approaches project planning can mitigate risk. One of the most often overlooked aspects of project management is contingency planning.

Army Green Berets are fond of the saying, “No plan survives contact with the enemy,” a quote attributed to the famous Prussian General Helmut Von Moltke. Von Moltke and his Prussian contemporaries were brilliant strategists and sought to perfect the “Theory of War.” Their ideas still serve as basis for our doctrine today. Moltke understood that battles can be broken down to a near infinite set of complex options, each of them in turn having still more options depending upon the enemy’s response. A plan only goes so far, at which point the enemy ultimately casts his vote. Von Moltke provided his subordinate commanders with his intentions (a method still taught today in military basic leadership courses), and held them responsible for extensively preparing for all plausible contingencies.
von_moltke

As a Special Forces Detachment Commander, once my team received our orders, we immediately began to plan. Along with the mission, we were given a specific deadline by which we needed to brief our commander. Regardless of the deadline, whether it was 6 hours or a week, we spent approximately 5% of the time coming up with a solid, practical, and safe plan. We spent the other 95% of the time “war-gaming” the plan. During the “war-gaming” portion, we would go through every step of the plan from the time we left our barracks to the time we got back. We thought of every possible thing that could go wrong. What if only two helicopters showed up and not four? What if the helicopter couldn’t land where we wanted it? What if someone got injured? What if more bad guys were on the objective than we had originally thought? We would break our mission down into distinct pieces, or milestones, and would work in teams to come up with solutions to every possible contingency. We would collaborate with our partners, the pilots for example, and let the subject matter experts take the lead in their scope of responsibility. Time permitting, we built mock-ups of the buildings we anticipated operating in, and conducted detailed rehearsals. The rehearsals revealed other contingencies we hadn’t planned for, such as equipment that was being carried by the wrong guy or individuals who needed to work together that weren’t located near each other. We did this because we knew the battlefield was going to be fluid and dynamic. Being prepared not only enabled us to be successful, it saved lives.

While the tech industry doesn’t deal in life or death, although it certainly might feel like that at times, Von Moltke’s wisdom still applies, especially in complex project development. What if equipment doesn’t arrive on time during a data center build? A critical engineer or developer gets sick during a launch or gets hired away? AKF Partners is a company heavily influenced by veterans and our collective experience on and off the battlefield. We encourage our clients to practice the AKF “5-95 Rule” in their Agile methodology. Having a solid plan is a great start but understanding the possible variations in the project’s execution will help ensure the projected is delivered on time and within the budget. Remember the AKF “5-95 Rule” and spend 5 percent of your time planning and 95 percent of your time developing contingencies.


1 comment

Risk Mitigation

We define risk as being comprised of three components: severity of impact, the probability of occurrence, and the ability to detect the event. The combination of these provide an overall risk assessment of a failure mode or a manner in which something can fail. We often apply this formula to code releases or for a more granular approach, we apply it to individual features within a release. By identifying the ways (failure modes) in which a feature could fail in production and giving that a score based on these three factors, we can quantify how much risk we have. We often teach this technique as a FMEA (failure mode effects analysis).

Risk = severity * probability * inability to detect

The most typical approach to risk mitigation has been an attempt to reduce the risk by reducing the probability. This is often done through rigorous requirements definition, thorough design reviews, and most often, lots of testing. The problem with this approach is that no matter how good we are, bugs will slip through. Users have different configurations than we have in our test environments or they use the product differently than we expect. A more recent approach to reducing the probability has been to deploy smaller changes. This was done by reducing development cycles or sprints to 1-2 weeks or in the extreme, employing continous deployment. The theory being the smaller the release, the fewer changes, and thus the less probability of failure.

A different approach to risk reduction and mitigation is by attempting to reduce the severity factor. There are several ways in which we can attempt to do this. The first is through the use of monitoring. By monitoring business metrics (checkouts, listings, signups, etc) we can quickly identify if there is a problem. Continuous deployment requires a rigorous approach to monitoring in order to quickly identify the problem and rollback the changes, thus reducing the severity or impact of the problem to your customers.

Another approach to reducing the severity is by pushing code changes to a small set of your users. Ideally this should be done through “swim lanes” but it can also be accomplished manually. In a process that we call “incremental rollout”, you would deploy new code to a small set of your servers (1-5%) and watch for issues. Once you’re satisfied with that there are no issues, roll to a larger set of your servers. Continue this “roll, pause, and observe” cycle until you have the release completely deployed. Teams that employ this strategy often take days to deploye code changes but by doing so they have much less risks of a customer impacting problem.

There are lots of ways to approach risk mitigation and we should continue to add these approaches to our toolkits. Some approaches work better than others based on the team culture and product or service being offered.


1 comment

Sensible Security

This post is the last in a 3 part series and will cover the last 2 points from our post entitled the Top 4 Failures in Corporate Information Security.  Here we are going to focus on why firewalls aren’t always the best solution to your problems and how to use your security team properly in your risk making processes.  We’ll end with a quick review.

Firewalls Can Be Bad Too

Firewalls can absolutely be overdone.  In fact, in our experience they are very often overdone.  Often firewalls are cited as being necessary to be compliant with certain regulatory requirements or industry standards (such as PCI compliance).  Sometimes companies feel they must put them in place simply because similar “comparison companies” have installed them.  Many times the driver of this need isn’t as much the requirement, standard or “comparison” company as it is misinformation on the part of firewall vendors or decisions made without complete information.

Firewalls, besides not being free either in terms of labor or capital (obviously), almost always reduce your availability and decrease your flexibility.  Like any other piece of hardware and software, they fail from time to time.  These failures often either lead to idle employees who cannot perform their work or even worse, the turning away of revenue generating customers from certain functions on your site.  There’s no way around it – if you put a firewall in the way of a transaction sooner or later it will cause a problem.  Sometimes this is both acceptable and advisable, such as the additional protection that a firewall provides a database that stores PII information such as credit cards.  Other times, it is just an unfortunate cost and burden such as when firewalls are used to protect static image servers that have very little valuable information on them and which are of little interest to money-focused bad guys.  And finally they can really harm employee productivity by stalling business initiatives.   It’s not unusual to spend thousands of dollars of labor several times a year troubleshooting why a new service won’t work or an why an old service quit working  before identifying that a port in a firewall needs to be opened or was recently closed.

Security Teams as Contributors – Not Decision Makers

Your security team very likely has a lofty and aggressive goal – to keep your company, your systems and your data (or your customer’s data) free from being abused by bad guys.  This goal doesn’t come cheaply and the only way to guarantee it is attained is to either go out of business or spend so much on your risk adjustment initiatives that you will never make a profit.

The security team rarely has the business background and overall business context to make business tradeoffs when it comes to risk.  While they may in fact have a number of people with advanced business degrees, their focus on reducing risk means that they are not focused on maximizing profits within the context of all of the available business levers.  And you may not want them to have such a broad business focus as some practitioners argue that you want your risk team focused singularly on the available risk options rather than making the risk tradeoff decisions.  The bottom line here is that the team should be involved in the decision process, but they are not necessarily the best decision makers for your risk management options.

Acting Sensibly

Treat your security and risk initiatives as you would your personal property and valuables.  Lock up and keep out of sight those things of significant value, but retain enough flexibility to allow you and your team to do your jobs quickly.  You probably don’t put deadlocks on every bedroom in your house as it just doesn’t make sense and you probably don’t need to put firewalls on every LAN segment in your network for the same reason.  Add passive detection advices such as intrusion detection systems to increase your level of security.

We covered four failures in corporate information security:

1)      Fear rather than Risk and Profit driving decisions

2)      Teams not understanding financial drivers of the “enemy”

3)      Overemphasis on Firewalls

4)      Security decisions made by the wrong team

By understanding what motivates your enemy, approaching security with risk and profit rather than fear as a driver, acting sensibly when it comes to risk mitigation and making risk decisions at the appropriate level you can both decrease risk and increase profitability.


Comments Off on Sensible Security

The Financial Drivers of Security

Our last post, Top 4 Failures in Corporate Information Security, kicked off a 3 part series on security addressing some of the most common themes from our work with clients.  This post will cover the first 2 failures from our last post in greater detail.  The first section will focus on why financial concerns, and not fear, should drive your security decisions.  The last section focuses on the financial motivations of potential thieves and the ramifications to your security architecture and design.

Focus on Finance (or profits) and not Fear

Has your security team (or have you) ever presented a project justification that something has to be done “or else we will be horribly exposed?”  Or maybe the proposal was worded such that the project must be done or you risk “irreparably tarnishing our brand”.  Or how about “Our front doors are basically wide open, nearly anyone can walk in and take whatever they’d like”.  The problem with all of these statements is that not only are they difficult to prove or disprove, they are positioned to elicit a fear response for the purposes of attaining a goal.  How does one quantify fear and evaluate it against other business initiatives?  Our position is that one can’t and that one shouldn’t.

Our jobs as managers and executives are to make sound business decisions that maximize shareholder wealth.  The appropriate management of risk is an example of such a decision.  We spend money on risk management initiatives to offset potential future losses associated with the realization of that risk.  We might invest in fraud detection systems for instance to reduce future potential losses.  In doing so we pay the expense and capital of putting such a system in place, and potentially lose some revenue through the “false positive” identification of fraud within our revenue stream in order to significantly reduce the amount of real fraud going on within our systems.  Similarly, we might decide to put firewalls in certain places to reduce the probability of a penetration and associated brand damage at the expense of the labor to put those firewalls in place, the capital to purchase the firewalls, and the decrease in availability and scalability those firewalls might present.  Those firewalls also might slow our time to market for certain initiatives or increase the cost of those initiatives by adding steps in order to put new rules in place for new applications, etc.

On a project level, the point at which we should stop adjusting risk in any given area is the point at which the incremental cost of effort of risk adjustment exceeds the incremental value.  On a portfolio level, the cost and value of the risk adjustment above should be compared to all other capital and effort based projects.  Just because the project has a return, doesn’t mean it is the best use of our time and resources.  So, if we add a $10M fraud system that is only likely to return $8M in total benefits over 3 years have we made the right decision?  What if it returns $10M in 3 years?  The point here is that the initiative should be couched in business terms and compared appropriately against other business initiatives in terms of its financial benefit.  Don’t let fear motivate your decisions.

Bad Guys Like to Make Money Too

Sun Tzu is attributed with saying “If you know both yourself and your enemy, you can win a hundred battles without a single loss.”  How well do you know your enemy?  While some of your enemies are out to brag about their accomplishments , a majority of your enemies are out to make money.  The people who perpetrate technology crimes are generally skilled and intelligent (though morally bereft) people who see the perceived benefit of stealing your data as being significantly greater than the perceived cost.  It is this equation that we are going to address in this section.

In our equation Perceived Benefit (PB) > Perceived Cost (PC) the word “perceived” is very important.  We need elements in our security architecture that decrease the perceived benefit of a potential security breach.  This might be one-way encryption of sensitive data such that it can’t be used by someone stealing it, or it might be hiding our data and valuables so that “passers-by” don’t ever perceive any value in attacking you.  Maybe you can develop marking technologies for your data or “beacons” such that the data can be tracked if used.

Elements of perceived cost include the perceived cost of obtaining the data and the perceived cost of getting caught.  This implies that not everything need to have the same “actual” cost of protection as it makes little sense to spend money protecting something that has little perceived value.  The perceived cost of getting caught is at least partially influenced by your track record with catching would-be thieves as well as how well you publicize your successes.  If I am choosing between attacking site A and site B, each of them equivalently physically protected and of equivalent value to me, I will likely choose the site that appears to me to be the least likely to catch or prosecute me.

In our next post, we will discuss who should make security decisions and why firewalls aren’t always a good thing.


Comments Off on The Financial Drivers of Security