This article is one of a series of security anti-patterns commonly found in our experience and during technical due diligences for investors, and workshops with our clients.
A security architectural anti-pattern we almost always see with private hardware is multiple layers of firewalls (and with older companies, we also see this mimicked within cloud infrastructure as well).
Having spent much of my career in the physical security world with a CPP certification and multiple audits for SOC2, PCI, and others, I understand well the concept of concentric circle protection.
The problem, however, is that having multiple layers of firewalls isn’t concentric security. It is just layers and layers of the same thing with added latency, cost, support obligations, and trouble-shooting complexity. At the end of the day, a firewall is only as good as the rule sets and the team maintaining the rules, and having more of them with the same rules doesn't provide additional security.
An iconic TV moment depicted in the meme above was when US TV personality Oprah convinced the US car maker Pontiac to give a free car to everyone in the audience. Additionally, Oprah footed the bill for the registration and other fees. But the gift tax required by the US government was not covered. Many people went to the press to complain that the “free car” wasn’t free (even though they could have taken a cash payout instead of getting the car).
When you dole out firewalls throughout your infrastructure like a bag of candy on Halloween, each one comes with a gift tax of latency, multiple effects of failure, unknown circuitous routes, and higher cost to maintain and upkeep.
We also covered this anti-pattern in our book Scalability Rules in Rule #15: Firewalls, Firewalls Everywhere!
definition
The firewall frenzy anti-pattern occurs when there are multiple layers of firewalls within an infrastructure architecture which do not offer additional security.
Why firewall frenzy is a bad idea
Unlike physical security concentric circles, adding layers of firewalls doesn’t necessarily add additional value. An even worse variant of this anti-pattern we see is that the layers of firewalls aren’t from the same manufacturer, are a mix of cloud and on-prem hardware and VMs, and/or different teams manage each layer. Complexity makes it difficult to understand the true level of security. As the number of firewalls increases, the combinations and
permutations make it hard to declaratively state a secure environment exists. As a result, we must rely on testing alone to ensure all is safe as we cannot prove it due to complexity.
To illustrate the complexity caused during trouble shooting, we’ve seen multiple clients turning at least one of the pairs of firewall's rules to “any/any” because a different team is managing the firewall rules and no one person/team knows all the configurations deployed. And this “any/any” stays on for a long time … so the firewalls deployed in high availability pairs are consuming electricity, requiring cooling, and adding some latency but “unlocked” and not providing any additional protection.
Alternately, we've seen teams copy/paste rules from the gateway firewall to the new set of firewalls – again providing no additional protection while adding latency and complexity. However, they forget to copy/paste when changes are made ... back to configuring "any/any" override temporarily (or in one case multiple years ...) until they have time to troubleshoot the problem. Firewall rules tend to grow organically based on demands when they should be calculated.
“Have firewalls” is just checkbox compliance in this case.
Common Denominators of Firewall Frenzy Anti-Pattern
- Everything is behind a firewall, including static images and information that is neither confidential nor needs protection.
- Security response to why everything is behind a firewall fails to take into account business requirements and usually ends with “because that is our policy.”
- Security teams do not have cloud experience and/or engineering teams may also lack cloud experience.
- Team lacks an experieced chief architect (or lacks the position altogether) who understands the entire information flow and how different teams and components interact.
- Security team is a separate team and not embedded in delivery teams and require ticket submission by engineering instead of partnering for success.
- There are cross data center/cloud environment calls.
- The architecture is aged and in need of being refreshed.
- Delivery teams are neither cross-functional, nor autonomous and empowered to make decisions.
Having multiple firewalls between the end user and the needed data is akin to having multiple locksets on a door with the same key for all of the locks. It takes more time to open the door, but doesn't add any customer protection or value.
Correct Pattern
Security should:
- Work closely with architects to separate out information by security level.
- Rationalize all data traversing a firewall and only send requests for secure information through a firewall.
- Do not put static images and other non-confidential information behind a firewall, cache at the edge.
- Create read only replicas for any data needed outside of the website hosting environment – DO NOT make cross-data center calls. This reduces availability, adds latency, and increases the security cost and risk environment.
- Appropriately size the main highly available firewalls.
- If multiple firewalls are needed to keep attack vectors/surfaces smaller, load balance calls out to each area with their own dedicated firewall, but all calls should be contained within that security zone and pair of firewalls.
- Send security engineers and architects to cloud training if needed and/or (for smaller organizations) ensure chief architects have modern cloud training.
Summary
Successful security should make sense and have business logic behind each decision – and be made as a team. This engenders trust between information Security and the business in healthy organizations and security moves from checkbox compliance and ticket-based approach to becoming a part of the company culture – the holy grail of successful security efforts! We focus on many things, but can help get you pointed in the right direction if you are having security problems. Give us a call, we can help!