Migrating More than 250 Billion Daily Connections to AWS Network Firewall

Early in its history, AWS hosted services in a traditional data center environment using a dedicated physical network. It extended this network in the cloud in 2009 and 2011, with the introduction of Amazon Virtual Private Cloud (Amazon VPC), a service used to define and launch AWS resources in a logically isolated virtual network, and AWS Direct Connect, which establishes a dedicated network connection to AWS. In the years since, AWS migrated nearly all its services and network functions to the cloud. However, some functionality remained in the physical legacy network, including firewalls protecting specific network paths.

The time required to procure, deploy, and manage the hardware firewalls was substantial. Over time, AWS used different hardware firewall platforms, each with unique challenges ranging from monitoring to configuring resources. AWS also needed dedicated teams for firewall management, which added to the operational overhead. Scaling firewalls became increasingly difficult as the company grew. Each change was time-intensive and expensive to maintain because many services shared the same firewalls.

With the introduction of AWS Network Firewall in late 2020, the AWS Networking team had the tools to migrate all remaining internal traffic passing through hardware-based firewalls—totaling 10 Tbps—to native AWS Cloud services.

AWS began planning its migration to AWS Network Firewall in early 2022. To support the migration, the team collaborated with various AWS product teams, developing new capabilities to reduce management complexity and costs for both AWS and its customers who rely on NAT gateways and AWS Network Firewall services. For example, AWS used multiple IP address support to minimize the number of NAT gateways needed in larger environments. AWS also added Transmission Control Protocol (TCP) Reject and TCP reset functionalities to AWS Network Firewall, improving the performance of latency-sensitive applications by eliminating prolonged TCP time-out periods during traffic shifts or network failover.

By 2023, AWS successfully completed one of the largest migrations in its history, shifting TCP/IP traffic for the internal microservices that power various parts of AWS to AWS Network Firewall. Within a few months, it migrated more than 10 Tbps of traffic and more than 250 billion connections per day globally to the cloud. A total of 2,600 NAT gateways and 1,300 AWS Network Firewalls were deployed.

In the new cloud architecture, AWS adopted a strategy of deploying multiple small firewalls dedicated to specific network segments instead of using large firewall clusters to handle all traffic. This approach provides granular visibility into each service and shrinks the size of the fault domain from the entire Availability Zone to individual services within the Availability Zone. “Traditional firewall-monitoring technologies would show only one great body of traffic, making it challenging to distinguish the individual voices,” says Andy Lemin, senior development engineer at AWS. “Using AWS Network Firewall, every customer has its own firewall. When a customer has an issue, the logs show the individual voices within that customer’s traffic. This significantly improves fault diagnosis and identification.”

The implementation of a more scalable infrastructure means that, as AWS grows and onboards more customers, the firewalls can expand without overhauls or downtime. AWS Network Firewall infrastructure automatically scales and provisions resources on demand, eliminating the need for customers to plan for firewall capacity. Additionally, AWS Network Firewall uses the data from these workloads to optimize the automatic scaling process, providing better functionality for everyone.

Furthermore, cloud technologies facilitate agile updates. AWS can bypass the support ticket process with external vendors and solve potential issues in house, which has resulted in a 90 percent reduction in networking tickets. This allows the company to rapidly adapt and optimize AWS Network Firewall in response to evolving network challenges. This migration has not only reduced costs, time, and effort for AWS but also accelerated its pace of innovation.

“Expanding capacity in hardware firewall projects often took 6–12 months. With rigid, expensive hardware firewalls that require specialized equipment, migrating and scaling involves additional steps that don’t exist when using AWS Network Firewall,” says Wade Millican, senior manager at AWS, whose team is responsible for operating the company’s internal firewalls. “All these tasks—from procuring hardware to installing multiple racks—are now irrelevant. Using AWS Network Firewall, we can easily and proactively scale within hours, so we can focus on operating the business and deploying solutions.”

Now that it has migrated to AWS Network Firewall, AWS has minimized operating costs while simultaneously increasing network resiliency, capacity, flexibility, and observability. Looking ahead, the company plans to use insights from this migration to improve its services and add new features for the benefit of its customers.

“What AWS Network Firewall gains from this new workload is a direct connection to a set of internal customers with their own highly developed operational monitoring,” says Jamie Lavigne, senior software development engineer for AWS Network Firewall. “By working directly with internal AWS service owners, we have gained new insights into the long tail of performance-related challenges and how these specifically relate to the details of end-user applications. We will launch new capabilities in 2024 based on what we have learned through these deep dives.”