DENOG17

At Hetzner we’ve historically used an Open vSwitch based data plane for connecting hundred thousands of cloud servers to the network. This has served us well for many years and mostly still does. We have however reached some limitations and wanted to improve scalability, resiliency and flexibility with a more specialized data plane that's tailored to our needs while being easy to operate and building a strong foundation for new features.

When checking our options back in 2022, the team reached the conclusion that the best path to achieve this goal is to build and maintain our own highly specialized networking stack based on eBPF/XDP, and so we went on a journey to make it reality.

Today, roughly three years later, we’ve implemented a versatile network stack, called hcnet, which handles public and private cloud networking (using VXLAN encapsulation), stateful firewalls, and provides DHCP services as well as traffic capture tooling - all of this using XDP with a control plane written in Go.

To make operation’s life easier, the stack is collecting and exposing meaningful metrics and is designed to self-heal whenever possible. We’ve been using hcnet in our internal cloud for two years now, with every new feature getting its first real-world tests there on a daily basis, including customer-facing applications. We are looking forward to a public beta, once we have full feature parity with our existing stack.

In this talk we want to provide an overview of how we’ve built the new network stack, what challenges we’ve faced and where we're hitting current limitations of XDP. As of today the most pressing challenges are support for offloading and driver maturity in general.