Firezone logo light
Thomas Eizinger
Thomas Eizinger

Pushing WireGuard on macOS to 5+ Gbit/s

A story of networking fundamentals and what we learned about the XNU kernel.

We are on a never-ending quest to offer the fastest WireGuard VPN possible, on every platform. Historically, performance improvements to VPNs have often been limited to Linux as it offers the most versatile and most optimized APIs for fast I/O. We'll get into the details as to how that matters later in this post.

Much to the dismay of many Linux enthusiasts, even 2026 isn't really the year of the Linux desktop. Especially in the enterprise world where VPNs are used extensively to protect company networks, most users connect from Windows or Apple devices. As such, when building a fast VPN you really ought to focus on Apple and Windows devices too, and in our humble opinion, your VPN shouldn't limit you in how fast you can transfer data. If you have a 5 Gbit/s Internet connection, you should be able to use that completely.

Unlike our Linux and Windows builds, Firezone on macOS and iOS wasn't performing as well as we would have liked, achieving 1 Gbit/s download speed at best and upload speeds often being crippled to a few hundred Mbit/s. The problem got worse with higher latency, which was a hint that packets might be getting lost somewhere.

In this post, we are going to talk through how we optimized the Firezone macOS Client from around 1 Gbit/s to over 5 Gbit/s. It is a mix of networking theory and hunting through the XNU source code to uncover how exactly TUN and UDP work on Darwin, the core of Apple's operating system that macOS and iOS build upon.

What even is the maximum throughput?

A network like the Internet is built from many intermediary devices and paths. Unsurprisingly, the maximum throughput you can get between two endpoints is limited by the slowest device or link along the path. What is perhaps surprising is how we even know what the maximum throughput is. Throughput isn't something that gets advertised and then both ends of the connection adhere to it. Instead, throughput is something you measure and adapt to dynamically.

The Internet Protocol (IP) was designed to tolerate packet loss. An IP packet is sent as fire-and-forget and it is the job of protocols built on top of it to deal with lost or untransmitted packets.

Protocols like TCP and QUIC therefore come with a congestion controller. It is the congestion controller's job to track what got sent, what the remote received (i.e. acknowledged), how long it took to send the data and from there calculate how fast it should send data. This continuous feedback loop is the reason why a speed test doesn't just immediately report your maximum speed and it first has to "warm up".

Essentially, the congestion controller starts with a default window size, sends the data, waits for it to be acknowledged by the remote, measures the round-trip time (RTT) and then starts to increase the window size. The bandwidth-delay product defines how these variables interact. At an RTT of 20 ms, a window size of 1 MB gives us an upper bound of 400 Mbit/s of throughput. Increasing the window size to 2 MB allows for a maximum throughput of 800 Mbit/s.

By varying the amount of data that is in flight (and thus unacknowledged), the congestion controller steers the speed of the transfer.

On modern Windows (10+), Linux and macOS, the default congestion controller for TCP is CUBIC. CUBIC's algorithm is loss-based, which means it will reduce the window size (and thus the send rate) as soon as it concludes from the remote's acknowledgments that packets got lost.

In TCP for example, this happens as part of the TCP header. TCP numbers every byte it sends and the receiver's acknowledgment says "I have everything before byte N". Based on this feedback, the sending side knows which packets are missing. If the gap doesn't close within a short reordering window, it retransmits them and the congestion controller treats the gap as loss.

TCP acknowledgments around a lost segmentA sender and a receiver exchange numbered segments. Segment 1 arrives and the receiver answers ACK 2: it has everything before segment 2. Segment 2 is lost on the way. Segment 3 arrives, but the receiver's ACK stays at 2 because segment 2 is still missing. The sender later retransmits segment 2. Once it arrives, the receiver answers ACK 4: it now has everything before segment 4.SenderReceiversegment 1ACK 2segment 2lostsegment 3ACK 2segment 2, againACK 4
For illustrative purposes, segment numbers are simplified here. ACK 4 doesn't get sent until segment 2 has been received.

This exchange happens between the sender and the receiver of the connection. Devices along the path do not explicitly negotiate a send rate with the sender. They can signal congestion by dropping packets or, with Explicit Congestion Notification (ECN), by marking them. In both cases, it is the sender that adjusts the rate based on feedback from the receiving end.

For physical network devices and links, this makes a lot of sense. An intermediary device has physical constraints in how many packets it can forward based on e.g. its clock speed. Packets arriving faster than they can be processed get dropped, which is how the sender learns that it has been sending packets too quickly.

Given some ramp-up time, this is how the congestion controller finds a send rate the path can sustain by varying the size of the congestion window.

The congestion window can shrink while the link stays busyAn illustrative graph of CUBIC's congestion window over time. The window grows during slow start, then each loss reduces it to 70% before it grows again. A lower reference marks the window needed to keep this link busy. In this example, every reduced window remains above that reference, so queued packets can keep the link busy even while the sender backs off. The graph shows the congestion window, not the throughput rate.timecongestion windowwindow at congestionwindow needed to keep link busyslow startpacket losswindow reduced by 30%
Here, the window remains large enough to keep the link busy after each loss. Queued packets can sustain throughput while the sender backs off.

When CUBIC detects packet loss, it reduces the congestion window by 30%. This sounds aggressive at first but does not directly translate into a 30% dip in the transfer rate. After all, data that is still in flight is still being drained and delivered. The congestion window only controls at what rate we are sending new data.

Most importantly, all of this works based on feedback from the receiver, which is why it takes longer to settle on the effective throughput as latency to the receiver increases.

Losing packets is expensive

What does that mean for an overlay network like a VPN? The congestion controller doesn't know that the interface it is sending packets into is a virtual one. It treats it like any other network device. It is therefore important that we play by the congestion controller's rules: If it controls throughput by measuring packet loss, we better not lose or reorder any packets in our software.

Why does the order matter? To the sender, a packet that arrives late initially looks just like a lost one: there is a gap in the acknowledgments. Networks can legitimately reorder packets, for example when they spread them across parallel paths. That is why TCP waits for a short reordering window before declaring a packet lost. But if a packet arrives later than that, the sender has already retransmitted it and likely reduced its send rate for nothing. So, to get the best performance, we really want to avoid reordering packets and we definitely must avoid dropping packets.

The cost of an avoidable packet dropAn illustrative comparison of a sender's rate over time, with and without an avoidable packet drop. Both rates start below the physical path's capacity. Without the additional loss, the rate continues to increase. When the sender detects the extra loss, it backs off and takes time to recover, even though the physical path has spare capacity. The shaded area between the rates represents data the sender could have sent during that time.timesend ratephysical path capacitywithout additional losspacket loss detecteddata we couldhave sent
An avoidable packet drop makes the sender back off even when the physical link has spare capacity.

If we drop packets while we are below the possible bandwidth capacity of the link, the congestion controller will reduce the window size prematurely. Effectively, this results in lower throughput numbers than our processing speed would allow.

Backpressure beats packet loss

Similar to a physical network link, the code that makes up our virtual one can be thought of as a queue. Packets go in on one side and come out on the other.

Internally, a packet has to go through multiple stages. In Firezone, those stages run in different threads which again are connected by queues / channels.

Firezone's thread layoutFirezone's data plane runs on several threads connected by bounded channels. On the way out, the TUN rx thread reads an IP packet from the TUN device and hands it to the event loop thread, which does routing, policy and NAT. The event loop passes the packet as a job to one of several crypto worker threads, which encrypt it with WireGuard and hand the datagram to the UDP threads (one each for IPv4 and IPv6), which send it to the network. On the way in, the UDP threads receive a datagram and hand it to the event loop, which passes it to a crypto worker for decryption. The decrypted packet comes back to the event loop, which hands it to the TUN tx thread to be written to the TUN device.TUN deviceNetworkTUN rx threadread from TUN fdTUN tx threadwrite to TUN fdEvent loop threadroutingpolicyNATUDP threadsendreceiveCrypto worker threadencryptdecrypt
Each arrow between two threads is a bounded channel.

All of these channels are bounded, which is important for two reasons:

  1. We need to make sure that our overall memory use is bounded and it is not possible to OOM a Firezone Client by forcing packets into it.
  2. Just like the slowest link in a network should slow down the others, the slowest processing stage should signal to the previous stage that it is at capacity.

Bounded channels achieve both of these: If a channel is full, the writing task will suspend and - in Rust terms - register a waker. Whenever the receiver takes out an element and finds a registered waker, it then uses it to notify the sender that there is now capacity for another element.

A processing task that takes elements from one channel and sends them into another will therefore automatically "pass on" the backpressure signal of a slower, later stage.

Backpressure through a bounded channelA sender task and a receiver task are connected by a bounded channel with room for four items. The receiver is the slower stage, so the channel is full. When the sender tries to send into the full channel, it leaves its waker in the channel and parks. The receiver takes an item out, finds the sender's waker and uses it to wake the sender. The sender sends its next item, which fills the channel again, so on its next attempt it leaves its waker and parks once more. This repeats for as long as the receiver is the slower stage: the sender only ever sends as fast as the receiver makes room, and no item is dropped.SenderparkedwokensendingparkedBounded channelsender's wakerno wakersender's wakerReceiverslower stagesendfull: parkreceivewake
A full channel parks the sender until the receiver makes room and wakes it.

Applying this approach consistently throughout the code ensures that no packets get lost within Firezone even when a given stage starts to become the bottleneck.

You might wonder now: Why are we even bothering with this when we've just established that the most commonly used congestion controllers are loss-based?

The reason is latency. Loss-based feedback is delayed whereas local backpressure is essentially instant and has the same effect, assuming we can create a backpressure chain all the way back to the user's socket.

For example, say the user wants to upload a file through the browser to a website that gets accessed via Firezone. What happens underneath is that the browser opens a TCP stream to the server and starts writing to the provided file descriptor. To establish the connection, the kernel's network stack creates a TCP SYN packet and routes it to the interface, which, in this case, is Firezone's TUN device. Firezone then receives an IP packet and performs the necessary encryption and routing to send it out as a UDP packet containing a WireGuard data packet. Once the TCP handshake has finished, more IP packets with TCP segments start to stream in.

One IP packet in, one UDP datagram outAn IP packet carrying a TCP segment comes out of the TUN device. Firezone encrypts it and sends it through the UDP socket as one UDP datagram: an IP header, a UDP header and a WireGuard header, followed by the original IP packet, encrypted, and an authentication tag.TUN deviceFirezoneUDP socketIPTCPdataencryptIPUDPWGIPTCPdatatagthe IP packet, encrypted
Every IP packet Firezone reads from the TUN device leaves as exactly one UDP datagram, with the original packet encrypted inside.

Within the kernel, the TCP socket has a send buffer that gets filled by the syscalls of the browser handling the upload. Data only leaves that buffer once the server has acknowledged it, because TCP may have to retransmit it. Crucially, the kernel rejects the syscall if the buffer is full. On Linux for example, in the case of a non-blocking socket, this results in EWOULDBLOCK being returned as the error code.

The same idea applies to the UDP socket that carries the WireGuard datagrams. Each IP packet passed to Firezone by the kernel maps directly to one UDP datagram carrying a WireGuard data packet. Similar to TCP, a UDP socket also has a send buffer where the data passed to write initially lands. From there, the kernel passes the datagrams to the corresponding NIC driver which ultimately sends them out via the network.

The main difference from TCP is that UDP doesn't define any flow or congestion control. In other words, it doesn't have a congestion controller that modulates the rate at which packets leave your device. There are no acknowledgments that we have to wait for. Instead, the packets essentially go out at line rate.

Backpressure, however, still applies. If the send buffer is full, the kernel will reject the syscall and, through that, link the backpressure signal from the UDP socket to the sending application.

Most likely, the rate at which UDP packets can leave your device is the limiting factor for throughput as your CPU can create new packets at a much faster rate than your network connection can carry.

The one caveat in this system is the TUN transmit queue, i.e. the queue that Firezone reads new IP packets from. That one does not have a backpressure signal based on queue size alone and therefore needs to have enough capacity to absorb the packets TCP (or QUIC) can still send after we stop reading. If it can't, we are back to loss as the congestion signal: the kernel drops packets at the full queue, and the congestion controller slows down only once it notices the missing acknowledgments. If it is large enough, TCP runs out of window before the queue fills up and holds off sending until more ACKs arrive.

Backpressure from the UDP socket to the browserA browser uploading a file writes into a TCP socket in the kernel. The kernel forms IP packets from the socket's send buffer and routes them to the TUN device, where Firezone's TUN thread reads them. Firezone encrypts each packet and its UDP thread writes it as a UDP datagram into the UDP socket's send buffer in the kernel, which hands datagrams to the NIC to go out at line rate. When the NIC cannot keep up, the UDP send buffer fills and Firezone's next write returns EWOULDBLOCK. Firezone's internal channels fill up behind it, so the TUN thread stops reading, and packets accumulate in the TUN device's queue. TCP exhausts its available window and waits for acknowledgments. Unacknowledged data remains in the TCP send buffer, which fills as the browser continues writing. Once the send buffer is full, the browser's next write returns EWOULDBLOCK, and the browser waits. With sufficient TUN queue capacity, this happens without dropping packets.BrowseruploadKernelTCP send bufferTUN deviceFirezoneTUNUDPwritereadstops readingTCP window exhaustedEWOULDBLOCKNICline rateKernelUDP send bufferwriteEWOULDBLOCKchannels full
A full UDP send buffer holds back Firezone, and from there every stage back to the browser's write.

In summary, what we are relying on here are the contracts of a non-blocking I/O system:

  1. When a buffer is full, a write gets rejected with "would block".
  2. After subscribing to "write-readiness", we receive a notification that the descriptor is writable again.

On Linux, (1) is implemented as the EWOULDBLOCK error code and (2) is implemented via the epoll syscall. Reactors like mio are then built on top of this system to offer building blocks for non-blocking I/O.

As mentioned above, this design had been working great in our Windows and Linux builds. On macOS, however, upload performance was nowhere near those other platforms with throughput often crippled to a few hundred Mbit/s or even less as latency increased.

Diving into the XNU source code

XNU is to Darwin what Linux is to Fedora or Ubuntu: the kernel. What may come as a surprise to some people is that Apple actually publishes the XNU source code on GitHub. It is usually delayed by a few months from the release of a new macOS or iOS version but it does land there eventually.

XNU's networking stack has its roots in FreeBSD, and its socket API goes back as far as 4.2BSD in 1983. Linus adopted the same API when he created Linux, which is why the two look alike on the surface with their syscalls of bind, connect, read and write. Underneath, however, they work very differently and, as it turns out, XNU diverges quite heavily in its behavior and API when it comes to non-blocking I/O for network sockets.

UDP sockets do not use a send buffer

When debugging throughput problems, one of the first things to try is to increase buffer sizes. The reason is that larger buffers help in absorbing bursts and scheduling delays which would otherwise lead to packet loss. That didn't help in our case and looking at the source code revealed why.

XNU does allocate a send buffer for every UDP socket, but it effectively remains unused. Instead of being buffered, the packet is handed directly to the network interface's transmit queue and the buffer's fill level always remains 01.

So what did we configure when we tried to increase the buffer size via SO_SNDBUF? It turns out that configuring the send buffer really only sets a high-water mark that enforces how big the datagram is allowed to be2. Somewhat amusingly, this is also documented as part of the send buffer's default value3.

With the fill level always being at 0, the check for space succeeds no matter how full the transmit queue is4. For other protocols like TCP, this is where XNU returns EWOULDBLOCK in case the buffer is full.

TCP and UDP send paths on XNUTwo paths from an application's write to the network interface on XNU, one for TCP and one for UDP. Each goes through the socket's send buffer, then the interface's transmit queue, then the network interface, which sends packets out at line rate. For TCP, written data waits in the send buffer. Once that buffer is full, the application's next write fails with EWOULDBLOCK. For UDP, the send buffer stays empty: each packet bypasses it and goes straight into the interface's transmit queue. Because the empty buffer always has space, the application's write succeeds. When the transmit queue is full, the packet is dropped there, and the application never finds out.ApplicationSend bufferTransmit queueNetwork interfaceTCPwriteline rateEWOULDBLOCKUDPwriteline ratewrite succeedsdropped
TCP data waits in the socket's send buffer, so a full buffer makes the next write fail with EWOULDBLOCK. UDP packets skip that buffer and go straight to the interface's transmit queue, where they are dropped once it is full while the write still succeeds.

In other words: on macOS, UDP backpressure is broken in practically every async runtime. Reactors and language runtimes rely on the behavior that, on a Unix system, a write to a non-blocking socket fails with EWOULDBLOCK when there isn't enough space for the message. On XNU, that never happens for UDP: the socket happily accepts every message and reports no error, while the packets are dropped at the interface transmit queue.

Connected sockets are different

Going through this code path revealed something else. A connected UDP socket is treated differently from an unconnected one.

What is a connected socket? On a protocol level, UDP is stateless. Each packet merely describes source and destination attributes. UDP has no handshake, sequence numbers, or message types. By default, a single UDP socket can be used to send to and receive from many peers at the same time.

Calling connect on a UDP socket restricts the destination to the provided peer and only routes messages from that peer to the socket. That is convenient if you need to manage several long-lived UDP "connections" without doing your own demultiplexing of inbound datagrams. Instead of a single socket, you create one socket per destination peer, call connect on it, and handle the sockets in different tasks / threads.

On XNU, connecting a UDP socket has additional side effects. Most importantly, using connected sockets allows us to obtain a backpressure signal, although it isn't what typical reactors and language runtimes expect. Once the interface transmit queue is full, further sends fail with ENOBUFS instead of EWOULDBLOCK. Runtimes treat ENOBUFS as an error, so using it for backpressure needs special handling.

Despite the different error code, registering for write-readiness after receiving ENOBUFS does produce a wake-up as soon as the queue has room again.

Restoring backpressure

This explains the slow upload speeds we had been seeing. On average, we were sending well below the capacity of the link, but the interface's transmit queue only holds a limited number of packets. Whenever draining it was briefly delayed, for example because the CPU was busy with other work, the queue filled up and XNU dropped every further datagram while still reporting the write as successful. To the TCP stream inside the tunnel, this looked like congestion, so CUBIC shrank its congestion window even though the link had capacity to spare: exactly the premature reduction we described earlier. And because the sender only learns about loss a round-trip later, the higher the latency, the longer it took to recover from each of these drops.

By moving to connected sockets and special-casing the ENOBUFS error, we were able to implement a dedicated code path that suspends our UDP transmit thread upon a full interface queue, restoring backpressure. The details of this work are in PR 13725. Now such a delay merely pauses our transmit thread until the queue has room again, and a longer stall propagates through our bounded channels back to the TUN device instead of turning into packet loss.

Amortization, the holy grail of throughput

So far, we've focused on not losing packets in order to not upset the congestion controller. That is a good foundation, but it doesn't make anything faster by itself. Ironically, measuring speed-ups while some part of the system still drops packets often produces worse results because you are still losing packets, now just faster.

To make packet processing faster, it helps to split the cost into two parts. There is the per-packet work: encrypting and decrypting, NATing towards a new IP, updating checksums, recording flow logs. Because it is per-packet, the CPU time we spend on this grows with the number of packets.

And there is per-call cost: syscalls, thread wakeups, allocations. It is paid once per operation, regardless of how many packets that operation carries.

That is what makes amortization powerful. Read 32 packets in one syscall and the syscall costs 1/32nd per packet. Hand 100 packets to another thread at once and the wakeup costs 1/100th per packet.

The same packets, fewer callsFour packets pass through packet processing, an I/O wrapper, and the kernel. Passing packets one at a time makes four calls at each boundary between layers. Passing the four packets together makes one call at each boundary. The per-call overhead is shared across the batch.One packet at a timePacket processingI/O wrapperKernel4 calls per boundaryOne batch at a timePacket processingI/O wrapperKernel1 call per boundary
Batching shares the cost of each call across more packets.

Thus, the answer for faster packet processing is to do it in batches, and XNU exposes several ways for us to do that.

sendmsg_x and recvmsg_x

These are the equivalent of Linux's sendmmsg and recvmmsg syscalls. Instead of only passing a single packet / buffer, you can pass several in one call and have the kernel process them all at once. Unlike on Linux, these syscalls work both on a network socket and the TUN file descriptor!

This allows us to pass packets in batches between the kernel and userspace, thus splitting the cost of the syscall across several packets. These APIs still require you to pass an individual buffer for each packet, meaning we mostly amortize the syscall overhead itself.

On Linux and Windows, we can make use of syscalls that not only amortize the syscall overhead but also the actual processing in the kernel. This is enabled by an API that accepts a single contiguous buffer instead of a list of buffers, together with a known segment length that specifies where the buffer must be split to obtain the individual datagrams.

Just like a userspace program, the kernel is also composed of many layers. Passing a single but large buffer all the way through to the NIC driver therefore amortizes that processing cost compared to passing each individual datagram.

Unfortunately, XNU doesn't offer this capability for BSD sockets. Nevertheless, amortizing at least the syscall overhead is better than nothing and offers a good performance improvement.

Combining sendmsg_x with connected sockets

A nice side effect of the connected sockets we've already built is that when used together with sendmsg_x, the kernel implements a fast path5 where it does some of the socket setup, like updating the socket's policies and checking for buffer space, once per batch instead of once per datagram. And because the socket already knows its destination, the kernel also skips the per-datagram address and route lookup.

The idea is the same as using sendmsg_x to avoid the syscall overhead, now simply in the kernel. Anything that we can do once per operation instead of once per packet frees up processing capacity for more packets.

Coalescing TCP segments

When configuring a TUN device, we not only need to set the IPv4 and IPv6 addresses but also the MTU (maximum transmission unit) of the link. The MTU tells the kernel how big we'd like the IP packets to be that it creates for outbound traffic that is sent through Firezone.

Firezone's TUN device is configured with an MTU of 1280 across all platforms. This matches the minimum MTU for IPv6 and leaves sufficient room for WireGuard's encapsulation overhead on paths with the common 1500-byte Ethernet MTU.

An interesting finding we've made in digging through XNU is that the kernel accepts packets larger than its MTU on the write path, i.e. when Firezone delivers packets back to the kernel. Internally, it stores each packet in a fixed-size slot6, and on the Macs we tested, the NetworkExtension sizes those slots at 4096 bytes, which is about three times the 1280-byte MTU.

This slot size can't be changed after the TUN device has been created, but it can be queried! That allows us to merge up to 3 consecutive MTU-sized segments of the same TCP flow into a single, larger segment before we write it to the TUN device. This only works for TCP because it is a byte stream: the receiving application never sees where one segment ends and the next one begins. UDP, on the other hand, is message-based. Every datagram reaches the application as a separate message, so merging two would deliver them as a single message which the receiver then cannot parse.

Coalescing TCP packets requires us to recalculate the checksum, which means a full pass over the entire payload. That cost is, however, outweighed by the savings in the kernel's per-segment processing, which now happens up to three times less often.

Results

The real world is messy. We didn't make these discoveries in the "right" order: some of the packet processing optimizations landed before we had working backpressure. In particular, Firezone had been using the batched send and receive syscalls sendmsg_x and recvmsg_x on its UDP sockets for a long time before we discovered that backpressure on XNU needs special handling. As such, it is tricky to point to historical benchmark results.

That doesn't mean we can't measure the impact of these individual improvements, however.

The following results have been measured in our continuous benchmarking environment, which for the macOS Client consists of a pair of M1 Macs, directly connected via Thunderbolt at 40 Gbit/s. On these M1 Macs, Firezone achieves 5.87 Gbit/s upload and 5.76 Gbit/s download.

For the purpose of the following numbers, we selectively disabled the optimizations discussed in this post to show how much impact each one has. In each run, we measure a single TCP stream three times per direction with iperf3. The displayed number is the median of those three measurements.

Throughput as each optimization is addedUpload and download throughput for each step, in gigabits per second. Bars show the median receiver throughput across repeated measurements. Whiskers show the smallest and largest observed throughput, not confidence intervals.UploadDownload0123456Gbit/s1. BaselineUpload1.16Download1.162. + connected sockets + backpressureUpload1.73Download1.813. + UDP batching (sendmsg_x, recvmsg_x)Upload2.63Download2.604. + TUN batching (sendmsg_x, recvmsg_x)Upload4.95Download5.075. + 8 MiB TUN receive bufferUpload5.18Download5.506. + TCP coalescing on TUN deviceUpload5.87Download5.76
Median receiver throughput with the observed minimum and maximum.

What's next

Achieving 5+ Gbit/s throughput is great and should mean that for the majority of people's workloads, Firezone is not going to be the bottleneck. Yet we'd like Firezone to be faster.

The measured rate translates to ~600k packets per second at our MTU of 1280. Profiling shows that the busiest thread is now our UDP thread, and it already spends most of its time inside the kernel. The key thing that it reveals is that each UDP packet, despite the use of sendmsg_x, is handled individually as soon as it reaches the IP layer of XNU.

On Linux and Windows, segmentation offloads drastically reduce the per-packet cost here. Instead of passing an individual buffer per packet, we hand the kernel one large buffer together with a segment size, and it splits that buffer into individual datagrams as late as possible, ideally on the network card. This allows the cost of forming and routing an IP packet to be amortized across all packets in the buffer.

If Apple were to expose such a capability through its BSD socket API, the throughput numbers could very well be 2x or 3x higher and thus provide a great boost for any UDP-heavy workloads on macOS, userspace QUIC stacks included.

References

Footnotes

  1. XNU bsd/net/dlil.c, lines 3222 to 3241. ↩

  2. XNU bsd/kern/uipc_socket.c, lines 1943 to 1946. ↩

  3. XNU bsd/netinet/udp_usrreq.c, line 2327. ↩

  4. XNU bsd/kern/uipc_socket.c, lines 1947 to 1965. ↩

  5. XNU bsd/kern/uipc_syscalls.c, lines 1906 to 1918. ↩

  6. XNU bsd/net/if_utun.c, lines 144 to 162. ↩

Firezone Newsletter

Sign up with your email to receive roadmap updates, how-tos, and product announcements from the Firezone team.

By checking the box below, you agree to receive communications from Firezone. You can unsubscribe anytime.

To deliver your service, we need your permission to store and process your personal data. We care about your privacy. Learn how we handle your data in our Privacy Policy.