AIStore 4.7 adds a zero-copy fast path for eligible GET requests using the Linux sendfile system call. Instead of copying object data through userspace on the regular transmit path, sendfile lets the kernel move file data directly from the local filesystem to the client socket, reducing target-side CPU overhead while preserving the same client-facing response.

In the benchmarks, sendfile reduced CPU time per GiB transferred by up to 70% on AIS target nodes serving GET responses. When network bandwidth is not the bottleneck, those CPU savings translate directly into higher throughput, with benchmarks showing up to a 3x increase. Even during a network bottleneck, the reclaimed CPU can be used for colocated work such as ETL, jobs, and checksum validation.

Table of Contents

Background

For every GET request to AIStore, it first reaches an AIS proxy, which redirects the request to the target that owns the object. On a warm GET, where the object is already local, the target serves the response through the regular transmit path:

  1. Load the object’s local metadata.
  2. Open a reader for the local object.
  3. Construct the HTTP response headers from the object metadata.
  4. Read object data from the local file into a reusable userspace buffer.
  5. Perform any required processing on the buffered data.
  6. Write the buffer to the response writer, which sends the bytes to the client socket as the HTTP response body.
  7. Repeat until the entire object has been transmitted.

This process is illustrated in the diagram below, but a more in-depth overview is available in AIStore documentation.

Transmit Path

By passing object data through userspace, the transmit path supports additional request handling such as HTTPS and chunked object reads, though not every GET request needs that flexibility. When a request can be served as a plain HTTP transfer of a large, local, monolithic object, copying data through a userspace buffer becomes unnecessary overhead.

The sendfile path removes that extra copy. AIS still performs the same request handling, but the CPU no longer spends additional cycles copying every byte through a userspace buffer. This reduction in CPU usage within the target’s data transfer component provides significant savings for the overall GET request.

SendFile Path

Benchmarks

Setup

The benchmarks used a 3-node Kubernetes cluster with three AIS targets and three proxies. Each node ran Ubuntu 20.04 on a 24-core machine with about 1 TB of RAM and a 100 Gb/s NIC (corresponding to a maximum bandwidth of around 11.6 GiB/s).

The two paths were compared using separate AIS node container images: one built with sendfile enabled and one with it disabled. Before each run, the target aisnode process was traced with strace to confirm it was taking the expected path (sendfile vs userspace read/write on the response body).

All benchmark objects were prepopulated into AIS buckets before the GET tests so that the results reflect what sendfile directly impacts: warm GETs on local objects. Every GET targeted objects owned by a single chosen target node, and that node was monitored for the results reported below.

The reported CPU and throughput numbers come from target-node sar data collected and processed offline to filter out unrelated cluster activity during each run. Alongside those numbers, visualizations from the AIS monitoring stack are also included to provide a high-level overview of cluster performance during each run.

Object Size

Object size is one of the largest factors in the effectiveness of sendfile. Each GET carries the same fixed per-request overhead (object lookup and locking, metadata load, HTTP header construction) while sendfile only reduces the cost of copying the object bytes into the response body, so the benefit is easier to see when the body is large enough for that copy cost to dominate the fixed per-request work.

To measure this effect, the regular transmit path was compared with the sendfile path for object sizes ranging from 64 KiB to 4 GiB, increasing by a factor of 4 at each step. Each size was tested for 3 minutes with 120 concurrent workers.

Transmit Object Benchmark

Sendfile Object Benchmark

Both the transmit and sendfile object-size runs quickly reached a network ceiling of roughly 11.3 GiB/s. Because throughput was capped by the network, this test did not reveal any significant throughput differences between the two paths. Instead, the useful comparison is CPU cost at equal throughput: at comparable network throughput, the target node consumed meaningfully less CPU with sendfile.

The plots below normalize target CPU usage as CPU-seconds per GiB transferred and show the relative CPU reduction of sendfile compared with transmit.

Object CPU Sec per Gib

Object CPU Reduction Rate

For 64 KiB objects, sendfile showed no benefit in this run as the measured CPU reduction was slightly negative. That is expected for very small responses because the fixed HTTP and AIS request costs dominate the body-copy savings. The CPU reduction rates peaked at around 20% for object sizes over 256 MiB, which suggests that per-request overhead is small relative to body transfer beyond that point.

There were minor run-to-run fluctuations from variations in network latency, but the data did show a consistent trend: sendfile is more CPU-efficient than the transmit path, especially for larger object sizes.

Throughput Gains

The object-size benchmark did show CPU savings, but the network ceiling was masking potential throughput gains. To eliminate the network bottleneck, a second benchmark was conducted with the AIS client colocated on the same node as the AIS proxy and target. This setup isolates the test from network limits and shows the pure performance improvement that sendfile brings.

While running the AIS client on the same node as the AIS proxy and target is not a typical AIStore deployment, such a setup simulates environments where target nodes have enough network capacity that CPU becomes the limiting resource.

In the second benchmark, each test ran for 5 minutes against a bucket of 1 GiB objects, with worker count swept from 50 to 800 (doubling at each step).

Transmit Worker Benchmark

Sendfile Worker Benchmark

In this CPU-bound setup, the results show a significant difference in throughput. The transmit path achieved 15–17 GiB/s while the sendfile path reached 44–50 GiB/s at roughly the same high CPU utilization. These results indicate that without network limitations, the reduced CPU cost translates directly into higher GET throughput.

Worker CPU Sec per Gib

Worker CPU Reduction Rate

The normalized CPU plots show a 62-72% reduction in target CPU-seconds per GiB for sendfile compared with transmit. The reduction percentage is higher here than in the previous benchmark because the CPU-bound setup produces a compounding effect: transmit spends more CPU per delivered GiB, while sendfile keeps CPU cost lower and converts the saved CPU into additional throughput.

Limitations

The sendfile path is intentionally narrow. AIS only uses it when the request can be served as a correctly framed HTTP response whose body is a direct view of a local file. When AIS needs to inspect, transform, encrypt, or reassemble bytes before sending them, it falls back to the regular transmit path.

TLS

sendfile cannot be used for HTTPS traffic. With TLS, object bytes must be encrypted before reaching the client, which means AIS must continue reading data into userspace so the TLS stack can encrypt it before writing it to the socket.

Chunked Objects

AIS chunked objects are represented by a manifest and multiple backing chunks, not by one continuous local file that can be passed directly to sendfile. For chunked object reads, AIS has to resolve the object layout, open the relevant chunks, and stream them in the correct order. That reassembly work requires AIS-controlled readers, so the request remains on the transmit path.

Range Reads

Range reads can use sendfile when AIS only needs to validate the HTTP Range header, set the 206 Partial Content response headers, seek to the requested offset, and transfer the requested byte range directly. When AIS must compute or validate a checksum for the requested range, it may need to read the range first, materialize it, and then send it. In those cases, the transfer is no longer a pure file-backed response body, so AIS has to fall back to the transmit path.

Transformed Responses

sendfile does not apply when the response body is not the original local object file. Examples include archive extraction, ETL-transformed reads, AIS Select-style processing, and other cases where AIS must generate a response from object data. For these requests, AIS needs to inspect or transform the bytes before sending them to the client. Since the response is produced by AIS rather than being a direct view of the local file, the userspace transmit path remains necessary.

Conclusion

On eligible warm GETs, sendfile lowered target CPU-seconds per GiB by as much as 72% and raised throughput by roughly 3x. Those peaks reflect the best-case scenario from the benchmarks: CPU-bound deployments serving large, local, monolithic objects. In network-bound runs, sendfile delivered comparable throughput while reducing target CPU cost by about 20% for large objects.

Despite the performance gains, the sendfile fast path is reserved only when the response body is a direct view of the requested file and can be transferred without userspace processing. All other GET requests automatically fall back to the transmit path. That narrow eligibility is the main limitation on sendfile in AIStore today. Additionally, the benchmarks showed that even eligible GETs do not benefit equally, as object size and network saturation significantly influence the effectiveness of sendfile.

References

AIStore

Linux