CPU-GPU I/O-Aware LLM Inference Reduces Latency in GPUs by Optimizing CPU-GPU Interactions

Optimizing inference latency in large language models (LLMs) is crucial for achieving faster and more efficient processing, especially when using GPUs for high-performance computation. In LLM inference, latency refers to the time it takes for a model to process input data and produce an output. Reducing this latency is essential, particularly for real-time applications like conversational agents or other interactive AI systems, where delays can negatively affect user experience.

One of the key factors influencing inference latency is the interaction between the CPU and GPU. GPUs are designed for parallel processing and are highly efficient at performing large-scale matrix operations, which are central to LLM computations. However, the interaction between the CPU and GPU—particularly how data is moved between them—can create significant bottlenecks. Memory bandwidth, for example, plays a critical role in this process. The speed at which data is transferred from the CPU to GPU memory can determine how efficiently the model performs during inference.

To optimize this interaction, techniques such as memory caching, model parallelism, and reducing kernel launch overhead are employed. By optimizing the CPU-GPU communication, it's possible to minimize delays, making inference faster and more cost-effective. These optimizations help ensure that GPUs can be used to their full potential, ultimately reducing latency and improving the overall performance of LLMs.

LLMs are driving significant advancements in research and development today, with a notable shift in research goals and methodologies towards LLM-centric approaches. However, the high costs associated with these models make them inaccessible for many, especially for large-scale applications. As a result, reducing the latency of operations, particularly in dynamic applications requiring high responsiveness, presents a major challenge.

The KV cache plays a crucial role in autoregressive decoding in LLMs. It stores key-value pairs in the multi-headed attention mechanism during the pre-filling phase of inference, and during the decoding stage, new KV pairs are appended to the cache. This approach reduces the complexity of the attention mechanism from quadratic to linear order, improving efficiency. However, as the KV cache grows with batch size, sequence length, and model size, it can exceed the GPU's handling capacity. Transferring the cache to the CPU introduces bottlenecks that increase latency and reduce throughput.

The limitations of PCIe interfaces become particularly apparent when transferring the KV cache between the CPU and GPU for computation. Slow PCIe performance can cause latency to spike, leading to significant GPU idle time, which worsens overall performance.

While previous work has aimed to mitigate the issue of slow PCIe performance, these solutions often fall short due to mismatched data transfer and computation times, especially with large batches and context sizes. Some approaches also rely on CPU resources, which can also become a bottleneck. This article discusses a new method for optimizing PCIe and GPU utilization.

Researchers at the University of Southern California have proposed an efficient CPU-GPU I/O-aware LLM inference technique that optimizes PCIe utilization. This method involves partial KV cache recomputation and asynchronous overlapping, addressing the bottleneck caused by transferring large KV caches. Instead of transferring the entire KV cache to the GPU, smaller segments of the activation cache are moved, allowing the GPU to reconstruct the full cache. This technique ensures minimal information loss by carefully computing attention scores.

The authors introduce an automated approach to optimize recomputation and communication splits in LLM applications, featuring three key modules designed to minimize GPU latency:

1. Profiler Module: Gathers hardware details, such as PCIe bandwidth and GPU processing speeds, to assess system capabilities.

2. Scheduler Module: Uses linear programming to determine the ideal KV split point, aiming to maximize the overlap between computation and communication processes, based on system data and user configurations.

3. Runtime Module: Manages memory allocation and coordinates data transfer between the CPU and GPU.

The Scheduler Module works in two ways:

• Row-by-Row Schedule: This approach reduces latency by executing row-by-row, where the GPU starts reconstructing the KV cache while the remaining activations load asynchronously.

• Column-by-Column Schedule: Optimizes throughput, particularly for large batch inferences, by reusing model weights across batches and overlapping KV cache and activation transfers with MHA (multi-headed attention) computations.

Additionally, the Runtime Module enhances efficiency using a six-process communication parallelism strategy, allowing for concurrent GPU computation and CPU-GPU communication​.

The authors evaluated the proposed framework for efficient LLM inference using an NVIDIA A100 GPU connected to a CPU via a PCIe 4.0 x16 interface. The experiments were designed to assess performance based on two key metrics:

1. Latency-Oriented Workload: The proposed method demonstrated a significant improvement, reducing latency by 35.8%, outperforming baseline approaches.

2. Throughput-Oriented Workload: The method achieved up to a 29% increase in throughput compared to the baseline, highlighting its efficiency in handling larger data volumes.

These results suggest that the framework successfully optimizes both latency and throughput, making it a promising solution for LLM inference on high-performance systems.

The CPU-GPU I/O-aware LLM inference method efficiently reduces latency while increasing throughput in LLM inference. It leverages partial KV cache recomputation and overlaps it with data transmission to minimize idle GPU time and enhance efficiency.

The Challenges of LLM Inference

In the context of large language model (LLM) inference, GPU memory limitations are a critical challenge due to the enormous size of the models involved. These models often have billions of parameters, requiring vast amounts of memory to store the weights, activations, and intermediate results during inference. However, modern GPUs, even those with substantial memory like NVIDIA A100s, are still constrained by memory bandwidth and capacity, which can hinder the throughput and efficiency of LLMs when processing long sequences or handling large batches.

One common strategy to mitigate memory constraints during inference is the use of key-value (KV) caching. KV caching stores intermediate query-key-value pairs from attention layers so that they can be reused in subsequent computations. This reduces redundant calculations and allows the model to avoid recalculating these values for each token in a sequence. Instead, only the new tokens need to be processed, leveraging the stored values, thus saving both time and memory.

To further optimize this process, various memory architectures and management techniques are applied. For example, systems like CaR (Cache-aware Resource scheduling) use multi-tier cache systems that integrate GPU memory with external memory solutions like CXL. This enables the KV cache to be stored in external memory when not actively used, allowing it to be loaded back into the GPU’s high-bandwidth memory (HBM) on-demand. This approach not only reduces latency but also minimizes the overhead associated with recomputing key-value pairs. Additionally, advancements such as multi-query attention (MQA) improve the efficiency of KV caching by sharing key-value pairs across multiple attention heads, reducing memory usage and improving computational throughput.

By combining KV caching with advanced memory management strategies, it's possible to significantly alleviate the memory bottleneck in LLM inference, improving both performance and scalability without sacrificing accuracy.

When the key-value (KV) cache exceeds GPU memory, offloading the cache to CPU memory presents significant challenges, particularly in terms of both latency and bandwidth.

The GPU is typically used for high-throughput computation due to its massively parallel architecture. However, when the KV cache grows too large for the GPU to accommodate, it needs to be offloaded to CPU memory, which is slower than GPU memory. This can introduce significant latency, as data must be transferred back and forth between the CPU and GPU during inference. This data transfer is bottlenecked by the PCIe bandwidth, which is orders of magnitude slower than memory access on the GPU itself.

Additionally, this shift requires more complex memory management strategies to ensure the system doesn’t stall due to frequent memory accesses. Large language models (LLMs) rely heavily on efficient memory access patterns to maintain performance, and moving the KV cache to CPU memory disrupts these patterns. Without careful optimization, offloading leads to inefficient memory utilization, as the CPU struggles to handle the increased load and the GPU spends more time waiting for data.

To mitigate these challenges, researchers have proposed techniques such as KV cache compression and selective offloading, where only the most critical data is moved between memory types, and non-essential parts are discarded. For example, compression strategies for the KV cache, including adaptive methods based on the token’s attention, can help reduce the memory footprint. Another approach uses hybrid offloading methods that combine GPU and CPU memory more effectively, ensuring minimal disruption to inference speed.

These strategies are crucial for maintaining throughput and minimizing latency in LLM applications that rely on large caches and batch processing.

Traditional methods that attempt to address I/O bottlenecks in GPU computation, such as overlapping GPU computation with data transfers, are limited by several factors, most notably the constraints of the PCIe interface. While overlapping computation with I/O is effective in theory, it often falls short due to several key challenges:

1. PCIe Bandwidth: The bandwidth of PCIe, even with newer versions like PCIe 4.0, often becomes a limiting factor in high-demand applications such as large language model (LLM) inference. This is because PCIe, while fast, cannot keep up with the volume of data transfer required when the entire key-value (KV) cache is offloaded from GPU to CPU memory. The limited bandwidth between the CPU and GPU means that, even with overlapping computation and data transfer, significant latency remains as the system struggles to manage the data movement.

2. Data Movement and Synchronization: Effective computation-communication overlap requires careful synchronization of data transfers and GPU computation. A common method for achieving this is using separate streams for computation and communication, which allows the GPU to continue computation while data is being transferred. However, the challenge lies in efficiently managing these streams, as data movement can still cause delays. In some cases, using multiple CUDA streams or over-decomposing tasks can improve overlap, but it often still does not fully mitigate the delays due to the inherent limitations of PCIe​.

3. Dependency on CPU Resources: Many of the existing methods that involve CPU-GPU heterogeneous execution rely heavily on CPU resources to manage I/O tasks. However, this can introduce another bottleneck, as the CPU must handle additional data management tasks, potentially leading to its own performance limitations. The reliance on CPU capabilities further complicates achieving optimal performance, especially in systems where CPU-GPU coordination is suboptimal​.

   
   

Thus, while overlapping GPU computation with I/O is a promising strategy for mitigating latency, the PCIe bandwidth and the additional complexity of managing data movement and synchronization pose significant limitations. Solutions that go beyond traditional methods, such as partial KV cache recomputation or more advanced memory management strategies, are often required to achieve higher performance levels.

I/O-Aware LLM Inference: An Efficient Solution

I/O-aware inference refers to optimizing the data transfer between the CPU and GPU in large language models (LLMs), particularly for tasks involving the key-value (KV) cache. It focuses on balancing computation and communication to reduce latency and improve throughput during model inference.

In the context of LLMs, the KV cache holds key-value pairs that are critical for operations such as attention mechanisms. Efficient management of the KV cache can significantly impact both computation performance and memory usage. I/O-aware inference techniques optimize the transfer of this cache by recomputing parts of it during active data transfers, thus minimizing waiting times for memory access and improving GPU utilization.

One approach to I/O-aware inference involves splitting the KV cache so that portions of it are recomputed while the rest is transferred between the CPU and GPU. This allows for overlapping computation with communication, thus reducing bottlenecks related to PCIe bandwidth and memory access. A key challenge in this area is dynamically managing memory allocation and transfer scheduling without negatively affecting the overall throughput of the system. Systems like vLLM and PagedAttention address this by using dynamic memory profiling and managing non-contiguous memory storage, which helps mitigate queuing delays and optimize resource utilization.

The concept also links closely with techniques like the LayerKV system, which focuses on layer-wise KV cache management to better align computation with data transfer, ensuring that new requests can be processed without significant delays caused by memory access conflicts.

To minimize idle GPU time and maximize throughput, the method proposed in recent research leverages an intelligent balance between computation and data transfer during LLM inference. The approach reduces latency and boosts throughput by strategically handling the overlap between GPU computation and I/O operations, specifically focusing on key-value (KV) cache management. By offloading portions of the KV cache to CPU memory and recomputing only partial segments of the cache when needed, the system can avoid transferring the entire cache back and forth, which would otherwise result in significant delays due to the limited bandwidth of the PCIe connection. This partial recomputation, done concurrently with the transfer of the remaining KV cache, ensures that the GPU remains active and maximizes its throughput rather than remaining idle during the I/O phase.

Furthermore, the approach integrates a scheduler that optimizes the division of computational and data transfer tasks. This ensures that the data movement and computation phases are efficiently coordinated, with minimal waiting times for the GPU. Experimental results demonstrate that this method not only reduces latency by as much as 35.8% but also improves throughput by up to 46.2% compared to state-of-the-art methods, highlighting its efficiency in managing both GPU compute cycles and I/O operations.

How It Works

The integration of a profiler, scheduler, and runtime module for automating the optimization of Large Language Model (LLM) inference involves several complex processes, each targeting specific aspects of system performance. By utilizing these components, LLM inference services can be dynamically adjusted to accommodate varying system hardware and input characteristics.

1. Profiler: This module collects detailed performance data from the system, including metrics like time-to-first-token (TTFT) and time-per-output-token (TPOT)​. The profiler assesses how system configuration, such as batch size or GPU types (e.g., NVIDIA A100), impacts performance, providing insights into potential bottlenecks. This data is critical for understanding how different parameters influence LLM inference efficiency, helping identify the optimal configuration​.

   
   

2. Scheduler: The scheduler dynamically allocates tasks based on the available system resources, ensuring that the workload is balanced across multiple GPUs or CPUs. It may also factor in the characteristics of the input, such as the type of LLM (e.g., Llama-3 or Meta-Llama) and specific workload demands (e.g., conversational bots or text-to-sql applications). Given that different applications require different performance metrics (e.g., throughput for offline tasks, low latency for interactive tasks), the scheduler must account for these variations to optimize the service's service-level objectives (SLOs)​.

   
   

3. Runtime Module: The runtime component is responsible for executing the optimized configurations generated by the profiler and scheduler. This module automates the inference process by applying the most suitable parameters to the running system. It adjusts the parameters in real-time based on changing workload demands, ensuring that system constraints are respected. For instance, certain parameter combinations may not work well due to hidden constraints, such as when specific memory settings lead to crashes or degraded performance​. By integrating these modules, the optimization process becomes continuous, automatically tuning the system based on incoming tasks and performance data.

   
   

This automated framework allows for efficient, high-performance LLM inference without manual tuning, significantly reducing the overhead of experimentation and configuration​.

A dynamic execution plan plays a pivotal role in optimizing LLM (Large Language Model) inference performance, particularly in environments where real-time adjustments are necessary. These dynamic plans are designed to adapt to the fluctuating demands of the system, ensuring that resources are efficiently utilized throughout the computation and data transfer process.

The key component of a dynamic execution plan is its ability to adjust resource allocation in real-time. For instance, when performing inference tasks, the system continuously monitors and analyzes the workload. Based on factors such as GPU and CPU utilization, batch size, sequence length, and available memory, it makes instant decisions on how to divide the work between the CPU and GPU. This real-time adjustment maximizes throughput while minimizing latency.

Such plans are essential for handling large-scale, complex models where the computational demands can vary dramatically during different stages of the inference process. For example, in an LLM system utilizing key-value (KV) cache, as the size of the model grows, the memory required for KV storage increases linearly, posing a challenge to maintaining throughput. A dynamic execution plan can manage this by intelligently segmenting the KV cache and optimizing the flow of data between the CPU and GPU. During the decoding phase, the system dynamically schedules the transfer of KV cache segments, ensuring that the GPU is always engaged in computations while the CPU asynchronously loads additional data.

The integration of dynamic execution strategies allows the system to avoid the common bottleneck of mismatched data transfer and GPU computation times. As a result, the overall system performance is significantly improved, achieving faster response times and more efficient use of resources. Dynamic execution plans are particularly beneficial when dealing with large batch sizes and complex models, where a static approach would lead to resource contention and inefficiencies.

Performance Improvements

Experimental results from a recent study on I/O-aware LLM inference methods demonstrate a significant reduction in latency and an improvement in throughput compared to traditional approaches. Specifically, the method introduced in the study involved an efficient strategy for handling Key-Value (KV) caches by recomputing partial caches on the GPU while concurrently transferring the rest from CPU memory over PCIe. This method effectively overlaps the GPU recomputation with data transfer, minimizing idle GPU time and thus improving performance.

The study's results showed a substantial 35.8% reduction in latency and a 46.2% increase in throughput during decoding when compared to state-of-the-art techniques. The improvement in throughput was achieved by optimizing the distribution of computation and communication workloads through an integrated system of profiler, scheduler, and runtime modules. These modules automate the process, leveraging system hardware characteristics and user input to optimize the KV cache distribution and maximize the overlap between computation and data transfer, thus enhancing the efficiency of the inference process.

The method you referenced, which involves I/O-aware partial KV cache recomputation, has been shown to significantly improve LLM inference performance. The framework, designed to optimize memory and computational efficiency during auto-regressive decoding, delivers impressive results:

1. Latency Reduction: By utilizing CPU-GPU I/O-aware techniques, the system reduces latency by up to 35.8%, addressing the bottlenecks associated with KV cache transfer over PCIe links.

2. Throughput Increase: The method also achieves a 46.2% higher throughput compared to conventional approaches. This improvement is driven by better optimization of the GPU computation alongside data transfer, leveraging the simultaneous execution of recomputation and data transfer.

These gains are achieved by integrating a fully automated pipeline consisting of a profiler module to collect system information, a scheduler module to optimize computational and I/O workloads, and a runtime module that implements the execution plan effectively​. This approach avoids excessive data movement and optimizes resource usage, marking a notable advancement in efficient LLM inference.

Real-World Applications

The integration of efficient key-value (KV) cache mechanisms has profound implications for AI systems, particularly in high-demand environments such as cloud computing and real-time AI applications. One of the most promising methods is leveraging **prefix caching**, which optimizes the retrieval of key-value pairs by storing previously computed values for re-use in subsequent operations. This method addresses challenges related to high latency and the massive memory bandwidth required by large language models (LLMs) during inference.

In practical applications, these optimizations can drastically reduce the computational overhead and resource consumption of AI systems. For example, **multi-query attention (MQA)** has been shown to improve compute utilization by sharing keys and values across multiple attention heads while maintaining accuracy. This reduction in memory demands allows for higher throughput, particularly beneficial in environments where large-scale data is processed, such as in cloud-based AI services. These techniques minimize the need for repeated computation of the same values, ultimately improving inference efficiency, reducing energy consumption, and enabling the scaling of AI models in cloud infrastructure.

The use of disk-based storage for KV caches is another key advancement, particularly in large-scale systems. For example, services like Deepseek have adopted this approach, allowing for a dramatic reduction in inference costs—up to 90%—by leveraging prefix caching on disk. While GPU memory offers the fastest access times, it is expensive and limited in capacity. Disk storage, particularly high-performance SSDs, provides a more cost-effective solution that balances memory and bandwidth requirements, particularly for long-context models. The trade-off between different storage mediums highlights the critical role of I/O efficiency in reducing the bottlenecks typically seen with large-scale inference systems.

Moreover, these advancements have substantial implications for **real-time AI applications**. By optimizing cache management and using techniques such as **chunk prefill**—which breaks down large inputs into smaller chunks for sequential processing—AI systems can handle longer input sequences without monopolizing resources. This capability is crucial for applications that require immediate responses, such as in autonomous vehicles or real-time decision-making systems. Additionally, leveraging **tensor parallelism** and **sequence parallelism** in model training and inference can further enhance computational efficiency. These strategies allow for the parallel execution of operations across different devices, leading to better memory distribution and faster processing times.

In summary, the real-world applications of these optimizations are vast. They not only improve the efficiency and responsiveness of AI systems but also reduce the environmental impact of running large-scale models, which is especially relevant in cloud computing. By improving I/O operations and optimizing memory and computation, AI systems can be made more scalable and cost-effective, facilitating their deployment in diverse, resource-constrained environments.

Conclusion

In modern Large Language Model (LLM) inference, achieving efficient computation across distributed systems requires balancing several competing factors, including throughput, latency, and memory constraints. To address these challenges, multiple optimization techniques, such as CPU-GPU pipelining, memory offloading, and parallelism, have been proposed.

A notable advancement in reducing memory bottlenecks involves offloading parts of the computational tasks, particularly attention calculations and key-value (KV) caches, from GPUs to CPUs. This approach allows the system to maintain a higher throughput by expanding the effective batch size, which is often constrained by GPU memory limitations. Research like that presented in NEO has shown that by offloading certain tasks to CPUs, throughput can be increased by up to 79.3% without sacrificing latency, especially when paired with higher-performance CPUs.

Tensor parallelism has also emerged as an effective method to manage the trade-off between latency and throughput. By distributing the model across multiple GPUs, tensor parallelism can lower the latency of individual inference requests, although this may come at the cost of reduced throughput. The optimal configuration often depends on specific latency goals; for example, higher tensor parallelism (TP) settings are beneficial for low-latency tasks, but data parallelism should be considered when throughput is more critical.

Further advancements involve speculative decoding, a technique where a smaller model is used to predict tokens in parallel, which are then validated by the full model. This approach significantly reduces the time to first token (TTFT), benefiting high-concurrency, low-latency applications like real-time chatbots and autonomous systems.

In summary, optimizing LLM inference involves a nuanced balance of various strategies that leverage the strengths of both GPUs and CPUs. Techniques like CPU offloading and speculative decoding, when combined with tensor parallelism and efficient batching, enable significant improvements in throughput and latency, offering substantial gains in large-scale deployment scenarios.