Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo

Shadow engine recovery, available as a preview feature in NVIDIA Dynamo, moves most of this recovery work off the serving path. It keeps a fully initialized shadow engine idle on the same GPUs as the active engine. The GPU Memory Service (GMS) shares the existing weights between the engines without creating another copy in HBM. If the active process fails, the shadow takes over within seconds. Re-initialization occurs in the background entirely off the serving path.

We measured the impact by deliberately terminating one worker in a two-worker GLM-5.2 deployment. Without shadow engine recovery, the remaining worker served all incoming traffic during the 283-second cold restart, increasing TTFT and reducing per-user decode rate throughout the outage. With shadow engine recovery, a second worker resumed serving in 7.3 seconds, nearly 39 times faster, minimizing disruption to service quality.

Figure 1. Time until a second worker resumes serving after one of two workers fails. A cold restart reloads weights, sizes the KV cache, autotunes, and recaptures CUDA graphs. A preinitialized shadow engine can begin serving much sooner

Why LLM inference recovery is slow: two core problems[](#why_llm_inference_recovery_is_slow_two_core_problems)

Production LLM engines commonly experience recoverable software faults, including process crashes, recoverable CUDA errors, and transient collective failures. In these cases, the hardware, drivers, and node remain healthy; only the process holding the corrupted state is lost, and a replacement engine can typically start on the same GPUs.

So why can’t that fresh engine skip the initialization cost? Two problems stand in the way:

• Weights are tied to the engine process. GPU memory is linked to the engine’s CUDA context, which is itself tied to the engine process. When the process exits, the driver releases all resources, including weights already resident in GPU memory. Consequently, a replacement engine process must repeat the full weight-loading procedure.

• Some initialization states are non-transferable. NCCL and torch.distributed communicators bind to the specific running process, and CUDA graphs are fixed to the virtual addresses present during capture. These states can’t be handed off from a previous engine and must be recreated during every restart.

Shadow engine recovery addresses each problem with a targeted optimization: decoupling weight lifetime from the engine process, and completing non-transferable initialization before a failure.

How shadow engine recovery works[](#how_shadow_engine_recovery_works)

Shadow engine recovery combines persistent GPU memory, a pre-warmed standby engine, and worker-level coordination to recover without a cold restart.

GPU Memory Service: Persistent GPU memory for LLM inference[](#gpu_memory_service_persistent_gpu_memory_for_llm_inference)

The GPU Memory Service (GMS) manages specific memory regions, such as weights, independently of the engine process. By using a process distinct from the engine to own these regions, weights remain resident in memory even as engines are restarted. As a result, a new engine on the same GPU can attach to existing memory.

GMS is a per-GPU sidecar that owns physical GPU memory on behalf of inference engines. It is mostly dormant and has no CUDA context of its own; it allocates physical pages, hands out handles to them, and arbitrates which engines may read or write at any time. Engines connect, import handles, and map the underlying pages at virtual addresses in their own CUDA contexts. Mapping happens once, at startup, and GMS is not involved in any subsequent access.

This functionality is built on the CUDA Virtual Memory Management API. With this API, physical GPU memory and its associated virtual addresses can have independent lifetimes. Since physical allocations are reference-counted, they survive as long as any process maintains a mapping. Two engines mapping the same weight tensor access the same physical bytes, each using virtual addresses local to its context. A kernel reading a weight dereferences an ordinary pointer into the same HBM the weight would have occupied anyway, so a GMS-backed read costs no more than an engine-allocated one.

Figure 2. Each engine maps the weights into its own address space, but there is only one physical copy in HBM. GMS holds the allocation and hands out the handles; it does not sit between an engine and the memory it reads from

This architecture provides two benefits. First, weights persist beyond engine failure. While the kernel removes the failed engine’s CUDA context, the GMS reference ensures the physical pages stay resident so a fresh engine can immediately map them. Second, weights can be shared between concurrent engines; therefore, a secondary engine on the same GPU incurs zero marginal weight cost.

Integrating GMS into inference frameworks requires only a narrow change. vLLM, SGLang, and NVIDIA TensorRT-LLM each integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight memory pool. From inside the engine, weights remain ordinary torch.Tensors. Adopting GMS is as simple as flipping a flag at startup.

GMS is not limited to weights. The current preview does not support using GMS for the KV cache, but that capability is under active development. The goal is for a promoted shadow to map the outgoing engine’s cache instead of rebuilding it as traffic arrives.

Shadow engines: preinitialized standbys with zero marginal weight cost[](#shadow_engines_preinitialized_standbys_with_zero_marginal_weight_cost)

A shadow engine is a fully initialized engine process that remains idle and co-resident on the same GPUs as the active engine. Weight sharing makes this configuration feasible: without it, the second engine would require another full copy of the weights, substantially reducing the memory available for processing requests.

A shadow runs through the same startup path as an active engine. On each of its GPUs, it connects to the local GMS and imports the weight mappings, establishes communicators (NCCL and NIXL for KV transfer between workers), captures CUDA graphs, and performs any necessary warm-up. By the end of the startup, it is ready to serve. Then, instead of serving, it parks: it releases the materializable parts of its memory and blocks, waiting for its turn.

What a shadow has precomputed before it parks:

• CUDA context, captured graphs, and communicators. These non-transferable components are ready when the shadow engine is activated because they cannot be inherited from another process.

• Weight mappings. The GMS handles are already imported, so waking the shadow requires only remapping them to virtual addresses established during initialization.

What it has deferred:

• KV cache materialization. The KV cache is the largest reclaimable allocation held by an engine. The shadow reserves its address range without physical backing while parked and materializes the cache when promoted.

A parked shadow therefore retains only its CUDA context, captured graphs, communicators, and weight mappings—no separate copy of the weights and no KV cache. This footprint is small enough for the shadow to remain alongside an active engine on the same devices, enabling recovery within seconds.

The worker: a single deployable unit[](#the_worker_a_single_deployable_unit)

The following figure shows how these components fit within each worker and scale behind a shared router.

Figure 3. A fleet of workers behind a single router. The two-engine layout is internal to each worker, so the router, frontend, and orchestrator need no changes to benefit from it

These foundations are integrated into one pod. The worker holds two engine containers, a GMS sidecar to mediate GPU memory access, and a shared lock to elect the active engine.

At steady state, one engine holds the lock and remains awake, connected to GMS, holding a materialized KV cache, and registered with the frontend router. The other remains fully initialized and connected to GMS but is dormant, holds no KV cache, and waits on the lock.

Recovery in depth[](#recovery_in_depth)

The following sections trace the recovery sequence and explain the synchronization and memory-management mechanisms that make it reliable.

Sequence[](#sequence)

A worker moves through four phases before returning to steady state.

Figure 4. The four phases of a recovery. Active and shadow roles swap between engine A and engine B, and the worker returns to steady state without either engine reloading weights

• T₀ Steady. Engine A holds the lock and is awake, registered with the router. Engine B is dormant, blocked on the lock.

• T₁ Failure. Engine A’s process exits, either because it crashed outright or because a liveness probe found it hung and killed it. Either way, the kernel releases its lock as the process is reaped. The worker is briefly unroutable, until the shadow registers.

• T₂ Cutover. Engine B acquires the lock, wakes, remaps weights through GMS, materializes its KV cache, and re-registers with the router. Engine A’s container is restarted by the orchestrator.

• T₃ Restarted. Engine A finishes initialization and enters the shadow state. The system returns to a steady state, with the roles swapped.

The shadow’s advantage is that it enters T₂ already initialized. The only work on the critical path is acquiring the lock, remapping weights, and materializing the KV cache.

Synchronization[](#synchronization)

The worker requires both mutual exclusion, ensuring only one engine is awake at a time, and reliable release to ensure the standby engine takes over if the active one fails. A POSIX flock on a shared file provides these guarantees. When the active process exits due to a shutdown, segfault, or SIGKILL, the kernel reaps its file descriptors, and the shadow engine acquires the lock to begin serving.

Each engine’s startup path is therefore a short leader election:

await engine.initialize() # weight load, torch.compile, autotune, CUDA graph capture
...
# put the engine to sleep while we wait on the lock
await engine.sleep()
lock = FlockFailoverLock(lock_path)
await lock.acquire(engine_id=engine.id) # wait on the lock to wake
await engine.wake()