vLLM v1 + Mooncake Store Architecture

Focus: pure vLLM v1 scheduler path, non-V2 GPUModelRunner with KVConnectorModelRunnerMixin, and MooncakeStoreConnector. Code pointers are from /home/zhewen/repos/vllm at commit 1c78f76c29.

1. Mental model

2. Source map

3. Engine loop

The engine loop is the backbone. It asks the scheduler for a SchedulerOutput, sends that object to workers, gets a ModelRunnerOutput, then lets the scheduler update request state.

EngineCore.step()
  scheduler_output = scheduler.schedule()
  future = model_executor.execute_model(scheduler_output, non_block=True)
  model_output = future.result() or model_executor.sample_tokens(...)
  scheduler.update_from_output(scheduler_output, model_output)

4. Core scheduler hooks

In pure v1, the core scheduler calls three connector scheduler-side methods during scheduling, and one method when a request finishes. It later calls update_connector_output() when worker output comes back.

5. V1 model runner mixin

If use_v2_model_runner is false, gpu_worker.py instantiates vllm.v1.worker.gpu_model_runner.GPUModelRunner. That class inherits KVConnectorModelRunnerMixin. The mixin is the bridge between SchedulerOutput and worker-side connector logic.

KVConnectorModelRunnerMixin._get_kv_connector_output(...)
  output = KVConnectorOutput()
  kv_connector = get_kv_transfer_group()
  kv_connector.bind_connector_metadata(scheduler_output.kv_connector_metadata)
  kv_connector.start_load_kv(get_forward_context())
  try:
      yield output                 # model forward happens here
  finally:
      kv_connector.wait_for_save()
      output.finished_sending, output.finished_recving =
          kv_connector.get_finished(scheduler_output.finished_req_ids)
      output.kv_cache_events = kv_connector.get_kv_connector_kv_cache_events()
      kv_connector.clear_connector_metadata()

6. Mooncake Store pieces

Data objects

7. External prefix lookup

Lookup begins in the core scheduler, but it is serviced by a worker-local ZMQ server on rank 0. The scheduler asks, "how many prefix tokens already exist in Mooncake Store for this request's block hashes?" The worker answers with a scheduler-safe prefix length.

8. Scheduler-to-worker metadata

MooncakeStoreConnectorMetadata is not just a list of current scheduled requests. It also carries unfinished request IDs and preemptions, and each ReqMeta tells the worker whether to load, save, both, or only poll completion.

MooncakeStoreConnectorMetadata
  unfinished_request_ids: set[str]
  preempted_req_ids: set[str]
  requests: list[ReqMeta]

ReqMeta
  req_id
  token_len_chunk
  block_ids: tuple[list[int], ...]      # one list per KV cache group
  block_hashes
  can_save
  load_spec
  is_last_chunk
  current_event
  token_ids
  original_block_size

9. Worker I/O path

Mooncake Store intentionally centralizes I/O issue and completion polling in worker-side get_finished(). The method does three things in one pass: enqueue loads, enqueue stores, and poll completions from prior work.

Load path

1. get_finished() finds each request with load_spec.can_load.
2. It sets load_spec.token_len to the external cached length.
3. It enqueues the request on KVCacheStoreRecvingThread.
4. The recv thread uses coord.load_mask() to skip chunks the local KV spec would not populate.
5. It turns each chunk into Mooncake keys and GPU address/size lists.
6. It calls batch_get_into_multi_buffers() and marks the request finished.

Store path

1. get_finished() records a CUDA event when any request can save.
2. Each save request receives that event and is enqueued on KVCacheStoreSendingThread.
3. The send thread aligns token length to coord.lcm_block_size.
4. It uses coord.store_mask() to avoid storing chunks that no future hit would need.
5. It deduplicates with batch_is_exist().
6. It synchronizes the CUDA event before reading GPU memory.
7. It calls batch_put_from_multi_buffers() and decrements outstanding store jobs.

10. Completion feedback

Completion has two meanings: finished_recving means remote KV is now loaded into local GPU blocks, while finished_sending means blocks whose free was delayed for store can now be released.

11. HMA and block-size frames

Mooncake Store with HMA is easiest to reason about if you keep three token frames separate. Mixing these frames is the usual source of wrong conclusions.

The coordinator is the HMA bridge. It mirrors vLLM cache-manager hit logic over an ExternalCachedBlockPool, not over local allocated blocks. Lookup feeds it an existence set from Mooncake keys. Load and store paths ask it for masks so each group only transfers chunks that the corresponding KV spec would actually use.

MooncakeStoreCoordinator
  find_longest_cache_hit(block_hashes, max_length, ExternalCachedBlockPool)
    -> (load_mask_per_group, hit_length)

  load_mask(block_hashes, token_len)
    -> which chunks to load locally

  store_mask(aligned_token_len)
    -> which chunks are worth storing for future hits

12. Suggested read order

1. Start with vllm/v1/engine/core.py:141-156 and 202-211 to see how scheduler/hash block sizes and request hashing are wired into v1.
2. Read the scheduler connector calls in vllm/v1/core/sched/scheduler.py:597-603, 746-751, 887-922, and 2127-2154.
3. Read vllm/v1/worker/kv_connector_model_runner_mixin.py:85-119 to understand exactly when worker-side connector methods are invoked.
4. Read mooncake/store/connector.py:80-147 to see how one connector class splits into scheduler and worker role objects.
5. Read mooncake/store/scheduler.py:84-127 for lookup and 163-366 for metadata construction.
6. Read mooncake/store/data.py:70-149 for key/address translation, then 195-290 for metadata payload shape.
7. Read mooncake/store/worker.py:1090-1186 first, then jump back to 393-623 and 626-759 for send/recv thread internals.
8. Finish with mooncake/store/coordinator.py:55-284; it explains why HMA lookup/load/store are not just flat block-list operations.