Benchmark Local LLMs vs Cloud APIs for VS Code Code Completion

Cloud-based inline code completion tools like GitHub Copilot or Tabnine send your proprietary codebase to remote servers every time you press Enter, raising immediate privacy flags for enterprise developers. While running local Large Language Models via Ollama or llama.cpp keeps your data safely on your machine, it is nearly impossible to tell if the local setup actually matches the sub-second latency and completion quality of cloud APIs without empirical data. You are going to build a repeatable benchmarking pipeline to measure time-to-first-token, token generation speed, and structural accuracy, allowing you to objectively decide whether local inference can replace your cloud subscription.

The Testing Harness Stack

To benchmark inline completion fairly, you cannot rely on manual keystrokes. You need a test harness that feeds identical partial code snippets to both a local endpoint (Ollama running on http://localhost:11434) and a cloud endpoint (such as the OpenAI API or Anthropic API), while recording exact performance metrics.

Setting Up the Benchmark Script

Create a new directory for your benchmarking suite, initialize a Python virtual environment, and install the required dependencies for async HTTP requests and metrics calculation.

mkdir llm-completion-benchmark
cd llm-completion-benchmark
python3 -m venv venv
source venv/bin/activate
pip install httpx pydantic tabulate

Create a file named benchmark.py and populate it with the harness logic. This script sends a prompt simulating an active editor state, measures the time until the first byte arrives, and captures the total completion time and token count.

import asyncio
import time
import httpx
from pydantic import BaseModel
from tabulate import tabulate

class BenchmarkResult(BaseModel):
    provider: str
    model: str
    ttft_ms: float
    total_time_ms: float
    tokens_generated: int
    tokens_per_sec: float
    output_text: str

PROMPT_SNIPPET = """def calculate_checksum(file_path: str, chunk_size: int = 8192) -> str:
    """

async def benchmark_ollama(model_name: str) -> BenchmarkResult:
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model_name,
        "prompt": PROMPT_SNIPPET,
        "stream": True,
        "options": {
            "temperature": 0.1,
            "num_predict": 64
        }
    }
    
    async with httpx.AsyncClient(timeout=30.0) as client:
        start_time = time.perf_counter()
        first_token_time = None
        full_text = ""
        token_count = 0
        
        async with client.stream("POST", url, json=payload) as response:
            async for line in response.aiter_lines():
                if not line:
                    continue
                import json
                data = json.loads(line)
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                
                chunk = data.get("response", "")
                full_text += chunk
                token_count += 1
                if data.get("done", False):
                    break
                    
        end_time = time.perf_counter()
        
    ttft = (first_token_time - start_time) * 1000 if first_token_time else 0
    total_time = (end_time - start_time) * 1000
    tps = (token_count / (total_time / 1000)) if total_time > 0 else 0
    
    return BenchmarkResult(
        provider="Ollama (Local)",
        model=model_name,
        ttft_ms=ttft,
        total_time_ms=total_time,
        tokens_generated=token_count,
        tokens_per_sec=tps,
        output_text=full_text.strip()
    )

async def benchmark_openai(api_key: str, model_name: str) -> BenchmarkResult:
    url = "https://api.openai.com/v1/chat/completions"
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": model_name,
        "messages": [{"role": "user", "content": f"Complete this Python function inline. Return only the completion code:\n{PROMPT_SNIPPET}"}],
        "max_tokens": 64,
        "temperature": 0.1,
        "stream": True
    }
    
    async with httpx.AsyncClient(timeout=30.0) as client:
        start_time = time.perf_counter()
        first_token_time = None
        full_text = ""
        token_count = 0
        
        async with client.stream("POST", url, json=payload, headers=headers) as response:
            async for line in response.aiter_lines():
                if not line.startswith("data: "):
                    continue
                data_str = line[6:].strip()
                if data_str == "[DONE]":
                    break
                import json
                try:
                    data = json.loads(data_str)
                    delta = data["choices"][0]["delta"].get("content", "")
                    if delta:
                        if first_token_time is None:
                            first_token_time = time.perf_counter()
                        full_text += delta
                        token_count += 1
                except json.JSONDecodeError:
                    pass
                    
        end_time = time.perf_counter()
        
    ttft = (first_token_time - start_time) * 1000 if first_token_time else 0
    total_time = (end_time - start_time) * 1000
    tps = (token_count / (total_time / 1000)) if total_time > 0 else 0
    
    return BenchmarkResult(
        provider="OpenAI (Cloud)",
        model=model_name,
        ttft_ms=ttft,
        total_time_ms=total_time,
        tokens_generated=token_count,
        tokens_per_sec=tps,
        output_text=full_text.strip()
    )

async def main():
    import os
    results = []
    
    # Test Local Model (Ensure Ollama is running qwen2.5-coder:7b)
    try:
        res_local = await benchmark_ollama("qwen2.5-coder:7b")
        results.append(res_local)
    except Exception as e:
        print(f"Local benchmark failed: {e}")
        
    # Test Cloud Model (Set OPENAI_API_KEY environment variable)
    api_key = os.getenv("OPENAI_API_KEY")
    if api_key:
        try:
            res_cloud = await benchmark_openai(api_key, "gpt-4o-mini")
            results.append(res_cloud)
        except Exception as e:
            print(f"Cloud benchmark failed: {e}")
            
    table_data = [
        [r.provider, r.model, f"{r.ttft_ms:.2f} ms", f"{r.total_time_ms:.2f} ms", f"{r.tokens_per_sec:.2f}", r.output_text[:40] + "..."]
        for r in results
    ]
    print(tabulate(table_data, headers=["Provider", "Model", "TTFT", "Total Time", "Tokens/Sec", "Sample Output"]))

if __name__ == "__main__":
    asyncio.run(main())

Execute your script with your OpenAI API key set in your environment to run the comparison.

python benchmark.py

Expected output structure:

Provider         Model           TTFT      Total Time    Tokens/Sec  Sample Output
---------------  --------------  --------  ------------  ------------  ----------------------------------------
Ollama (Local)   qwen2.5-coder   142.30 ms  850.12 ms     75.18        import hashlib\n\n    hash_sha25...
OpenAI (Cloud)   gpt-4o-mini     312.45 ms  1210.89 ms    52.80        import hashlib\n\n    sha256 = hashli...

Configuring VS Code for Local LLM Completions

Once your benchmarks confirm your local model hits the target metrics (ideally under 200ms Time-To-First-Token and above 40 tokens per second), configure VS Code to use the local endpoint for inline completions instead of a commercial extension.

Installing and Pointing Continue.dev

Instead of extensions that lock you into cloud telemetry, install Continue, an open-source AI code assistant that natively supports local OpenAI-compatible endpoints like Ollama, llama.cpp, or vLLM.

1. Open the VS Code Extensions view (Cmd+Shift+X on macOS, Ctrl+Shift+X on Windows/Linux).
2. Search for Continue and click Install.
3. Open the Continue sidebar icon on the left panel.
4. Click the gear icon at the bottom right of the Continue sidebar to open ~/.continue/config.json.

Update your config.json to route inline completions (tab-autocomplete) directly to your local Ollama instance running a specialized coding model.

{
  "models": [
    {
      "title": "Qwen 2.5 Coder 7B",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen 2.5 Coder 1.5B",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b"
  }
}

Note: For inline autocomplete, use a smaller 1.5B or 3B parameter model rather than a 7B or 14B model. Inline suggestions require instantaneous responses; larger models introduce noticeable UI lag while typing.

Validating Ollama Performance Parameters

If your local benchmark shows poor tokens-per-second performance, your local inference server is likely running on your CPU or lacking optimal hardware acceleration flags. Ensure Ollama utilizes your GPU by checking your server logs or setting explicit thread limits.

Stop any running background instance of Ollama and start it with explicit environment variables to maximize hardware utilization.

# For Apple Silicon Macs (Metal is enabled by default)
OLLAMA_NUM_PARALLEL=2 ollama serve

# For Linux with NVIDIA GPUs (ensure CUDA toolkit is installed)
CUDA_VISIBLE_DEVICES=0 OLLAMA_NUM_PARALLEL=2 ollama serve

Verify that Ollama offloaded all layers to the GPU by running a model inspection command in a separate terminal tab:

ollama ps

Expected output indicating full GPU acceleration:

NAME                  ID           SIZE     PROCESSOR    UNTIL
qwen2.5-coder:1.5b    f2c03387...  1.9 GB   100% GPU     4 minutes from now

Troubleshooting Common Local Completion Bottlenecks

Local LLM setups for inline editing frequently break due to resource contention or misconfigured context windows. Use this troubleshooting guide to resolve the most common blockers.

High Time-To-First-Token (TTFT > 500ms)

If your model generates text quickly once it starts, but pauses half a second before showing any completion, your context window or prompt prefix is too large.

1. Open your ~/.continue/config.json.
2. Add a contextWindow and maxOutputTokens constraint to your tabAutocompleteModel block:

"tabAutocompleteModel": {
  "title": "Qwen 2.5 Coder 1.5B",
  "provider": "ollama",
  "model": "qwen2.5-coder:1.5b",
  "contextWindow": 2048,
  "maxOutputTokens": 64
}

VS Code UI Freezing During Typing

If your entire editor stutters when you type rapidly, synchronous extension calls are blocking the main VS Code rendering thread.

1. Open VS Code Settings (Cmd+, or Ctrl+,).
2. Search for Continue: Enable Tab Autocomplete.
3. Ensure that debouncing is enabled and set to at least 350ms so completions do not fire on every single keystroke.

Alternatively, add these properties directly to your VS Code settings.json:

{
  "continue.enableTabAutocomplete": true,
  "continue.autocompleteDebounce": 350
}

Maintaining and Scaling Your Setup

Benchmarking is not a one-time task; model weights update, and VS Code extensions introduce performance regressions with every minor release. Re-run your benchmark.py script monthly against new model releases such as DeepSeek-Coder-V2 or updated Qwen iterations.

If your hardware cannot sustain local 7B model speeds, do not fall back entirely to cloud APIs. Instead, route your inline completions through a smaller local model (like starcoder2:3b) for instant boilerplate generation, and configure a cloud fallback model strictly for heavy chat refactoring where latency is acceptable.