Setup

• Machine: one Runcrate 4x RTX A6000 (48 GB each) VM, US Midwest, $2.64/hour for the whole machine at launch time. Two separate machines of the same type over the morning: the first for the baseline, the second for the fixes.
• Topology (nvidia-smi topo -m): every GPU pair is PHB, i.e. GPU to GPU traffic crosses the CPU's PCIe host bridge. No NVLink. PCIe gen 3 x16 per card.
• Model: a GPT-2 style decoder written in plain PyTorch, two sizes: 124M parameters (12 layers, 768 wide) and 355M (24 layers, 1024 wide). Random tokens, sequence 1024, micro-batch 8 per GPU, bf16 autocast, fused AdamW.
• Software: torch 2.8.0+cu128 (first machine) and 2.11.0+cu128 (second), NCCL as shipped in the wheel.
• Metric: tokens/s from the median step time over 40 steps after 10 warm-up steps. Weak scaling: each GPU always gets its own micro-batch of 8.

The commands

# baseline: 1, 2 and 4 processes on the same machine
for N in 1 2 4; do torchrun --standalone --nproc_per_node $N train_ddp.py --size small; done

# all-reduce bandwidth, nccl-tests style (bus bandwidth = algbw x 2(n-1)/n)
torchrun --standalone --nproc_per_node 4 allreduce.py

# the fixes
torchrun --standalone --nproc_per_node 4 train_ddp.py --size small --hook bf16            # bf16 gradient compression
torchrun --standalone --nproc_per_node 4 train_ddp.py --size small --accum 4              # all-reduce every 4 micro-batches
torchrun --standalone --nproc_per_node 4 train_ddp.py --size small --accum 4 --hook bf16  # both

The two fixes are a few lines each in the training loop:

from torch.distributed.algorithms.ddp_comm_hooks import default_hooks
ddp = DDP(model, device_ids=[local_rank], gradient_as_bucket_view=True)
ddp.register_comm_hook(None, default_hooks.bf16_compress_hook)   # send gradients as bf16

for k in range(accum):                                            # gradient accumulation
    ctx = ddp.no_sync() if k < accum - 1 else contextlib.nullcontext()
    with ctx, torch.autocast("cuda", dtype=torch.bfloat16):
        loss = model_loss(ddp, next_batch()) / accum
        loss.backward()
opt.step(); opt.zero_grad(set_to_none=True)

Baseline: 2x from 4 GPUs

The all-reduce explains it. NCCL bus bandwidth between the four cards was 4.3 GB/s (2 GPUs: 3.9 GB/s), flat from 8 MB to 1 GB messages. DDP all-reduces fp32 gradients: about 500 MB per step for the 124M model. At 2.9 GB/s of algorithm bandwidth that is roughly 170 ms of communication against a 130 ms compute step, and 4 GPUs measured about 255 ms per step. The GPUs spend more time talking than computing.

The fixes, on 4 GPUs

(1-GPU tokens/s with accumulation 4: 66,189 and 27,893, so the "both" rows are 88% and 89% scaling efficiency against the same recipe on one GPU.)

What surprised me

1. The interconnect, not the GPU, set the speed. 4.3 GB/s of all-reduce bandwidth is a fraction of what PCIe gen 3 x16 can carry. With every pair going through the host bridge, plain DDP on four cards only doubled throughput.
2. Forcing peer-to-peer made it worse. NCCL_P2P_LEVEL=SYS (use P2P even across the host bridge) halved the all-reduce bandwidth to 2.3 GB/s and dropped 124M training from 129k to 80k tokens/s. NCCL's default choice was right; don't override it on a PHB machine.
3. Two cheap changes recovered most of it. bf16 gradient compression halves the bytes; accumulating 4 micro-batches per all-reduce cuts the number of all-reduces by 4. Together: 3.5x to 3.7x on four GPUs. Accumulation changes your effective batch size (32 per GPU here), so retune the learning rate if you adopt it.
4. Per token, one GPU still won. Even with both fixes, a token cost about 10% more on four GPUs than on one, because four GPUs at 88% efficiency is still four GPUs. Rent the 4x machine when wall-clock time matters (a 3.5x shorter run), not to save money.
5. The bigger model scaled slightly better, as expected: more compute per byte of gradient.

Caveats

• Random tokens and a synthetic model: this measures throughput, not convergence. bf16 gradient compression slightly changes numerics.
• Two machines of the same type, one run per configuration. Topology varies between machines; run nvidia-smi topo -m and a 30-second all-reduce test before a long job.
• Machines with NVLink behave completely differently; this is a PCIe-only result.
• Prices are on-demand at launch time on 2026-10-26.

Reproduce it

1. Launch a 4x GPU instance through POST /api/v1/instances with max_runtime_minutes set.
2. uv venv /opt/env && . /opt/env/bin/activate && uv pip install torch numpy --torch-backend=cu128.
3. Copy train_ddp.py and allreduce.py (about 100 lines together; the core is the snippet above) and run the torchrun commands.
4. Delete the instance.

Both 4x machines together, including setup and every configuration above, were billed $0.92.

If you want to try it on your own model: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the runs described above; raw logs are kept with the post.

What I measured

For each machine, seconds since the create call returned:

• API running: the instance API reports running and hands out the IP.
• First SSH: my script's first successful ssh root@ip echo ok (it retries every 2 s once it has the IP).
• First CUDA op: after SSH, install uv, create a venv, uv pip install torch --torch-backend=cu128, then multiply two 4096x4096 matrices on the GPU and synchronize.

I also read two clocks on the machine itself, because they tell you where the time goes: kernel boot (uptime -s) and when ssh.service came up (systemctl show ssh -p ActiveEnterTimestamp).

The commands

# 1. launch (instance_type_id from GET /api/v1/instances/types)
curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"cold-l40s","instance_type_id":"<id>","ssh_key_id":"<key id>","max_runtime_minutes":30}'

# 2. poll every 10 s until status is running and ip is set
curl -s https://www.runcrate.ai/api/v1/instances/<id> -H "Authorization: Bearer $RUNCRATE_API_KEY"

# 3. on the machine
curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv -q --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install -q torch --torch-backend=cu128
python -c "import torch,time;t=time.time();x=torch.randn(4096,4096,device='cuda');y=x@x;torch.cuda.synchronize();print(time.time()-t)"

# 4. delete
curl -s -X DELETE https://www.runcrate.ai/api/v1/instances/<id> -H "Authorization: Bearer $RUNCRATE_API_KEY"

Results

All VMs, Ubuntu image, one GPU each, launched within 4 seconds of each other at 09:02 UTC.

And from the machines' own clocks, same runs:

The torch step on its own (venv + wheel install + first matmul): 10 to 48 s for the install, 0.2 to 1.0 s for the first matmul, which is mostly CUDA context creation.

What surprised me

1. The spread is 4x, and it is not about the GPU. The L40S was doing math in about 3 minutes, the A6000 in almost 12. Same API, same image, same script. Cold start depends on where the machine comes from far more than on which card is in it, so if you launch often, measure the specific region you use.
2. On three types, the machine was ready long before the API said so. The V100, RTX 6000 Ada and A6000 booted and had sshd listening in 1 to 2 minutes, but the API reported them running about 8 minutes later. Nothing in my script could have used them earlier, because the IP arrives with running. The upside: billing starts at running, which is why the bills in the table are a cent or two.
3. The newest PyTorch wheels no longer run on a V100. uv pip install torch --torch-backend=cu128 resolved to torch 2.11.0, which warns that it was built for compute capability 7.5 and up, and the matmul then failed with cudaErrorNoKernelImageForDevice. The V100 is 7.0. On an earlier V100 run the same command had resolved to torch 2.8.0, which worked. If you rent Volta cards, pin torch (torch==2.8.* worked here) instead of taking the latest.
4. Not every create call works. Over the day I made 18 create calls. Two (an RTX 4090 and an RTX 5090 VM in the same region) came back within 7 seconds with 500 provider_unavailable, and one type was gone between listing it and launching it (400 ... no longer available). Both failures are fast and cost nothing, so the right client behaviour is: read /instances/types right before launching, and fall back to the next option on a 4xx/5xx.
5. A container is a different animal. In an earlier batch I launched an RTX 5090 as a container instead of a VM: its clock said it started 4 seconds after the create call. If startup time matters more than a full VM, check the deployment_type column.
6. My own polling made my first batch useless. I first polled nine machines every 2 seconds, hit the API's rate limit (HTTP 429) for several minutes, and the "running" times from that batch were wrong by minutes. This table is from a second batch polling every 10 seconds. Poll slowly.

Caveats

• One launch per GPU type in this table, at one time of day. Cold start depends on what is in stock at that moment; treat it as a sample, not a promise.
• "First CUDA op" includes downloading a 900 MB torch wheel. A pre-built image or a volume with a cached venv removes most of the 10 to 48 s.
• The prices are the on-demand prices at launch time and move daily.

Reproduce it

1. Create an API key, add an SSH key (POST /api/v1/ssh-keys).
2. For each type you care about: launch with the curl above, poll every 10 s or slower, SSH in as soon as there is an IP, run the torch block, delete.
3. Read uptime -s and systemctl show ssh -p ActiveEnterTimestamp on the machine if you want to know where the time went.

The six machines in the table were billed $0.07 in total.

If you want to time it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the runs described above; raw logs are kept with the post.

Setup

• Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization.
• Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --max-model-len 4096 --gpu-memory-utilization 0.90, everything else default.
• Load: vllm bench serve with random prompts, 512 tokens in and exactly 256 tokens out (--ignore-eos), at concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42. The client ran on the same machine as the server, so there is no network in these numbers.
• Machines (on-demand price at launch time):

Cost per 1M output tokens below is simply hourly price / (output tokens per second x 3600) x 1,000,000. It counts only generated tokens; the 512 prompt tokens per request are free riders in this math.

The commands

Launch (one call per GPU; the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"vllm-a6000","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":45}'

max_runtime_minutes is the safety net: the machine stops itself even if my script dies. Then, on the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' --torch-backend=cu128

vllm serve Qwen/Qwen2.5-7B-Instruct --dtype bfloat16 --max-model-len 4096 \
  --gpu-memory-utilization 0.90 --port 8000 &

for C in 1 8 32 64; do
  case $C in 1) N=16;; 8) N=64;; 32) N=192;; 64) N=320;; esac
  vllm bench serve --backend vllm --model Qwen/Qwen2.5-7B-Instruct \
    --dataset-name random --random-input-len 512 --random-output-len 256 --ignore-eos \
    --num-prompts $N --max-concurrency $C --seed 42 \
    --save-result --result-dir results --result-filename bench_c$C.json
done

Then DELETE /api/v1/instances/<id> and a check that the list is empty.

Results

"Per-user tok/s" is 1000 / median time per output token, i.e. how fast one stream feels to the person reading it.

How long each machine took before it could serve anything (measured on the machine):

What surprised me

1. The L40S is barely faster than the A6000 at decoding. It has roughly twice the bf16 compute on paper, but single-stream decode on a 7B model is limited by memory bandwidth, and the two cards are close there. 48 vs 42 tokens/s at concurrency 1, 1,887 vs 1,727 at 64. Per dollar, the A6000 wins at every load level I tried.
2. The H100 is not the per-token deal, it is the latency deal. It produced 2.6x the A6000's throughput at concurrency 64 for 4.8x the hourly price. But every user still saw 79 tokens/s at that load, against 32 on the other two cards, and the p99 time to first token stayed under a second. If your product is a chat UI with real people waiting, that is what you are paying for.
3. At concurrency 1, every GPU is expensive. $4.86 to $9.47 per million tokens. Batching is where the money is: going from 1 to 64 concurrent requests cut the cost per token by about 40x on all three cards. If your traffic can't keep the batch full, a smaller, older card idles more cheaply.
4. p99 TTFT at concurrency 64 on the A6000 was 3.3 s, while the median was 259 ms. Some requests wait for a whole batch slot. Look at tail latency before you take the cost-per-token number at face value.
5. Setup time varied more than benchmark time. The 15 GB model download took 13 s on one machine and 103 s on another. For short experiments, that download is a real part of the bill; put the weights on a volume if you will start the server more than once.

Caveats

• One run per GPU, on one morning. Treat differences under about 10% as noise.
• Random prompts, fixed 512/256 shape. Real chat traffic has prefix reuse and varied lengths; prefix caching would help all three cards.
• These are PCIe cards, one per machine. No tensor parallelism, no FP8, no speculative decoding. An FP8 checkpoint would likely change the H100 math more than the others.
• Prices are the on-demand prices at launch time on 2026-10-26 and move daily.

Reproduce it

1. Create an API key and add an SSH key (POST /api/v1/ssh-keys).
2. Pick an instance type from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set.
3. Run the setup and benchmark block, copy results/ back, delete the instance.

The three machines, including setup time, were billed $1.07 together ($0.15 for the A6000, $0.29 for the L40S, $0.64 for the H100).

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Every RAG project hits the same fork: self-host the embedding model on a GPU, or pay an embeddings API per token. The usual arguments are about privacy and control. This post is about the boring part, time and money, measured on one real corpus with the same model on both sides.

The corpus and the model

• Corpus: Runcrate's own documentation, 146 MDX pages, split on paragraph boundaries into 527 chunks of up to about 1,200 characters (501,843 characters in total).
• Model: BAAI/bge-m3 (1024-dimensional) on both sides, so quality is not a variable.
• GPU side: one NVIDIA RTX A4000 (16 GB) as a Runcrate job at $0.20/hour (the live price on 2026-10-24 at 21:35 UTC), sentence-transformers in fp16. A job runs one command on a fresh machine and stops billing when the command exits.
• API side: Runcrate's OpenAI-compatible /v1/embeddings with the same model, listed at $0.01 per 1M input tokens in GET /v1/models at the time of the run.

Both sides ran on our own platform. That is convenient for a fair test (same model, same day, same bill format), but keep it in mind. Prices on 2026-10-24 at 22:13 UTC, re-read for this post from the live feed: the A4000 is listed from $0.20 to $1.04 per hour depending on region (/api/gpu-pricing, see GPU pricing), and bge-m3 at $0.01 per 1M tokens (/v1/models).

GPU side: the job

The job uploads a folder with the corpus and one script. The command creates a venv with uv, installs torch for CUDA 12.8 and sentence-transformers, and runs:

import json, time, torch
from sentence_transformers import SentenceTransformer

docs = [json.loads(l)["text"] for l in open("corpus.jsonl")]
model = SentenceTransformer("BAAI/bge-m3", device="cuda", model_kwargs={"torch_dtype": torch.float16})
model.max_seq_length = 512

def timed_encode(texts, bs):
    torch.cuda.synchronize(); t0 = time.time()
    v = model.encode(texts, batch_size=bs, normalize_embeddings=True, convert_to_numpy=True)
    torch.cuda.synchronize()
    return v, time.time() - t0

model.encode(docs[:256], batch_size=64, normalize_embeddings=True)   # warm-up
vecs, s = timed_encode(docs, 64)                                      # the real corpus, once
for bs in (16, 64, 128):                                              # steady state: corpus x20
    _, s20 = timed_encode(docs * 20, bs)

Submitted with the Python SDK, with a hard cap on runtime:

from runcrate import Runcrate

rc = Runcrate()
up = rc.jobs.uploads.upload("./code")
job = rc.jobs.run(command=open("command.sh").read(), gpu_type=["A4000", "A6000"],
                  max_runtime_minutes=20, template="ubuntu-train", code={"upload_id": up.upload_id})

GPU results

Peak GPU memory was 2.6 GB, and the job's per-minute GPU metrics read 69 to 95% utilization while the x20 passes ran. Bigger batches were slower here, not faster.

The clock and the bill, from the job's event log and the billing transaction:

API side

from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

api = OpenAI(base_url="https://api.runcrate.ai/v1", api_key=os.environ["RUNCRATE_API_KEY"])
batches = [docs[i:i + 32] for i in range(0, len(docs), 32)]

with ThreadPoolExecutor(max_workers=4) as pool:
    out = list(pool.map(lambda b: api.embeddings.create(model="BAAI/bge-m3", input=b), batches))

Are they the same embeddings? For the first 100 chunks, 99 of the API vectors have cosine similarity of at least 0.9998 with the GPU vectors. The one outlier (0.92) is a 3,533-character table chunk: our GPU script truncates at 512 tokens, the API embedded all of it. That is a setting, not a model difference.

So which is cheaper?

Per token, the GPU is far cheaper once it is running: at batch 16 the A4000 did 50.6k tokens/s for $0.20/hour, which is $0.0011 per 1M tokens against the API's $0.0105. About 9.5x.

But a job has a fixed cost before the first token: here 67.8 s of machine setup, 26.1 s to import and load the model, and 12.1 s to shut down, about 106 billed seconds, or $0.0059. That alone is about 3.4 times what the API charged for the whole corpus.

Putting the two together:

• GPU cost ≈ $0.0059 + tokens × $0.0011 / 1M
• API cost ≈ tokens × $0.0105 / 1M
• Break-even: about 630k tokens, nearly four times the size of these docs.

The rows past the first are arithmetic from the measured throughput and overhead, not separate runs.

What we'd do

• Under about a million tokens, or anything interactive: the API. The money is trivial either way, and 5 seconds beats a 10-minute round trip, most of which was waiting for the machine.
• Re-embedding a big corpus (tens of millions of tokens) on a schedule: a GPU job. The fixed cost disappears into the total and you pay about a tenth per token. Use batch 16 to 64 for this kind of text, not the biggest batch that fits.
• Measure your own overhead. The 106 seconds above include a template where the pip installs finished in about a second. If your job installs dependencies from scratch or downloads a large model every time, the break-even moves up; cache the model on a volume.

Everything here is from one run on 2026-10-24. Prices move; check /api/gpu-pricing and /v1/models on the day you decide.

The job API we used is documented in Jobs.

Written with AI assistance. Every number above was measured on 2026-10-24 and checked against the job log, the event timeline and the billing records.