The night before, our lab's render pipeline encountered a strange and time-consuming symptom: flux jobs on GPU1 (RTX 5060 Ti 16GB) took over 2 minutes per step, while the same jobs on GPU0 (same card model) completed in ~54 seconds for the entire batch. We reproduced this 3 times — it wasn't random.
Read the signature correctly first
The first thing I did was capture nvidia-smi samples during 6 stuck jobs. The results formed a signature that told everything:
| Value during stuck job | Measured |
|---|---|
| SM utilization | 96–99% (looks like "working hard") |
| Memory utilization | 0–1% (no data transfer at all) |
| Card power | 38W (abnormally low for sampling) |
| Time per step | >120 seconds (normal ~4 seconds) |
Full SM but zero memory + low power = the card isn't actually computing. It's spinning in a spin-wait. This is a runtime livelock, not model load. Compare with normal jobs where the real signature is sm 96–99% + mem-util 18–41% + power 115–143W.
Only one variable actually changed
GPU0 and GPU1 use the same card model, same model (flux1-dev fp8), but differ in runtime:
- GPU0 (:8188) — imgfactory python_embeded, torch 2.13.0+cu130
- GPU1 (:8190) — venv Python 3.12, torch 2.7.1+cu128
That night's single hypothesis: runtime 2.7.1+cu128 doesn't mesh with the 5060 Ti's Blackwell architecture under heavy dequant workloads. Proof was simple — lift GPU0's proven runtime to run on GPU1 via CUDA_VISIBLE_DEVICES=1 without touching the model or workload at all.
Results after switching runtime
After switching to start_gpu1_v2 (imgfactory python_embeded, torch 2.13.0+cu130, completely separate output/input/db directories to avoid collisions with 8188):
| Gate | Result |
|---|---|
| Single job (job 85917) | 54.4 seconds — back to normal |
| 8 consecutive jobs | 50.7–127.4 seconds, all passed |
| 10 total jobs | 10/10 PASS · median 54.15 seconds |
| vs GPU0 same period | parity 100.3% (band ±50%) |
| Side effects on GPU0 | Zero — 8188 continued running and publishing images during the same period |
Measured actual throughput of GPU1 after fix = 30.5 jobs/hour (10 jobs / 1,182 seconds).
What we didn't do (and shouldn't have)
- Didn't delete old venv — left it in E:\ComfyUI per the "move don't delete" rule, just changed schtask to point to new launcher
- Didn't touch the old idle_watchdog that used to help reboot until twins appeared — removed it and handled with on-demand manual heal
- Didn't change anything else: workload, parameters, model, queue order — everything identical byte-for-byte
Lab lesson
If you encounter a GPU "slowing down," don't blame the model or add VRAM first — always read these three values: SM, memory utilization, and watts. If sm is full but memory is zero, the problem is almost entirely in the runtime. And the correct proof is to switch one variable (runtime), leave everything else as-is, then measure 10 jobs — not guess and start hunting for new drivers from scratch.
