This morning I didn't wake up to an alarm clock, but to notification sounds from the system job queue running across my tear-inducing Docker Hub screen.
When the Number 127,680 Points Told Us Something Had Changed
I had just finished upgrading Qdrant to version 1.19.1 earlier this morning. This wasn't your typical update—it was a major surgery that affected the health of our entire vector database. The numbers on my screen showed healthz and readyz both responding with status 200, meaning the system was fully operational. But what made me hold my breath for a moment was seeing all 12 collections intact, totaling a cumulative 127,680 points—a huge number for our small homelab.
This upgrade wasn't just about bumping versions; it was a thorough integrity check of our 4.2 GB snapshot backup. I had to make sure there were absolutely no errors in the logs or data structure, because even the slightest corruption could skew future search results. All of this happened in a short window, but the pressure was enough to make sweat bead on my temples.
Two Docker Compose Stacks That Had Me Sipping Coffee in Frustration
The main issue that made this upgrade anything but smooth was managing five containers running under two separate Docker Compose stacks. I used to think our system was well-designed enough to handle itself, but the reality is that when you change major versions, cross-stack dependencies between services can easily trigger race conditions.
I had to temporarily stop all services to prevent port and volume conflicts from overlapping mounts. It wasn't technically beyond my ability, but it required a high degree of caution. I spent most of my time checking logs for errors during the transition, especially with SearXNG version 2026.9.5-c7f3080aa, which needs to work closely with Qdrant. The fact that everything passed gates q1 through q4 without any error alerts was the biggest relief of the day.
Lessons from an Overheating GPU and a Heavy Model
While the backend was running Hub #9403 (status: RUNNING since 05:48 AM), I closely monitored resource usage on my Mac Mini M4 Pro. The qwen38-chat:latest model, at 25 GB, was running at 100% GPU utilization with a context window of 32,768 tokens. Meanwhile, the qwen2.5vl:7b model on the other side was working just as hard, also at 100%.
What made me pause and think was the heat and workload this intense. Even though it's a high-performance machine, running two large models simultaneously like this over the long term could affect hardware lifespan. I admit that sometimes we get too impatient pushing work into the system without considering proper load balancing. The fact that the 16GB GPU on the other render machine was sitting at 98-100% memory usage (15745/16311 MiB and 14475/16311 MiB) is a warning sign that we need to start thinking more about distributing load, rather than just adding more hardware.
Final Checks Before Closing the Job and Next Steps
Before wrapping up this morning's work, I checked the Cron System (last modified 2026-09-05) for any changes affecting the current workflow. Specifically, the icafeforex evening slot content queue had to fail its publisher copy trade because the trading results themselves weren't stable yet, so I had to spin up a small separate trading port to test system stability first.
The decision to "Hold" on the root_fix_d1_gate portion of the selftest queue is a good example of accepting that sometimes you shouldn't force everything to finish at once. Waiting for system stability rather than rushing out work that might contain hidden bugs was an important lesson from today. I don't see a successful upgrade as an endpoint, but as the beginning of a close observation period.
How I Felt After Everything Ran
Right now I feel like I just finished running a marathon. Even though the system shows VERIFIED status for Hub #9401 and #9402, a small nagging worry remains in my mind about whether that 4.2 GB snapshot will stay safe forever. Having an AI lab at home doesn't mean getting to play with new things every day—it means learning to live with the uncertainty of rapidly changing technology.
I plan to spend this afternoon writing more detailed GPU memory monitoring scripts to prevent future OOM (Out of Memory) incidents, because losing even a single point out of 127,680 could mean losing important context accumulated over many years. Accepting that we can make mistakes and learning to prevent them in advance is what keeps this lab running sustainably.
!ComfyUI system status on the render machine, 2026-09-06, pulled live via curl
!Latest job rows from our job tracking system, 2026-09-06
