You type a question into your own chat model, hit enter, and a few seconds later there's an answer — private, free, no token meter ticking. You lean back feeling pretty clever. You stopped renting AI; you own it now.
Then the electricity bill shows up, and the jump is bigger than a handful of seconds a day could explain. Here's the twist: those bursts of typing barely cost a thing. The real charge is racking up in the hours you're not even at the keyboard.
This one's about where the power is really being used — the idle "kept warm, just in case" hours, the one GPU trick that cuts power without slowing your answers, and how to stop choosing between letting the PC sleep and watching an overnight job die.
Where the watts actually go
A local-AI machine has three states. The expensive ones are “model loaded, GPU idle” and “waiting, 24/7,” not active inference:
- Active inference. A high-end card at stock settings can pull 300–450 W while generating (diffusion and prompt processing push the high end). Sounds bad — but each request only takes seconds.
- Model loaded, GPU idle. Between requests, the model can sit resident in VRAM — for minutes or indefinitely, depending on your keep-alive settings. The GPU idles noticeably higher than an empty desktop — and a multi-monitor, high-refresh setup raises that floor further.
- Waiting, 24/7. If the box serves an API for your phone, your editor plugin, or Home Assistant, it can never sleep — so the whole machine (CPU, fans, board, drives) idles around the clock.
Unless you keep it generating all day, your "AI server" spends most of its time with the model loaded and the GPU idle, or waiting 24/7, drawing power to do nothing. The time it actually spends on inference is very short next to those idle hours.
The power-limit trick (you lose less speed than you think)
r/LocalLLaMA has known this one for a while: LLM inference is mostly memory-bandwidth-bound, not compute-bound. Cap the GPU's power and the clocks drop, but tokens/sec barely moves. Community tests on cards like the 3090 and 4090 consistently find that cutting the power limit by 25–35% costs only a few percent in tokens/sec — while cutting heat and fan noise along with the watts.
The manual version, in an admin shell:
nvidia-smi -pl 300— cap the card at 300 W (your card's valid range:nvidia-smi -q -d POWER).- Benchmark your own setup before and after — five minutes with your usual model tells you the real cost. Image generation is more compute-bound than LLMs, so expect a bigger hit there and cap less aggressively.
- Note: the limit resets on reboot, so it needs a startup task to stick.
The sleep dilemma (the other half of the problem)
Power limits handle the watts. Sleep is where Windows gets actively hostile to local AI:
- Long jobs die. A fine-tune, a big batch run, an overnight embedding job — Windows' sleep timer counts keyboard and mouse, not CUDA. No input for 30 minutes → "idle" → sleep → hours of GPU work gone.
- Serving dies too. If the box sleeps, your Ollama endpoint goes dark and every client breaks.
So everyone applies the same fix: disable sleep entirely. Which is how a privacy decision quietly becomes a machine that burns full idle power forever. Each step was rational; the sum is the worst of both worlds.
The static-settings treadmill, AI edition
You can wire all of this up by hand — a startup task for nvidia-smi, sleep disabled, maybe a scheduled script that tightens the power limit at night and loosens it for daytime use. It works, the same way the homelab version works, until the day you're gaming at 8 PM on a GPU still capped for overnight serving, or another program switches your power plan. The script never notices, and the only place it shows up is a later electricity bill.
Automating it
This is exactly the kind of problem PowerDoze exists for. The pieces, mapped to local AI:
- Keep-awake whitelist — while a program on the list is running, the PC stays awake. So add the program that actually runs the long job, such as the one running your training or batch-generation script; for a Python script, that's usually
python(if you also keep other Python programs open all the time, the PC will stay awake for those too). Don't addollama: on Windows, Ollama starts when you sign in and keeps running in the background, so whitelisting it means the PC never sleeps. To add a program, open Whitelist in the left sidebar and click Add App. If the program is running, click it on the Running tab; if not, switch to the Manual tab, type its name (no .exe needed), and click Add. Then click Add Selected. Once the job finishes and the program exits, the PC goes back to your normal sleep settings. Long jobs keep running, and the PC doesn't waste power on idle nights. - Power modes with an NVIDIA power limit built in — bundle GPU watts, CPU cap, and cooling policy into modes, then schedule them: full power during the day, a capped "serving mode" overnight. The GPU limit is re-applied automatically after every reboot.
- Analytics (part of Pro) — estimated watts, kWh, and cost over time, so "did capping the GPU actually matter?" gets a number instead of a feeling.
Honest boundary: PowerDoze doesn't schedule your AI jobs, doesn't touch VRAM or model loading, and doesn't make inference faster. It manages the part around the AI — sleep, power limits, schedules, and measurement — which happens to be where the waste lives.
Schedules, power modes, the whitelist, and monitoring all come in the free version, no account required. The free version allows up to 3 schedule rules and 2 power modes. This post's setup uses one mode for daytime and one for overnight, so 2 modes is enough. Everything runs locally — which, if you're running local AI for privacy reasons, is presumably a feature you care about.
Frequently asked questions
How long does Ollama keep a model loaded in VRAM?
Ollama's documentation says 5 minutes by default: after 5 minutes with no new request, the model is unloaded, which ends the “model loaded, GPU idle” state. To change it, quit Ollama from the taskbar, search Windows for “environment variables”, choose “Edit environment variables for your account”, add OLLAMA_KEEP_ALIVE (for example 1m for one minute, 0 to unload right after each answer, or -1 to keep it loaded), then start Ollama again from the Start menu.
How do I see how many watts my GPU is drawing right now?
Run nvidia-smi --query-gpu=power.draw --format=csv -l 1 in Command Prompt; it prints the card's current power draw once a second, so you can watch it while your model runs. Press Ctrl+C to stop.
Can I follow this with an AMD or Intel GPU?
Partly. nvidia-smi only works with NVIDIA cards, and PowerDoze only shows the GPU Power Limit setting in a power mode on PCs with an NVIDIA card; the CPU cap, cooling policy, and schedules work either way. For the GPU's own power limit, use your GPU maker's software — we haven't tested that part.
Who doesn't need this
If your models run in the cloud, your power bill is someone else's problem. If you do inference on a laptop, the battery already polices waste. And if your AI box is a dedicated headless Linux server, PowerDoze can't help — it's Windows-only; nvidia-smi plus a systemd timer is your friend. This is for the most common setup of all: the Windows desktop that is both your daily machine and your AI box — the one that needs full power at 2 PM and almost none at 2 AM, and currently gets full power at both.
Rather not write startup tasks and scheduled scripts yourself? PowerDoze's power modes and schedule rules are free: full power by day, capped overnight, and switched for you every day once you set it up.
