The switch that turns off every GPU

Our server at home has fourteen graphics cards in it. With nothing loaded they still draw about 295 watts between them, around 20 watts a card, all day. One command takes every one of them down to nothing while the server keeps running everything else. This is how that works, what went wrong while we built it, and the arithmetic that changed how we use it.

One relay at the head of a chain

The cards have power supplies of their own, four of them, beside the one that runs the server. The four are chained: each one starts the next through a small ADD2PSU adapter board, which switches a supply on when it sees 12 volts from the one before it. The cards sit on powered risers, so with the chain off they get nothing at all, not even through the slot.

At the head of the chain is a Zigbee relay. It sits in the 12 volt line from the server’s supply to the first adapter. Closed, the 12 volts pass and the four supplies come up one after another. Open, the first adapter sees nothing and the whole chain stays off.

The relay is one more device on the house’s Zigbee network, so turning the cards off is one message:

mosquitto_pub -t 'zigbee2mqtt/GPU Chain/set' -m '{"state":"OFF"}'

Smart plugs on the same network report what the supplies really draw, which is where the numbers in this post come from.

The order matters

There is one rule: never cut power to a card that its driver is still using. Everything in the script exists to make that true first. gpu-power-switch detach-all does this, in order:

1. Scales down every workload that uses a card, and records how many copies of each were running.
2. Waits until nothing has a card open. If something still does after four minutes, it stops there. The cards stay powered and the work stays stopped, so it can simply be run again.
3. Unbinds the driver from each card, then removes the cards from the system.
4. Opens the relay.

gpu-power-switch attach closes the relay and restarts the server. When it comes back up, a small service puts every workload back the way step 1 recorded it.

What went wrong

It only stopped what it could see. The first versions found workloads by asking the cluster which ones had requested a card. Some of ours pin a card by its ID instead and request nothing, so they were invisible, and a development server that restarts itself grabbed the cards again halfway through. The unbind then hung inside the kernel, where nothing can interrupt it. The script now also looks for pinned cards, and stops whole process groups, not single processes.

It did not wait long enough. It gave the holders about nine seconds, printed a warning and unbound anyway. A program with a model loaded needs far longer than that to exit, and an unbind that starts too early does not recover when the program finally leaves. That is why step 2 is now a hard stop: giving up early costs nothing, and a hung unbind costs a restart.

A hung write could hang everything. One stuck write once held up a shutdown for about 36 minutes. Every write to the kernel now runs with a 30 second limit, so the script can report a stuck card and move on.

Unloading the driver hung too. On the first run where all fourteen unbinds worked, unloading the driver module hung instead, and the script never reached the relay. That left fourteen cards powered with no driver, which is the worst state of all: with no driver there is no power management. Unloading is now tried for 45 seconds and skipped if it will not finish, because by then the cards are already let go and safe to switch off.

Turning them back on takes a restart

We tried hard to bring the cards back without restarting the server. It cannot be done on this board. A card that has been fully powered off needs its reset line released after its power is steady, and the board only does that when it starts up. Rescanning found 5 of the 14 cards, and none of those finished starting. Resetting each slot’s bridge found 5 of 14 again, not the same five, running at the slowest link speed there is. So attach restarts the server. The boot alone takes about eight and a half minutes, and the work is back a few minutes after that.

Two things from those attempts are worth keeping. nvidia-smi -L prints “No devices found” and still exits with success, so count the cards it lists and never trust the exit code. And to prove a card is really off, read its vendor ID with setpci: a card with no power answers ffff.

The arithmetic that changed the plan

The first idea was a schedule: cards off at 6 PM and back at 9 PM, the summer hours when our power costs the most. Measured at the plugs:

Nearly all of the saving comes from the first step down. Stopping the work saves about 1,610 watts, and that needs no relay and no restart. Cutting the power as well saves the last 295. Over a year of peak hours that is about 230 kilowatt-hours: about $45 a year at the price we used when we did the sum, and about $61 at today’s peak price. Against that stood a restart every night of everything else the server runs, and every way of hanging described above.

So on 1 August 2026 we took the schedule out. The relay and the script are still there, used by hand, for when we want every card dark.

One trap in between: do not let go of the driver and leave the cards powered as a middle setting. Cards with no driver draw more than idle cards with one.

If you build one

• Do the arithmetic before the engineering. Ours said to stop the work and leave the power alone.
• Make “is anything still using a card” a hard stop, with a long wait in front of it.
• Put a time limit on every write to the kernel.
• Count working cards. Do not trust exit codes.
• Plan for a restart to bring the cards back, and check what else that restart takes down with it.

The same peak prices drive what the rest of the house does. That is in A house that gets ahead of peak prices.