APC SmartUPS 750, apcupsd and power failure: what exactly happens, where “UPS Rbt off” and “UPS StayOff” come from and which delays are configurable and which are not

In theory, an Uninterruptable Power Supply (UPS)’s workings such as the APC SmartUPS 750 in connection with an attached PC are trivial: when power fails and battery capacity runs low, the PC is shut down. When power returns, the PC is turned on again. In reality, there can be confusing messages such as UPS Rbt off by USB in: XXs and UPS StayOff by USB for: XXs that are not only completely undocumented in APC’s manuals but amazingly also on the Internet.

A fact that rewards writing down what I learned as a blog post. So here’s what happens step by step, based on my findings with apcupsd 3.14.14 and an APC SmartUPS 750 UPS (firmware 7.x):

1. The power fails, apcupsd will log that and mail us but nothing else happens until either the limit for remaining battery capacity, BATTERYLEVEL or the limit for estimated remaining runtime, MINUTES has been reached or TIMEOUT seconds have passed since apcupsd noticed the UPS had to transfer to powering the PC from batteries.

2. 

   apcupsd executes /etc/apcupsd/apccontrol doshutdown which initiates either a shutdown or hibernation. With SUSE distributions this is set in /etc/sysconfig/apcupsd. We will keep examining the shutdown scenario here.
3. At the end of shutdown systemd executes a shutdown script (with SUSE distributions /usr/lib/systemd/system-shutdown/apcupsd_shutdown) which usually checks for presence of /etc/apcupsd/powerfail, which has been created beforehand by apcupsd. The file exists, thus the script calls /etc/apcupsd/apccontrol killpower.
4. The apccontrol script, now called with the killpower argument, once again warns that power to the PC is going to be killed. After a delay of 10 seconds it calls apcupsd --killpower, which tries to program the UPS to shutdown, detail of which we will see below.
5. Since we’re in the “modern” ATX world, the attached PC turns off itself. Well, soft-off, not physically off as in flipping a power switch.
6. The UPS will wait some time before it really kills the power to the PC and show on its display “UPS Rbt off by USB in: XXs” — a message totally undocumented just like the fact that this delay is not configurable in apcupsd! This delay is instead configurable through the USV’s menu: Control -> Main group outlets -> Turn Off Delay (the SmartUPS 750 has only one controllable outlet group). Also note that the UPS will ignore if the power returns during this countdown.

7. 

   The UPS shuts down, meaning: the main group outlets will be turned off, in case that wasn’t clear. The UPS itself will still be controllable via its display and keys.
8. The UPS will then show another weird message: “UPS StayOff by USB for: XXs”. Again, you can try googling it but it’s hard to find any explanation of this message, so I will have to jump in: this is a delay during which the UPS will for whatever reason stay off, further ignoring any possibly returning power. This delay is also neither configurable in apcupsd nor at the unit itself, especially it’s neither “Turn On Delay” nor “Minimum Return Runtime”. Instead it’s the result of the earlier mentioned apcupsd trying to program the UPS. I had to read its sources to find out that a.) a value of 10 seconds is hard-coded into apcupsd and b.) some/most (?) UPSes apparently round up that value to the next 60 seconds. So, no, you can’t actually modify this delay.
9. Now after this delay, should the power have returned the UPS will turn on the main group outlets immediately. No, again, it will not honor the configured Turn On Delay.
10. If you configured the attached PC correctly to power on after power loss, it will now boot again automatically.

So basically this all will work just fine even if you’re physically away and the power fails for a longer time that the UPS’s batteries last, provided that when the power returns it stays available. If the power fails when the system boots again, the then remaining battery capacity hopefully lasts long enough for apcupsd to immediately shutdown the system again.

But what’s the use of the Power On Delay then? It seems to be only used when you use the UPS’s interface and go to Control -> Power On with Delay. Wow. Impressive. Maybe that’s when Minimum Return Runtime is also actually taken into consideration…

Some of my observations may be specific to my particular UPS and firmware versions, but I guess we can once again thank APC for such a brilliant undocumented interface. Read more about all the fun in apcupsd’s usb driver sourcecode lines 805ff, e.g. line 963/964.