1
Desc: Configuring automatic UPS shutdowns
4
Auth: Russell Kroll <rkroll@exploits.org>
9
When your UPS batteries get low, the operating system needs to be brought
10
down cleanly. Also, the UPS load should be turned off so that all devices
11
that are attached to it are forcibly rebooted.
13
Here are the steps that occur when a critical power event happens:
15
1. The UPS goes on battery
17
2. The UPS reaches low battery (a "critical" UPS)
19
3. The upsmon master notices and sets "FSD" - the "forced shutdown"
20
flag to tell all slave systems that it will soon power down the load.
22
(If you have no slaves, skip to step 6)
24
4. upsmon slave systems see "FSD" and:
26
- generate a NOTIFY_SHUTDOWN event
27
- wait FINALDELAY seconds - typically 5
28
- call their SHUTDOWNCMD
29
- disconnect from upsd
31
5. The upsmon master system waits up to HOSTSYNC seconds (typically 15)
32
for the slaves to disconnect from upsd. If any are connected after
33
this time, upsmon stops waiting and proceeds with the shutdown
38
- generates a NOTIFY_SHUTDOWN event
39
- waits FINALDELAY seconds - typically 5
40
- creates the POWERDOWNFLAG file - usually /etc/killpower
41
- calls the SHUTDOWNCMD
43
7. On most systems, init takes over, kills your processes, syncs and
44
unmounts some filesystems, and remounts some read-only.
46
8. init then runs your shutdown script. This checks for the
47
POWERDOWNFLAG, finds it, and tells the UPS driver(s) to power off
50
9. The system loses power.
52
10. Time passes. The power returns, and the UPS switches back on.
54
11. All systems reboot and go back to work.
59
1. Make sure your POWERDOWNFLAG setting in upsmon.conf points somewhere
60
reasonable. Specifically, that filesystem must be mounted when your
63
2. Edit your shutdown scripts to check for the POWERDOWNFLAG so they know
64
when to power off the UPS. You must check for this file, as you don't
65
want this to happen during normal shutdowns!
67
You can use upsdrvctl to start the shutdown process in your UPS
68
hardware. Use this script as an example, but change the paths to
71
if (test -f /etc/killpower)
73
echo "Killing the power, bye!"
74
/usr/local/ups/bin/upsdrvctl shutdown
78
# uh oh... the UPS poweroff failed!
79
# you probably should reboot here to avoid getting stuck
80
# *** see the section on power races below ***
83
Make sure the filesystem containing upsdrvctl, ups.conf and your UPS
84
driver(s) is mounted when the system gets to this point. Otherwise
85
it won't be able to figure out what to do.
90
NOTE: If you run any sort of RAID equipment, make sure your arrays
91
are either halted (if possible) or switched to "read-only" mode.
92
Otherwise you may suffer a long resync once the system comes back up.
94
The kernel may not ever run its final shutdown procedure, so you
95
must take care of all array shutdowns in userspace before upsdrvctl
98
If you use software RAID (md) on Linux, get mdadm and try using
99
'mdadm --readonly' to put your arrays in a safe state. This has to
100
happen after your shutdown scripts have remounted the filesystems.
102
On hardware RAID or other kernels, you have to do some detective work.
103
It may be necessary to contact the vendor or the author of your
104
driver to find out how to put the array in a state where a power loss
105
won't leave it "dirty".
107
My understanding is that 3ware devices on Linux will be fine unless
108
there are pending writes. Make sure your filesystems are remounted
109
read-only and you should be covered.
111
Multiple UPS shutdowns
112
======================
114
If you have multiple UPSes connected to your system, chances are that you
115
need to shut them down in a specific order. The goal is to shut down
116
everything but the one keeping upsmon alive at first, then you do that one
119
To set the order in which your UPSes receive the shutdown commands, define
120
the "sdorder" value in your ups.conf.
137
The order runs from 0 to the highest number available. So, for this
138
configuration, the order of shutdowns would be misc, littleguy, and then
141
If you have a UPS that shouldn't be shutdown when running "upsdrvctl
142
shutdown", set the sdorder to -1.
147
To see how upsdrvctl will behave without actually turning off power, use
148
the -t argument. It will display the sequence without actually calling
154
You may delete the POWERDOWNFLAG in the startup scripts, but it is not
155
necessary. upsmon will clear that file for you when it starts.
157
Remember that some operating systems unmount a good number of filesystems
158
when going into read-only mode. If the UPS software is installed to /usr
159
and it's not mounted, your shutdowns will fail. If this happens, either
160
make sure it stays mounted at shutdown, or install to another partition.
165
There is a situation where the power may return during the shutdown
166
process. This is known as a race. Here's how we handle it.
168
"Smart" UPSes typically handle this by using a command that forces the UPS
169
to power the load off and back on. This way, you are assured that the
170
systems will restart even if the power returns at the worst possible
173
Contact closure units (ala genericups), on the other hand, have the
174
potential for a race when feeding multiple systems. This is due to the
175
design of most contact closure UPSes. Typically, the "kill power" line
176
only functions when running on battery. As a result, if the line power
177
returns during the shutdown process, there is no way to power down the
180
The workaround is to force your systems to reboot after some
181
interval. This way, they won't be stuck in the halted state with the UPS
182
running on line power.
187
The easiest way to see if your configuration will handle a power race
188
successfully is to do 'upsmon -c fsd'. This will force the UPS software
189
to shut down as if it had a OB+LB situation, and your shutdown script
190
should call the UPS driver(s) in shutdown mode.
192
If everything works correctly, the computer will be forcibly powered off,
193
may remain off for a few seconds to a few minutes (depending on the
194
driver and UPS type), then will power on again.
196
If your UPS just sits there and never resets the load, you are vulnerable
197
to the above power race and should add the "reboot after timeout" hack
203
UPS equipment varies from manufacturer to manufacturer and even within
204
model lines. You should test the shutdown sequence on your systems before
205
leaving them unattended. A successful sequence is one where the OS halts
206
before the battery runs out, and the system restarts when power returns.
211
If your UPS powers up immediately after a power failure instead of
212
waiting for the batteries to recharge, you can rig up a little hack to
213
handle it in software.
215
Essentially, you need to test for the POWERDOWNFLAG in your *startup*
216
scripts while the filesystems are still read-only. If it's there, you
217
know your last shutdown was caused by a power failure and the UPS
218
battery is probably still quite weak.
220
In this situation, your best bet is to sleep it off. Pausing in your
221
startup script to let the batteries recharge with the filesystems in a
222
safe state is recommended. This way, if the power goes out again, you
223
won't face a situation where there's not enough battery capacity left
224
for upsmon to do its thing.
226
Exactly how long to wait is a function of your UPS hardware, and will
227
require careful testing.
229
If this is too evil for you, buy another kind of UPS that will either
230
wait for a minimum amount of charge, a minimum amount of time, or both.