1
Fault injection capabilities infrastructure
2
===========================================
4
See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
7
Available fault injection capabilities
8
--------------------------------------
12
injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
16
injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
20
injects disk IO errors on devices permitted by setting
21
/sys/block/<device>/make-it-fail or
22
/sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
26
injects MMC data errors on devices permitted by setting
27
debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
29
Configure fault-injection capabilities behavior
30
-----------------------------------------------
34
fault-inject-debugfs kernel module provides some debugfs entries for runtime
35
configuration of fault-injection capabilities.
37
- /sys/kernel/debug/fail*/probability:
39
likelihood of failure injection, in percent.
42
Note that one-failure-per-hundred is a very high error rate
43
for some testcases. Consider setting probability=100 and configure
44
/sys/kernel/debug/fail*/interval for such testcases.
46
- /sys/kernel/debug/fail*/interval:
48
specifies the interval between failures, for calls to
49
should_fail() that pass all the other tests.
51
Note that if you enable this, by setting interval>1, you will
52
probably want to set probability=100.
54
- /sys/kernel/debug/fail*/times:
56
specifies how many times failures may happen at most.
57
A value of -1 means "no limit".
59
- /sys/kernel/debug/fail*/space:
61
specifies an initial resource "budget", decremented by "size"
62
on each call to should_fail(,size). Failure injection is
63
suppressed until "space" reaches zero.
65
- /sys/kernel/debug/fail*/verbose
68
specifies the verbosity of the messages when failure is
69
injected. '0' means no messages; '1' will print only a single
70
log line per failure; '2' will print a call trace too -- useful
71
to debug the problems revealed by fault injection.
73
- /sys/kernel/debug/fail*/task-filter:
76
A value of 'N' disables filtering by process (default).
77
Any positive value limits failures to only processes indicated by
78
/proc/<pid>/make-it-fail==1.
80
- /sys/kernel/debug/fail*/require-start:
81
- /sys/kernel/debug/fail*/require-end:
82
- /sys/kernel/debug/fail*/reject-start:
83
- /sys/kernel/debug/fail*/reject-end:
85
specifies the range of virtual addresses tested during
86
stacktrace walking. Failure is injected only if some caller
87
in the walked stacktrace lies within the required range, and
88
none lies within the rejected range.
89
Default required range is [0,ULONG_MAX) (whole of virtual address space).
90
Default rejected range is [0,0).
92
- /sys/kernel/debug/fail*/stacktrace-depth:
94
specifies the maximum stacktrace depth walked during search
95
for a caller within [require-start,require-end) OR
96
[reject-start,reject-end).
98
- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
100
Format: { 'Y' | 'N' }
101
default is 'N', setting it to 'Y' won't inject failures into
102
highmem/user allocations.
104
- /sys/kernel/debug/failslab/ignore-gfp-wait:
105
- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
107
Format: { 'Y' | 'N' }
108
default is 'N', setting it to 'Y' will inject failures
109
only into non-sleep allocations (GFP_ATOMIC allocations).
111
- /sys/kernel/debug/fail_page_alloc/min-order:
113
specifies the minimum page allocation order to be injected
118
In order to inject faults while debugfs is not available (early boot time),
124
mmc_core.fail_request=<interval>,<probability>,<space>,<times>
126
How to add new fault injection capability
127
-----------------------------------------
129
o #include <linux/fault-inject.h>
131
o define the fault attributes
133
DECLARE_FAULT_INJECTION(name);
135
Please see the definition of struct fault_attr in fault-inject.h
138
o provide a way to configure fault attributes
142
If you need to enable the fault injection capability from boot time, you can
143
provide boot option to configure it. There is a helper function for it:
145
setup_fault_attr(attr, str);
149
failslab, fail_page_alloc, and fail_make_request use this way.
152
fault_create_debugfs_attr(name, parent, attr);
156
If the scope of the fault injection capability is limited to a
157
single kernel module, it is better to provide module parameters to
158
configure the fault attributes.
160
o add a hook to insert failures
162
Upon should_fail() returning true, client code should inject a failure.
164
should_fail(attr, size);
169
o Inject slab allocation failures into module init/exit code
174
echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
175
echo 10 > /sys/kernel/debug/$FAILTYPE/probability
176
echo 100 > /sys/kernel/debug/$FAILTYPE/interval
177
echo -1 > /sys/kernel/debug/$FAILTYPE/times
178
echo 0 > /sys/kernel/debug/$FAILTYPE/space
179
echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
180
echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
184
bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
189
echo "Usage: $0 modulename [ modulename ... ]"
196
faulty_system modprobe $m
199
faulty_system modprobe -r $m
202
------------------------------------------------------------------------------
204
o Inject page allocation failures only for a specific module
208
FAILTYPE=fail_page_alloc
213
echo "Usage: $0 <modulename>"
219
if [ ! -d /sys/module/$module/sections ]
221
echo Module $module is not loaded
225
cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
226
cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
228
echo N > /sys/kernel/debug/$FAILTYPE/task-filter
229
echo 10 > /sys/kernel/debug/$FAILTYPE/probability
230
echo 100 > /sys/kernel/debug/$FAILTYPE/interval
231
echo -1 > /sys/kernel/debug/$FAILTYPE/times
232
echo 0 > /sys/kernel/debug/$FAILTYPE/space
233
echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
234
echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
235
echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
236
echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
238
trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
240
echo "Injecting errors into the module $module... (interrupt to stop)"