1
.\" Copyright 1994 Dr. Greg Wettstein, Enjellic Systems Development.
2
.\" May be distributed under the GNU General Public License
3
.\" Sun Jul 30 01:35:55 MET: Martin Schulze: Updates
4
.\" Sun Nov 19 23:22:21 MET: Martin Schulze: Updates
5
.\" Mon Aug 19 09:42:08 CDT 1996: Dr. G.W. Wettstein: Updates
7
.TH RKLOGD 8 "03 JULY 2007" "Version 1.14.2 (devel)" "Linux System Administration"
9
rklogd \- Kernel Log Daemon
34
is a system daemon which intercepts and logs Linux kernel
40
Sets the default log level of console messages to \fIn\fR.
43
Enable debugging mode. This will generate \fBLOTS\fR of output to
47
Log messages to the specified filename rather than to the syslog facility.
50
Signal the currently executing rklogd daemon. Both of these switches control
51
the loading/reloading of symbol information. The \-i switch signals the
52
daemon to reload the kernel module symbols. The \-I switch signals for a
53
reload of both the static kernel symbols and the kernel module symbols.
56
Avoid auto-backgrounding. This is needed especially if the
58
is started and controlled by
62
Execute in 'one\-shot' mode. This causes \fBrklogd\fP to read and log
63
all the messages that are found in the kernel message buffers. After
64
a single read and log cycle the daemon exits.
67
Enable paranoia. This option controls when rklogd loads kernel module symbol
68
information. Setting this switch causes rklogd to load the kernel module
69
symbol information whenever an Oops string is detected in the kernel message
73
Force \fBrklogd\fP to use the system call interface to the kernel message
77
Use the specified file as the source of kernel symbol information.
80
Print version and exit.
83
Omits EIP translation and therefore doesn't read the System.map file.
86
When symbols are expanded, print the line twice. Once with addresses
87
converted to symbols, once with the raw text. This allows external
88
programs such as ksymoops do their own processing on the original
92
The functionality of rklogd has been typically incorporated into other
93
versions of syslogd but this seems to be a poor place for it. In the
94
modern Linux kernel a number of kernel messaging issues such as
95
sourcing, prioritization and resolution of kernel addresses must be
96
addressed. Incorporating kernel logging into a separate process
97
offers a cleaner separation of services.
99
In Linux there are two potential sources of kernel log information: the
101
file system and the syscall (sys_syslog) interface, although
102
ultimately they are one and the same. Klogd is designed to choose
103
whichever source of information is the most appropriate. It does this
104
by first checking for the presence of a mounted
106
file system. If this is found the
108
file is used as the source of kernel log
109
information. If the proc file system is not mounted
112
system call to obtain kernel messages. The command line switch
114
can be used to force rklogd to use the system call interface as its
117
If kernel messages are directed through the
118
.BR syslogd " daemon the " rklogd
119
daemon, as of version 1.1, has the ability to properly prioritize
120
kernel messages. Prioritization of the kernel messages was added to it
121
at approximately version 0.99pl13 of the kernel. The raw kernel messages
124
\<[0\-7]\>Something said by the kernel.
126
The priority of the kernel message is encoded as a single numeric
127
digit enclosed inside the <> pair. The definitions of these values is
128
given in the kernel include file kernel.h. When a message is received
129
from the kernel the rklogd daemon reads this priority level and assigns
130
the appropriate priority level to the syslog message. If file output
131
(\fB-f\fR) is used the prioritization sequence is left pre\-pended to the
136
daemon also allows the ability to alter the presentation of
137
kernel messages to the system console. Consequent with the
138
prioritization of kernel messages was the inclusion of default
139
messaging levels for the kernel. In a stock kernel the the default
140
console log level is set to 7. Any messages with a priority level
141
numerically lower than 7 (higher priority) appear on the console.
143
Messages of priority level 7 are considered to be 'debug' messages and
144
will thus not appear on the console. Many administrators,
145
particularly in a multi\-user environment, prefer that all kernel
146
messages be handled by rklogd and either directed to a file or to
147
the syslogd daemon. This prevents 'nuisance' messages such as line
148
printer out of paper or disk change detected from cluttering the
153
is given on the commandline the
155
daemon will execute a system call to inhibit all kernel messages from
156
being displayed on the console. Former versions always issued this
157
system call and defaulted to all kernel messages except for panics.
158
This is handled differently nowardays so
160
doesn't need to set this value anymore. The
161
argument given to the \fB\-c\fR switch specifies the priority level of
162
messages which will be directed to the console. Note that messages of
163
a priority value LOWER than the indicated number will be directed to
166
For example, to have the kernel display all messages with a
169
or more severe the following
170
command would be executed:
176
The definitions of the numeric values for kernel messages are given in
178
.IR kernel.h " which can be found in the " /usr/include/linux
179
directory if the kernel sources are installed. These values parallel
180
the syslog priority values which are defined in the file
181
.IR syslog.h " found in the " /usr/include/sys " sub\-directory."
183
The rklogd daemon can also be used in a 'one\-shot' mode for reading the
184
kernel message buffers. One shot mode is selected by specifying the
185
\fB\-o\fR switch on the command line. Output will be directed to either the
186
syslogd daemon or to an alternate file specified by the \fB-f\fR switch.
188
For example, to read all the kernel messages after a system
189
boot and record them in a file called krnl.msg the following
190
command would be given.
193
rklogd -o -f ./krnl.msg
196
.SH KERNEL ADDRESS RESOLUTION
197
If the kernel detects an internal error condition a general protection
198
fault will be triggered. As part of the GPF handling procedure the
199
kernel prints out a status report indicating the state of the
200
processor at the time of the fault. Included in this display are the
201
contents of the microprocessor's registers, the contents of the kernel
202
stack and a tracing of what functions were being executed at the time
206
.B EXTREMELY IMPORTANT
207
in determining what caused the internal error condition. The
208
difficulty comes when a kernel developer attempts to analyze this
209
information. The raw numeric information present in the protection
210
fault printout is of very little use to the developers. This is due
211
to the fact that kernels are not identical and the addresses of
212
variable locations or functions will not be the same in all kernels.
213
In order to correctly diagnose the cause of failure a kernel developer
214
needs to know what specific kernel functions or variable locations
215
were involved in the error.
217
As part of the kernel compilation process a listing is created which
218
specified the address locations of important variables and function in
219
the kernel being compiled. This listing is saved in a file called
220
System.map in the top of the kernel directory source tree. Using this
221
listing a kernel developer can determine exactly what the kernel was
222
doing when the error condition occurred.
224
The process of resolving the numeric addresses from the protection
225
fault printout can be done manually or by using the
227
program which is included in the kernel sources.
231
will attempt to resolve kernel numeric addresses to their symbolic
232
forms if a kernel symbol table is available at execution time. If you
233
require the original address of the symbol, use the
235
switch to preserve the numeric address. A
236
symbol table may be specified by using the \fB\-k\fR switch on the
237
command line. If a symbol file is not explicitly specified the
238
following filenames will be tried:
243
.I /usr/src/linux/System.map
246
Version information is supplied in the system maps as of kernel
247
1.3.43. This version information is used to direct an intelligent
248
search of the list of symbol tables. This feature is useful since it
249
provides support for both production and experimental kernels.
251
For example a production kernel may have its map file stored in
252
/boot/System.map. If an experimental or test kernel is compiled with
253
the sources in the 'standard' location of /usr/src/linux the system
254
map will be found in /usr/src/linux/System.map. When rklogd starts
255
under the experimental kernel the map in /boot/System.map will be
256
bypassed in favor of the map in /usr/src/linux/System.map.
258
Modern kernels as of 1.3.43 properly format important kernel addresses
259
so that they will be recognized and translated by rklogd. Earlier
260
kernels require a source code patch be applied to the kernel sources.
261
This patch is supplied with the sysrklogd sources.
263
The process of analyzing kernel protections faults works very well
264
with a static kernel. Additional difficulties are encountered when
265
attempting to diagnose errors which occur in loadable kernel modules.
266
Loadable kernel modules are used to implement kernel functionality in
267
a form which can be loaded or unloaded at will. The use of loadable
268
modules is useful from a debugging standpoint and can also be useful
269
in decreasing the amount of memory required by a kernel.
271
The difficulty with diagnosing errors in loadable modules is due to
272
the dynamic nature of the kernel modules. When a module is loaded the
273
kernel will allocate memory to hold the module, when the module is
274
unloaded this memory will be returned back to the kernel. This
275
dynamic memory allocation makes it impossible to produce a map file
276
which details the addresses of the variable and functions in a kernel
277
loadable module. Without this location map it is not possible for a
278
kernel developer to determine what went wrong if a protection fault
279
involves a kernel module.
282
has support for dealing with the problem of diagnosing protection
283
faults in kernel loadable modules. At program start time or in
284
response to a signal the daemon will interrogate the kernel for a
285
listing of all modules loaded and the addresses in memory they are
286
loaded at. Individual modules can also register the locations of
287
important functions when the module is loaded. The addresses of these
288
exported symbols are also determined during this interrogation
291
When a protection fault occurs an attempt will be made to resolve
292
kernel addresses from the static symbol table. If this fails the
293
symbols from the currently loaded modules are examined in an attempt
294
to resolve the addresses. At the very minimum this allows rklogd to
295
indicate which loadable module was responsible for generating the
296
protection fault. Additional information may be available if the
297
module developer chose to export symbol information from the module.
299
Proper and accurate resolution of addresses in kernel modules requires
302
be informed whenever the kernel module status changes. The
306
switches can be used to signal the currently executing daemon that
307
symbol information be reloaded. Of most importance to proper
308
resolution of module symbols is the
310
switch. Each time a kernel module is loaded or removed from the
311
kernel the following command should be executed:
319
switch can also be used to insure that module symbol information is up
320
to date. This switch instructs
322
to reload the module symbol information whenever a protection fault
323
is detected. Caution should be used before invoking the program in
324
\'paranoid\' mode. The stability of the kernel and the operating
325
environment is always under question when a protection fault occurs.
326
Since the rklogd daemon must execute system calls in order to read the
327
module symbol information there is the possibility that the system may
328
be too unstable to capture useful information. A much better policy
329
is to insure that rklogd is updated whenever a module is loaded or
330
unloaded. Having uptodate symbol information loaded increases the
331
probability of properly resolving a protection fault if it should occur.
333
Included in the sysrklogd source distribution is a patch to the
334
modules-2.0.0 package which allows the
339
utilities to automatically signal
341
whenever a module is inserted or removed from the kernel. Using this
342
patch will insure that the symbol information maintained in rklogd is
343
always consistent with the current kernel state.
348
will respond to eight signals:
349
.BR SIGHUP ", " SIGINT ", " SIGKILL ", " SIGTERM ", " SIGTSTP ", "
350
.BR SIGUSR1 ", "SIGUSR2 " and " SIGCONT ". The"
351
.BR SIGINT ", " SIGKILL ", " SIGTERM " and " SIGHUP
352
signals will cause the daemon to close its kernel log sources and
353
terminate gracefully.
356
.BR SIGTSTP " and " SIGCONT
357
signals are used to start and stop kernel logging. Upon receipt of a
359
signal the daemon will close its
360
log sources and spin in an idle loop. Subsequent receipt of a
362
signal will cause the daemon to go through its initialization sequence
363
and re-choose an input source. Using
364
.BR SIGSTOP " and " SIGCONT
365
in combination the kernel log input can be re-chosen without stopping and
366
restarting the daemon. For example if the \fI/proc\fR file system is to be
367
un-mounted the following command sequence should be used:
378
Notations will be made in the system logs with
381
documenting the start/stop of logging.
384
.BR SIGUSR1 " and " SIGUSR2
385
signals are used to initiate loading/reloading of kernel symbol information.
388
signal will cause the kernel module symbols to be reloaded. Signaling the
391
will cause both the static kernel symbols and the kernel module symbols to
394
Provided that the System.map file is placed in an appropriate location the
395
signal of generally greatest usefulness is the
397
signal. This signal is designed to be used to signal the daemon when kernel
398
modules are loaded/unloaded. Sending this signal to the daemon after a
399
kernel module state change will insure that proper resolution of symbols will
400
occur if a protection fault occurs in the address space occupied by a kernel
407
One Source for kernel messages
410
.I /var/run/rklogd.pid
411
The file containing the process id of
414
.I /boot/System.map, /System.map, /usr/src/linux/System.map
415
Default locations for kernel system maps.
418
Probably numerous. Well formed context diffs appreciated.
423
was originally written by Steve Lord (lord@cray.com), Greg Wettstein
424
made major improvements.
428
Dr. Greg Wettstein (greg@wind.enjellic.com)
430
Enjellic Systems Development
435
Oncology Research Divsion Computing Facility
437
Roger Maris Cancer Center
added
removed
rklogd.8
