Commit | Line | Data |
---|---|---|
dc851a0f DW |
1 | ================================================================ |
2 | Documentation for Kdump - The kexec-based Crash Dumping Solution | |
b089f4a6 VG |
3 | ================================================================ |
4 | ||
dc851a0f DW |
5 | This document includes overview, setup and installation, and analysis |
6 | information. | |
b089f4a6 | 7 | |
dc851a0f DW |
8 | Overview |
9 | ======== | |
b089f4a6 | 10 | |
dc851a0f DW |
11 | Kdump uses kexec to quickly boot to a dump-capture kernel whenever a |
12 | dump of the system kernel's memory needs to be taken (for example, when | |
13 | the system panics). The system kernel's memory image is preserved across | |
14 | the reboot and is accessible to the dump-capture kernel. | |
b089f4a6 | 15 | |
dc851a0f DW |
16 | You can use common Linux commands, such as cp and scp, to copy the |
17 | memory image to a dump file on the local disk, or across the network to | |
18 | a remote system. | |
b089f4a6 | 19 | |
ee8bb9ea | 20 | Kdump and kexec are currently supported on the x86, x86_64, ppc64 and ia64 |
dc851a0f | 21 | architectures. |
b089f4a6 | 22 | |
dc851a0f DW |
23 | When the system kernel boots, it reserves a small section of memory for |
24 | the dump-capture kernel. This ensures that ongoing Direct Memory Access | |
25 | (DMA) from the system kernel does not corrupt the dump-capture kernel. | |
26 | The kexec -p command loads the dump-capture kernel into this reserved | |
27 | memory. | |
b089f4a6 | 28 | |
dc851a0f DW |
29 | On x86 machines, the first 640 KB of physical memory is needed to boot, |
30 | regardless of where the kernel loads. Therefore, kexec backs up this | |
31 | region just before rebooting into the dump-capture kernel. | |
b089f4a6 | 32 | |
dc851a0f DW |
33 | All of the necessary information about the system kernel's core image is |
34 | encoded in the ELF format, and stored in a reserved area of memory | |
35 | before a crash. The physical address of the start of the ELF header is | |
36 | passed to the dump-capture kernel through the elfcorehdr= boot | |
37 | parameter. | |
38 | ||
39 | With the dump-capture kernel, you can access the memory image, or "old | |
40 | memory," in two ways: | |
41 | ||
42 | - Through a /dev/oldmem device interface. A capture utility can read the | |
43 | device file and write out the memory in raw format. This is a raw dump | |
44 | of memory. Analysis and capture tools must be intelligent enough to | |
45 | determine where to look for the right information. | |
46 | ||
47 | - Through /proc/vmcore. This exports the dump as an ELF-format file that | |
48 | you can write out using file copy commands such as cp or scp. Further, | |
49 | you can use analysis tools such as the GNU Debugger (GDB) and the Crash | |
50 | tool to debug the dump file. This method ensures that the dump pages are | |
51 | correctly ordered. | |
52 | ||
53 | ||
54 | Setup and Installation | |
55 | ====================== | |
56 | ||
9c61a446 VG |
57 | Install kexec-tools |
58 | ------------------- | |
dc851a0f DW |
59 | |
60 | 1) Login as the root user. | |
61 | ||
62 | 2) Download the kexec-tools user-space package from the following URL: | |
63 | ||
ea112bd5 H |
64 | http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/kexec-tools-testing.tar.gz |
65 | ||
66 | This is a symlink to the latest version, which at the time of writing is | |
67 | 20061214, the only release of kexec-tools-testing so far. As other versions | |
68 | are made released, the older onese will remain available at | |
69 | http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/ | |
dc851a0f | 70 | |
9c61a446 | 71 | Note: Latest kexec-tools-testing git tree is available at |
dc851a0f | 72 | |
9c61a446 VG |
73 | git://git.kernel.org/pub/scm/linux/kernel/git/horms/kexec-tools-testing.git |
74 | or | |
75 | http://www.kernel.org/git/?p=linux/kernel/git/horms/kexec-tools-testing.git;a=summary | |
dc851a0f | 76 | |
9c61a446 | 77 | 3) Unpack the tarball with the tar command, as follows: |
dc851a0f | 78 | |
ea112bd5 | 79 | tar xvpzf kexec-tools-testing.tar.gz |
dc851a0f | 80 | |
ea112bd5 | 81 | 4) Change to the kexec-tools directory, as follows: |
dc851a0f | 82 | |
ea112bd5 | 83 | cd kexec-tools-testing-VERSION |
dc851a0f | 84 | |
9c61a446 | 85 | 5) Configure the package, as follows: |
dc851a0f DW |
86 | |
87 | ./configure | |
88 | ||
9c61a446 | 89 | 6) Compile the package, as follows: |
dc851a0f DW |
90 | |
91 | make | |
92 | ||
9c61a446 | 93 | 7) Install the package, as follows: |
dc851a0f DW |
94 | |
95 | make install | |
96 | ||
97 | ||
9c61a446 VG |
98 | Build the system and dump-capture kernels |
99 | ----------------------------------------- | |
100 | There are two possible methods of using Kdump. | |
101 | ||
102 | 1) Build a separate custom dump-capture kernel for capturing the | |
103 | kernel core dump. | |
104 | ||
105 | 2) Or use the system kernel binary itself as dump-capture kernel and there is | |
106 | no need to build a separate dump-capture kernel. This is possible | |
107 | only with the architecutres which support a relocatable kernel. As | |
108 | of today i386 and ia64 architectures support relocatable kernel. | |
109 | ||
110 | Building a relocatable kernel is advantageous from the point of view that | |
111 | one does not have to build a second kernel for capturing the dump. But | |
112 | at the same time one might want to build a custom dump capture kernel | |
113 | suitable to his needs. | |
dc851a0f | 114 | |
9c61a446 VG |
115 | Following are the configuration setting required for system and |
116 | dump-capture kernels for enabling kdump support. | |
dc851a0f | 117 | |
9c61a446 VG |
118 | System kernel config options |
119 | ---------------------------- | |
dc851a0f DW |
120 | |
121 | 1) Enable "kexec system call" in "Processor type and features." | |
122 | ||
123 | CONFIG_KEXEC=y | |
124 | ||
125 | 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo | |
126 | filesystems." This is usually enabled by default. | |
127 | ||
128 | CONFIG_SYSFS=y | |
129 | ||
130 | Note that "sysfs file system support" might not appear in the "Pseudo | |
131 | filesystems" menu if "Configure standard kernel features (for small | |
132 | systems)" is not enabled in "General Setup." In this case, check the | |
133 | .config file itself to ensure that sysfs is turned on, as follows: | |
134 | ||
135 | grep 'CONFIG_SYSFS' .config | |
136 | ||
137 | 3) Enable "Compile the kernel with debug info" in "Kernel hacking." | |
138 | ||
139 | CONFIG_DEBUG_INFO=Y | |
140 | ||
141 | This causes the kernel to be built with debug symbols. The dump | |
142 | analysis tools require a vmlinux with debug symbols in order to read | |
143 | and analyze a dump file. | |
144 | ||
9c61a446 VG |
145 | Dump-capture kernel config options (Arch Independent) |
146 | ----------------------------------------------------- | |
dc851a0f | 147 | |
9c61a446 VG |
148 | 1) Enable "kernel crash dumps" support under "Processor type and |
149 | features": | |
dc851a0f | 150 | |
9c61a446 | 151 | CONFIG_CRASH_DUMP=y |
dc851a0f | 152 | |
9c61a446 | 153 | 2) Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems". |
b089f4a6 | 154 | |
9c61a446 VG |
155 | CONFIG_PROC_VMCORE=y |
156 | (CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.) | |
dc851a0f | 157 | |
9c61a446 VG |
158 | Dump-capture kernel config options (Arch Dependent, i386) |
159 | -------------------------------------------------------- | |
160 | 1) On x86, enable high memory support under "Processor type and | |
dc851a0f DW |
161 | features": |
162 | ||
163 | CONFIG_HIGHMEM64G=y | |
164 | or | |
165 | CONFIG_HIGHMEM4G | |
166 | ||
9c61a446 | 167 | 2) On x86 and x86_64, disable symmetric multi-processing support |
dc851a0f DW |
168 | under "Processor type and features": |
169 | ||
170 | CONFIG_SMP=n | |
9c61a446 | 171 | |
dc851a0f DW |
172 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line |
173 | when loading the dump-capture kernel, see section "Load the Dump-capture | |
174 | Kernel".) | |
175 | ||
9c61a446 VG |
176 | 3) If one wants to build and use a relocatable kernel, |
177 | Enable "Build a relocatable kernel" support under "Processor type and | |
178 | features" | |
dc851a0f | 179 | |
9c61a446 | 180 | CONFIG_RELOCATABLE=y |
dc851a0f | 181 | |
9c61a446 VG |
182 | 4) Use a suitable value for "Physical address where the kernel is |
183 | loaded" (under "Processor type and features"). This only appears when | |
184 | "kernel crash dumps" is enabled. A suitable value depends upon | |
185 | whether kernel is relocatable or not. | |
186 | ||
187 | If you are using a relocatable kernel use CONFIG_PHYSICAL_START=0x100000 | |
188 | This will compile the kernel for physical address 1MB, but given the fact | |
189 | kernel is relocatable, it can be run from any physical address hence | |
190 | kexec boot loader will load it in memory region reserved for dump-capture | |
191 | kernel. | |
192 | ||
193 | Otherwise it should be the start of memory region reserved for | |
194 | second kernel using boot parameter "crashkernel=Y@X". Here X is | |
195 | start of memory region reserved for dump-capture kernel. | |
196 | Generally X is 16MB (0x1000000). So you can set | |
197 | CONFIG_PHYSICAL_START=0x1000000 | |
198 | ||
199 | 5) Make and install the kernel and its modules. DO NOT add this kernel | |
200 | to the boot loader configuration files. | |
dc851a0f | 201 | |
9c61a446 VG |
202 | Dump-capture kernel config options (Arch Dependent, x86_64) |
203 | ---------------------------------------------------------- | |
204 | 1) On x86 and x86_64, disable symmetric multi-processing support | |
205 | under "Processor type and features": | |
206 | ||
207 | CONFIG_SMP=n | |
208 | ||
209 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line | |
210 | when loading the dump-capture kernel, see section "Load the Dump-capture | |
211 | Kernel".) | |
dc851a0f | 212 | |
9c61a446 | 213 | 2) Use a suitable value for "Physical address where the kernel is |
dc851a0f DW |
214 | loaded" (under "Processor type and features"). This only appears when |
215 | "kernel crash dumps" is enabled. By default this value is 0x1000000 | |
216 | (16MB). It should be the same as X in the "crashkernel=Y@X" boot | |
9c61a446 | 217 | parameter. |
dc851a0f | 218 | |
9c61a446 | 219 | For x86_64, normally "CONFIG_PHYSICAL_START=0x1000000". |
dc851a0f | 220 | |
9c61a446 VG |
221 | 3) Make and install the kernel and its modules. DO NOT add this kernel |
222 | to the boot loader configuration files. | |
dc851a0f | 223 | |
9c61a446 VG |
224 | Dump-capture kernel config options (Arch Dependent, ppc64) |
225 | ---------------------------------------------------------- | |
dc851a0f | 226 | |
9c61a446 | 227 | - Make and install the kernel and its modules. DO NOT add this kernel |
dc851a0f DW |
228 | to the boot loader configuration files. |
229 | ||
9c61a446 VG |
230 | Dump-capture kernel config options (Arch Dependent, ia64) |
231 | ---------------------------------------------------------- | |
ee8bb9ea H |
232 | |
233 | - No specific options are required to create a dump-capture kernel | |
234 | for ia64, other than those specified in the arch idependent section | |
235 | above. This means that it is possible to use the system kernel | |
236 | as a dump-capture kernel if desired. | |
237 | ||
238 | The crashkernel region can be automatically placed by the system | |
239 | kernel at run time. This is done by specifying the base address as 0, | |
240 | or omitting it all together. | |
241 | ||
242 | crashkernel=256M@0 | |
243 | or | |
244 | crashkernel=256M | |
245 | ||
246 | If the start address is specified, note that the start address of the | |
247 | kernel will be aligned to 64Mb, so if the start address is not then | |
248 | any space below the alignment point will be wasted. | |
9c61a446 VG |
249 | |
250 | ||
251 | Boot into System Kernel | |
252 | ======================= | |
253 | ||
254 | 1) Make and install the kernel and its modules. Update the boot loader | |
255 | (such as grub, yaboot, or lilo) configuration files as necessary. | |
256 | ||
257 | 2) Boot the system kernel with the boot parameter "crashkernel=Y@X", | |
258 | where Y specifies how much memory to reserve for the dump-capture kernel | |
259 | and X specifies the beginning of this reserved memory. For example, | |
260 | "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory | |
261 | starting at physical address 0x01000000 (16MB) for the dump-capture kernel. | |
262 | ||
263 | On x86 and x86_64, use "crashkernel=64M@16M". | |
264 | ||
265 | On ppc64, use "crashkernel=128M@32M". | |
dc851a0f | 266 | |
ee8bb9ea H |
267 | On ia64, 256M@256M is a generous value that typically works. |
268 | The region may be automatically placed on ia64, see the | |
269 | dump-capture kernel config option notes above. | |
270 | ||
dc851a0f DW |
271 | Load the Dump-capture Kernel |
272 | ============================ | |
273 | ||
9c61a446 VG |
274 | After booting to the system kernel, dump-capture kernel needs to be |
275 | loaded. | |
276 | ||
277 | Based on the architecture and type of image (relocatable or not), one | |
278 | can choose to load the uncompressed vmlinux or compressed bzImage/vmlinuz | |
279 | of dump-capture kernel. Following is the summary. | |
280 | ||
281 | For i386: | |
282 | - Use vmlinux if kernel is not relocatable. | |
283 | - Use bzImage/vmlinuz if kernel is relocatable. | |
284 | For x86_64: | |
285 | - Use vmlinux | |
286 | For ppc64: | |
287 | - Use vmlinux | |
288 | For ia64: | |
ee8bb9ea H |
289 | - Use vmlinux or vmlinuz.gz |
290 | ||
9c61a446 VG |
291 | |
292 | If you are using a uncompressed vmlinux image then use following command | |
293 | to load dump-capture kernel. | |
dc851a0f | 294 | |
9c61a446 | 295 | kexec -p <dump-capture-kernel-vmlinux-image> \ |
dc851a0f | 296 | --initrd=<initrd-for-dump-capture-kernel> --args-linux \ |
9c61a446 | 297 | --append="root=<root-dev> <arch-specific-options>" |
dc851a0f | 298 | |
9c61a446 VG |
299 | If you are using a compressed bzImage/vmlinuz, then use following command |
300 | to load dump-capture kernel. | |
dc851a0f | 301 | |
9c61a446 VG |
302 | kexec -p <dump-capture-kernel-bzImage> \ |
303 | --initrd=<initrd-for-dump-capture-kernel> \ | |
304 | --append="root=<root-dev> <arch-specific-options>" | |
305 | ||
ee8bb9ea H |
306 | Please note, that --args-linux does not need to be specified for ia64. |
307 | It is planned to make this a no-op on that architecture, but for now | |
308 | it should be omitted | |
309 | ||
9c61a446 VG |
310 | Following are the arch specific command line options to be used while |
311 | loading dump-capture kernel. | |
312 | ||
ee8bb9ea | 313 | For i386, x86_64 and ia64: |
9c61a446 VG |
314 | "init 1 irqpoll maxcpus=1" |
315 | ||
316 | For ppc64: | |
317 | "init 1 maxcpus=1 noirqdistrib" | |
dc851a0f | 318 | |
9c61a446 VG |
319 | |
320 | Notes on loading the dump-capture kernel: | |
dc851a0f DW |
321 | |
322 | * By default, the ELF headers are stored in ELF64 format to support | |
323 | systems with more than 4GB memory. The --elf32-core-headers option can | |
324 | be used to force the generation of ELF32 headers. This is necessary | |
325 | because GDB currently cannot open vmcore files with ELF64 headers on | |
326 | 32-bit systems. ELF32 headers can be used on non-PAE systems (that is, | |
327 | less than 4GB of memory). | |
328 | ||
329 | * The "irqpoll" boot parameter reduces driver initialization failures | |
330 | due to shared interrupts in the dump-capture kernel. | |
331 | ||
332 | * You must specify <root-dev> in the format corresponding to the root | |
333 | device name in the output of mount command. | |
334 | ||
335 | * "init 1" boots the dump-capture kernel into single-user mode without | |
336 | networking. If you want networking, use "init 3." | |
337 | ||
9c61a446 VG |
338 | * We generally don' have to bring up a SMP kernel just to capture the |
339 | dump. Hence generally it is useful either to build a UP dump-capture | |
340 | kernel or specify maxcpus=1 option while loading dump-capture kernel. | |
dc851a0f DW |
341 | |
342 | Kernel Panic | |
343 | ============ | |
344 | ||
345 | After successfully loading the dump-capture kernel as previously | |
346 | described, the system will reboot into the dump-capture kernel if a | |
347 | system crash is triggered. Trigger points are located in panic(), | |
348 | die(), die_nmi() and in the sysrq handler (ALT-SysRq-c). | |
349 | ||
350 | The following conditions will execute a crash trigger point: | |
351 | ||
352 | If a hard lockup is detected and "NMI watchdog" is configured, the system | |
353 | will boot into the dump-capture kernel ( die_nmi() ). | |
354 | ||
355 | If die() is called, and it happens to be a thread with pid 0 or 1, or die() | |
356 | is called inside interrupt context or die() is called and panic_on_oops is set, | |
357 | the system will boot into the dump-capture kernel. | |
358 | ||
670e9f34 | 359 | On powererpc systems when a soft-reset is generated, die() is called by all cpus and the system will boot into the dump-capture kernel. |
dc851a0f DW |
360 | |
361 | For testing purposes, you can trigger a crash by using "ALT-SysRq-c", | |
362 | "echo c > /proc/sysrq-trigger or write a module to force the panic. | |
363 | ||
364 | Write Out the Dump File | |
365 | ======================= | |
366 | ||
367 | After the dump-capture kernel is booted, write out the dump file with | |
368 | the following command: | |
b089f4a6 VG |
369 | |
370 | cp /proc/vmcore <dump-file> | |
371 | ||
dc851a0f DW |
372 | You can also access dumped memory as a /dev/oldmem device for a linear |
373 | and raw view. To create the device, use the following command: | |
b089f4a6 | 374 | |
dc851a0f | 375 | mknod /dev/oldmem c 1 12 |
b089f4a6 | 376 | |
dc851a0f DW |
377 | Use the dd command with suitable options for count, bs, and skip to |
378 | access specific portions of the dump. | |
b089f4a6 | 379 | |
dc851a0f | 380 | To see the entire memory, use the following command: |
b089f4a6 | 381 | |
dc851a0f | 382 | dd if=/dev/oldmem of=oldmem.001 |
a7e670d8 | 383 | |
dc851a0f DW |
384 | |
385 | Analysis | |
b089f4a6 | 386 | ======== |
b089f4a6 | 387 | |
dc851a0f DW |
388 | Before analyzing the dump image, you should reboot into a stable kernel. |
389 | ||
390 | You can do limited analysis using GDB on the dump file copied out of | |
391 | /proc/vmcore. Use the debug vmlinux built with -g and run the following | |
392 | command: | |
393 | ||
394 | gdb vmlinux <dump-file> | |
b089f4a6 | 395 | |
dc851a0f DW |
396 | Stack trace for the task on processor 0, register display, and memory |
397 | display work fine. | |
b089f4a6 | 398 | |
dc851a0f DW |
399 | Note: GDB cannot analyze core files generated in ELF64 format for x86. |
400 | On systems with a maximum of 4GB of memory, you can generate | |
401 | ELF32-format headers using the --elf32-core-headers kernel option on the | |
402 | dump kernel. | |
b089f4a6 | 403 | |
dc851a0f DW |
404 | You can also use the Crash utility to analyze dump files in Kdump |
405 | format. Crash is available on Dave Anderson's site at the following URL: | |
a7e670d8 | 406 | |
dc851a0f DW |
407 | http://people.redhat.com/~anderson/ |
408 | ||
409 | ||
410 | To Do | |
411 | ===== | |
a7e670d8 | 412 | |
dc851a0f DW |
413 | 1) Provide a kernel pages filtering mechanism, so core file size is not |
414 | extreme on systems with huge memory banks. | |
a7e670d8 | 415 | |
dc851a0f DW |
416 | 2) Relocatable kernel can help in maintaining multiple kernels for |
417 | crash_dump, and the same kernel as the system kernel can be used to | |
418 | capture the dump. | |
b089f4a6 | 419 | |
dc851a0f DW |
420 | |
421 | Contact | |
b089f4a6 | 422 | ======= |
dc851a0f | 423 | |
b089f4a6 | 424 | Vivek Goyal (vgoyal@in.ibm.com) |
d58831e4 | 425 | Maneesh Soni (maneesh@in.ibm.com) |
dc851a0f DW |
426 | |
427 | ||
428 | Trademark | |
429 | ========= | |
430 | ||
431 | Linux is a trademark of Linus Torvalds in the United States, other | |
432 | countries, or both. |