~ubuntu-branches/ubuntu/precise/linux-lowlatency/precise

<listitem><para>Enter the kernel debugger manually or by waiting for an oops or fault. There are several ways you can enter the kernel debugger manually; all involve using the sysrq-g, which means you must have enabled CONFIG_MAGIC_SYSRQ=y in your kernel config.</para>

386

387

<listitem><para>When logged in as root or with a super user session you can run:</para>

388

<para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>

389

<listitem><para>Example using minicom 2.2</para>

390

<para>Press: <constant>Control-a</constant></para>

391

<para>Press: <constant>f</constant></para>

392

<para>Press: <constant>g</constant></para>

393

</listitem>

394

<listitem><para>When you have telneted to a terminal server that supports sending a remote break</para>

395

<para>Press: <constant>Control-]</constant></para>

396

<para>Type in:<constant>send break</constant></para>

397

<para>Press: <constant>Enter</constant></para>

398

<para>Press: <constant>g</constant></para>

399

</listitem>

400

</itemizedlist>

401

</listitem>

402

<listitem><para>From the kdb prompt you can run the "help" command to see a complete list of the commands that are available.</para>

403

<para>Some useful commands in kdb include:

404

405

<listitem><para>lsmod -- Shows where kernel modules are loaded</para></listitem>

406

<listitem><para>ps -- Displays only the active processes</para></listitem>

407

<listitem><para>ps A -- Shows all the processes</para></listitem>

408

<listitem><para>summary -- Shows kernel version info and memory usage</para></listitem>

409

<listitem><para>bt -- Get a backtrace of the current process using dump_stack()</para></listitem>

410

<listitem><para>dmesg -- View the kernel syslog buffer</para></listitem>

411

<listitem><para>go -- Continue the system</para></listitem>

412

</itemizedlist>

413

</para>

414

</listitem>

415

416

<para>When you are done using kdb you need to consider rebooting the

417

system or using the "go" command to resuming normal kernel

418

execution. If you have paused the kernel for a lengthy period of

419

time, applications that rely on timely networking or anything to do

420

with real wall clock time could be adversely affected, so you

421

should take this into consideration when using the kernel

422

debugger.</para>

423

</listitem>

424

</orderedlist></para>

425

</sect1>

426

427

<title>Quick start for kdb using a keyboard connected console</title>

428

<para>This is a quick example of how to use kdb with a keyboard.</para>

429

430

<listitem><para>Boot kernel with arguments:

431

432

<listitem><para><constant>kgdboc=kbd</constant></para></listitem>

433

</itemizedlist></para>

434

435

<para>Configure kgdboc after the kernel booted:

436

437

<listitem><para><constant>echo kbd > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>

438

</itemizedlist>

439

</para>

440

</listitem>

441

442

443

<listitem><para>When logged in as root or with a super user session you can run:</para>

444

<para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>

445

<listitem><para>Example using a laptop keyboard</para>

446

<para>Press and hold down: <constant>Alt</constant></para>

447

<para>Press and hold down: <constant>Fn</constant></para>

448

<para>Press and release the key with the label: <constant>SysRq</constant></para>

449

<para>Release: <constant>Fn</constant></para>

450

<para>Press and release: <constant>g</constant></para>

451

<para>Release: <constant>Alt</constant></para>

452

</listitem>

453

<listitem><para>Example using a PS/2 101-key keyboard</para>

454

<para>Press and hold down: <constant>Alt</constant></para>

455

<para>Press and release the key with the label: <constant>SysRq</constant></para>

456

<para>Press and release: <constant>g</constant></para>

457

<para>Release: <constant>Alt</constant></para>

458

</listitem>

459

</itemizedlist>

460

</listitem>

461

462

<para>Now type in a kdb command such as "help", "dmesg", "bt" or "go" to continue kernel execution.</para>

463

</listitem>

464

</orderedlist></para>

465

</sect1>

466

</chapter>

467

468

<title>Using kgdb / gdb</title>

469

<para>In order to use kgdb you must activate it by passing

470

configuration information to one of the kgdb I/O drivers. If you

471

do not pass any configuration information kgdb will not do anything

472

at all. Kgdb will only actively hook up to the kernel trap hooks

473

if a kgdb I/O driver is loaded and configured. If you unconfigure

474

a kgdb I/O driver, kgdb will unregister all the kernel hook points.

475

</para>

476

<para> All kgdb I/O drivers can be reconfigured at run time, if

477

<symbol>CONFIG_SYSFS</symbol> and <symbol>CONFIG_MODULES</symbol>

478

are enabled, by echo'ing a new config string to

479

<constant>/sys/module/<driver>/parameter/<option></constant>.

480

The driver can be unconfigured by passing an empty string. You cannot

481

change the configuration while the debugger is attached. Make sure

482

to detach the debugger with the <constant>detach</constant> command

483

prior to trying to unconfigure a kgdb I/O driver.

484

</para>

485

486

<title>Connecting with gdb to a serial port</title>

487

488

<listitem><para>Configure kgdboc</para>

489

<para>Boot kernel with arguments:

490

491

<listitem><para><constant>kgdboc=ttyS0,115200</constant></para></listitem>

492

</itemizedlist></para>

493

494

<para>Configure kgdboc after the kernel booted:

495

496

<listitem><para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>

497

</itemizedlist></para>

498

</listitem>

499

500

<para>Stop kernel execution (break into the debugger)</para>

501

<para>In order to connect to gdb via kgdboc, the kernel must

502

first be stopped. There are several ways to stop the kernel which

503

include using kgdbwait as a boot argument, via a sysrq-g, or running

504

the kernel until it takes an exception where it waits for the

505

debugger to attach.

506

507

<listitem><para>When logged in as root or with a super user session you can run:</para>

508

<para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>

509

<listitem><para>Example using minicom 2.2</para>

510

<para>Press: <constant>Control-a</constant></para>

511

<para>Press: <constant>f</constant></para>

512

<para>Press: <constant>g</constant></para>

513

</listitem>

514

<listitem><para>When you have telneted to a terminal server that supports sending a remote break</para>

515

<para>Press: <constant>Control-]</constant></para>

516

<para>Type in:<constant>send break</constant></para>

517

<para>Press: <constant>Enter</constant></para>

518

<para>Press: <constant>g</constant></para>

519

</listitem>

520

</itemizedlist>

521

</para>

522

</listitem>

523

524

<para>Connect from from gdb</para>

525

<para>

526

Example (using a directly connected port):

527

</para>

528

529

% gdb ./vmlinux

530

(gdb) set remotebaud 115200

531

(gdb) target remote /dev/ttyS0

532

</programlisting>

533

<para>

534

Example (kgdb to a terminal server on TCP port 2012):

535

</para>

536

537

% gdb ./vmlinux

538

(gdb) target remote 192.168.2.2:2012

539

</programlisting>

540

<para>

541

Once connected, you can debug a kernel the way you would debug an

542

application program.

543

</para>

544

<para>

545

If you are having problems connecting or something is going

546

seriously wrong while debugging, it will most often be the case

547

that you want to enable gdb to be verbose about its target

548

communications. You do this prior to issuing the <constant>target

549

remote</constant> command by typing in: <constant>set debug remote 1</constant>

550

</para>

551

</listitem>

552

</orderedlist>

553

<para>Remember if you continue in gdb, and need to "break in" again,

554

you need to issue an other sysrq-g. It is easy to create a simple

555

entry point by putting a breakpoint at <constant>sys_sync</constant>

556

and then you can run "sync" from a shell or script to break into the

557

debugger.</para>

558

</sect1>

559

</chapter>

560

561

<title>kgdb and kdb interoperability</title>

562

<para>It is possible to transition between kdb and kgdb dynamically.

563

The debug core will remember which you used the last time and

564

automatically start in the same mode.</para>

565

<sect1>

566

<title>Switching between kdb and kgdb</title>

567

<sect2>

568

<title>Switching from kgdb to kdb</title>

569

<para>

570

There are two ways to switch from kgdb to kdb: you can use gdb to

571

issue a maintenance packet, or you can blindly type the command $3#33.

572

Whenever kernel debugger stops in kgdb mode it will print the

573

message <constant>KGDB or $3#33 for KDB</constant>. It is important

574

to note that you have to type the sequence correctly in one pass.

575

You cannot type a backspace or delete because kgdb will interpret

576

that as part of the debug stream.

577

578

<listitem><para>Change from kgdb to kdb by blindly typing:</para>

579

580

<listitem><para>Change from kgdb to kdb with gdb</para>

581

<para><constant>maintenance packet 3</constant></para>

582

<para>NOTE: Now you must kill gdb. Typically you press control-z and

583

issue the command: kill -9 %</para></listitem>

584

</orderedlist>

585

</para>

586

</sect2>

587

<sect2>

588

<title>Change from kdb to kgdb</title>

589

<para>There are two ways you can change from kdb to kgdb. You can

590

manually enter kgdb mode by issuing the kgdb command from the kdb

591

shell prompt, or you can connect gdb while the kdb shell prompt is

592

active. The kdb shell looks for the typical first commands that gdb

593

would issue with the gdb remote protocol and if it sees one of those

594

commands it automatically changes into kgdb mode.</para>

595

596

<listitem><para>From kdb issue the command:</para>

597

598

<para>Now disconnect your terminal program and connect gdb in its place</para></listitem>

599

<listitem><para>At the kdb prompt, disconnect the terminal program and connect gdb in its place.</para></listitem>

600

</orderedlist>

601

</sect2>

602

</sect1>

603

<sect1>

604

<title>Running kdb commands from gdb</title>

605

<para>It is possible to run a limited set of kdb commands from gdb,

606

using the gdb monitor command. You don't want to execute any of the

607

run control or breakpoint operations, because it can disrupt the

608

state of the kernel debugger. You should be using gdb for

609

breakpoints and run control operations if you have gdb connected.

610

The more useful commands to run are things like lsmod, dmesg, ps or

611

possibly some of the memory information commands. To see all the kdb

612

commands you can run <constant>monitor help</constant>.</para>

613

<para>Example:

614

615

(gdb) monitor ps

616

1 idle process (state I) and

617

27 sleeping system daemon (state M) processes suppressed,

618

use 'ps A' to see all.

619

Task Addr Pid Parent [*] cpu State Thread Command

620

621

0xc78291d0 1 0 0 0 S 0xc7829404 init

622

0xc7954150 942 1 0 0 S 0xc7954384 dropbear

623

0xc78789c0 944 1 0 0 S 0xc7878bf4 sh

624

(gdb)

625

</programlisting></informalexample>

626

</para>

627

</sect1>

628

</chapter>

629

630

<title>kgdb Test Suite</title>

631

<para>

632

When kgdb is enabled in the kernel config you can also elect to

633

enable the config parameter KGDB_TESTS. Turning this on will

634

enable a special kgdb I/O module which is designed to test the

635

kgdb internal functions.

636

</para>

637

<para>

638

The kgdb tests are mainly intended for developers to test the kgdb

639

internals as well as a tool for developing a new kgdb architecture

640

specific implementation. These tests are not really for end users

641

of the Linux kernel. The primary source of documentation would be

642

to look in the drivers/misc/kgdbts.c file.

643

</para>

644

<para>

645

The kgdb test suite can also be configured at compile time to run

646

the core set of tests by setting the kernel config parameter

647

KGDB_TESTS_ON_BOOT. This particular option is aimed at automated

648

regression testing and does not require modifying the kernel boot

649

config arguments. If this is turned on, the kgdb test suite can

650

be disabled by specifying "kgdbts=" as a kernel boot argument.

651

</para>

652

</chapter>

653

654

<title>Kernel Debugger Internals</title>

655

656

<title>Architecture Specifics</title>

657

<para>

658

The kernel debugger is organized into a number of components:

659

660

<listitem><para>The debug core</para>

661

<para>

662

The debug core is found in kernel/debugger/debug_core.c. It contains:

663

664

<listitem><para>A generic OS exception handler which includes

665

sync'ing the processors into a stopped state on an multi-CPU

666

system.</para></listitem>

667

<listitem><para>The API to talk to the kgdb I/O drivers</para></listitem>

668

<listitem><para>The API to make calls to the arch-specific kgdb implementation</para></listitem>

669

<listitem><para>The logic to perform safe memory reads and writes to memory while using the debugger</para></listitem>

670

<listitem><para>A full implementation for software breakpoints unless overridden by the arch</para></listitem>

671

<listitem><para>The API to invoke either the kdb or kgdb frontend to the debug core.</para></listitem>

672

<listitem><para>The structures and callback API for atomic kernel mode setting.</para>

673

<para>NOTE: kgdboc is where the kms callbacks are invoked.</para></listitem>

674

</itemizedlist>

675

</para>

676

</listitem>

677

<listitem><para>kgdb arch-specific implementation</para>

678

<para>

679

This implementation is generally found in arch/*/kernel/kgdb.c.

680

As an example, arch/x86/kernel/kgdb.c contains the specifics to

681

implement HW breakpoint as well as the initialization to

682

dynamically register and unregister for the trap handlers on

683

this architecture. The arch-specific portion implements:

684

685

<listitem><para>contains an arch-specific trap catcher which

686

invokes kgdb_handle_exception() to start kgdb about doing its

687

work</para></listitem>

688

<listitem><para>translation to and from gdb specific packet format to pt_regs</para></listitem>

689

<listitem><para>Registration and unregistration of architecture specific trap hooks</para></listitem>

690

<listitem><para>Any special exception handling and cleanup</para></listitem>

691

<listitem><para>NMI exception handling and cleanup</para></listitem>

692

<listitem><para>(optional)HW breakpoints</para></listitem>

693

</itemizedlist>

694

</para>

695

</listitem>

696

<listitem><para>gdbstub frontend (aka kgdb)</para>

697

<para>The gdbstub is located in kernel/debug/gdbstub.c. It contains:</para>

698

699

<listitem><para>All the logic to implement the gdb serial protocol</para></listitem>

700

</itemizedlist>

701

</listitem>

702

<listitem><para>kdb frontend</para>

703

<para>The kdb debugger shell is broken down into a number of

704

components. The kdb core is located in kernel/debug/kdb. There

705

are a number of helper functions in some of the other kernel

706

components to make it possible for kdb to examine and report

707

information about the kernel without taking locks that could

708

cause a kernel deadlock. The kdb core contains implements the following functionality.</para>

709

710

<listitem><para>A simple shell</para></listitem>

711

<listitem><para>The kdb core command set</para></listitem>

712

<listitem><para>A registration API to register additional kdb shell commands.</para>

713

714

<listitem><para>A good example of a self-contained kdb module

715

is the "ftdump" command for dumping the ftrace buffer. See:

716

kernel/trace/trace_kdb.c</para></listitem>

717

<listitem><para>For an example of how to dynamically register

718

a new kdb command you can build the kdb_hello.ko kernel module

719

from samples/kdb/kdb_hello.c. To build this example you can

720

set CONFIG_SAMPLES=y and CONFIG_SAMPLE_KDB=m in your kernel

721

config. Later run "modprobe kdb_hello" and the next time you

722

enter the kdb shell, you can run the "hello"

723

command.</para></listitem>

724

</itemizedlist></listitem>

725

<listitem><para>The implementation for kdb_printf() which

726

emits messages directly to I/O drivers, bypassing the kernel

727

log.</para></listitem>

728

<listitem><para>SW / HW breakpoint management for the kdb shell</para></listitem>

729

</itemizedlist>

730

</listitem>

731

<listitem><para>kgdb I/O driver</para>

732

<para>

733

Each kgdb I/O driver has to provide an implementation for the following:

734

735

<listitem><para>configuration via built-in or module</para></listitem>

736

<listitem><para>dynamic configuration and kgdb hook registration calls</para></listitem>

737

<listitem><para>read and write character interface</para></listitem>

738

<listitem><para>A cleanup handler for unconfiguring from the kgdb core</para></listitem>

739

<listitem><para>(optional) Early debug methodology</para></listitem>

740

</itemizedlist>

741

Any given kgdb I/O driver has to operate very closely with the

742

hardware and must do it in such a way that does not enable

743

interrupts or change other parts of the system context without

744

completely restoring them. The kgdb core will repeatedly "poll"

745

a kgdb I/O driver for characters when it needs input. The I/O

746

driver is expected to return immediately if there is no data

747

available. Doing so allows for the future possibility to touch

748

watch dog hardware in such a way as to have a target system not

749

reset when these are enabled.

750

</para>

751

</listitem>

752

</orderedlist>

753

</para>

754

<para>

755

If you are intent on adding kgdb architecture specific support

756

for a new architecture, the architecture should define

757

<constant>HAVE_ARCH_KGDB</constant> in the architecture specific

758

Kconfig file. This will enable kgdb for the architecture, and

759

at that point you must create an architecture specific kgdb

760

implementation.

761

</para>

762

<para>

763

There are a few flags which must be set on every architecture in

764

their <asm/kgdb.h> file. These are:

765

766

767

<para>

768

NUMREGBYTES: The size in bytes of all of the registers, so

769

that we can ensure they will all fit into a packet.

770

</para>

771

<para>

772

BUFMAX: The size in bytes of the buffer GDB will read into.

773

This must be larger than NUMREGBYTES.

774

</para>

775

<para>

776

CACHE_FLUSH_IS_SAFE: Set to 1 if it is always safe to call

777

flush_cache_range or flush_icache_range. On some architectures,

778

these functions may not be safe to call on SMP since we keep other

779

CPUs in a holding pattern.

780

</para>

781

</listitem>

782

</itemizedlist>

783

</para>

784

<para>

785

There are also the following functions for the common backend,

786

found in kernel/kgdb.c, that must be supplied by the

787

architecture-specific backend unless marked as (optional), in

788

which case a default function maybe used if the architecture

789

does not need to provide a specific implementation.

790

</para>

791

!Iinclude/linux/kgdb.h

792

</sect1>

793

794

<title>kgdboc internals</title>

795

<sect2>

796

<title>kgdboc and uarts</title>

797

<para>

798

The kgdboc driver is actually a very thin driver that relies on the

799

underlying low level to the hardware driver having "polling hooks"

800

which the to which the tty driver is attached. In the initial

801

implementation of kgdboc it the serial_core was changed to expose a

802

low level UART hook for doing polled mode reading and writing of a

803

single character while in an atomic context. When kgdb makes an I/O

804

request to the debugger, kgdboc invokes a callback in the serial

805

core which in turn uses the callback in the UART driver.</para>

806

<para>

807

When using kgdboc with a UART, the UART driver must implement two callbacks in the <constant>struct uart_ops</constant>. Example from drivers/8250.c:<programlisting>

808

#ifdef CONFIG_CONSOLE_POLL

809

.poll_get_char = serial8250_get_poll_char,

810

.poll_put_char = serial8250_put_poll_char,

811

#endif

812

</programlisting>

813

Any implementation specifics around creating a polling driver use the

814

<constant>#ifdef CONFIG_CONSOLE_POLL</constant>, as shown above.

815

Keep in mind that polling hooks have to be implemented in such a way

816

that they can be called from an atomic context and have to restore

817

the state of the UART chip on return such that the system can return

818

to normal when the debugger detaches. You need to be very careful

819

with any kind of lock you consider, because failing here is most likely

820

going to mean pressing the reset button.

821

</para>

822

</sect2>

823

824

<title>kgdboc and keyboards</title>

825

<para>The kgdboc driver contains logic to configure communications

826

with an attached keyboard. The keyboard infrastructure is only

827

compiled into the kernel when CONFIG_KDB_KEYBOARD=y is set in the

828

kernel configuration.</para>

829

<para>The core polled keyboard driver driver for PS/2 type keyboards

830

is in drivers/char/kdb_keyboard.c. This driver is hooked into the

831

debug core when kgdboc populates the callback in the array

832

called <constant>kdb_poll_funcs[]</constant>. The

833

kdb_get_kbd_char() is the top-level function which polls hardware

834

for single character input.

835

</para>

836

</sect2>

837

838

<title>kgdboc and kms</title>

839

<para>The kgdboc driver contains logic to request the graphics

840

display to switch to a text context when you are using

841

"kgdboc=kms,kbd", provided that you have a video driver which has a

842

frame buffer console and atomic kernel mode setting support.</para>

843

<para>

844

Every time the kernel

845

debugger is entered it calls kgdboc_pre_exp_handler() which in turn

846

calls con_debug_enter() in the virtual console layer. On resuming kernel

847

execution, the kernel debugger calls kgdboc_post_exp_handler() which

848

in turn calls con_debug_leave().</para>

849

<para>Any video driver that wants to be compatible with the kernel

850

debugger and the atomic kms callbacks must implement the

851

mode_set_base_atomic, fb_debug_enter and fb_debug_leave operations.

852

For the fb_debug_enter and fb_debug_leave the option exists to use

853

the generic drm fb helper functions or implement something custom for

854

the hardware. The following example shows the initialization of the

855

.mode_set_base_atomic operation in

856

drivers/gpu/drm/i915/intel_display.c:

857

858

859

static const struct drm_crtc_helper_funcs intel_helper_funcs = {

860

[...]

861

.mode_set_base_atomic = intel_pipe_set_base_atomic,

862

[...]

863

};

864

</programlisting>

865

</informalexample>

866

</para>

867

<para>Here is an example of how the i915 driver initializes the fb_debug_enter and fb_debug_leave functions to use the generic drm helpers in

868

drivers/gpu/drm/i915/intel_fb.c:

869

870

871

static struct fb_ops intelfb_ops = {

872

[...]

873

.fb_debug_enter = drm_fb_helper_debug_enter,

874

.fb_debug_leave = drm_fb_helper_debug_leave,

875

[...]

876

};

877

</programlisting>

878

</informalexample>

879

</para>

880

</sect2>

881

</sect1>

882

</chapter>

883

884

<title>Credits</title>

885

<para>

886

The following people have contributed to this document:

887

888

<listitem><para>Amit Kale<email>amitkale@linsyssoft.com</email></para></listitem>

889

<listitem><para>Tom Rini<email>trini@kernel.crashing.org</email></para></listitem>

890

</orderedlist>

891

In March 2008 this document was completely rewritten by:

892

893

<listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem>

894

</itemizedlist>

895

In Jan 2010 this document was updated to include kdb.

896

897

<listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem>

898

</itemizedlist>

899

</para>

900

</chapter>

901

</book>

902

Older »