kdb(1M) kdb(1M)
NAME
kdb - kernel debugger
SYNOPSIS
kdb
DESCRIPTION
KDB is a kernel debugger that works like a Reverse Polish
Notation (RPN) calculator. KDB can set breakpoints, display
kernel stack traces and various kernel structures, and modify
the contents of memory, I/O, and registers. The debugger
supports basic arithmetic operations, conditional execution,
variables, and macros. KDB does conversions from a kernel
symbol name to its virtual address, from a virtual address to
the value at that address, and from a virtual address to the
name of the nearest kernel symbol. You have a choice of
different numeric bases, address spaces, and operand sizes.
This is an advanced tool, only for those who are thoroughly
familiar with the UNIX kernel. Because UNIX systems differ,
you could possibly damage your system by following some of the
examples in this discussion.
You can invoke the debugger by using the kdb command or the
sysi86(SI86TODEMON) system call on all systems, CTRL-ALT-d
(from the console only) on an AT-bus system, or the interrupt
character (from the console only) on a Multibus system. In
addition, KDB is entered automatically under various
conditions, such as panics and breakpoint traps. Any time the
kdb>> prompt appears, you are in the debugger. I/O is done
via the console (kd), or a serial terminal.
To exit the debugger, type CTRL-d or q.
When you exit and re-enter the debugger, its state is
preserved, including the contents of the value stack.
USING KDB AS A CALCULATOR
KDB operates as an RPN calculator, similar to dc(1). This
calculator has a 32-level value stack for storing results and
intermediate values. Commands and values you enter operate on
the value stack, which is an internal data structure in KDB.
It has no connection with the kernel stack or any other stack
in the system.
Copyright 1994 Novell, Inc. Page 1
kdb(1M) kdb(1M)
To use KDB, at the kdb>> prompt type one or more items (values
or commands) on a line. Separate items with spaces or tabs.
Press ENTER to end a line and send its contents to KDB for
processing. Each item is processed separately, from left to
right.
The values can be:
Numbers
Use positive or negative integers. Numbers must begin
with a digit, or a minus sign for negative numbers.
Begin octal numbers with ``0o'' and hex numbers with
``0x.'' Otherwise, numbers are assumed to be in the
default base - the default is hex, unless you change
it. (See ``Resetting the Numeric Base'' for
instructions.)
Character constants
You can have KDB convert characters to a number by
entering one to four characters inside single quotes.
C-style escapes are supported in character constants.
Strings
Use C-style strings, enclosed in double quotes.
Kernel symbol names
When you type a kernel symbol name, its address is
pushed onto the value stack.
When you enter a number or a string, it is pushed onto the
value stack, becoming the new TOS (Top Of Stack). Values
remain on the value stack until they are popped off as a
result of a command.
In the descriptions below, [TOS] means the value on the top of
the stack and
[TOS-1] means the value just below it (pushed previously).
Stack Operations
KDB provides these commands for examining or changing the
value stack:
stk print all values on the stack
p print [TOS]
Copyright 1994 Novell, Inc. Page 2
kdb(1M) kdb(1M)
dup push [TOS]
pop pop 1 value
clrstk
pop all values
stk For example, starting with an empty value stack, this
input:
5 "xyzzy" 7 stk
displays the entire stack:
5
"xyzzy"
7
p At this point, the input:
p
displays the top value on the stack, which is:
7
The next example uses the p command to display the
address of a kernel symbol. The input:
lbolt p
produces an address something like this:
D01821BC
dup This command is useful when you want to use a value
twice in a calculation. For example:
5 3 * dup 2 + * p
would produce the output:
FF
which is the value of (((5 * 3) + 2) * (5 * 3)).
pop This command removes the top value from the value stack.
For example, if this is the stack:
5
"xyzzy"
7
the input:
pop stk
Copyright 1994 Novell, Inc. Page 3
kdb(1M) kdb(1M)
removes the top value from the stack and displays the
resulting stack:
5
"xyzzy"
clrstk
This command clears the value stack. Remember that the
contents of the stack are saved when you exit and re-
enter KDB.
Arithmetic Operations
You can perform arithmetic operations on the top values on the
stack:
+ compute [TOS-1] + [TOS]; pop 2; push result
- compute [TOS-1] - [TOS]; pop 2; push result
* compute [TOS-1] * [TOS]; pop 2; push result
/ compute [TOS-1] / [TOS]; pop 2; push result
% compute [TOS-1] % [TOS]; pop 2; push result
>> compute [TOS-1] >> [TOS]; pop 2; push result
<< compute [TOS-1] << [TOS]; pop 2; push result
< compute [TOS-1] < [TOS]; pop 2; push result
> compute [TOS-1] > [TOS]; pop 2; push result
== compute [TOS-1] == [TOS]; pop 2; push result
!= compute [TOS-1] != [TOS]; pop 2; push result
& compute [TOS-1] & [TOS]; pop 2; push result
| compute [TOS-1] | [TOS]; pop 2; push result
^ compute [TOS-1] ^ [TOS]; pop 2; push result
&& compute [TOS-1] && [TOS]; pop 2; push result
|| compute [TOS-1] || [TOS]; pop 2; push result
! replace [TOS] with ![TOS]
++ replace [TOS] with [TOS] + 1
-- replace [TOS] with [TOS] - 1
For example, this input (subtracting 5 from 7):
7 5 - p
would produce this output:
2
The power of KDB's calculator feature lies in its ability to
evaluate expressions like this:
Copyright 1994 Novell, Inc. Page 4
kdb(1M) kdb(1M)
callout 16 +
This pushes the address of the callout table on the stack and
adds 16 to it. If the size of a callout table entry is 16
bytes, the result of the calculation is the address of the
second entry in the callout table. (Use the size command of
crash(1M) to find the sizes of common system tables.)
Caution: Make sure the divide operator (slash character) is
both preceded and followed by spaces. If any other character
appears next to the slash, it indicates a suffix instead of
division.
READING AND WRITING TO MEMORY
These commands still operate like an RPN calculator, but they
perform specific debugging operations instead of calculations.
To examine and set the contents of memory (and I/O) use the
commands:
r replace [TOS] with the value at virtual address
[TOS]
w write [TOS-1] into virtual address [TOS]; pop 2
dump show [TOS] bytes starting at virtual address
[TOS-1]; pop 2
fdump show [TOS-1] formatted items at [TOS-2] with
format [TOS]; pop 3
r For example, you can find the value of the (long) kernel
variable, lbolt, by typing:
lbolt r p
This puts the virtual address of lbolt on the stack,
replaces it with the value found at that address, and
prints the result.
w To change the value of lbolt to 2000, type:
2000 lbolt w
This writes 2000 at lbolt's virtual address.
You could increment lbolt by typing:
lbolt r ++ lbolt w
This puts the virtual address of lbolt on the stack,
replaces it with the value found at that address, adds 1
to the value, and writes the result at lbolt's virtual
Copyright 1994 Novell, Inc. Page 5
kdb(1M) kdb(1M)
address.
dump This command displays a range of memory, both in hex and
ASCII. For example, if you typed:
putbuf 10 dump
which shows 10 bytes, starting at the virtual address of
putbuf, you would see something like:
........ ........ ........ 61746F74 D0108C50 ............tota
6572206C 6D206C61 726F6D65 ........ D0108C60 l real memor....
In each line, the block of four values on the left shows
the values of 16 bytes, displayed as four 4-byte
longwords in hex. The dots represent values outside of
the requested range. (dump may also display question
marks here; that means the address is invalid). The
next column is the address of the first of the 16 bytes.
The last column is the same 16 bytes displayed in ASCII.
Dots here represent values outside the requested range
or unprintable characters.
fdump This command displays an arbitrary memory structure in a
formatted fashion according to a format string. The
syntax for fdump is:
<address> <count> <format string> fdump
This displays <count> items starting at address
<address>. fdump keeps track of a memory address called
``dot'' while it is processing the format string; some
format commands change the value of dot; others use dot
to access memory; dot is initially set to <address>.
The format string is a (double-quoted) string which is a
sequence of any of the command characters listed below.
Any digits (0-9) in the format string are interpreted as
the ``current number'' (num), which can modify the
following command; most commands reset num after they're
done. Any other character in the format string is
printed literally.
Number Commands
x Print value at dot in hex (base 16),
unsigned.
Copyright 1994 Novell, Inc. Page 6
kdb(1M) kdb(1M)
o Print value at dot in octal (base 8),
unsigned.
d Print value at dot in decimal (base 10),
signed.
u Print value at dot in decimal (base 10),
unsigned.
Size Modifiers (Prefix to Number Commands)
B byte (char)
H half (short)
I int
L long
Other Print Commands
c Print value at dot as a character.
s Print value at dot as a (null-terminated)
string.
i Print value at dot as a machine instruction.
p Print value at dot symbolically.
a Print dot symbolically (hex if num non-
zero).
n Print a newline character.
t Print a tab character.
Register Modification (256 registers; first 200 are general purpose)
m Move dot to register num.
M Move register num to dot.
r Move num2 to register num.
Copyright 1994 Novell, Inc. Page 7
kdb(1M) kdb(1M)
R Move register num to num.
. Move num to dot.
/ Move num to num2.
Indirection
* Push dot; move value at dot to dot.
^ Pop dot from indirection stack.
Quoting
\ Print the next character literally.
`` '' Print the characters between the quotes
literally.
Other Commands
+ Increment dot (by num if non-zero).
- Decrement dot (by num if non-zero).
; Repeat next command num times.
( ) Group commands together.
All print commands advance dot by the size of the
object printed. Print commands are modified by
non-zero num and/or num2 as follows: For number
and instruction commands, num specifies the output
field width. For character and string commands,
num specifies the style of character printing:
default is that all non-printable characters show
as `.'; 1 means to use ``control-character'' style
which prints control characters as a caret (`^')
followed by the control-character letter. For
string commands, num2 specifies the maximum number
of characters to be printed.
Note that the stack used for pushes and pops in
the indirection commands is not the KDB value
stack; it is a special stack just for those
commands.
Copyright 1994 Novell, Inc. Page 8
kdb(1M) kdb(1M)
As an example of using fdump, consider the
following kernel data structure:
struct cdevsw {
int (*d_open)();
int (*d_close)();
int (*d_read)();
int (*d_write)();
int (*d_ioctl)();
int (*d_mmap)();
int (*d_segmap)();
int (*d_poll)();
int (*d_msgio)();
struct streamtab *d_str;
char *d_name;
int *d_flag;
int d_cpu;
} cdevsw[];
The following fdump command could be used to print
out three elements from this cdevsw array:
cdevsw 3 "7;(p, )nt3;(p, )'*s', ^4+*x, ^4+d" fdump
The output from this command would be something
like this:
wdopen, wdclose, wdread, wdwrite, nodev, nodev, nodev,
nodev, nodev, 0x00000000, 'wd', 112, 0
nulldev, nulldev, nodev, nodev, nodev, nodev, nodev,
nodev, nodev, scinfo, 'sc', 100, 0
mmopen, mmclose, mmread, mmwrite, mmioctl, mmmap, nodev,
nodev, nodev, 0x00000000, 'mm', 100, 0
Suffixes
Suffixes can be appended to many KDB commands. They always
begin with the slash character (/).
Caution: Don't leave spaces before or after the slash
character. When the slash is preceded and followed by a
space, it indicates division instead of a suffix.
Operand-size suffixes
The r, w and dump commands can also work with units of
bytes and words, as well as the default longs. To do
this, append one of these suffixes to the command:
/b byte
Copyright 1994 Novell, Inc. Page 9
kdb(1M) kdb(1M)
/w word (2 bytes)
/l long (4 bytes)-this is the default
For example, to display the value of a short (2-byte)
variable at address 0xD0008120, type:
0xD0008120 r/w p
Entering the dump command with /b displays 16 1-byte
values per line, with /w displays eight 2-byte values
per line, and with /l (or nothing) displays four 4-byte
values per line.
Address-space suffixes
The r, w and dump commands, by default, work with kernel
virtual addresses. You can change to physical
addresses, I/O addresses, or user process virtual
addresses by appending one of these suffixes to the
command:
/k kernel virtual-this is the default
/p physical
/io I/O port
/u# user process number # virtual (# is a process
slot number in hex)
/cpu# cpu number # (# is in hex)
/c# same as /cpu#
/p For example, to dump 40 (hex) bytes in longword
format from physical address 2000, type:
2000 40 dump/p
The default address is kernel virtual, so the /p
suffix is required for the physical address. Note
that an operand-size suffix is not required,
because long is the default.
/io For example, to read from port 300 (in bytes) and
display the result, type:
300 r/io/b p
/u# For example, to dump 20 longwords from process
16's u area at an offset of 1000, type:
Copyright 1994 Novell, Inc. Page 10
kdb(1M) kdb(1M)
1000 u + 20 dump/u16
/cpu# For example, to print a stack-trace of cpu 5's
stack, type:
stack/cpu5
or
stack/c5
Suffix formats
Address-space suffixes can be combined with
operand-size suffixes; only the first slash is
required. For example, to do the read from I/O
port 300 shown above, any of these command lines
is acceptable:
300 r/io/b
300 r/b/io
300 r/iob
300 r/bio
Suffixes can also be attached directly to an
address as shorthand for ``read and print.''
Thus, 2000 r/p p can be shortened to 2000/p.
Since the default address-space is kernel virtual,
the common operation of ``read and print from
kernel virtual'' can be even further shortened.
Type lbolt/ to read and print the value of the
(long) kernel variable, lbolt.
DISPLAYING AND WRITING TO REGISTERS
You can examine the CPU's general registers (and some pseudo-
registers) with these commands:
%eax push the contents of 32-bit register eax
%ebx push the contents of 32-bit register ebx
%ecx push the contents of 32-bit register ecx
%edx push the contents of 32-bit register edx
%esi push the contents of 32-bit register esi
%edi push the contents of 32-bit register edi
%ebp push the contents of 32-bit register ebp
%esp push the contents of 32-bit register esp
%eip push the contents of 32-bit register eip
%efl push the contents of 32-bit register efl
Copyright 1994 Novell, Inc. Page 11
kdb(1M) kdb(1M)
%cs push the contents of 16-bit register cs
%ds push the contents of 16-bit register ds
%es push the contents of 16-bit register es
%fs push the contents of 16-bit register fs
%gs push the contents of 16-bit register gs
%ax push the contents of 16-bit register ax
%bx push the contents of 16-bit register bx
%cx push the contents of 16-bit register cx
%dx push the contents of 16-bit register dx
%si push the contents of 16-bit register si
%di push the contents of 16-bit register di
%bp push the contents of 16-bit register bp
%sp push the contents of 16-bit register sp
%ip push the contents of 16-bit register ip
%fl push the contents of 16-bit register fl
%al push the contents of 8-bit register al
%ah push the contents of 8-bit register ah
%bl push the contents of 8-bit register bl
%bh push the contents of 8-bit register bh
%cl push the contents of 8-bit register cl
%ch push the contents of 8-bit register ch
%dl push the contents of 8-bit register dl
%dh push the contents of 8-bit register dh
%trap
push the trap number
%ipl push the interrupt priority level
You can modify the values of general-purpose registers with
these commands:
w%eax
write [TOS] into 32-bit register eax; pop 1
w%ebx
write [TOS] into 32-bit register ebx; pop 1
w%ecx
write [TOS] into 32-bit register ecx; pop 1
w%edx
write [TOS] into 32-bit register edx; pop 1
w%esi
write [TOS] into 32-bit register esi; pop 1
w%edi
write [TOS] into 32-bit register edi; pop 1
w%ebp
write [TOS] into 32-bit register ebp; pop 1
Copyright 1994 Novell, Inc. Page 12
kdb(1M) kdb(1M)
w%esp
write [TOS] into 32-bit register esp; pop 1
w%eip
write [TOS] into 32-bit register eip; pop 1
w%efl
write [TOS] into 32-bit register efl; pop 1
w%cs write [TOS] into 16-bit register cs; pop 1
w%ds write [TOS] into 16-bit register ds; pop 1
w%es write [TOS] into 16-bit register es; pop 1
w%fs write [TOS] into 16-bit register fs; pop 1
w%gs write [TOS] into 16-bit register gs; pop 1
w%ax write [TOS] into 16-bit register ax; pop 1
w%bx write [TOS] into 16-bit register bx; pop 1
w%cx write [TOS] into 16-bit register cx; pop 1
w%dx write [TOS] into 16-bit register dx; pop 1
w%si write [TOS] into 16-bit register si; pop 1
w%di write [TOS] into 16-bit register di; pop 1
w%bp write [TOS] into 16-bit register bp; pop 1
w%sp write [TOS] into 16-bit register sp; pop 1
w%ip write [TOS] into 16-bit register ip; pop 1
w%fl write [TOS] into 16-bit register fl; pop 1
w%al write [TOS] into 8-bit register al; pop 1
w%ah write [TOS] into 8-bit register ah; pop 1
w%bl write [TOS] into 8-bit register bl; pop 1
w%bh write [TOS] into 8-bit register bh; pop 1
w%cl write [TOS] into 8-bit register cl; pop 1
w%ch write [TOS] into 8-bit register ch; pop 1
w%dl write [TOS] into 8-bit register dl; pop 1
w%dh write [TOS] into 8-bit register dh; pop 1
w%trap
write [TOS] into the trap number pseudo-register;
pop 1
Register Sets
The commands listed above can also be used to access specific
register sets. Multiple sets of general registers may have
been saved on the kernel stack (one for each interrupt, trap,
and so on). For more information see ``Printing Kernel Stack
Traces.''
Register sets are numbered from 0 to 19, with 0 being the
current (most recent) set. By default, the general-register
commands use register set 0, but you can override this with a
register-set suffix:
/rs# register set number #
Copyright 1994 Novell, Inc. Page 13
kdb(1M) kdb(1M)
Note that by combining suffixes, you can access any register
of any process. For example, you can get the eax register
from process 5's register set 1 by typing:
%eax/u5rs1
This command will push the contents of that register (%eax) in
register set 1 (/rs1) of user process 5 (/u5).
CPU Control Registers
In addition to the general registers, you can examine the
values of CPU control registers with these commands:
cr0 push the contents of register cr0
cr2 push the contents of register cr2
cr3 push the contents of register cr3
CREATING DEBUGGER VARIABLES
KDB allows you to create named variables that are stored in
the debugger and hold debugger values (numbers or strings).
Two KDB commands apply to variables:
= variable store [TOS] in [variable]; pop 1
vars show values of debugger variables
= variable
This command assigns a value to a debugger variable.
For example:
5 = abc
creates the variable abc if it does not exist, and sets
the variable equal to 5. Now whenever you use the
variable name, its value is pushed onto the stack. For
example:
abc abc + 2 - p
(5 + 5 - 2) will yield 8.
Note that variable names share the same namespace as
debugger macros and kernel global symbols.
vars To look at all the existing variables, use the vars
command. Variables are shown in the following format:
name = value
Copyright 1994 Novell, Inc. Page 14
kdb(1M) kdb(1M)
The vars command also lists macros, in this format:
name :: value
SETTING BREAKPOINTS
Set and modify breakpoints with these commands:
B set breakpoint #[TOS] at address [TOS-1]; pop 2
-or-
set breakpoint #[TOS] at address [TOS-2] with
command string [TOS-1]; pop 3
b set first free breakpoint address [TOS]; pop 1
-or-
set first free breakpoint at address [TOS-1] with
command string [TOS]; pop 2
bn set breakpoint (like b) and push breakpoint #
brkoff
disable breakpoint #[TOS]; pop 1
brkon re-enable breakpoint #[TOS]; pop 1
brksoff
disable all breakpoints
brkson
re-enable all (disabled) breakpoints
trace set breakpoint #[TOS] trace count to [TOS-1]; pop
2
clrbrk
clear breakpoint #[TOS]; pop 1
clrbrks
clear all breakpoints
clraddrbrks
clear all breakpoints for address [TOS]; pop 1
curbrk
push the current breakpoint number, or -1 if not
entered from a breakpoint
Copyright 1994 Novell, Inc. Page 15
kdb(1M) kdb(1M)
?brk show current breakpoint settings
You can have up to 20 breakpoints, numbered 0 through 19, set
at one time.
B, b, and bn
The B command lets you set specific breakpoints, while
the b command automatically picks the first un-set
breakpoint.
This example sets breakpoint 3 at a specific address:
0xD0125098 3 B
Normally, you'll just set a breakpoint at a certain
address. For example:
read b
This sets an instruction breakpoint at the beginning of
the kernel read routine, using the next available
breakpoint number. When the specified address is
executed (after exiting from the debugger), you enter
the debugger again, with a message indicating which
breakpoint was triggered.
Debugger command strings can be added to the breakpoint
commands. Enter a quoted string of commands after the
address:
read "stack" b
which is used as a series of debugger commands that are
executed when the breakpoint is triggered. If there are
several items in the string, separate them with spaces:
ie6unitdata_req "300 r/bio p" b
After these commands are executed, you are prompted for
debugger commands, as usual, unless the q (quit) command
is executed in the command string.
The bn command works like b except that it leaves the
breakpoint number on the value stack. This is useful
for macros.
You can append breakpoint-type suffixes to the
breakpoint commands (B, b, and bn). By default,
breakpoints are ``instruction'' breakpoints, which
trigger when the specified address is executed. The
Copyright 1994 Novell, Inc. Page 16
kdb(1M) kdb(1M)
suffixes cause breakpoints to trigger on data accesses
instead. The breakpoint-type suffixes are:
/a data access breakpoint
/m data modify breakpoint
/i instruction execution breakpoint-this is
the default
With access and modify breakpoints, you can also use
operand-size suffixes to control the size of the address
range that will trigger the breakpoint. The default is
/l (4 bytes); you can also use /w (word) and /b (byte).
(See the earlier discussion of suffixes under "Reading
and Writing to Memory" for more information.)
brkoff and brkon
These commands let you temporarily disable and re-enable
a breakpoint, instead of clearing it with clrbrk and
then re-entering it later. This is especially handy for
breakpoints with command strings.
trace This command sets a trace count for a breakpoint. This
causes the debugger to just print a message and
decrement the count when the breakpoint is triggered,
instead of entering the debugger, until the count
reaches zero. Commands attached to the breakpoint are
not executed.
?brk Use this command to determine the current breakpoint
settings. Each set breakpoint is displayed, with (1)
the breakpoint number, the address (both (2) in hex and
(3) symbolic), (4) the current state, and (5) the type:
0: 0xD003907C(read) ON /i
1 2 3 4 5
The possible states are:
ON set and enabled
DISABLED
set, but currently disabled
OFF un-set (these breakpoints are not displayed
by ?brk)
Copyright 1994 Novell, Inc. Page 17
kdb(1M) kdb(1M)
The possible types (in this example /i) are the same as
the breakpoint-type suffixes described earlier.
If a breakpoint has a non-zero trace count, that is
displayed after the breakpoint state. If a breakpoint
has a command string, it is displayed at the end of the
line. For example, with a count of 5 and a stack
command, the above breakpoint would display as:
0: 0xD003907C(read) ON 0x5 /i "stack"
SINGLE-STEPPING THROUGH INSTRUCTIONS
You can use these commands for single-stepping:
s single step 1 instruction
ss single step [TOS] instructions; pop 1
S single step 1 instruction (passing calls)
SS single step [TOS] instructions (passing calls);
pop 1
s and ss single-step all instructions. S and SS single-step
all instructions except call instructions. They don't step
down into the called routine, but instead skip ahead to the
return from the call, treating the whole subroutine sequence
as a single instruction.
EXAMINING KERNEL DATA STRUCTURES
KDB provides commands for looking at certain kernel
structures:
ps show process information
The ps command shows information about each active process in
the system. This information includes process IDs, flags, and
command names, on one line, with an additional line per LWP
within the process with LWP IDs, flags, states, and scheduling
information. The CPU field of an LWP line is blank for non-
running LWPs; otherwise it is the CPU number on which that LWP
is currently running.
PRINTING KERNEL STACK TRACES
KDB provides the following commands to look at kernel stack
traces:
stack kernel stack trace for the current process
Copyright 1994 Novell, Inc. Page 18
kdb(1M) kdb(1M)
lstack kernel stack trace for LWP [TOS]; pop 1
tstack ``try'' kernel stack trace from [TOS]; pop 1
stackargs set max # arguments in stack trace to [TOS];
pop 1
Note that the argument to lstack can be specified either as
the address of the LWP structure or -1 for the current LWP for
the given CPU. (-1 lstack is equivalent to the stack
command.)
tstack is sometimes useful if lstack or stack fail to give a
full stack trace for some reason. (For example, another CPU
might be hung and fails to respond to the stack-trace
request.) It takes a single argument, which is a stack
pointer value, and attempts to find a potentially valid trace
starting from some stack address greater than or equal to that
value.
The output of stack, lstack and tstack have the same format.
A typical stack trace (for the current process, entered using
CTRL-ALT-d) looks like this:
(current) idle stack:
DEBUGGER ENTERED FROM USER REQUEST
kdcksysrq(D101FD40 D00DE624 81 20)........esp:FFFE9C94 ret:D008F592
*kdintr+0x186(1)...........................esp:FFFE9CD8 ret:D0011A3A
INTERRUPT TO devint0+0x78 from 8:D001218A (r0ptr:FFFE9CEC)
eax: 8 ebx: ------- ecx:FFFFFFFF edx: 8 efl: 246
esi: ------- edi: ------- esp:FFFE9D04 ebp: ------- regset: 0
idle(0 D00EDDD0 D106BC00).................esp:FFFE9D0C ret:D006F11F
*swtch(0 0 0)..............................esp:FFFE9D40 ret:D002464C
>use_private+0xAB()
The stack trace shows a history of which routine called which
other routine, up until the point the debugger was entered (or
in the case of a non-current process, until the process was
context-switched out).
The most-recently-entered routine is shown on the first line.
In the example, the debugger was entered from kdcksysrq,
which, in turn, was called by kdintr; idle was called from
swtch, and so on. The stack trace ends at the point the
kernel was entered from user mode. In the case of a system
process or an idle stack (as shown here) where there is no
user mode, the stack trace ends at the top-level routine
(use_private in this case). (An idle stack is a per-CPU
Copyright 1994 Novell, Inc. Page 19
kdb(1M) kdb(1M)
private stack which is used when the CPU is idle or is
otherwise not running an LWP.)
Routine Trace Format
The trace for each routine has four parts: (1) its address,
(2) the arguments passed to it, (3) the value of its ebp
register, and (4) its return address. For example:
kdcksysrq(D101FD40 D00DE624 81 20)........esp:FFFE9C94 ret:D008F592
1 ------------2---------- ----3--- ---4----
Address
The address that was called usually appears in symbolic
form. A routine name may also include:
An offset (a plus sign (+) and a hex number): *kdintr+0x186
The offset may mean that the actual address called
was somewhere past the start of the indicated
routine. This will most likely happen if a
subroutine was declared "static." Since the
debugger only has access to global symbols, it
finds the nearest preceding global symbol.
The offset may also mean that the exact address
called cannot be determined. The address
displayed in this case is the return address into
this routine from the routine it called. This
will most likely happen if this routine was called
indirectly via a function pointer.
An asterisk (*): *kdintr+0x186
This means the routine was called indirectly.
There is insufficient information in the stack
format to be 100% sure of the correctness of
indirect call traces.
Whenever you see an asterisk in a stack trace,
there is a small chance that some part of the
stack trace from that point on is incorrect.
Arguments
The arguments passed to the routine appear as a list of
hex numbers, enclosed in parentheses. Since the actual
number of arguments passed cannot be determined, KDB
assumes that each routine has no more than a certain
Copyright 1994 Novell, Inc. Page 20
kdb(1M) kdb(1M)
maximum number of arguments. The default is three, but
you can change it with the stackargs command. If a
routine actually has:
Fewer arguments than displayed:
Only the first ones are real. In rare cases when
the debugger can deduce that a routine could not
have been called with the maximum number of
arguments (because there isn't enough room on the
stack), it displays only the maximum possible
number of arguments. In the above stack trace,
the call to kdintr is shown with only one argument
(1).
More arguments than displayed:
Increase the number with stackargs and then
display the stack trace again, or dump out a
portion of the stack directly in order to see all
the arguments (continue to the next section for
details).
esp register
The value of the esp register inside the routine is
shown as a hex number following esp:. This value can be
used as a ``frame pointer'' to access arguments and
local variables for the routine. The following diagram
illustrates the stack layout.
| . . . |
+-------------------------+
[ESP] + 8 | argument 2 |
+-------------------------+
[ESP] + 4 | argument 1 |
+-------------------------+
[ESP] ---> | return address |
+-------------------------+
[ESP] - 4 | local or saved register |
+-------------------------+
[ESP] - 8 | local or saved register |
+-------------------------+
| . . . |
For example, if you want to see all the arguments to a
routine that takes five arguments, find its esp value
from the stack trace - say 0xD2473CD4 - and enter these
commands:
0xD2473CD4 4 + 5 4 * dump
Copyright 1994 Novell, Inc. Page 21
kdb(1M) kdb(1M)
or, more succinctly:
0xD2473CD8 14 dump
Any ebp value in parentheses is a computed value (see
the ebp values for idle and switch in the example). In
these cases, due to code optimization or partial
execution, the ebp value has not been set up for one or
more routines. KDB computes the value ebp ought to have
had and displays it in parentheses.
Return address
This is the address this routine returns to in its
caller. It is shown as a hex number following ret:.
Trap Frames
In addition to lines for each routine, stack traces will often
include ``trap frames'' created when an event causes
suspension of current processing, saving all register values
on the stack. Typical events are interrupts, hardware
exceptions, and system calls. Trap frames are three lines
each, starting with an upper-case, non-indented keyword (like
INTERRUPT in the example). The next two lines contain the
values of the registers at the time the event occurred. The
first line of a trap frame is in one of these formats:
INTERRUPT TO devint0+0x78 from 8:D001218A (r0ptr:E0000D84)
TRAP 0x1 from 8:D001218A (r0ptr:E0000D94)
TRAP EVENT from 17:830676D (r0ptr: E0000D94, ss:esp: 1F:80468E8)
SYSTEM CALL from 17:830676D (r0ptr: E0000D94, ss:esp: 1F:80468E8)
SIGNAL RETURN from 17:830676D (r0ptr: E0000D94, ss:esp: 1F:80468E8)
These represent interrupts, hardware exception traps, trap
event processing, system calls, and returns from old-style
signal handlers, respectively. The number after TRAP is the
hardware exception number; the most common are 0x1 for
breakpoint traps and 0xE for page faults. See
/usr/include/sys/trap.h for a full list of trap numbers.
The colon-separated numbers after the word from are the
segment and offset (cs and eip) at the time the event
occurred. The values in parentheses show the r0ptr value for
the beginning of the trap frame (for example, the address of
the saved %eax register; see /usr/include/sys/reg.h for the
layout of saved registers), and the user stack pointer segment
and offset at the time the event occurred. The user stack
information is only displayed if the trap frame is for an
Copyright 1994 Novell, Inc. Page 22
kdb(1M) kdb(1M)
entry into the kernel from user mode.
RESETTING THE NUMERIC BASE
If you don't start numbers with "0o" (for octal) or "0x" (for
hex), KDB assumes they are in the default numeric base.
Initially, the defaults for both input and output are set to
16 (hex), but you can use these commands to change them:
ibase set default input base to [TOS]; pop 1
ibinary
set default input base to 2
ioctal set default input base to 8
idecimal
set default input base to 10
ihex set default input base to 16
obase set output base to [TOS]; pop 1
ooctal set output base to 8
odecimal
set output base to 10
ohex set output base to 16
CONVERTING ADDRESS SPACES
Use these commands to convert a virtual address to a physical
address:
kvtop convert kernel virtual address [TOS] to physical
uvtop convert user proc #[TOS] address [TOS-1] to
physical; pop 1
PERFORMING CONDITIONAL EXECUTION
KDB provides two commands for conditional execution:
then if [TOS] = 0, skip to endif; pop 1
endif end scope of then command
In other words, a sequence like:
<condition> then <commands> endif
executes <commands> if and only if the <condition> is true
(non-zero).
These are mostly useful for macros and breakpoint command
strings. For example, imagine you wish to set a breakpoint
for when the function inb is called with 2E as its first
argument. Use the following command:
Copyright 1994 Novell, Inc. Page 23
kdb(1M) kdb(1M)
inb "%esp 4 + r 2E != then q" b
This says to set a breakpoint at inb, but enter the debugger
only if the contents of (%esp+4) are equal to 2E. This works
because esp points to the return address on the stack, and the
longword after that is the first argument. For the second
argument, you would add 8 instead of 4 (see the "Printing
Kernel Stack Traces" section for details of the stack layout).
If you do a ?brk command, the display for that breakpoint
includes the string of debugger commands:
0: 0xD003907C(inb) ON /i "%esp 4 + r 2E != then q"
CALLING A KERNEL FUNCTION
Use this command to call an arbitrary kernel function:
call call the function at address [TOS-1] with [TOS]
arguments, given by [TOS-([TOS]+1)], . . . [TOS-2]; pop
[TOS]+2
vcall call the function at address [TOS-1] with [TOS]
arguments, given by [TOS-([TOS]+1)], . . . [TOS-2]; pop
[TOS]+2
Use the call command to call a function which returns a value;
this value will be the [TOS] value on the value stack after
the call command. Use the vcall (void call) command to call a
function with no return value.
To call psignal() with two arguments, the current process and
9, type:
curproc r 9 psignal 2 call
curproc r gives the value of the current process, the first
argument, and 9 is the second argument. psignal is converted
into the address at which that function can be called, and 2
specifies the number of arguments to pass to psignal().
PERFORMING A SYSTEM DUMP
Some systems support dumping all of memory to disk with a
sysdump() kernel function. For these systems, you can take a
dump by calling this function with the command:
Copyright 1994 Novell, Inc. Page 24
kdb(1M) kdb(1M)
sysdump 0 vcall
All of memory and the current state is dumped to the dump
partition on the disk, so you can use crash(1M) to do a
postmortem.
MISCELLANEOUS COMMANDS
Some miscellaneous KDB commands are:
findsym print kernel symbol with address closest to
[TOS]; pop 1
dis disassemble [TOS] instructions starting at
address [TOS-1]; pop 2
nonverbose
turn verbose mode off
verbose turn verbose mode on
newdebug
switch to another debugger on next debugger
entry
help print a help message
? print a help message (same as help)
cmds print a list of all debugger commands
WRITING MACROS
KDB provides the ability to assign a string of commands to a
single new command name, called a macro. When a debugging
task involves repeating the same set of commands many times
(possibly doing other things in between), it is easier to
define a macro and use it in place of the whole set of
commands.
These commands are used for macros:
:: macro
define [macro] as command string [TOS]; pop 1
P print [TOS] in raw form; pop 1
PP print [TOS] values in raw form, from [TOS-
[TOS]], ... [TOS-1]; pop [TOS]+1
vars show values of debugger macros and variables
:: macro
Use this command to define macros. For example:
"curproc r 16 - p" :: newaddr
Copyright 1994 Novell, Inc. Page 25
kdb(1M) kdb(1M)
Note that macro names share the same namespace as
debugger variables and kernel global symbols.
P and PP
These commands are provided to aid in writing macros. P
and PP print values in raw form, without the
embellishments provided by the p command, such as quotes
around strings and automatic newlines after each value.
This allows complete control over formatting. For
example, the input:
"The value of curproc is " curproc r ".\n" 3 PP
might produce the output:
The value of curproc is 0xD1011E80.
To put something like this into a macro means putting
strings inside strings, so you'll have to escape the
inner quotes:
"\"The value of curproc is \" curproc r \".\n\" 3 PP" :: pcurproc
vars Use this command to show the macro definitions. Macros
are shown in this format:
name :: value
Note that the vars command also shows the values of
variables, in this format:
name = value
OUTPUT CONTROL
Some KDB commands send a great deal of output to the screen.
There are various ways you can slow the output down if
necessary.
First, XON/XOFF flow-control is supported, using CTRL-q/CTRL-s
keys.
Second, output may be interrupted (and command execution
aborted) with either the DEL or CTRL-c key.
Third, output is automatically paginated. That is, after each
screenful of output, a [MORE]--- prompt is output, and KDB
waits for an input key before continuing. The number of lines
per screen is controlled by the db_nlines global variable.
You can change the number of lines to, for example, 60, with:
Copyright 1994 Novell, Inc. Page 26
kdb(1M) kdb(1M)
60 db_nlines w
EXECUTING DEBUGGER COMMANDS AT BOOT TIME
KDB allows you to specify an arbitrary command sequence to be
executed at boot time, when the system is coming up. You can
do this by writing the commands into the file
$ROOT/$MACH/etc/conf/cf.d/kdb.rc, then rebuilding the kernel
with idbuild.
Instead of rebuilding the kernel with idbuild, you can modify
the KDB information in an already-built kernel by typing the
command:
unixsyms -i /etc/conf/cf.d/kdb.rc /unix
At boot time, after the (possibly blank) string is executed,
the system enters KDB at the kdb>> prompt, unless a q command
was executed as part of the string - just like conditional
breakpoints. (A non-existent or zero-length kdb.rc file acts
as a single q command, so KDB is not entered.)
USING A SERIAL TERMINAL
KDB can be used from a serial terminal as well as the console.
This is particularly useful if you are trying to debug a
scenario that involves graphics or multiple virtual terminals
on the console.
Before you attempt to use the debugger from a serial terminal,
make sure there is a getty or ttymon running on it. It may be
either logged in or waiting at the login prompt. This ensures
that the baud rate and other parameters are properly set.
You can switch from the console to a terminal, and vice-versa,
with the newterm command. This immediately switches you to
the new terminal. The debugger continues to use this terminal
until you give it the newterm command again, even if you exit
and re-enter KDB.
The newterm command takes two arguments. The first argument
is a string which is the device name of a console-capable
device driver (for example, "iasy"). The second argument is
the unit number of the actual device (usually the minor device
number) you wish to use.
Copyright 1994 Novell, Inc. Page 27
kdb(1M) kdb(1M)
Once you exit from KDB, you can invoke it again from either
the console or a serial terminal. Use the kdb command to
invoke the debugger from a terminal; CTRL-ALT-d only works
from the console. Regardless of where you invoke KDB, its I/O
appears where you directed it during the last KDB session.
ENTERING THE DEBUGGER FROM A DRIVER
If you are debugging a device driver or another part of the
kernel, you can directly invoke the kernel debugger by
including this code in your driver:
#include <sys/xdebug.h>
(*cdebugger) (DR_OTHER, NO_FRAME);
DR_OTHER tells the debugger that the reason for entering is
``other.'' See sys/xdebug.h for a list of other reason codes.
Note that this mechanism cannot be used for debugging early
kernel startup code or driver init routines, since the
debugger cannot be used until its init routine (kdb_init) has
been called.
CONFIGURING KDB
The Kernel Debugger (KDB) can be configured two ways: it can
be enabled at all times, or it can be enabled only on demand.
If demand mode is chosen, KDB must be manually enabled before
it can be invoked.
Enabling KDB
KDB can be invoked by a console key sequence, by a system
crash, or by the kdb command. However, it can only be invoked
if it is enabled. KDB is enabled when it is loaded into the
kernel.
If you choose to enable it always, KDB will be loaded into the
kernel at all times and can be invoked at any time.
If you choose to enable it on demand, KDB must first be loaded
with the modadmin -l kdb command before it can be invoked.
KDB can be unloaded with the modadmin -U kdb command.
The advantage to having KDB always enabled is that it will be
ready even for unanticipated use. The disadvantage is that
whenever it is enabled, KDB consumes about 250K of main
memory. Most of this memory is used for the kernel symbol
table, which is locked in memory whenever KDB is loaded,
Copyright 1994 Novell, Inc. Page 28
kdb(1M) kdb(1M)
unless the PAGESYMTAB tunable is set to 2. This will affect
system performance on systems with small memories.
SECURITY CHECKS
KDB also provides optional security checks which prohibit an
unauthorized user from invoking the kernel debugger. If you
don't turn on these security checks, the kernel debugger can
be entered from a key sequence on the console, presenting a
potential security breach if your console is not physically
secure.
Disabling The Ctrl-Alt-d SEQUENCE
If the security check is not enabled, a user can type a key
sequence at the console without having to log into your
computer and enter the kernel debugger. Unless your console is
in a room that is locked or accessible to only a controlled
group of people, this security check should be enabled to
prevent a security breach. KDB can only be called from the
console using CTRL-ALT-d if the kdb_security flag was set to 0
when the kernel was built. To disable the CTRL-ALT-d key
sequence, reset the kdb_security flag by using
/etc/conf/bin/idtune to change the KDBSECURITY tunable to 1.
Note that the flag setting does not affect the kdb command.
In order to enter the debugger after disabling this key
sequence, the user must log into the computer using a
privileged account and then type the kdb command.
COMMAND SUMMARY
+ compute [TOS-1] + [TOS]; pop 2; push result
- compute [TOS-1] - [TOS]; pop 2; push result
* compute [TOS-1] * [TOS]; pop 2; push result
/ compute [TOS-1] / [TOS]; pop 2; push result
% compute [TOS-1] % [TOS]; pop 2; push result
>> compute [TOS-1] >> [TOS]; pop 2; push result
<< compute [TOS-1] << [TOS]; pop 2; push result
< compute [TOS-1] < [TOS]; pop 2; push result
> compute [TOS-1] > [TOS]; pop 2; push result
== compute [TOS-1] == [TOS]; pop 2; push result
!= compute [TOS-1] != [TOS]; pop 2; push result
& compute [TOS-1] & [TOS]; pop 2; push result
| compute [TOS-1] | [TOS]; pop 2; push result
^ compute [TOS-1] ^ [TOS]; pop 2; push result
&& compute [TOS-1] && [TOS]; pop 2; push result
|| compute [TOS-1] || [TOS]; pop 2; push result
Copyright 1994 Novell, Inc. Page 29
kdb(1M) kdb(1M)
! replace [TOS] with ![TOS]
++ replace [TOS] with [TOS] + 1
-- replace [TOS] with [TOS] - 1
%eax push the contents of 32-bit register eax
%ebx push the contents of 32-bit register ebx
%ecx push the contents of 32-bit register ecx
%edx push the contents of 32-bit register edx
%esi push the contents of 32-bit register esi
%edi push the contents of 32-bit register edi
%ebp push the contents of 32-bit register ebp
%esp push the contents of 32-bit register esp
%eip push the contents of 32-bit register eip
%efl push the contents of 32-bit register efl
%cs push the contents of 16-bit register cs
%ds push the contents of 16-bit register ds
%es push the contents of 16-bit register es
%fs push the contents of 16-bit register fs
%gs push the contents of 16-bit register gs
%ax push the contents of 16-bit register ax
%bx push the contents of 16-bit register bx
%cx push the contents of 16-bit register cx
%dx push the contents of 16-bit register dx
%si push the contents of 16-bit register si
%di push the contents of 16-bit register di
%bp push the contents of 16-bit register bp
%sp push the contents of 16-bit register sp
%ip push the contents of 16-bit register ip
%fl push the contents of 16-bit register fl
%al push the contents of 8-bit register al
%ah push the contents of 8-bit register ah
%bl push the contents of 8-bit register bl
%bh push the contents of 8-bit register bh
%cl push the contents of 8-bit register cl
%ch push the contents of 8-bit register ch
%dl push the contents of 8-bit register dl
%dh push the contents of 8-bit register dh
%trap push the trap number
%ipl push the interrupt priority level
= variable store [TOS] in [variable]; pop 1
:: macro define [macro] as command string [TOS]; pop 1
? print a help message (same as help)
?brk show current breakpoint settings
B set breakpoint #[TOS] at address [TOS-1]; pop 2
-or-
set breakpoint #[TOS] at address [TOS-2]
w/command string [TOS-1]; pop 3
Copyright 1994 Novell, Inc. Page 30
kdb(1M) kdb(1M)
b set 1st free breakpoint address [TOS]; pop 1
-or-
set 1st free breakpoint at address [TOS-1]
w/command string [TOS]; pop 2
bn set breakpoint (like b) and push breakpoint #
brkoff disable breakpoint #[TOS]; pop 1
brkon re-enable breakpoint #[TOS]; pop 1
brksoff disable all breakpoints
brkson re-enable all (disabled) breakpoints
call call the function at address [TOS-1] with [TOS]
arguments, given by [TOS-([TOS]+1)], ... [TOS-
2]; pop [TOS]+2; push function return value
clraddrbrks clear all breakpoints for address [TOS]; pop 1
clrbrk clear breakpoint #[TOS]; pop 1
clrbrks clear all breakpoints
clrstk pop all values
cmds print a list of all debugger commands
cr0 push the contents of register cr0
cr2 push the contents of register cr2
cr3 push the contents of register cr3
curbrk push the current breakpoint #, or -1 if not
entered from a breakpoint
dis disassemble [TOS] instructions starting at
address [TOS-1]; pop 2
dump show [TOS] bytes starting at virtual address
[TOS-1]; pop 2
dup push [TOS]
endif end scope of then command
findsym print kernel symbol with address closest to
[TOS]; pop 1
fdump show [TOS-1] formatted items at [TOS-2] with
format [TOS]; pop 3
help print a help message
ibase set default input base to [TOS]; pop 1
ibinary set default input base to 2
idecimal set default input base to 10
ihex set default input base to 16
ioctal set default input base to 8
kvtop convert kernel virtual addr [TOS] to physical
lstack kernel stack trace for LWP [TOS]; pop 1
newterm switch KDB console I/O to device [TOS-1] unit
number [TOS]; pop 2
newdebug switch to another debugger on next debugger
entry
Copyright 1994 Novell, Inc. Page 31
kdb(1M) kdb(1M)
nonverbose turn verbose mode off
obase set output base to [TOS]; pop 1
odecimal set output base to 10
ohex set output base to 16
ooctal set output base to 8
P print [TOS] in raw form; pop 1
p print [TOS]
PP print [TOS] values in raw form, from [TOS-
[TOS]], ... [TOS-1]; pop [TOS]+1
pop pop 1 value
ps show process information
q quit-exit from the debugger
r replace [TOS] with the value at virtual address
[TOS]
S single step 1 instruction (passing calls)
s single step 1 instruction
SS single step [TOS] instructions (passing calls);
pop 1
ss single step [TOS] instructions; pop 1
stack kernel stack trace for the current process
stackargs set max # arguments in stack trace to [TOS];
pop 1
stk print all values on the stack
then if [TOS] = 0, skip to endif; pop 1
trace set breakpoint #[TOS] trace count to [TOS-1];
pop 2
tstack ``try'' kernel stack trace from [TOS]; pop 1
uvtop convert user process #[TOS] address [TOS-1] to
physical; pop 1
vars show values of debugger variables
vcall call the function at address [TOS-1] with [TOS]
arguments, given by [TOS-([TOS]+1)], ... [TOS-
2]; pop [TOS]+2
verbose turn verbose mode on
w write [TOS-1] into virtual address [TOS]; pop 2
w%eax write [TOS] into 32-bit register eax; pop 1
w%ebx write [TOS] into 32-bit register ebx; pop 1
w%ecx write [TOS] into 32-bit register ecx; pop 1
w%edx write [TOS] into 32-bit register edx; pop 1
w%esi write [TOS] into 32-bit register esi; pop 1
w%edi write [TOS] into 32-bit register edi; pop 1
w%ebp write [TOS] into 32-bit register ebp; pop 1
w%esp write [TOS] into 32-bit register esp; pop 1
w%eip write [TOS] into 32-bit register eip; pop 1
Copyright 1994 Novell, Inc. Page 32
kdb(1M) kdb(1M)
w%efl write [TOS] into 32-bit register efl; pop 1
w%cs write [TOS] into 16-bit register cs; pop 1
w%ds write [TOS] into 16-bit register ds; pop 1
w%es write [TOS] into 16-bit register es; pop 1
w%fs write [TOS] into 16-bit register fs; pop 1
w%gs write [TOS] into 16-bit register gs; pop 1
w%ax write [TOS] into 16-bit register ax; pop 1
w%bx write [TOS] into 16-bit register bx; pop 1
w%cx write [TOS] into 16-bit register cx; pop 1
w%dx write [TOS] into 16-bit register dx; pop 1
w%si write [TOS] into 16-bit register si; pop 1
w%di write [TOS] into 16-bit register di; pop 1
w%bp write [TOS] into 16-bit register bp; pop 1
w%sp write [TOS] into 16-bit register sp; pop 1
w%ip write [TOS] into 16-bit register ip; pop 1
w%fl write [TOS] into 16-bit register fl; pop 1
w%al write [TOS] into 8-bit register al; pop 1
w%ah write [TOS] into 8-bit register ah; pop 1
w%bl write [TOS] into 8-bit register bl; pop 1
w%bh write [TOS] into 8-bit register bh; pop 1
w%cl write [TOS] into 8-bit register cl; pop 1
w%ch write [TOS] into 8-bit register ch; pop 1
w%dl write [TOS] into 8-bit register dl; pop 1
w%dh write [TOS] into 8-bit register dh; pop 1
w%trap write [TOS] into the trap number pseudo-
register; pop 1
Command Suffixes
Operand size
/b byte
/w word (2 bytes)
/l long (4 bytes)-this is the default
Address space
/k kernel virtual-this is the default
/p physical
/io I/O port
/u# user process number # virtual
/cpu# cpu number #
/c# cpu number # (same as cpu#)
Register set
/rs# register set number #
Breakpoint type
/a data access breakpoint
/m data modify breakpoint
Copyright 1994 Novell, Inc. Page 33
kdb(1M) kdb(1M)
/i instruction execution breakpoint-this is the
default
REFERENCES
crash(1M), dc(1)
Copyright 1994 Novell, Inc. Page 34