The lkdebug utility is a generic port-independent utility which can be used to help diagnose locking and hanging issues with the Distributed Lock Manager (DLM) in an Oracle Parallel Server (OPS) environment.
The user-interface to lkdebug is very similar to that of it's predecessor, lkdump. This note serves as a quick lkdebug reference; for a detailed example on debugging the DLM, see [NOTE:45410.1]. In fact the lkdebug dump examples are essentially taken from the above note.
The note also references the DLM-related (G)V$ views which externalize information dumped by lkdebug.
Usage:lkdebug [options]
-l [r|p] 'lock pointer' Lock Object
-r 'resource pointer' Resource Object
-p 'process id' DLM client pid
-d 'rdomain pointer' Rdomain Object
-P 'process pointer' Process Object
-D 'rdomain name' Rdomain name(string)
-O 'i1 i2 types' Oracle Format resname
-a 'res/lock/proc/pres/rdom' all pointers
-a convlock all converting lock (pointers)
-a convres all res ptr with converting locks
The following sections demonstrate how to dump information from the
DLM.
Use the -O option to dump a resource by it's Oracle name.
For example, by monitoring V$LOCK it can be seen that a session is
waiting on a TX lock:
SVRMGR> select * from v$lock where request > 0;
ADDR KADDR SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK
-------- -------- ----- -- ------- ------- ----- ------- ------ -----
813B6138 813B6148 10 TX 262224 14 0 6 15 2
V$LOCK shows no local holder of this lock.
Using the hex values of ID1 and ID2 values from V$LOCK, we can dump the
DLM resource covering this lock by resource name:
SVRMGR> oradebug lkdebug -O 0x40050 0xe TX
The output, written to user_dump_dest, is:
----------------resource---------------------
resname : [0x40050][0xe],[TX]
Local node : 1
dir_node : 0
Persistent Res: N
master_node : 0
Held mode : KJUSERNL
Cvt mode : KJUSEREX
Next Cvt mode : KJUSERNL
grant_bits : KJUSERNL
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 0
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
resp : 0x817648b4
On Scan_q? : N
Total accesses: 2161
Imm. accesses: 12
Granted_locks : 0
Cvting_locks : 1
grant queue :NULL
convert queue :0x81644504
GRANTED_Q:
CONVERT_Q:
Lock Held Req Resource Remote_lkp Master Pid BAST Convert_opt
0x81644504 KJUSERNL KJUSEREX 0x817648b4 0x81639798 0 26071 0 KJUSERGETVALUE
***************** End of lkdebug output *************************
Definitions:
resname : Obvious
Local node : Node requesting the lock resource dump
dir_node : Directory node for this resource
Persistent Res: Is this a persistent resource? Currently only DBA PCM
locks are persistent
master_node : Master node for this resource
Held mode : Lock held in this mode (in grant queue)
Cvt mode : Lock requested in this mode (in convert queue)
Next Cvt mode : @ tbd
grant_bits : Currently granted modes
grant mode : Number of locks (see "count") against this resource
for each mode
count : # of locks granted in each of the "grant mode" modes
for this resource
val_state : State of the resource value block
valblk : Value of the resource value block
resp : Resource pointer for this resource (see lkdebug -r)
On Scan_q? : Is it about to be processed by the DLM
Total accesses: Total # of accesses for this resource
Imm. accesses: Total # of accesses completed without delay
Granted_locks : # locks in grant queue
Cvting_locks : # locks in convert queue
grant queue : Array of locks on grant queue for this resource (extra
information supplied in GRANTED_Q below)
convert queue : Array of locks on convert queue for this resource (extra
information supplied in CONVERT_Q below)
GRANTED_Q : Details about locks on grant queue for this resource
CONVERT_Q : Details about locks on convert queue for this resource
Lock : Lock pointer (see lkdebug -l)
Held : Mode held
Req : Mode requested
Resource : Resource pointer (lkdebug -r)
Remote_lkp : @ tbd
Master : Master node for this resource
Pid : Process ID of lock requestor
BAST : @ tbd (Blocking AST sent? 1=Y, 0=N)
Convert_opt : Lock conversion options
The (G)V$DLM_LOCKS view externalizes most of the above information for all
locks that are either blocked or blocking other resource acquisitions.
Knowing the lock pointer, it is possible to dump the lock details:
SVRMGR> oradebug lkdebug -l 0x81644504
----------------lock-----------------------
Owner node : 1
grant_level : KJUSERNL
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : 0
resp : 0x817648b4
procp : 0x813ee4d0
remote_lockp : 0x81639798
pid : 26071
gid : 1
xid : 131085 2
dd_time : 49.0 secs
dd_count : 7
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : Y
lock_state : KJUSEROPENING
Open Options : KJUSERDEADLOCK
Convert options : KJUSERGETVALUE
***************** End of lkdebug output *************************
Definitions:
Owner node : Current node of requestor
grant_level : Current grant level
req_level : Request level
bast_level : Level log blocking request, if any
notify_func : Has this lock's BAST fired (0=N,1=Y)
resp : Pointer to resource protected by this lock (lkdebug -p)
procp : Process pointer (lkdebug -P)
remote_lockp : Pointer to lock on remote node (0 if blocker)
pid : Process ID of blocked client process (0 if blocker)
gid : Group ID for group-owned locks
xid : Transaction ID for group-owned locks
dd_time : Deadlock detection time (to next deadlock check)
dd_count : # deadlock checks
timeout : Absolute time for timeout of convert request (0=infinity)
On_timer_q? : Slated for timeout processing? (Y/N)
On_dd_q? : Slated for deadlock detection? (Y/N)
lock_state : State of lock
Open Options : Lock grant options
Convert options : Lock conversion options
Note, for group-owned locks the blocker pid will show as 0 in the lkdebug -l
dump.
@ As of 8.0.6/8.1.6 there will hopefully be a new field in the lock structure
@ to show the process which last performed an operation on the lock -
@ effectively who is holding it. Unknown yet whether this will be
@ externalized in lkdebug.
@ Currently this makes life difficult in trying to find the lock holder.
@ You will have to resort to V$LOCK or state dumps in order to get this
@ information.
Knowing the process pointer, the process resource can be dumped:
SVRMGR> oradebug lkdebug -P 0x813ee4d0
----------------proc-----------------------
Local node : 1
gid : 1
pid : 26071
lkp_node : 1
Rec Domain Name : V815
Rec Domain Ptr : 0x81c2abbc
Total accesses : 37990
Imm. accesses : 37968
Locks on ASTQ : 0
Locks Pending AST : 1
Granted locks : 0
AST_Q :NULL
PENDING_Q :0x81644504
GRANTED_Q :NULL
AST_Q:
PENDING_Q:
Lock Held Req Resource Remote_lkp Master Pid BAST Convert_opt
0x81644504 KJUSERNL KJUSEREX 0x817648b4 0x81639798 0 26071 0 KJUSERGETVALUE
GRANTED_Q:
***************** End of lkdebug output *************************
Definitions:
Local node : Current node of requestor
gid : Group ID
pid : PID of client process
lkp_node : Node where this process resides
Rec Domain Name : Recovery domain name
Rec Domain Ptr : Recovery domain pointer
Total accesses : Total # accesses (of DLM?) by this (Oracle) process
Imm. accesses : Total # accesses completed without delay
Locks on ASTQ : # locks on AST queue
Locks Pending AST : # Locks with AST pending
Granted locks : # locks granted to this process
AST_Q : List of locks with ASTs to be processed
PENDING_Q : List of this process's AST pending locks
GRANTED_Q : List of locks granted to this process
Dumps a resource by it's object pointer:
SVRMGR> oradebug lkdebug -r 0x817648b4
The resource pointer value can be found in various other lkdebug dumps:
lkdebug -O (look for "resp", or "Resource" in the GRANTED_Q or CONVERT_Q)
lkdebug -l (look for "resp")
The lkdebug -r option dumps essentially the same information as the -O
(Oracle resource name) option.
Dumps a process by it's operating system PID:
SVRMGR> oradebug lkdebug -p 26071.
This command dumps exactly the same information as lkdebug -P.
Dumps all locks participating in recovery, by domain name:
SVRMGR> oradebug lkdebug -D V815
----------------rdomain----------------------
Rec Domain Name : V815
Rdomain pointer : 0x81c2abbc
Rdomain state : V
Rdomain status : KJPR_RDOM_RR_NO_RV
reference count : 9
rec. in progress : Remote recovery done
Register Queue : NULL
Validate Queue : NULL
***************** End of lkdebug output *************************
Definitions:
Rec Domain Name : Recovery domain name
Rdomain pointer : Recovery domain pointer
Rdomain state : Recovery domain state
Rdomain status : Recovery domain status
reference count : @ tbd
rec. in progress : Recovery in progress
Register Queue : @ tbd
Validate Queue : @ tbd
The domain name can be found in the process (-p and -P) lkdebug dumps
(look for "Rec Domain Name").
Dumps all locks participating in recovery, by domain pointer:
SVRMGR> oradebug lkdebug -d 0x81c2abbc
The domain pointer can be found in the process (-p and -P) lkdebug dumps
(look for "Rec Domain Ptr"). The output is identical to that of the -D
(Rdomain name) dump.
The lkdebug -a
SVRMGR> oradebug lkdebug -a res
nresp = 523 : # resources
resp 0 0x81776e9c : List of resource pointers
resp 1 0x817737c4
resp 2 0x81769774
resp 3 0x8176e850
resp 4 0x817758d0
<< cut here >>
resp 520 0x81764fbc
resp 521 0x81776410
resp 522 0x817774f0
SVRMGR> oradebug lkdebug -a lock
nlockp = 725 : # locks
lockp 0 0x81653600 : List of lock pointers
lockp 1 0x8164e25c
lockp 2 0x81641d44
lockp 3 0x8164824c
<< cut here >>
lockp 722 0x816527ec
lockp 723 0x81649558
lockp 724 0x81653e48
SVRMGR> oradebug lkdebug -a proc
nproc = 16 : # processes
proc 0 0x813ee4d0 : List of process pointers
proc 1 0x813ee2ac
proc 2 0x813ef814
proc 3 0x813efc5c
<< cut here >>
proc 13 0x813ee088
proc 14 0x813ede64
proc 15 0x813edc40
SVRMGR> oradebug lkdebug -a pres
npresp = 54 : # persistent resources
presp 0 0x817672e4 : List of persistent resource pointers
presp 1 0x81771820
presp 2 0x8177392c
presp 3 0x81764cec
<< cut here >>
presp 51 0x81767e24
presp 52 0x817605e8
presp 53 0x81764fbc
SVRMGR> oradebug lkdebug -a rdom
nrdomp = 1 : # resource domains in the DLM
rdp 0 0x81c2abbc : Resource domain pointer
SVRMGR> oradebug lkdebug -a convlock
nlockp = 1 : # locks on the convert queue
lockp 0 0x81644504 : Lock pointer(s)
This lock pointer is the address of the lock covering the resource that we
have requested in exclusive mode (KJUSEREX) but have been granted in NULL
mode (KJUSERNL), and hence it is in the convert queue.
SVRMGR> oradebug lkdebug -a convres
nresp = 1 : # resources on the convert queue
resp 0 0x817648b4 : Resource pointer(s)
This resource pointer is the address of the resource that we have
requested in exclusive mode (KJUSEREX) but have been granted in NULL
mode (KJUSERNL), and hence it is in the convert queue.
The following (G)V$ views can also be used to view DLM-related information: (G)V$DLM_ALL_LOCKS : All locks in the DLM (G)V$DLM_CONVERT_LOCAL : Statistics on local lock conversions (G)V$DLM_CONVERT_REMOTE : Statistics on remote lock conversions (G)V$DLM_LOCKS : All blocking or blocked locks. Useful in diagnosing OPS hangs as the output is similar to that dumped by lkdebug -O (G)V$DLM_MISC : DLM statistics (G)V$DLM_RESS : All DLM resources - 1 record per resource (G)V$RESOURCE_LIMIT : SGA resources (including INITIAL, CURRENT, and MAXIMUM utilization). Useful for determining whether DLM LM_% resources have been set correctly.