我们在学习Oracle基础知识的时候肯定了解过PMON后台进程的功能，包括轮训其他ORACLE PROCESS，清理cleanup dead process并释放锁release enqueue lock ，及清理cleanup latch。 虽然这些理论在你耳边萦绕了千百回， 但你是否有亲眼见证过一次Pmon cleanup dead process并release lock呢？大多数人可能没有。 微观Oracle=> MicroOracle， Maclean带领你见证微观视角下的Oracle behavior:
SQL> select * from v$version;BANNER--------------------------------------------------------------------------------Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit ProductionPL/SQL Release 11.2.0.3.0 - ProductionCORE 11.2.0.3.0 ProductionTNS for Linux: Version 11.2.0.3.0 - ProductionNLSRTL Version 11.2.0.3.0 - ProductionSQL> select * from global_name;GLOBAL_NAME--------------------------------------------------------------------------------www.oracledatabase12g.com SQL> select pid,program from v$process; PID PROGRAM---------- ------------------------------------------------ 1 PSEUDO 2 oracle@vrh1.oracle.com (PMON) 3 oracle@vrh1.oracle.com (PSP0) 4 oracle@vrh1.oracle.com (VKTM) 5 oracle@vrh1.oracle.com (GEN0) 6 oracle@vrh1.oracle.com (DIAG) 7 oracle@vrh1.oracle.com (DBRM) 8 oracle@vrh1.oracle.com (PING) 9 oracle@vrh1.oracle.com (ACMS) 10 oracle@vrh1.oracle.com (DIA0) 11 oracle@vrh1.oracle.com (LMON) 12 oracle@vrh1.oracle.com (LMD0) 13 oracle@vrh1.oracle.com (LMS0) 14 oracle@vrh1.oracle.com (RMS0) 15 oracle@vrh1.oracle.com (LMHB) 16 oracle@vrh1.oracle.com (MMAN) 17 oracle@vrh1.oracle.com (DBW0) 18 oracle@vrh1.oracle.com (LGWR) 19 oracle@vrh1.oracle.com (CKPT) 20 oracle@vrh1.oracle.com (SMON) 21 oracle@vrh1.oracle.com (RECO) 22 oracle@vrh1.oracle.com (RBAL) 23 oracle@vrh1.oracle.com (ASMB) 24 oracle@vrh1.oracle.com (MMON) 25 oracle@vrh1.oracle.com (MMNL) 26 oracle@vrh1.oracle.com (MARK) 27 oracle@vrh1.oracle.com (D000) 28 oracle@vrh1.oracle.com (SMCO) 29 oracle@vrh1.oracle.com (S000) 30 oracle@vrh1.oracle.com (LCK0) 31 oracle@vrh1.oracle.com (RSMN) 32 oracle@vrh1.oracle.com (TNS V1-V3) 33 oracle@vrh1.oracle.com (W000) 34 oracle@vrh1.oracle.com (TNS V1-V3) 35 oracle@vrh1.oracle.com (TNS V1-V3) 37 oracle@vrh1.oracle.com (ARC0) 38 oracle@vrh1.oracle.com (ARC1) 40 oracle@vrh1.oracle.com (ARC2) 41 oracle@vrh1.oracle.com (ARC3) 43 oracle@vrh1.oracle.com (GTX0) 44 oracle@vrh1.oracle.com (RCBG) 46 oracle@vrh1.oracle.com (QMNC) 47 oracle@vrh1.oracle.com (TNS V1-V3) 48 oracle@vrh1.oracle.com (TNS V1-V3) 49 oracle@vrh1.oracle.com (Q000) 50 oracle@vrh1.oracle.com (Q001) 51 oracle@vrh1.oracle.com (GCR0)SQL> drop table maclean;Table dropped.SQL> create table maclean(t1 int);Table created.SQL> insert into maclean values(1);1 row created.SQL> commit;Commit complete.
以上构建了示例环境， 可以看到在该实例中：
PID=2 PMON PID=11 LMON PID=18 LGWR PID=20 SMON PID=12 LMD 我们尝试构造2个存在"enq: TX – row lock contention"冲突的事务，之后使KILL掉第一个事务，这会导致需要PMON来recover dead process并release TX lock:
PROCESS A:QL> select addr,spid,pid from v$process where addr = ( select paddr from v$session where sid=(select distinct sid from v$mystat));ADDR SPID PID---------------- ------------------------ ----------00000000BD516B80 17880 46SQL> select distinct sid from v$mystat; SID---------- 22SQL> update maclean set t1=t1+1;1 row updated.PROCESS BSQL> select addr,spid,pid from v$process where addr = ( select paddr from v$session where sid=(select distinct sid from v$mystat));ADDR SPID PID---------------- ------------------------ ----------00000000BD515AD0 17908 45SQL> update maclean set t1=t1+1;HANG..............PROCESS B 因为"enq: TX – row lock contention"而HANG住
我们使用PROCESS C启用 对SMON的10500 event trace 和对PMON的KST TRACE:
SQL> set linesize 200 pagesize 1400SQL> select * from v$lock where sid=22;ADDR KADDR SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK---------------- ---------------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------00000000BDCD7618 00000000BDCD7670 22 AE 100 0 4 0 48 200007F63268A9E28 00007F63268A9E88 22 TM 77902 0 3 0 32 200000000B9BB4950 00000000B9BB49C8 22 TX 458765 892 6 0 32 1PROCESS A holde的ENQUEUE LOCK有三个 AE、TM、TXSQL> alter system switch logfile;System altered.SQL> alter system checkpoint;System altered.SQL> alter system flush buffer_cache;System altered.SQL> alter system set "_trace_events"='10000-10999:255:2,20,33';System altered.SQL> ! kill -9 17880KILL PROCESS A 并等待PROCESS B完成update对PMON 和 PROCESS B 做errorstack 将KST TRACE记录到磁盘SQL> oradebug setorapid 2;Oracle pid: 2, Unix process pid: 17533, image: oracle@vrh1.oracle.com (PMON)SQL> oradebug dump errorstack 4;Statement processed.SQL> oradebug tracefile_name/s01/orabase/diag/rdbms/vprod/VPROD1/trace/VPROD1_pmon_17533.trcSQL> oradebug setorapid 45;Oracle pid: 45, Unix process pid: 17908, image: oracle@vrh1.oracle.com (TNS V1-V3)SQL> oradebug dump errorstack 4;Statement processed.SQL>oradebug tracefile_name/s01/orabase/diag/rdbms/vprod/VPROD1/trace/VPROD1_ora_17908.trc
分析PMON的 KST TRACE:
2012-05-18 10:37:34.557225 :8001ECE8:db_trace:ktur.c@5692:ktugru(): [10444:2:1] next rollback uba: 0x00000000.0000.002012-05-18 10:37:34.557382 :8001ECE9:db_trace:ksl2.c@16009:ksl_update_post_stats(): [10005:2:1] KSL POST SENT postee=18 num=4 loc='ksa2.h LINE:285 ID:ksasnd' id1=0 id2=0 name= type=02012-05-18 10:37:34.557514 :8001ECEA:db_trace:ksq.c@8540:ksqrcli(): [10704:2:1] ksqrcl: release TX-0007000d-0000037c mode=X2012-05-18 10:37:34.558819 :8001ECF0:db_trace:ksl2.c@16009:ksl_update_post_stats(): [10005:2:1] KSL POST SENT postee=45 num=5 loc='kji.h LINE:3418 ID:kjata: wake up enqueue owner' id1=0 id2=0 name= type=02012-05-18 10:37:34.559047 :8001ECF8:db_trace:ksl2.c@16009:ksl_update_post_stats(): [10005:2:1] KSL POST SENT postee=12 num=6 loc='kjm.h LINE:1224 ID:kjmpost: post lmd' id1=0 id2=0 name= type=02012-05-18 10:37:34.559271 :8001ECFC:db_trace:ksq.c@8826:ksqrcli(): [10704:2:1] ksqrcl: SUCCESS2012-05-18 10:37:34.559291 :8001ECFD:db_trace:ktu.c@8652:ktudnx(): [10813:2:1] ktudnx: dec cnt xid:7.13.892 nax:0 nbx:02012-05-18 10:37:34.559301 :8001ECFE:db_trace:ktur.c@3198:ktuabt(): [10444:2:1] ABORT TRANSACTION - xid: 0x0007.00d.0000037c2012-05-18 10:37:34.559327 :8001ECFF:db_trace:ksq.c@8540:ksqrcli(): [10704:2:1] ksqrcl: release TM-0001304e-00000000 mode=SX2012-05-18 10:37:34.559365 :8001ED00:db_trace:ksq.c@8826:ksqrcli(): [10704:2:1] ksqrcl: SUCCESS2012-05-18 10:37:34.559908 :8001ED01:db_trace:ksq.c@8540:ksqrcli(): [10704:2:1] ksqrcl: release AE-00000064-00000000 mode=S2012-05-18 10:37:34.559982 :8001ED02:db_trace:ksq.c@8826:ksqrcli(): [10704:2:1] ksqrcl: SUCCESS2012-05-18 10:37:34.560217 :8001ED03:db_trace:ksfd.c@15379:ksfdfods(): [10298:2:1] ksfdfods:fob=0xbab87b48 aiopend=02012-05-18 10:37:34.560336 :GSIPC:kjcs.c@4876:kjcsombdi(): GSIPC:SOD: 0xbc79e0c8 action 3 state 0 chunk (nil) regq 0xbc79e108 batq 0xbc79e1182012-05-18 10:37:34.560357 :GSIPC:kjcs.c@5293:kjcsombdi(): GSIPC:SOD: exit cleanup for 0xbc79e0c8 rc: 1, loc: 0x3032012-05-18 10:37:34.560375 :8001ED04:db_trace:kss.c@1414:kssdch(): [10809:2:1] kssdch(0xbd516b80 = process, 3) 1 0 exit2012-05-18 10:37:34.560939 :8001ED06:db_trace:kmm.c@10578:kmmlrl(): [10257:2:1] KMMLRL: Entering: flg(0x0) rflg(0x4)2012-05-18 10:37:34.561091 :8001ED07:db_trace:kmm.c@10472:kmmlrl_process_events(): [10257:2:1] KMMLRL: Events: succ(3) wait(0) fail(0)2012-05-18 10:37:34.561100 :8001ED08:db_trace:kmm.c@11279:kmmlrl(): [10257:2:1] KMMLRL: Reg/update: flg(0x0) rflg(0x4)2012-05-18 10:37:34.563325 :8001ED0B:db_trace:kmm.c@12511:kmmlrl(): [10257:2:1] KMMLRL: Update: ret(0)2012-05-18 10:37:34.563335 :8001ED0C:db_trace:kmm.c@12768:kmmlrl(): [10257:2:1] KMMLRL: Exiting: flg(0x0) rflg(0x4)2012-05-18 10:37:34.563354 :8001ED0D:db_trace:ksl2.c@2598:kslwtbctx(): [10005:2:1] KSL WAIT BEG [pmon timer] 300/0x12c 0/0x0 0/0x0 wait_id=78 seq_num=79 snap_id=1
PMON发现dead process A后首先释放了其手上的TX Lock: ksqrcl: release TX-0007000d-0000037c mode=X 接着它异步Post Process B，告诉Process B 你acquire的TX lock已经被我释放了: KSL POST SENT postee=45 num=5 loc='kji.h LINE:3418 ID:kjata: wake up enqueue owner' id1=0 id2=0 name= type=0 Process B在收到PMON通知后立即继续了工作 ksl2.c@14563:ksliwat(): [10005:45:151] KSL POST RCVD poster=2 num=5 loc='kji.h LINE:3418 ID:kjata: wake up enqueue owner' id1=0 id2=0 name= type=0 fac#=3 posted=0x3 may_be_posted=1 kslwtbctx(): [10005:45:151] KSL WAIT BEG [latch: ges resource hash list] 3162668560/0xbc827e10 91/0x5b 0/0x0 wait_id=14 seq_num=15 snap_id=1 kslwtectx(): [10005:45:151] KSL WAIT END [latch: ges resource hash list] 3162668560/0xbc827e10 91/0x5b 0/0x0 wait_id=14 seq_num=15 snap_id=1 在RAC中它还会POST LMD(lock Manager)进程，通知其改变相关的GES信息: 2012-05-18 10:37:34.559047 :8001ECF8:db_trace:ksl2.c@16009:ksl_update_post_stats(): [10005:2:1] KSL POST SENT postee=12 num=6 loc='kjm.h LINE:1224 ID:kjmpost: post lmd' id1=0 id2=0 name= type=0 之后ksqrcl: release TX的工作才真正成功： ksq.c@8826:ksqrcli(): [10704:2:1] ksqrcl: SUCCESS 之后PMON abort Process A原有的Transaction 2012-05-18 10:37:34.559291 :8001ECFD:db_trace:ktu.c@8652:ktudnx(): [10813:2:1] ktudnx: dec cnt xid:7.13.892 nax:0 nbx:0 2012-05-18 10:37:34.559301 :8001ECFE:db_trace:ktur.c@3198:ktuabt(): [10444:2:1] ABORT TRANSACTION - xid: 0x0007.00d.0000037c 释放Process A原本持有的maclean表的TM lock: ksq.c@8540:ksqrcli(): [10704:2:1] ksqrcl: release TM-0001304e-00000000 mode=SX ksq.c@8826:ksqrcli(): [10704:2:1] ksqrcl: SUCCESS 释放Process A原本持有的AE ( Prevent Dropping an edition in use) lock: ksq.c@8540:ksqrcli(): [10704:2:1] ksqrcl: release AE-00000064-00000000 mode=S ksq.c@8826:ksqrcli(): [10704:2:1] ksqrcl: SUCCESS 清理cleanup process A kjcs.c@4876:kjcsombdi(): GSIPC:SOD: 0xbc79e0c8 action 3 state 0 chunk (nil) regq 0xbc79e108 batq 0xbc79e118 GSIPC:kjcs.c@5293:kjcsombdi(): GSIPC:SOD: exit cleanup for 0xbc79e0c8 rc: 1, loc: 0x303 kss.c@1414:kssdch(): [10809:2:1] kssdch(0xbd516b80 = process, 3) 1 0 exit 0xbd516b80是原PROCESS A 的paddr 起始地址 kssdch函数的主要作用是 删除process相关的state object SO KSS: delete children of state obj.
PMON 调用kmmlrl()函数更新instance goodness统计update for session drop delta kmmlrl(): [10257:2:1] KMMLRL: Entering: flg(0x0) rflg(0x4) kmmlrl_process_events(): [10257:2:1] KMMLRL: Events: succ(3) wait(0) fail(0) kmmlrl(): [10257:2:1] KMMLRL: Reg/update: flg(0x0) rflg(0x4) kmmlrl(): [10257:2:1] KMMLRL: Update: ret(0) kmmlrl(): [10257:2:1] KMMLRL: Exiting: flg(0x0) rflg(0x4) 完成手头的工作后PMON再次进入 3s一次的"pmon timer"等待 kslwtbctx(): [10005:2:1] KSL WAIT BEG [pmon timer] 300/0x12c 0/0x0 0/0x0 wait_id=78 seq_num=79 snap_id=1