opatchauto打RU竟然无法自动关闭集群----打RU未成功该注意什么

近打补丁遇到一个问题，后续发现patchauto无法关闭集群，后续只能暂时关闭该节点集群，保持其他两个打好RU的节点负责对外服务。
下面记录下后续原厂的分析过程，还是值得记录下。
4月22号凌晨节点3次打补丁的时间点大概是1:19，详情如下：
1 从CRS alert.log可以看到从1:20开始，次opatchauto运行时，
CRS正常执行了stop动作；
2 从opatch日志opatchauto2022-04-22_01-19-03AM.log也可以看到从
1:20开始，次opatchauto运行时，调用了rootcrs.pl -prepatch，
而且CRS正常执行了stop动作；
3 从opatch日志opatchauto2022-04-22_01-19-03AM.log可以看到从
1:21开始，次opatchauto运行时，为/oracle/12.2这个
ORACLE_HOME打补丁遇到FAILED；
4 从opatch日志opatch2022-04-22_01-21-01AM_1.log可以看到从1:21
开始，次opatchauto运行时，失败的原因是文件
/oracle/12.2/lib/libclntsh.so.12.1和
/oracle/12.2/lib/libsqlplus.so被操作系统进程PID=377114锁定，根
据这2个文件名判断，当时有sqlplus进程正常运行，可能是某些定时监
控工具调用了sqlplus或有一直未退出的sqlplus进程，总之，是这些进程
锁定了文件，导致次opatchauto失败；
[Apr 22, 2022 1:21:20 AM] [INFO] Files in use by a process:
/oracle/12.2/lib/libclntsh.so.12.1 PID( 377114 ) /oracle/12.2/lib/libclntsh.so.12.1 PID( 377114 ) bclntsh.so.12.1 PID( 377114 )
[Apr 22, 2022 1:21:20 AM] [INFO] Start fuser command /sbin/fuser
/oracle/12.2/lib/libasmclntsh12.so at Fri Apr 22 01:21:20 CST 2022 :21:20 CST 2022
[Apr 22, 2022 1:21:20 AM] [INFO] Finish fuser command /sbin/fuser
/oracle/12.2/lib/libasmclntsh12.so at Fri Apr 22 01:21:20 CST 2022 :21:20 CST 2022
[Apr 22, 2022 1:21:20 AM] [INFO] Start fuser command /sbin/fuser
/oracle/12.2/lib/libsqlplus.so at Fri Apr 22 01:21:20 CST 2022 20 CST 2022
[Apr 22, 2022 1:21:20 AM] [INFO] Finish fuser command /sbin/fuser mmand /sbin/fuser
/oracle/12.2/lib/libsqlplus.so at Fri Apr 22 01:21:20 CST 2022 20 CST 2022
[Apr 22, 2022 1:21:20 AM] [INFO] Files in use by a process: /oracle/12.2/lib/libsqlplus.so PID( 377114 ) PID( 377114 ) <<<<<<文件被进程锁定
[Apr 22, 2022 1:21:20 AM] [INFO] Following active executables are not used by opatch e executables are not used by opatch
process : process :



Following activ Following active executables are used by opatch pro e executables are used by opatch process :
/oracle/12.2/lib/libclntsh.so.12.1 /oracle/12.2/lib/libclntsh.so.12.1 /oracle/12.2/lib/libclntsh.so.12.1
/oracle/12.2/li /oracle/12.2/lib/libsqlplus.so /oracle/12.2/lib/libsqlplus.so b/libsqlplus.so
[Apr 22, 2022 1:21:20 AM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables"
failed. failed.
The details a The details a The details are:

Following activ Following active executables are not used by opatch e executables are not used by opatch
process : process :

Following activ Following active executables are used by opatch pro e executables are used by opatch process :
/oracle/12.2/li /oracle/12.2/lib/libclntsh.so.12.1 /oracle/12.2/lib/libclntsh.so.12.1 b/libclntsh.so.12.1
/oracle/12.2/li /oracle/12.2/lib/libsqlplus.so /oracle/12.2/lib/libsqlplus.so b/libsqlplus.so
[Apr 22, 2022 1:21:20 AM] [SEVERE] OUI- [Apr 22, 2022 1:21:20 AM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check

后续尝试两次打RU
4月22号凌晨节点3后一次打补丁的时间点大概是3:17，详情如下：
 从CRS alert.log可以看到3:15开始CRS开始启动，3:16完成CRS启动，
3:46开始停止CRS，这说明后一次opatchauto运行时，CRS未执行
stop动作；

 从opatch日志opatchauto2022-04-22_01-19-03AM.log也可以看到从
1 3:19开始，后一次opatchauto运行时，调用了rootcrs.pl -
prepatch，但未看到CRS执行stop动作；
2 从opatch日志crspatch_n-pc-sr850-323_2022-04-22_03-19-
04AM.log可以看到从3:19开始，后一次opatchauto运行时，
prepatch检测到ROOTCRS_PREPATCH状态是SUCCESS，而不是
START/FAILED状态，参考MOS文档2703720.1，表明上一次（当晚
执行了多次opatchauto）执行opatchauto未成功完成postpatch，状态
残留未清除，所以，未调用CRS的stop动作；
3 从opatch日志opatchauto2022-04-22_01-19-03AM.log可以看到从
3:19开始，后一次opatchauto运行时，为/oracle/12.2这个
ORACLE_HOME打补丁遇到FAILED；
4 从opatch日志opatch2022-04-22_03-19-06AM_1.log可以看到从3:19
开始，后一次opatchauto运行时，失败的原因是文件
/oracle/12.2/bin/oracle被大量操作系统进程锁定，表明当时数据库未
停止，总之，由于上一次执行opatchauto未成功完成postpatch，后
一次opatchauto运行时未调用CRS的stop动作，因此未停止数据库，锁
定了文件，导致后一次opatchauto失败；
报错如下，此时说明 数据库实例没有停下来，后检查发现集群没有执行关闭操作。
[Apr 22, 2022 3:19:32 AM] [INFO] Following active executables are not used by opatch
process : process :
/oracle/12.2/bi /oracle/12.2/bin/oracle /oracle/12.2/bin/oracle n/oracle

Following active executables are use Following active executables are used by opatch pro d by opatch process :
[Apr 22, 2022 3:19:32 AM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables"
failed. failed.
The details are The details are: The details are:


Following activ Following active executables are not used by opatch e executables are not used by opatch
process : process :
/oracle/12.2/bi /oracle/12.2/bin/oracle /oracle/12.2/bin/oracle n/oracle

Oracle Advanced Customer Services 第15页
Following active executables are use Following active executables are used by opatch pro d by opatch process :

后分析如下
后一次opatchauto运行时，
prepatch检测到ROOTCRS_PREPATCH状态是SUCCESS，而不是
START/FAILED状态，参考MOS文档2703720.1，表明上一次（当晚
执行了多次opatchauto）执行opatchauto未成功完成postpatch，状态
残留未清除，所以，未调用CRS的stop动作；


22 03:19:05: The 'ROOTCRS_PREPATCH' status is SUCCESS
2022-04-22 03:19:05: prepatch has already been done successfully <<<<<prepatch已经被成功执行，但是执行opatchauto未成功完成postpatch

根据4月22号凌晨节点3后一次打补丁的失败原因，参考MOS文档
2703720.1，建议采用如下步骤为节点3打补丁
##停止节点 3 的 CRS，然后修复权限： ，然后修复权限：
su -root
# /grid/12.2/bin/crsctl stop crs - # /grid/12.2/bin/crsctl stop crs -f
# /grid/12.2/crs/install/rootcrs.sh - # /grid/12.2/crs/install/rootcrs.sh -init

##清理上一次打补丁失败后残留的状态信息： ##清理上一次打补丁失败后残留的状态信息：
su -grid
/grid/12.2/bin/cluutil -ckpt -oraclebase $ORACLE_BASE oraclebase -writeckpt -name ROOTCRS_PREPATCH -state START
/grid/12.2/bin/cluutil -ckpt -oraclebase $ORACLE_BASE oraclebase -chkckpt -name ROOTCRS_PREPATCH -status

##验证 prepatch/postpatch 是否可以正常执行： prepatch/postpatch 是否可以正常执行：
su -root
# /grid/12.2/crs/install/rootcrs.sh -prepatch prepatch prepatch
# ef|egrep "ora|grid|d.bin|sqlplus|rman|emagent" ##确认无残留进程
# crontab -l ##确认无 oracle 相关定时作业 oracle 相关定时作业

# /grid/12.2/crs/install/rootcrs.sh -postpatch

##重新打补丁 ##重新打补丁
su -root
# /grid/12.2/OPatch/opatchauto apply <patch_location> -oh /grid/12.2
# /oracle/12.2/OPatch/opatchauto apply <patch_location> -oh /oracle/12.2