Thursday, May 12, 2022

Cutover failed on 12.2

 

                                      

 adop phase=cutover restart=no

Statement handler reference is invalid.

Session killed.

 Note that this is normal in the online patching cutover phase.

 DB-Handle Error Code: 28

 DB-Handle Error Message: ORA-00028: your session has been killed (DBD ERROR: OCIStmtExecute/Describe)

 Reconnecting to the database.    [ERROR]     adop phase=cutover failed on Node: "sridev1"

    [ERROR]     adop phase=cutover failed on Node: "sridev2"

    [UNEXPECTED]adop phase=cutover on all the nodes returned failure.

    [UNEXPECTED]Unable to continue processing.

    [UNEXPECTED]Error running "adop phase=cutover" on node(s): sridev1 and sridev2.


Summary report for current adop session: 

    Node sridev1: Failed

       - Cutover status:    FS_CUTOVER_COMPLETED

    Node sridev2: Failed

       - Cutover status:    FORCE_STARTUP_COMPLETED

    For more details, run the command: adop -status -detail

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)



Solution:-

Please check the adop.log under cutover directory like below
vi /app/<dbname>/fs_ne/EBSapps/log/adop/35/12345_67890/cutover/sridev1/adop.log

[UNEXPECTED]Error occured submittimg compile_invalid_objects job.
[UNEXPECTED]Cutover phase has failed.

if you see above error please execute below steps

1. Check to see if the job is already scheduled:SELECT JOB_NAME, STATE FROM DBA_SCHEDULER_JOBS WHERE JOB_NAME = 'AD_COMPILE_INVALID_OBJECTS';If it returns:JOB_NAME STATE
-------------------------------------- ------------------------------------------
AD_COMPILE_INVALID_OBJECTS SCHEDULED2. Drop the job as apps user.BEGIN
dbms_scheduler.drop_job(job_name => 'AD_COMPILE_INVALID_OBJECTS');
END;3.Rerun the cutover:adop phase= cutover



Monday, February 7, 2022

 Golden gate Manager wonot start due to datastore corruption 

Manager process wouldnt start. /gg01/app/oracle/product/OGG122/dirrpt/MGR.rpt has below error:

/gg01app/oracle/product/OGG122/dirrpt/MGR.rpt2021-09-21 10:44:16 WARNING OGG-01934 Datastore repair failed.

2021-09-21 10:44:16 WARNING OGG-06530 Database encountered a critical error.

2021-09-21 10:44:16 WARNING OGG-00543 Unexpected threading library failure. Error code 16 (Device or resource busy).

Source Context :

SourceModule : [ggstd.multithreading]

SourceID : [/scratch/aime/adestore/views/aime_adc4150408/oggcore/OpenSys/src/gglib/ggstd/MultiThreading.cpp]

SourceFunction : [IssueError]

SourceLine : [689]

ThreadBacktrace : [9] elements

: [/gg01/app/oracle/product/OGG122/libgglog.so(CMessageContext::AddThreadContext()+0x1b) [0x7fda2813a0bb]]

: [/gg01/app/oracle/product/OGG122/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x134) [0x7fda28134254]]

: [/gg01/app/oracle/product/OGG122/libgglog.so(_MSG_ERR_PTHREAD_UNEXPECTED_FAILURE(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x3a) [0x7fda281200a5]]

: [./mgr(ggs::gglib::MultiThreading::Mutex::Unlock()+0xdd) [0x53c42d]]

: [./mgr(backgroundFailCheck(void*)+0x167) [0x5ed2d7]]

: [/gg01/app/oracle/product/OGG122/libggrepo.so(ggrepo::ggRepoThread_t::RunThread(void*)+0xf0) [0x7fda27dc3438]]

: [./mgr(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x147) [0x53e757]]

: [/lib64/libpthread.so.0 [0x7fda23f6083d]]

: [/lib64/libc.so.6(clone+0x6d) [0x7fda1eebb19d]]


2021-09-21 10:44:16 ERROR OGG-00542 Unexpected threading library failure. Error code 22 (Invalid argument).


2021-09-21 10:44:16 ERROR OGG-01668 PROCESS ABENDING.


Solution:

metalink note "OGG-00543 Unexpected Threading Library Failure. Error Code 16 (Device Or Resource Busy) (Doc ID 2149151.1)" to fix the issue:


1) I stopped all the services. Confirmed running "info all" command in ggsci:

GGSCI (hostname) 6> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED

JAGENT STOPPED


2) Take backup of /gg01/app/oracle/product/OGG122/dirbdb 

cd /gg01app/oracle/product/OGG122; tar cvfz dirbdb.tar.gz_19sep2018 dirbdb

3) connected to ggsci and executed delete datastore:

GGSCI (hostname) 1> delete datastore

Are you sure you want to delete the datastore? yes

BDB0118 shmat: id 1628405779: unable to attach to shared system memory region: Invalid argument

BDB0118 shmat: id 1628405779: unable to attach to shared system memory region: Invalid argument

2021-09-21T17:59:28Z INFO OGG-06492 Datastore deleted

GGSCI (hostname) 2>

4) Disconnect from ggsci

5) reconnect to ggsci and execute create datastore command:

GGSCI (hostname) 1> create datastore

Profile 'Trace' added.

2021-09-21T18:02:54Z INFO OGG-06489 Datastore created

GGSCI (hostname) 2>

6) Start mgr now and it should start fine:

GGSCI (hostname 3> start mgr

Manager started.

GGSCI (hostname) 4> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

JAGENT STOPPED