Golden gate Manager wonot start due to datastore corruption 

Manager process wouldnt start. /gg01/app/oracle/product/OGG122/dirrpt/MGR.rpt has below error:

/gg01app/oracle/product/OGG122/dirrpt/MGR.rpt2021-09-21 10:44:16 WARNING OGG-01934 Datastore repair failed.

2021-09-21 10:44:16 WARNING OGG-06530 Database encountered a critical error.

2021-09-21 10:44:16 WARNING OGG-00543 Unexpected threading library failure. Error code 16 (Device or resource busy).

Source Context :

SourceModule : [ggstd.multithreading]

SourceID : [/scratch/aime/adestore/views/aime_adc4150408/oggcore/OpenSys/src/gglib/ggstd/MultiThreading.cpp]

SourceFunction : [IssueError]

SourceLine : [689]

ThreadBacktrace : [9] elements

: [/gg01/app/oracle/product/OGG122/libgglog.so(CMessageContext::AddThreadContext()+0x1b) [0x7fda2813a0bb]]

: [/gg01/app/oracle/product/OGG122/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x134) [0x7fda28134254]]

: [/gg01/app/oracle/product/OGG122/libgglog.so(_MSG_ERR_PTHREAD_UNEXPECTED_FAILURE(CSourceContext*, int, char const*, CMessageFactory::MessageDisposition)+0x3a) [0x7fda281200a5]]

: [./mgr(ggs::gglib::MultiThreading::Mutex::Unlock()+0xdd) [0x53c42d]]

: [./mgr(backgroundFailCheck(void*)+0x167) [0x5ed2d7]]

: [/gg01/app/oracle/product/OGG122/libggrepo.so(ggrepo::ggRepoThread_t::RunThread(void*)+0xf0) [0x7fda27dc3438]]

: [./mgr(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x147) [0x53e757]]

: [/lib64/libpthread.so.0 [0x7fda23f6083d]]

: [/lib64/libc.so.6(clone+0x6d) [0x7fda1eebb19d]]

2021-09-21 10:44:16 ERROR OGG-00542 Unexpected threading library failure. Error code 22 (Invalid argument).

2021-09-21 10:44:16 ERROR OGG-01668 PROCESS ABENDING.

Solution:

metalink note "OGG-00543 Unexpected Threading Library Failure. Error Code 16 (Device Or Resource Busy) (Doc ID 2149151.1)" to fix the issue:

1) I stopped all the services. Confirmed running "info all" command in ggsci:

GGSCI (hostname) 6> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED

JAGENT STOPPED

2) Take backup of /gg01/app/oracle/product/OGG122/dirbdb 

cd /gg01app/oracle/product/OGG122; tar cvfz dirbdb.tar.gz_19sep2018 dirbdb

3) connected to ggsci and executed delete datastore:

GGSCI (hostname) 1> delete datastore

Are you sure you want to delete the datastore? yes

BDB0118 shmat: id 1628405779: unable to attach to shared system memory region: Invalid argument

BDB0118 shmat: id 1628405779: unable to attach to shared system memory region: Invalid argument

2021-09-21T17:59:28Z INFO OGG-06492 Datastore deleted

GGSCI (hostname) 2>

4) Disconnect from ggsci

5) reconnect to ggsci and execute create datastore command:

GGSCI (hostname) 1> create datastore

Profile 'Trace' added.

2021-09-21T18:02:54Z INFO OGG-06489 Datastore created

GGSCI (hostname) 2>

6) Start mgr now and it should start fine:

GGSCI (hostname 3> start mgr

Manager started.

GGSCI (hostname) 4> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

JAGENT STOPPED