While extending SGA size get out of memory error in Solaris

Please check the shmmax parameter under /etc/system and

Ensure shmmax kernel setting is equal or larger than the sga_max_size

set shmsys:shminfo_shmmax=<size in bytes>
set semsys:seminfo_semmni=1024 
set semsys:seminfo_semmsl=1024 
set shmsys:shminfo_shmmni=100 

if we want to increase the SGA size from 10 GB to 20 GB we need to maintain the shhmax value at OS level>=20 GB