Subject:  TECH: Unix Semaphores and Shared Memory Explained 
Creation Date:  06-OCT-1994 
 

Unix Semaphores and Shared Memory Explained
==========================================                                                                

            Understanding Oracle and Shared Memory/Semaphores

  Shared memory and semaphores are two important resources for an Oracle
  instance on Unix.  An instance cannot start if it is unable to allocate
  what it needs.  This paper primarily discusses the process Oracle goes through
  to allocate shared memory and semaphores at instance startup.  Other
  important points unrelated to startup as well as some troubleshooting
  information will be touched.

General
=======

  Shared memory is exactly that - a memory region that can shared between
  different processes.  Oracle uses shared memory for implementing the
  SGA, which needs to be visible to all database sessions.  Shared memory
  is also used in the implementation of the SQL*Net V1 Fast driver as a
  means of communicating between the application and shadow process.  On
  the RS/6000, each shadow process stores its PGA in a shared memory
  segment (however, only the shadow attaches this segment).  In the
  latter two cases, Oracle allocates the shared memory dynamically as
  opposed to the allocation of the SGA, which occurs at instance startup.
  This allocation will not be discussed in this paper.

  Semaphores can be thought of as flags (hence their name, semaphores).
  They are either on or off.  A process can turn on the flag or turn it off.
  If the flag is already on, processes who try to turn on the flag will
  sleep until the flag is off.  Upon awakening, the process will
  reattempt to turn the flag on, possibly suceeding or possibly sleeping
  again.  Such behaviour allows semaphores to be used in implementing a
  post-wait driver - a system where processes can wait for events (i.e.
  wait on turning on a semphore) and post events (i.e. turning of a
  semaphore).  This mechanism is used by Oracle to maintain concurrency
  control over the SGA, since it is writeable by all processes attached.
  Also, for the same reasons, use of the Fast Driver requires additional
  semaphores.  However, these semaphores will be allocated dynamically
  instead of at instance startup.  This allocation will not be discussed in
  this paper.

Instance startup
================

  On instance startup, the first things that the instance does is: 

    -Read the "init<SID>.ora"

    -Start the background processes

    -Allocate the shared memory and semphores required  

  The size of the SGA will be calculated from various "init.ora" parameters.
  This will be the amount of shared memory required.  The SGA is broken into 4 
  sections - the fixed portion, which is constant in size, the variable portion, 
  which varies in size depending on "init.ora" parameters, the redo block
  buffer, which has its size controlled by log_buffers, and the db
  block buffer, which has its size controlled by db_block_buffers.

  The size of the SGA is the sum of the sizes of the 4 portions.
  There is unfortunately no simple formula for determining the size
  of the variable portion.  Generally, the shared pool dominates all
  other parts of the variable portion, so as a rule of thumb, one can
  estimate the size as the value of shared_pool_size (in v6, one can
  ignore the size of the variable portion).  

  The number of semphores required is much simpler to determine. Oracle will 
  need exactly as many semaphores as the value of the processes "init.ora" 
  parameter.

  Note that the recommended kernel parameter values in the ICG are enough
  to support the default database (4M SGA, 50 processes), but may be
  insufficient to run a larger instance.  With the above estimations and the
  information which follows, a DBA should be able to build a kernel with
  appropriate settings to support the instance.


Shared memory allocation
========================

  Oracle has 3 different possible models for the SGA - one-segment,
  contiguous multi-segment, and non-contiguous multi-segment.
  When attempting to allocate and attach shared memory for the SGA, it
  will attempt each one, in the above order, until one succeeds or raises
  an ORA error.  On other, non-fatal, errors, Oracle simply cleans up and
  tries again using the next memory model.  The entire SGA must fit into
  shared memory, so the total amount of shared memory allocated under any
  model will be equal to the size of the SGA.  This calculated value will
  be referred to below as SGASIZE.

  The one-segment model is the simplest and first model tried. In this
  model, the SGA resides in only one shared memory segment.  Oracle attempts
  to allocate and attach one shared memory segement of size equal to total
  size of the SGA.  However, if the SGASIZE is larger than the configured
  SHMMAX, this will obviously fail (with EINVAL).  In this case, the SGA will
  need to be placed in multiple shared memory segments, and Oracle proceeds
  to the next memory model for the SGA.  If an error other than EINVAL occurs
  when allocating the shared memory with shmget(), Oracle will raise an
  ORA-7306.  If the segment was received (i.e. if SHMMAX > SGASIZE), Oracle
  attempts to attach it at the start address defined in ksms.o.  An error
  on the attach will raise an ORA-7307.

  With multiple segments there are two possibilities.  The segments
  can be attached contiguously, so that it appears to be one large
  shared memory segment, or non-contiguously, with gaps between the
  segments.  The former wastes less space that could be used for the stack
  or heap, but depending on alignment requirements for shared memory
  (defined by SHMLBA in the kernel), it may not be possible.

  At this point, Oracle needs to determine SHMMAX so it can determine how many
  segments will be required.  This is done via a binary search
  algorithm over the range [1...SGASIZE] (since Oracle is trying this
  model and not the one segment model it must be that SHMMAX<SGASIZE)
  The value of SHMMAX calculated will then be rounded to an even page size
  (on some machines, possibly to an even 2 or 4 page block).

  In the contiguous segment model, Oracle simply divides the SGA into
  SGASIZE/SHMMAX (rounded down) segments of size SHMMAX plus another segment
  of size SGASIZE modulo SHMMAX.  If more than SS_SEG_MAX segments are
  required total, an ORA-7329 is raised.  It then allocates and attaches
  one segment at a time, attaching the first segment at the start address
  defined in "ksms.o".  Subsequent segments are attached at an address equal
  to the previous segment's attach address plus the size of the previous
  segment so that they are contiguous in memory.  

  For example, if SHMMAX is 2M, SGASIZE is 5M, and the start address is 
  0xd0000000, there would be 3 segments, 2 of 2M and 1 of 1M.  They would be 
  attached at 0xd0000000, 0xd0000800 (0xd0000000+2M), and 0xd0001000 
  (0xd0000800+2M).  If Oracle receives an error allocating a shared memory 
  segment, an ORA-7336 is raised.

  If an error is raised on attaching a shared memory segement, Oracle checks
  the system error returned.  If it is EINVAL, the attach address used is most
  likely badly aligned (not a mulitple of SHMLBA).  In this case, Oracle tries
  the next model for SGA allocation, non-contiguous segments.  Otherwise, an
  ORA-7337 is raised.

  The last model Oracle will try is the non-contiguous model.  Here,
  things become a bit more complicated.  After calculating SHMMAX, Oracle
  first checks to see if it can put the fixed and variable portion into
  one shared memory segment just large enough to hold the two portions
  If it can, it allocates a segment just big enough to hold both portions.
  If it cannot, it will put them each into their own seperate segment just
  large enough to hold each portion.  If the fixed portion is larger than
  SHMMAX, an ORA-7330 will be raised.  If the variable portion is larger
  than SHMMAX, an ORA-7331 will be raised.  Then Oracle computes the number
  of redo block buffers it can fit in a segment (rounded down to an
  integral number of buffers - buffers cannot overlap segments).  An ORA-7332
  is raised is SHMMAX is smaller than the size of a redo block.  

  Similarly, the number of db block buffers per segment is calculated, with an 
  ORA-7333 raised if SHMMAX is too small to hold one db block.  Then Oracle can 
  compute the total number of segments required for both the redo and database 
  block buffers. This will be buffers/buffers per segment (rounded down) segments
  and one (if necessary) of buffers modulo buffers per segment size, calculated
  seperately for both the redo and db block buffers.  These segments will be
  of a size just large enough to hold the buffers (so no space is wasted).

  The total number of segments allocated will then be the number needed for
  the fixed and variable portions (1 or 2) plus the number needed for the
  redo block buffers plus the number of segments needed for the database block
  buffers.  If this requires more than SS_SEG_MAX segments, an ORA-7334 is
  raised. 

  Once the number of segments and their sizes is determined, Oracle
  then allocates and attaches the segments one at a time; first the fixed
  and variable portion segment(s), then the redo block buffer segment(s),
  then the db block buffer segment(s).  They will be attached non-contiguously,
  with the first segment attached at the start address in "ksms.o" and following
  segments being attached at the address equal to the attach address of the
  previous segment plus the size of the previous segment, rounded up to a
  mulitple of SHMBLA.  

  If Oracle receives an error allocating a shared memory segment, an ORA-7336 is 
  raised.  If an error is raised on attaching a shared memory segement, Oracle 
  checks the system error returned.  If it is EINVAL, normally another model 
  would be tried, but as there are no more models to try, an ORA-7310 is raised. 
  Other attach errors raise an ORA-7337.

  At this point, we have either attached the entire SGA or returned an
  ORA error.  The total size of segments attached is exactly SGASIZE;
  no space is wasted.  Once Oracle has the shared memory attached, Oracle
  proceeds to allocating the semaphores it requires.

Semaphore allocation
====================

  Semaphore allocation is much simpler than shared memory.  Oracle just
  needs to allocate a number of semaphores equal to the processes parameter
  in "init.ora".  PROCESSES will be used to refer to this value.  Note on
  machines with a post-wait kernel extension, Oracle does not need to allocate
  semaphores (because it doesn't need to implement its own post-wait mechanism).

  Oracle uses semaphores to control concurrency between all the  
  background processes (pmon, smon, dbwr, lgwr, and oracle shadows).  
  Semaphores are also used to control two-task communication between  
  the user process and shadow process if the fast (shared memory)  
  driver is used.  And in the Unix ports based on MIPS RISC  
  processors, Oracle uses a special semaphore to perform basic  
  test & set functions that are not provided by the processor. 

  Typing "ipcs -sb" will show you what semaphores are allocated to  
  your system at the moment.  This will display all the semaphore  
  sets allocated, their identifying number, the owner, the number  
  of semaphores in each set, and more.  
  
  Occasionally, unexpected termination of Oracle processes will  
  leave semaphore resources locked.  If your database is not  
  running, but "ipcs -sb" shows that semaphore sets owned by  
  oracle are still in use, then you need to deallocate (free)  
  them.  If you don't do this, then you may not be able to allocate  
  enough semaphores later to restart your database.  
  
  Freeing semaphore sets is done with the "ipcrm" command.  For  
  each set that oracle has allocated, type "ipcrm -s ID" where ID  
  is the set number you see from the "ipcs" output.  Semaphores can  
  also be freed by rebooting the system.  
  
  ORA-7250, ORA-7279, ORA-27146
  If the environment variable ORANSEMS is set, Oracle will use that value
  as the number it will allocate per set.  Oracle will attempt to allocate
  one set of size ORANSEMS.  If this fails, an ORA-7250 is raised.  If
  ORANSEMS is not set, Oracle tries to determine the maximum number of
  semaphores allowed per set (SEMMSL).  It does this by first trying to
  allocate a set of PROCESSES semaphores.  If this fails with EINVAL, it
  tries again, this time trying to get one fewer semaphore.  If this fails
  an ORA-7279 or ORA-27146 on 8.1.X or higher is raised. This process 
  continues until either the semget() succeeds, or when the 
  number of semaphores Oracle is attempting to allocate drops to zero. 
  Increase the kernel parameter SEMMNS if an ORA-7279 or ORA-27146 is 
  generated.  

  ORA-7251
  If the latter case occurs, an ORA-7251 will be raised.  Now Oracle begins 
  allocating sets of size SEMMSL (or ORANSEMS, as the case may be) until it has 
  at least PROCESSES semaphores.

  ORA-7252, ORA-7339
  All semaphore sets will be the same size, so if PROCESSES is not a multiple
  of SEMMSL (or ORANSEMS), there will be additional semaphores allocated that
  will not be used (or in other words, PROCESSES/SEMMSL, rounded up, sets of
  SEMMSL semaphores will be allocated).  Should an error occur trying to
  allocate a semaphore set, ORA-7252 will be raised.  If more than SS_SEM_MAX
  semaphore sets would be required, an ORA-7339 occurs.

  At this point, Oracle has either allocated at least PROCESSES semaphores
  or returned an ORA error.  All IPC resources required by Oracle on Unix
  have been allocated and the related information can be written into the
  sgadef file for this instance for later use by other processes which connect
  to the instance.

Connecting to an instance
=========================

  All shadow processes, when starting, attempt to attach the SGA.  Shadows
  will be started whenever there is a logon attempt (the connect command
  includes an implicit logoff, so it produces a new shadow).  The only
  exception is SQL*Dba in version 7 which immediately spawns a shadow process
  and where connect commands do not spawn a new shadow.  Also, since SQL*Dba
  is used to start up the database, errors encountered in attempting to attach
  the SGA will be discarded because the SGA may not have been allocated yet.
  When a startup command is issued later, the SGA and semaphores will be
  allocated.  Note that this applies only to version 7 and sqldba.

  What Oracle does when attempting to connect to the SGA depends on the
  version of Oracle.  In version 6, the "sgadef<SID>.dbf" file is used to
  get the necessary information.  In version 7, the SGA itself contains
  the information about the shared memory and semaphores (how the
  bootstrap works will be explained later).  In either case, the
  information stored is the same - the key, id, size, and attach
  address of each shared memory segment and the key, id, and size of
  each semaphore set.  Note that we need not do anything special to
  initialize the semaphores.  We can use them with the data structure
  we read in on connecting.

  The version 6, approach is rather simple.  It first tries to open the
  "sgadef<SID>.dbf" file.  If it cannot, an ORA-7318 is raised.  Once
  opened, the data written earlier on startup is read.  If an error
  occurs for some reason on the read, an ORA-7319 occurs.  Once all the
  data is read in, Oracle attaches each segment in turn.  
  
  First, it generates what it believes the key for the segment should be. It
  then gets that segment, returning ORA-7429 if it fails.  The key used
  and the key stored are then compared.  They should be equal, but if
  not, an ORA-7430 occurs.  Once the key is verified, the segment is
  attached.  A failure to attach the segment raises an ORA-7320.  If
  the segment is attached, but not at the address we requested, an
  ORA-7321 occurs. This process is repeated for all segments until the
  entire SGA is attached.

  Version 7 differs only in the first part, when the shared memory and
  semaphore data is read. Once that data is read in, Oracle proceeds in
  the same manner.  To fetch this data, Oracle generates what it thinks
  should be the key for the first segment of the SGA and attaches it
  as if it were the only segment.  Once it is attached, the data is
  copied from the SGA.  With this data, Oracle attaches any remaining
  segments for the SGA.  

  There is one possible problem.  If somehow two instances have a key 
  collision (i.e. they both generate the same key for their first segment), it 
  is possible to only have one of the two instances up at a time! Connection 
  attempts to either one will connect a user to whichever instance is up.  
  This is rare, but can happen.  Development is currently working on a better 
  key generation algorithm.


Attaching shared memory
=======================

  As seen in previous sections, shared memory must be received (this may
  mean allocating the shared memory, but not necessarily) and then
  attached, to be used.  Attaching shared memory brings the shared
  memory into the process' memory space.  There are some important
  things about attach addresses.  For one thing, they may need to be
  alligned on some boundary (generally defined by SHMLBA).  More
  importantly, shared memory must mapped to pages in the process'
  memory space which are unaccounted for.  Every process already has a
  text, a data, and a stack segment laid out as follows (in general):

               +---------+  high addresses
               |  stack  |
               |---------| -+
               |    |    |  |
               |    v    |  |
               |---------|  |
               | shm seg |  |- unused portion
               |---------|  |  These are valid pages for shared memory
               |    ^    |  |  Pages are allocated from this area
               |    |    |  |  as both the stack and heap(data) grow
               |---------| -+
               |   data  |
               |---------|
               |   text  |
               +---------+  low addresses

  So, valid attach addresses lie in the unused region between the stack
  and the data segments (a shared memory segment is drawn in the
  diagram to aid in visualization - not every process has shared memory
  attached!). Of course, the validity also depends on the
  size of the segment, since it cannot overlap another segment. Note
  that both the stack and data segments can grow during the life of a
  process.  Because segments must be contiguous and overlapping is not
  allowed, this is of some importance.  

  Attaching shared memory creates a limit on how much the stack or data segment 
  can grow.  Limiting the stack is typically not a problem, except when running 
  deeply recursive code.  Neither is limiting the data segment, but this does
  restrict the amount memory that can be dynamically allocated by a
  program.  It is possible (but seldom) that some applications
  running against the database may hit this limit in the shadow (since
  the shadow has the SGA attached).  This is the cause of ORA-7324 and
  ORA-7325 errors.  How to deal with these is discussed in the
  troubleshooting section.

  The SGA is attached, depending on the allocation model used, more or
  less contiguously (there may be gaps, but those can be treated as if
  they were part of the shared memory).  So where the beginning of the
  SGA can be attached depends on the SGA's size.  The default address
  which is chosen by Oracle is generally sufficient for most SGAs.
  However, it may be necessary to relocate the SGA for very large
  sizes.   It may also need to be changed if ORA-7324 or ORA-7325 errors
  are occuring.  The beginning attach address is defined in the file
  "ksms.s".  Changing the attach address requires recompilation of the
  Oracle kernel and should not be done without first consulting Oracle
  personnel.  Unfortunately, there is no good way to determine what a good
  attach address will be.  

  When changing the address to allow a larger SGA, a good rule of thumb is 
  taking the default attach address in "ksms.s" and subtracting the size of 
  the SGA.  The validity of an attach address can be tested with the Oracle 
  provided tstshm executable.  Using:

  tstshm -t <size of SGA> -b <new attach address>
   
  will determine if the address is usable or not.

 
Troubleshooting
===============

  Errors which might have multiple causes are discussed in this
  sections.  Errors not mentioned here generally have only one cause
  which has a typically obvious solution.

  ORA-7306, ORA-7336, ORA-7329
  Oracle received a system error on a shmget() call.  The system error
  should be reported.  There are a few possibilities:

  1) There is insufficient shared memory available.  This is
     indicated by the operating system error ENOSPC.  Most likely, SHMMNI
     is too small.  Alternatively, there may shared memory already
     allocated; if it is not attached, perhaps it can be freed.  Maybe
     shared memory isn't configured in the kernel.

  2) There is insufficient memory available.  Remember, shared memory
     needs pages of virtual memory.  The system error ENOMEM indicates there
     is insufficient virtual memory.  Swap needs to be increased, either by
     adding more or by freeing currently used swap (i.e. free other shared
     memory, kill other processes)

  3) The size of the shared memory segment requested is invalid.  In this
     case, EINVAL is returned by the system. This should be very rare - however, 
     it is possible.  This can occur if SHMMAX is not a mulitple of page
     size and Oracle is trying a multi-segment model.  Remember that Oracle
     rounds its calculation of SHMMAX to a page boundary, so it may have
     rounded it up past the real SHMMAX! (Whether this is a bug is
     debatable).

  4) The shared memory segment does not exist.  This would be indicated
     by the system error ENOENT.  This would never happen on startup; it
     only would happen on connects.  The shared memory most likely has been
     removed unexpectedly by someone or the instance is down.

  ORA-7307, ORA-7337, ORA-7320
  Oracle received a system error on a shmat() call.  The system should be
  reported.  There a a few possibilities:
  
  1) The attach address is bad.  If this is the cause, EINVAL is returned
     by the system.  Refer to the section on the attach address to see why
     the attach address might be bad.  This may happen after enlarging the
     SGA.

  2) The permissions on the segment do not allow the process to attach
     it.  The operating system error will be EACCES.  Generally the cause of
     this is either the setuid bit is not turned on for the oracle
     executable, or root started the database (and happens to own the shared
     memory).  Normally, this would be seen only on connects.

  3) The process cannot attach any more shared memory segments.  This
     would be accompanieed by the system error EMFILE.  SHMSEG is too
     small.  Note that as long as SHMSEG is greater than SS_SEG_MAX, you
     should never see this happen.

  ORA-7329, ORA-7334
  Oracle has determined the SGA needs too many shared memory segments.  Since you
  can't change the limit on the number of segments, you should instead increase
  SHMMAX so that fewer segments are required.

  ORA-7339
  Oracle has determined it needs too many semaphore sets.  Since you
  can't change the limit on the number of semaphore sets, you should
  increase SEMMSL so fewer sets are required.

  ORA-7250, ORA-7279, ORA-7252, ORA-27146
  Oracle received a system error on a semget() call.  The system error should be
  reported.  There should be only one system error ever returned with
  this, ENOSPC.  This can mean one of two things.  Either the system
  limit on sempahore sets has been reached or the system limit on the
  total number of semaphores has been reached.  Raise SEMMNI or SEMMNS,
  as is appropriate, or perhaps there are some semaphore sets which can
  be released.  In the case of ORA-7250, ORANSEMS may be set too high
  (>SEMMSL).  If it is, raise SEMMSL or decrease ORANSEMS.

  ORA-7251
  Oracle failed to allocate even a semaphore set of only one semaphore.  It is
  likely that semaphores are not configured in the kernel.

  ORA-7318
  Oracle could not open the sgadef file.  The system error number will be
  returned.  There are a few possible causes:

  1) The file doesn't exist.  In this case, the system error ENOENT is
     returned.  Maybe ORACLE_SID or ORACLE_HOME is set wrong so that Oracle
     is looking in the wrong place.  Possibly the file does not exist (in this 
     case, a restart is necessary to allow connections again).

  2) The file can't be accessed for reading.  The operating system error returned
     with this is EACCES.  The permissions on the file (or maybe
     directories) don't allow an open for reading of the sgadef file.  It
     might not be owned by the oracle owner.  The setuid bit might not be
     turned on for the oracle executable.

  ORA-7319
  Oracle did not find all the data it expected when reading the
  sgadef<SID>.dbf file.  Most likely the file has been truncated. The
  only recovery is to restart the instance.

  ORA-7430
  Oracle expected a key to be used for the segment which does not match the
  key stored in the shared memory and semaphore data structure.  This probably
  indicates a corruption of the sgadef file (in version 6) or
  the data in the first segment of the SGA (in version 7).  A restart of
  the instance is probably necessary to recover in that case.  It may
  also be a key collision problem and Oracle is attached to the wrong
  instance.

  ORA-7321
  Oracle was able to attach the segment, but not at the address it
  requested.  In most cases, this would be caused by corrupted data in
  the sgadef file (in version 6) or the first segment of the SGA (in
  version 7).  A restart of the database may be necessary to recover.

  ORA-7324, ORA-7325
  Oracle was unable to allocate memory.  Most likely, the heap (data
  segment) has grown into the bottom of the SGA.  Relocating the SGA to a
  higher attach address may help, but there may be other causes.  Memory
  leaks can cause this error.  The init.ora parameter sort_area_size may be
  too large, decreasing it may resolve the error.  The init.ora parameter
  context_incr may also be too large, decreasing it may resolve this

  ORA-7264, ORA-7265
  Oracle was unable to decrement/increment a semaphore.  This generally
  is accompanied by the system error EINVAL and a number which is the
  identifier of the semaphore set.  This is almost always because the
  semaphore set was removed, but the shadow process was not aware of it
  (generally due to a shutdown abort or instance crash).  This error
  is usually ignorable.

System Parameters
=================

SHMMAX - kernel parameter controlling maximum size of one shared memory
         segment
SHMMHI - kernel parameter controlling maximum number of shared memory segments
         in the system
SHMSEG - kernel parameter controlling maximum number of shared memory segments
         a process can attach
SEMMNS - kernel parameter controlling maximum number of semphores in
         the system
SEMMNI - kernel parameter controlling maximum number of semaphore
         sets.  Semphores in Unix are allocated in sets of 1 to SEMMSL.
SEMMSL - kernel parameter controlling maximum number of semaphores in a
         semphore set.
SHMLBA - kernel parameter controlling alignment of shared memory
         segments; all segments must be attached at multiples of this value.
         Typically, non-tunable.


System errors
=============

ENOENT - No such file or directory, system error 2
ENOMEM - Not enough core, system error 12
EACCES - Permission denied, system error number 13
EINVAL - Invalid argument, system error number 22
EMFILE - Too many open files, system error number 24
ENOSPC - No space left on device, system error number 28


Oracle parameters
=================

SS_SEG_MAX - Oracle parameter specified at compile time (therefore,
             unmodifiable without an Oracle patch) which defines maximum
             number of segements the SGA can reside in.  Normally set to 20.
SS_SEM_MAX - Oracle parameter specified at compile time (therefore,
             unmodifiable without an Oracle patch) which defined maximum
             number of semaphore sets oracle will allocate.  Normally set to 10.