Every new request to a web app will cause a Semaphore to be created... the problem is that it does not release the created semaphore. I have tried just about every combination of option suggested and none kills the semephore created.

This is how I resolved the issue.

Eventually you will get this error in your /var/log/httpd/error_log:

[crit] (28)No space left on device: Failed to create mutex

Restarting apache will not kill these either!

First check your Semaphore Settings:

ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 250
semaphore max value = 32767

!!NOTE!!  Make sure that this calculation below is always true:

max number of arrays * max semaphores per array = max semaphores system wide.

When the number of Semaphores reaches the "max semaphores per array" you will get the above mentioned error.

Here is my resolution:

1.  Increase "max semaphores per array" so the error happens less :

echo "kernel.sem=250 32000 250 128" >> /etc/sysctl.conf

2.  Update the /etc/init.d/httpd so a simple restart of Apache will kill the stray Semaphores:

add this to the begining of the Start()  process:

ipcs|grep apache|awk '{print $2}'|xargs -I {} ipcrm -s {};

3.  Use OSSEC to restart Apache

First Create a command in ossec.conf

nano /var/ossec/etc/ossec.conf

<!-- START Charles van den akker III custom commands-->
<command>
  <name>apache-restart</name>
  <executable>apache-restart.sh</executable>
  <expect></expect>
  <timeout_allowed>no</timeout_allowed>
</command>
<!-- END  Charles van den akker III custom commands-->

as well as an Active-Response

<!-- START Charles van den akker III custom Active Response-->
<active-response>
  <command>apache-restart</command>
  <location>server</location>
  <rules_id>100051</rules_id>
</active-response>
<!-- END Charles van den akker III custom Active Response-->

Open local_rules.xml

nano /var/ossec/rules/local_rules.xml

<rule id="100051" level="6">
<if_sid>1007</if_sid>
<match>[crit] (28)No space left on device: Failed to create mutex</match>
  <description>Apache Semaphore limit reached. Apache Restarted</description>
  <group>low_diskspace,</group>
</rule>
</group>

This is rule 1007 in syslog_rules.xml:

  <rule id="1007" level="7">
    <match>file system full|No space left on device</match>
    <description>File system full.</description>
    <group>low_diskspace,</group>
  </rule>

Create apache-restart.sh

nano /var/ossec/active-response/bin/apache-restart.sh

#!/bin/sh
# Restarts Apache.
# Requirements: none
# Author: Charles van den akker III

ACTION=$1
USER=$2
IP=$3
ALERTID=$4
RULEID=$5


LOCAL=`dirname $0`;
cd $LOCAL
cd ../
PWD=`pwd`

# Logging the call
echo "`date` $0 $1 $2 $3 $4 $5 $6 $7 $8" >> ${PWD}/../logs/apacherestarts.log


# Restart Apache
if [ "x${ACTION}" = "xadd" ]; then
   /etc/init.d/httpd restart
   exit 0;


# Invalid action
else
   echo "$0: invalid action: ${ACTION}"
fi

exit 1;

Set File Premissions

chown root:ossec /var/ossec/active-response/bin/apache-restart.sh;
chmod 755 /var/ossec/active-response/bin/apache-restart.sh;

Create logfile for apacherestarts:

echo "[START]" >> /var/ossec/logs/apacherestarts.log;
chown ossec:ossec /var/ossec/logs/apacherestarts.log;
chmod 750 /var/ossec/logs/apacherestarts.log;

Restart Ossec:

/var/ossec/bin/ossec-control restart

run a test:

logger "[crit] (28)No space left on device: Failed to create mutex '/tmp/mod_mono/whatever";tail -f /var/log/httpd/error_log;

you should see apache restarting.