Issue
Had this pop up today, been several years since the ugly Apache semaphore scenario reared its messy head. You’ll have an Apache http web service down, upon typical quick look and attempt to restart, Apache fails to kick off. No biggie, whether digging through logs, systemctl status and journalctl stuffs you’ll eventually come across something similar to:
[Sun Dec 23 15:22:11.048641 2018] [core:emerg] [pid 39427] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex [Sun Dec 23 14:59:38.593507 2018] [:emerg] [pid 33084] AH00020: Configuration Failed, exiting
Many times the last line will reference a config issue, don’t let that necessarily send you down the wrong path if your configs have not changed.
WTF though right, because your server has plenty of space, what’s this junk about no space ?!?!
An Apache semaphore is an inter-process communication tool that is used by Apache to communicate with its child processes. Semaphore marks memory locations as locked and may not release it upon completion of process. In most cases, if parent process dies before the child.
Due to the marking of too many memory locations as being used while they are not, the system will run out of memory locations after some time. This can happen in very busy servers with high uptime.
Let’s check, a few ways to check who and what’s using holding semaphores. If Apache is already stopped:
# ipcs -s<br /> ------ Semaphore Arrays --------<br /> key semid owner perms nsems<br /> 0x00000000 13795328 apache 600 1<br /> 0x00000000 13828097 apache 600 1<br /> 0x00000000 13860866 apache 600 1<br /> 0x00000000 21069827 apache 600 1<br /> 0x00000000 21102596 apache 600 1<br /> 0x00000000 21004293 apache 600 1<br /> 0x00000000 21037062 apache 600 1<br /> 0x00000000 21135367 apache 600 1<br /> 0x00000000 21168136 apache 600 1<br /> 0x00000000 21200905 apache 600 1<br /> 0x00000000 21233674 apache 600 1<br /> 0x00000000 21266443 apache 600 1
# ipcs -s | wc -l
33000
# ipcs -s | awk ‘{print $3}’ | uniq -c
33000 httpd
234 some_software
Resolution
To clear all semaphores owned by the apache user:
ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}'
Afterwards try restarting your Apache process to see if this resolved.
Now, if you need to make modifications and or tweaks to you settings, here’s a few great links depending on scenario: