No space left on device: Couldn’t create accept lock – Apache fails to start

Hey Guys,

This is a rare case of Apache issue but can happen particularly in a VPS environment. The HTTPd services will be found dead and it fails to start/restart. In the logs, you will find an error similar to the one in the snippet below.

[emerg] (28)No space left on device: Couldn't create accept lock
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[notice] Digest: generating secret for digest authentication ...
[notice] Digest: done
[warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[emerg] (28)No space left on device: Couldn't create accept lock

This issue is related to the semaphore. A semaphore is an inter-process communication tool that is used by Apache to communicate with its child processes. Semaphore allocate memory locations for Apache and mark it as locked. It may not release it once the child process is completed. Due to the allocation of  too many memory locations as being used while they are not actually used up,  the system will run out of memory allocations after some time. The apache server fails to allocate semaphores for new child processes.

The semaphores should normally free up the memory when the web server is restarted or completed the process. If the semaphores still exist after such a situation, there arises the issue.

The semaphores can be listed by the command

ipcs -s

[[email protected] logs]# ipcs -s
------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 32769      root       600        1
0x00000000 65538      root       600        1
0x00000000 98307      nobody     600        1
0x00000000 131076     nobody     600        1
0x00000000 163845     nobody     600        1
0x00000000 196614     nobody     600        1

If the semaphores exists when the webserver is down, we can clean it up to resolve the issue. The semaphores can be removed by using the command

ipcrm -s <semid>

[[email protected] logs]# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x00000000 65538      root       600        1         
0x00000000 622595     nobody     600        1         
0x00000000 655364     nobody     600        1         
0x00000000 688133     nobody     600        1         
0x00000000 720902     nobody     600        1         

[[email protected] logs]# ipcrm -s 688133

To destroy all the semaphores at once, use the command below

for semid in `ipcs -s | grep nobody | cut -f2 -d” “`; do ipcrm -s $semid; done

[[email protected] logs]# for semid in `ipcs -s | grep nobody | cut -f2 -d" "`; do ipcrm -s $semid; done
[[email protected] logs]# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x00000000 65538      root       600        1

To fix this permenantly, we need to increase the current limits. To increase, modify the file /etc/sysctl.conf and add the following lines.

kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

After modifying the files, run the command below to load the new settings.

sysctl -p

You can view the current kernel/semaphore parameters by using the command:

ipcs -l

Get more stuff like this
in your inbox

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for subscribing.

Something went wrong.

Leave a Reply

Your email address will not be published. Required fields are marked *

Get more stuff like this
in your inbox

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for subscribing.

Something went wrong.