A Big Thanks to “https://help.directadmin.com/item.php?id=110”

Restarting  Apache in one of the Servers failed with “No Space left on the Device ” Error and we have enough space in all available partitions.

This is something to deal with Server Kernel semaphores.

“On Linux, A semaphore is a System V IPC object that is used to control utilization of a particular process. Refer https://www.tldp.org/LDP/tlk/ipc/ipc.html

Semaphores are a shareable resource that take on a non-negative integer value. They are manipulated by the P (wait) and V (signal) functions, which decrement and increment the semaphore, respectively. When a process needs a resource, a “wait” is issued and the semaphore is decremented. When the semaphore contains a value of zero, the resources are not available and the calling process spins or blocks (as appropriate) until resources are available. When a process releases a resource controlled by a semaphore, it increments the semaphore and the waiting processes are notified.”

The system does n`t have actual resource available to serve the request, so either we need to configure the Kernel Semaphores or clear the Old entries to get it back.

kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI

SEMMSL      maximum number of semaphores per array
SEMMNS     maximum semaphores system-wide
SEMOPM     maximum operations per semop call
SEMMNI      maximum arrays

[root@r12 ~]# cat /etc/sysctl.conf |grep kernel.sem
kernel.sem = 256 32000 100 142
[root@r12 ~]#

[root@r12 ~]# cat /proc/sys/kernel/sem
256 32000 100 142
[root@r12 ~]#

[root@r12 ~]# ipcs -l | awk ‘FNR>=7 && FNR<=15’

—— Semaphore Limits ——–
max number of arrays = 142
max semaphores per array = 256
max semaphores system wide = 32000
max ops per semop call = 100
semaphore max value = 32767

—— Messages: Limits ——–
[root@r12 ~]#

[root@r12 ~]# ipcs -ls

—— Semaphore Limits ——–
max number of arrays = 142
max semaphores per array = 256
max semaphores system wide = 32000
max ops per semop call = 100
semaphore max value = 32767

To Clear ,  we can execute the following

ipcs | grep apache | awk ‘{print $2}’ > sem.txt
for i in `cat sem.txt`; do { ipcrm -s $i; }; done;