Pulse Node Becomes Inaccessible

Pulse Node Becomes Inaccessible When the acceldata User Exceeds the Process Limit

Issue

Pulse Node Becomes Inaccessible Due to Process Limit Exhaustion for the acceldata User.

Cause

The issue occurs when the acceldata user exceeds its configured process/thread (nproc) limit.

In the reported instance, axnserver spawned an unusually high number of processes/threads, causing the acceldata user to exceed the configured limit.

The condition that causes the axnserver process count to increase unexpectedly has not been consistently reproduced.

Resolution

1. Access the Pulse node using a privileged user.
2. Check the number of processes and threads running for the acceldata user:

ps -u acceldata -Lf | wc -l

3. Check the current nproc limit for the acceldata user:

su - acceldata -c 'ulimit -u'

4. If required, increase the nproc limit by adding or updating the following entries in /etc/security/limits.conf:

acceldata soft nproc <limit>
acceldata hard nproc <limit>

5. Replace <limit> with the required process limit. Log out and log back in for the updated limit to take effect.
6. Check the system-wide PID limit:

sysctl kernel.pid_max

7. If required, update it temporarily:

sudo sysctl -w kernel.pid_max=<value>

8. To persist the setting across system restarts, add the following entry to /etc/sysctl.conf:

kernel.pid_max=<value>

9. Then apply the configuration:

sudo sysctl -p

10. If axnserver is consuming an unusually high number of processes or threads, redeploy the axnserver add-on.
11. Verify that the acceldata user can access the node and run commands successfully.