Ulimit controls the maximum number of system resources that can be allocated to running processes. There is a number of settings. Their current values can be viewed by typing ulimit -a into the terminal window. The result could be this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31343
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31343
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Very often the default settings are not the best ones for production setups. Applications such as cassandra or elasticsearch require changing the defaults for open files or max memory size. The problem I stumbled upon was increasing these values for apps running in docker containers.

Apps running in containers will inherit the limits from their parent process - docker daemon. Typically it is started and managed by upstart (Ubuntu 14.x) or systemd (Ubuntu 16.x) so changing the values in /etc/limits.conf or /etc/security/limits.conf won’t affect them. They need to be set in the specific init scripts.

Upstart (Ubuntu 14.x)

Let’s say we want to increase the limit for max locked memory to unlimited and increase the limits for open files to 524288 and 1048576 respectively for soft limit and hard limit. To do this in upstart managed docker append to /etc/init/docker.conf file the lines:

limit nofile 524288 1048576
limit memlock unlimited unlimited

Then save the file, stop the containers if they are running and restart the docker daemon:

sudo service docker restart

To be sure that everything is ok we can check the process’ limits by issuing the command:

cat /proc/2238/limits

assuming that 2238 is pid of our docker daemon process.

Soft limit is the limit enforced by kernel and the hard limit is the ceiling for the resource value for unprivilged processess which can raise its soft limit only to this value. More info on limits

Systemd.service (Ubuntu 16.x)

Let’s say we want to set unlimited locked memory for docker using the systemd. Just append:

 LimitMEMLOCK=infinity

to the /lib/systemd/system/docker.service file. Then again just stop the containers and restart the daemon. You can check the new limits as described above.

TL;DR To increase the limits edit the docker daemon upstart (Ubuntu 14.x) or systemd (Ubuntu 16.x) init scripts.