mlockall: false despite being configured

[markwalkom]

Lots of details below, but the general problem here is that mlockall is not being applied despite all settings being applied as per the config docs.

[root@centos7 ~]# grep MAX_LOCKED_MEMORY /etc/sysconfig/elasticsearch
MAX_LOCKED_MEMORY=unlimited
[root@centos7 ~]# grep mlock /etc/elasticsearch/elasticsearch.yml
bootstrap.mlockall: true

[root@centos7 ~]# rpm -qa|grep elastic
elasticsearch-1.4.2-1.noarch
[root@centos7 ~]# java -version
java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.1.el7_0-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
[root@centos7 ~]# uname -a
Linux centos7 3.10.0-123.4.4.el7.x86_64 #1 SMP Fri Jul 25 05:07:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@centos7 ~]# cat /etc/redhat-release
CentOS Linux release 7.0.1406 (Core)

From the logs
[2015-01-18 17:32:48,613][WARN ][common.jna ] Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit).

From the _nodes API;
"max_file_descriptors" : 65535,

And the outcome is;

[vagrant@centos7 ~]$ curl -s localhost:9200/_nodes?pretty|grep mlockall
          "mlockall" : "true"
        "mlockall" : false

The last one is process.mlockall.

[drewr]

This can happen if the user starting ES doesn't have permission to lock the address space. In the parent environment root must have set ulimit -l unlimited before execing Java.

Is this with the RPM? With sysvinit or systemd?

[markwalkom]

This is a default install from repo, so RPM and sysvinit.

[markwalkom]

Looks like it uses systemd.

@masaruh pointed out that you can edit /usr/lib/systemd/system/elasticsearch.service and set;

LimitMEMLOCK=infinity

Then;

systemctl daemon-reload
service elasticsearch restart

That gives;

# curl -s localhost:9200/_nodes?pretty|grep mlock
          "mlockall" : "true"
        "mlockall" : true

[markwalkom]

Looks like we need to correctly set this via systemd, which is the default as of RHEL/CentOS 7.

[rsareth]

Hi,

I'm using the elasticsearch-1.4.2 RPM package and I'm dealing with the same problem on redhat-6.3 x86_64 / vmware vm with this version of java :

java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.1.el5_11-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

I've changed the property LimitMEMLOCK /usr/lib/systemd/system/elasticsearch.service hoping it would've worked. No effect as espected :-(

And this line in the log that I don't understand :

[2015-04-03 17:22:28,827][WARN ][common.jna ] unable to link C library. native methods (mlockall) will be disabled.

The jna.jar is present :-|

And I don't know how to make the OS using the LimitMEMLOCK parameter.

What's very weird :

• on manual installation (elasticsearch-1.4.4), I can see the mlockall parameters set to true,
• on VM, I've read that the mlockall doesn't work (http://grokbase.com/t/gg/elasticsearch/1344ea29rg/mlockall-fails#20130408pbjeniv37tm2og7cyxa7ifohie)

Any idea, please ?

Thank you

PS: I can't upgrade to ES-1.4.4. I'm stuck on ES-1.4.2 :-(

[lmgray]

I've been chasing this one also with latest elasticsearch on RHEL 6 -- kept getting the following on startup despite having "elasticsearch - memlock unlimited" in /etc/security/limits.conf, "MAX_LOCKED_MEMORY=unlimited" in /etc/sysconfig/elasticsearch:

Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit).

It seems if I cd /etc/rc.d/init.d and start elasticsearch using ./elasticsearch start, the script runs and limits/memlock stuff works as expected. But when it's run as /etc/rc.d/init.d/elasticsearch start, it uses systemctl and the limits are not applied.

I found /usr/lib/systemd/system/elasticsearch.service had the needed line commented out:

# See MAX_LOCKED_MEMORY in sysconfig, use "infinity" when MAX_LOCKED_MEMORY=unlimited and using bootstrap.mlockall: true
#LimitMEMLOCK=infinity

Uncommenting that, running "systemctl daemon-reload" and restarting elasticsearch allowed the memlock to work properly.

[rsareth]

Hi Imgray,

Thank you for your response. I've tested what you have done, but it hasn't worked for me.

I've checked :

• file /etc/security/limits.conf has the correct lines
• file /etc/sysconfig/elasticsearch has "MAX_LOCKED_MEMORY=unlimited"
• file /usr/lib/systemd/system/elasticsearch.service has the limitMEMLOCK uncommented

And I've restarted the service like you did with the script /etc/rc.d/init.d/elasticsearch

At the end, I've have the same result :

$ curl -s localhost:9200/_nodes?pretty|grep mlockall
"mlockall" : "true"
"mlockall" : false

[lmgray]

Have you tried:

cd /etc/rc.d/init.d

./elasticsearch start

I found running it that way it would run the init script vs. using systemd
and it would work.
That said, when I uncommented the limitMEMLOCK like in
/usr/lib/systemd/system/elasticsearch.service
systemd then worked so you seem to have a different problem than I had.

After you start it, try "cat /proc/pid/limits and see what the memory
limit there is?

[rsareth]

Hi Imgray,

Sorry for the delay. I've just tried your actions and it doesn't work too :-(

Here it is the limits for the current pid for elasticsearch :

$ cat /proc/33505/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 30449 processes
Max open files 64000 64000 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 30449 30449 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

As you can see, the "Max locked memory" is set to unlimited. Now, I think I can confirm that our problems are different.

[matejzero]

Why is LimitMEMLOCK commented in systemd service file anyway?
How can I make sure, this stays uncommented during upgrades? It would be nice to add an env variable in sysconfig/elasticsearch so that it would set this to the right value and stayed persistent during upgrades.

I also had this problem and commenting out LimitMEMLOCK in elasticsearch.service resolved the issue. That was on CentOS 7.

Matej

[tlrx]

@matejzero that's a nice suggestion but as far as I know env vars can be only be used in some SystemD settings like ExecStart but not in LimitMEMLOCK. I'll double check.