Memory overhead when using lua-nginx-module

[Zaynullin]

I have nginx with lua-nginx-module.
When I send range requests to nginx with lua-nginx-module and enabled http_slice_module, nginx takes all free memory and does not free even if all connections closed.
Without lua-nginx-module nginx does not use so much memory, one of worker processes can use some memory and free it during work.

$ nginx -V
nginx version: nginx/1.11.2
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-ef2b708510b1/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --add-module=ngx_devel_kit-0.3.0 --add-module=lua-nginx-module-0.10.6

libluajit-5.1-2 version:

$ ldd /usr/sbin/nginx | grep lua
    libluajit-5.1.so.2 => /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 
$ ls -al /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2
lrwxrwxrwx 1 root root 22 Jun  9  2013 /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 -> libluajit-5.1.so.2.0.2

With libluajit-5.1.so.2.0.4 memory usage same as with libluajit-5.1.so.2.0.2.

For example nginx with lua-nginx-module:

memory usage before test:

$ ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
 2712 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
13340 nginx: worker process
 3360 nginx: cache manager process
 2948 nginx: cache loader process
  944 grep --color=auto nginx:
Total memory =  223404

memory usage after test (all requests finished and all connections to nginx closed):

$ ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
 2372 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
2033368 nginx: worker process
1993736 nginx: worker process
2010688 nginx: worker process
2072408 nginx: worker process
1899844 nginx: worker process
2036096 nginx: worker process
2081336 nginx: worker process
1829648 nginx: worker process
1992792 nginx: worker process
1705992 nginx: worker process
2074204 nginx: worker process
1805180 nginx: worker process
2003932 nginx: worker process
1902636 nginx: worker process
2081448 nginx: worker process
2048644 nginx: worker process
 3008 nginx: cache manager process
  804 grep --color=auto nginx:
Total memory =  31578136

nginx config:

worker_processes  16;
worker_rlimit_nofile 1000000;

events {
}

http {
    output_buffers            1 1m;

    log_format main '$remote_addr [$time_local] "$request" $status $bytes_sent'
                    ' "$request_time" "$upstream_cache_status" "$http_range"';

    proxy_cache_path       /var/cache/nginx/proxy_cache levels=2:2 keys_zone=proxy_slice:500m inactive=3d use_temp_path=off;

    server {
      listen      127.0.0.1:8080 default_server backlog=65535;
      access_log  /var/log/nginx/proxy_slice.log main;

      slice             1m;
      proxy_cache       proxy_slice;
      proxy_cache_key   "$request_uri|$slice_range";
      proxy_cache_valid 200 206 1d;
      proxy_cache_valid any 0;
      proxy_set_header  Range    $slice_range;
      proxy_ignore_client_abort on;
      proxy_http_version 1.1;

      location / {
        proxy_pass http://127.0.0.1:8088;
      }
    }

    server {
      listen 127.0.0.1:8088;

      access_log /var/log/nginx/storage.log main;
      rewrite ^.*$ /1.data last;

      location / {
        root /var/data;
      }
    }
}

How to reproduce:

$ mkdir /var/data; 
$ dd if=/dev/urandom of=/var/data/1.data bs=1M count=4500

Command for test:

$ while true; do curl -H "Range: bytes=$(echo $(od -An -N3 -i /dev/urandom)-$(( 100 * $(od -An -N3 -i /dev/urandom) )))" "http://127.0.0.1:8080/$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13)" > /dev/null ; done

[gilgameshfreedom]

Hello!

I've got the same problem - after enabling slice memory is not released. When I reassembled nginx without lua-module everything is OK.

[agentzh]

@Zaynullin @gilgameshfreedom Well, it is not nginx itself holding the freed memory, but glibc on Linux.

Yes, this is a known problem of glibc on Linux that holds too much freed memory when it fails to use brk() to allocate memory in the heap. The work-around is to call malloc_trim() periodically to release such free memory blocks back to the OS.

I've just implemented the lua_malloc_trim N config directive in the malloc-trim github branch of this repo, that calls malloc_trim() every N main requests processed by the nginx core. By default, N is 1000. You can tune the number to fit your use cases (for testing purposes, you can set N to 1). Mind you, N = 0 disables the periodical trimming altogether.

For more details, please see

https://github.com/openresty/lua-nginx-module/tree/malloc-trim

f0b45946d

Please try this branch and directive out on your side. Feedback welcome!

[agentzh]

Oh, I must thank @yangshuxin for the workaround and suggestion here :)

[Zaynullin]

Hi all.
@agentzh @yangshuxin thank for your work.

But I had the same problem with your fixes of the module.
I used the same NGINX and libluajit versions.
For reference, my OS version:

# uname -a
Linux host 3.19.0-21-generic #21~14.04.1-Ubuntu SMP Sun Jun 14 18:45:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.2 LTS
Release:    14.04
Codename:   trusty

I got same memory overhead also without sending range requests.
In my new test I counted memory usage by NGINX, requested big file, waited 20 seconds and counted memory usage again.
I used same nginx.conf and added lua_malloc_trim 1; to http section.

Results of my tests when I used NGINX without lua-nginx-module:

# nginx -V
nginx version: nginx/1.11.2
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014 (running with OpenSSL 1.0.2g-fips  1 Mar 2016)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-ef2b708510b1/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --add-module=ngx_devel_kit-0.3.0

# ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}' ; curl -v "http://127.0.0.1:8080/test_file" > /dev/null ; ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}' ; sleep 20; ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
  800 grep --color=auto nginx:
 2240 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 3148 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3144 nginx: worker process
 3148 nginx: worker process
 2960 nginx: worker process
 2972 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3144 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3148 nginx: worker process
 3316 nginx: cache manager process
Total memory =  56352
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /test_file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx/1.11.2 is not blacklisted
< Server: nginx/1.11.2
< Date: Wed, 05 Oct 2016 12:42:16 GMT
< Content-Type: text/plain
< Content-Length: 4718592000
< Connection: keep-alive
< Last-Modified: Tue, 04 Oct 2016 09:26:31 GMT
< ETag: "57f375c7-119400000"
< Accept-Ranges: bytes
<
{ [data not shown]
100 4500M  100 4500M    0     0   313M      0  0:00:14  0:00:14 --:--:--  273M
* Connection #0 to host 127.0.0.1 left intact
  836 grep --color=auto nginx:
 2240 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4912 nginx: worker process
 4916 nginx: worker process
 4756 nginx: worker process
 4756 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
4712704 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 3316 nginx: cache manager process
Total memory =  4792512
  780 grep --color=auto nginx:
 2240 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4912 nginx: worker process
 4916 nginx: worker process
 4756 nginx: worker process
 4756 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 9328 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 4916 nginx: worker process
 3316 nginx: cache manager process
Total memory =  89080

Results of my tests when I used NGINX with lua-nginx-module from malloc-trim branch:

# nginx -V
nginx version: nginx/1.11.2
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014 (running with OpenSSL 1.0.2g-fips  1 Mar 2016)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --add-dynamic-module=debian/extra/njs-ef2b708510b1/nginx --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro' --add-module=ngx_devel_kit-0.3.0 --add-module=lua-nginx-module-0.10.6

# ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}' ; curl -v "http://127.0.0.1:8080/test_file" > /dev/null ; ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}' ; sleep 20; ps -e -orss=,args= | grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
 2440 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 3480 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3316 nginx: worker process
 3480 nginx: worker process
 3264 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3448 nginx: worker process
 3404 nginx: worker process
 3480 nginx: worker process
 3480 nginx: worker process
 3480 nginx: cache manager process
 3068 nginx: cache loader process
  796 grep --color=auto nginx:
Total memory =  64976
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /test_file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx/1.11.2 is not blacklisted
< Server: nginx/1.11.2
< Date: Wed, 05 Oct 2016 12:55:51 GMT
< Content-Type: text/plain
< Content-Length: 4718592000
< Connection: keep-alive
< Last-Modified: Tue, 04 Oct 2016 09:26:31 GMT
< ETag: "57f375c7-119400000"
< Accept-Ranges: bytes
<
{ [data not shown]
100 4500M  100 4500M    0     0   310M      0  0:00:14  0:00:14 --:--:--  295M
* Connection #0 to host 127.0.0.1 left intact
 2440 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 5356 nginx: worker process
 5352 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 5300 nginx: worker process
 5356 nginx: worker process
 5228 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
4710652 nginx: worker process
 5380 nginx: worker process
 5344 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 3480 nginx: cache manager process
 3068 nginx: cache loader process
  804 grep --color=auto nginx:
Total memory =  4800608
 2440 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
 5356 nginx: worker process
 5352 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 5300 nginx: worker process
 5356 nginx: worker process
 5228 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
4710652 nginx: worker process
 5380 nginx: worker process
 5344 nginx: worker process
 5356 nginx: worker process
 5356 nginx: worker process
 3480 nginx: cache manager process
 3068 nginx: cache loader process
  828 grep --color=auto nginx:
Total memory =  4800632

[yangshuxin]

Hi,

 As I figured out last night, the tip of branch has bug when Nginx 

is built without
debugging, and it's not yet fixed as of I write this email. Did you fix
the problem
by yourself?

 If no, you can just move the function call to malloc_trim(1) out of 

the macro,

i.e. from:

ngx_log_debug1(...., malloc_trim(1));

to:
rc = malloc_trim(1); // so it is still called if NGX_DEBUG is false.
ngx_log_debug1(....; rc);

Let me know if your problem still persist. Thanks!

Shuxin

On 10/5/16 6:53 AM, Zaynullin wrote:

Hi all.
@agentzh https://github.com/agentzh @yangshuxin
https://github.com/yangshuxin thank for your work.

But I had the same problem with your fixes of the module.
I used the same NGINX and libluajit versions.
For reference, my OS version:

|# uname -a Linux host 3.19.0-21-generic #21~14.04.1-Ubuntu SMP Sun Jun
14 18:45:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux # lsb_release -a
No LSB modules are available. Distributor ID: Ubuntu Description:
Ubuntu 14.04.2 LTS Release: 14.04 Codename: trusty |

I got same memory overhead also without sending range requests.
In my new test I counted memory usage by NGINX, requested big file,
waited 20 seconds and counted memory usage again.
I used same |nginx.conf| and added |lua_malloc_trim 1;| to |http|
section.

Results of my tests when I used NGINX without |lua-nginx-module|:

|# nginx -V nginx version: nginx/1.11.2 built by gcc 4.8.4 (Ubuntu
4.8.4-2ubuntu1~14.04.3) built with OpenSSL 1.0.1f 6 Jan 2014 (running
with OpenSSL 1.0.2g-fips 1 Mar 2016) TLS SNI support enabled configure
arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib/nginx/modules
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log
--pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_random_index_module --with-http_secure_link_module
--with-http_stub_status_module --with-http_auth_request_module
--with-http_xslt_module=dynamic
--with-http_image_filter_module=dynamic
--with-http_geoip_module=dynamic --with-http_perl_module=dynamic
--add-dynamic-module=debian/extra/njs-ef2b708510b1/nginx
--with-threads --with-stream --with-stream_ssl_module
--with-http_slice_module --with-mail --with-mail_ssl_module
--with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g
-O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security' --with-ld-opt='-Wl,-Bsymbolic-functions
-Wl,-z,relro' --add-module=ngx_devel_kit-0.3.0 # ps -e -orss=,args= |
grep nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total
memory = ", sum;}' ; curl -v "http://127.0.0.1:8080/test_file" >
/dev/null ; ps -e -orss=,args= | grep nginx: | awk
'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
; sleep 20; ps -e -orss=,args= | grep nginx: | awk
'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
800 grep --color=auto nginx: 2240 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf 3148 nginx: worker process
3148 nginx: worker process 3148 nginx: worker process 3148 nginx:
worker process 3148 nginx: worker process 3144 nginx: worker process
3148 nginx: worker process 2960 nginx: worker process 2972 nginx:
worker process 3148 nginx: worker process 3148 nginx: worker process
3148 nginx: worker process 3144 nginx: worker process 3148 nginx:
worker process 3148 nginx: worker process 3148 nginx: worker process
3316 nginx: cache manager process Total memory = 56352 * Hostname was
NOT found in DNS cache * Trying 127.0.0.1... % Total % Received %
Xferd Average Speed Time Time Time Current Dload Upload Total Spent
Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to
127.0.0.1 (127.0.0.1) port 8080 (#0) > GET /test_file HTTP/1.1 >
User-Agent: curl/7.35.0 > Host: 127.0.0.1:8080 > Accept: / > <
HTTP/1.1 200 OK * Server nginx/1.11.2 is not blacklisted < Server:
nginx/1.11.2 < Date: Wed, 05 Oct 2016 12:42:16 GMT < Content-Type:
text/plain < Content-Length: 4718592000 < Connection: keep-alive <
Last-Modified: Tue, 04 Oct 2016 09:26:31 GMT < ETag:
"57f375c7-119400000" < Accept-Ranges: bytes < { [data not shown] 100
4500M 100 4500M 0 0 313M 0 0:00:14 0:00:14 --:--:-- 273M * Connection
#0 to host 127.0.0.1 left intact 836 grep --color=auto nginx: 2240
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf 4916
nginx: worker process 4916 nginx: worker process 4916 nginx: worker
process 4916 nginx: worker process 4916 nginx: worker process 4912
nginx: worker process 4916 nginx: worker process 4756 nginx: worker
process 4756 nginx: worker process 4916 nginx: worker process 4916
nginx: worker process 4916 nginx: worker process 4712704 nginx: worker
process 4916 nginx: worker process 4916 nginx: worker process 4916
nginx: worker process 3316 nginx: cache manager process Total memory =
4792512 780 grep --color=auto nginx: 2240 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf 4916 nginx: worker process
4916 nginx: worker process 4916 nginx: worker process 4916 nginx:
worker process 4916 nginx: worker process 4912 nginx: worker process
4916 nginx: worker process 4756 nginx: worker process 4756 nginx:
worker process 4916 nginx: worker process 4916 nginx: worker process
4916 nginx: worker process 9328 nginx: worker process 4916 nginx:
worker process 4916 nginx: worker process 4916 nginx: worker process
3316 nginx: cache manager process Total memory = 89080 |

Results of my tests when I used NGINX with |lua-nginx-module| from
|malloc-trim| branch:

|# nginx -V nginx version: nginx/1.11.2 built by gcc 4.8.4 (Ubuntu
4.8.4-2ubuntu1~14.04.3) built with OpenSSL 1.0.1f 6 Jan 2014 (running
with OpenSSL 1.0.2g-fips 1 Mar 2016) TLS SNI support enabled configure
arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib/nginx/modules
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log
--pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_random_index_module --with-http_secure_link_module
--with-http_stub_status_module --with-http_auth_request_module
--with-http_xslt_module=dynamic
--with-http_image_filter_module=dynamic
--with-http_geoip_module=dynamic --with-http_perl_module=dynamic
--add-dynamic-module=debian/extra/njs-ef2b708510b1/nginx
--with-threads --with-stream --with-stream_ssl_module
--with-http_slice_module --with-mail --with-mail_ssl_module
--with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g
-O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security' --with-ld-opt='-Wl,-Bsymbolic-functions
-Wl,-z,relro' --add-module=ngx_devel_kit-0.3.0
--add-module=lua-nginx-module-0.10.6 # ps -e -orss=,args= | grep
nginx: | awk 'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total
memory = ", sum;}' ; curl -v "http://127.0.0.1:8080/test_file" >
/dev/null ; ps -e -orss=,args= | grep nginx: | awk
'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
; sleep 20; ps -e -orss=,args= | grep nginx: | awk
'BEGIN{sum=0;}{sum+=$1; print $0; }END{print "Total memory = ", sum;}'
2440 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
3480 nginx: worker process 3480 nginx: worker process 3480 nginx:
worker process 3480 nginx: worker process 3316 nginx: worker process
3480 nginx: worker process 3264 nginx: worker process 3480 nginx:
worker process 3480 nginx: worker process 3480 nginx: worker process
3480 nginx: worker process 3480 nginx: worker process 3448 nginx:
worker process 3404 nginx: worker process 3480 nginx: worker process
3480 nginx: worker process 3480 nginx: cache manager process 3068
nginx: cache loader process 796 grep --color=auto nginx: Total memory
= 64976 * Hostname was NOT found in DNS cache * Trying 127.0.0.1... %
Total % Received % Xferd Average Speed Time Time Time Current Dload
Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > GET
/test_file HTTP/1.1 > User-Agent: curl/7.35.0 > Host: 127.0.0.1:8080 >
Accept: / > < HTTP/1.1 200 OK * Server nginx/1.11.2 is not
blacklisted < Server: nginx/1.11.2 < Date: Wed, 05 Oct 2016 12:55:51
GMT < Content-Type: text/plain < Content-Length: 4718592000 <
Connection: keep-alive < Last-Modified: Tue, 04 Oct 2016 09:26:31 GMT
< ETag: "57f375c7-119400000" < Accept-Ranges: bytes < { [data not
shown] 100 4500M 100 4500M 0 0 310M 0 0:00:14 0:00:14 --:--:-- 295M *
Connection #0 to host 127.0.0.1 left intact 2440 nginx: master process
/usr/sbin/nginx -c /etc/nginx/nginx.conf 5356 nginx: worker process
5352 nginx: worker process 5356 nginx: worker process 5356 nginx:
worker process 5300 nginx: worker process 5356 nginx: worker process
5228 nginx: worker process 5356 nginx: worker process 5356 nginx:
worker process 5356 nginx: worker process 5356 nginx: worker process
4710652 nginx: worker process 5380 nginx: worker process 5344 nginx:
worker process 5356 nginx: worker process 5356 nginx: worker process
3480 nginx: cache manager process 3068 nginx: cache loader process 804
grep --color=auto nginx: Total memory = 4800608 2440 nginx: master
process /usr/sbin/nginx -c /etc/nginx/nginx.conf 5356 nginx: worker
process 5352 nginx: worker process 5356 nginx: worker process 5356
nginx: worker process 5300 nginx: worker process 5356 nginx: worker
process 5228 nginx: worker process 5356 nginx: worker process 5356
nginx: worker process 5356 nginx: worker process 5356 nginx: worker
process 4710652 nginx: worker process 5380 nginx: worker process 5344
nginx: worker process 5356 nginx: worker process 5356 nginx: worker
process 3480 nginx: cache manager process 3068 nginx: cache loader
process 828 grep --color=auto nginx: Total memory = 4800632 |

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#872 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGie-uKeh81whwcmjQjTgZVjwe9FBNAfks5qw6vMgaJpZM4KE3fN.

[agentzh]

@Zaynullin Yes, as @yangshuxin pointed out, lua_malloc_trim took no effect when --with-debug was off. I've just fixed the bug in the malloc-trim branch as commit bc0d84e. Please try upgrading to the latest malloc-trim branch.

BTW, since you are testing this with artificial load, please configure lua_malloc_trim 1; in your nginx.conf so that you can see it immediately after your load is gone (otherwise you'll need a bit less than 1000 "small" requests to trigger the trimming by default). But it comes with higher overhead. Read the docs in that branch to ensure that you understand what this directive does.

[Zaynullin]

Hi all.
@agentzh @yangshuxin thank again.

I had tested your fixes.
It works well when I request different URLs.
But when I request same URL a few times, memory have been eaten by NGINX.
For example, I executed tests (I changed output for short):

user@host:~$ function memory_calc { ps -e -orss=,args= | grep nginx: | awk 'BEGIN { sum=0; } { sum+=$1; } END { print "Total nginx memory = ", sum; }'; } ; echo -n "Before tests: "; memory_calc; for i in $(seq 1 10); do echo -n "Test #$i: "; curl -v "http://127.0.0.1:8080/test_file" 2 &> /dev/null; memory_calc; done;
Before tests: Total nginx memory =  63864
Test #1: Total nginx memory =  4790144
Test #2: Total nginx memory =  9464388
Test #3: Total nginx memory =  14138596
Test #4: Total nginx memory =  18812796
Test #5: Total nginx memory =  23487060
Test #6: Total nginx memory =  28161212
Test #7: Total nginx memory =  32834292
Test #8: Total nginx memory =  37497148
Test #9: Total nginx memory =  37494272
Test #10: Total nginx memory =  37431872

Part of long form of output (you can see that different worker processes used free memory on different iterations of loop without cleaning it):

Test #3:  
2496 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
4679128 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
4679188 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 3488 nginx: cache manager process
 3456 nginx: cache loader process
  824 grep --color=auto nginx:
Total nginx memory =  9437180
Test #4:  
2496 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
4679128 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
4679188 nginx: worker process
 4900 nginx: worker process
4679128 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 3488 nginx: cache manager process
  768 grep --color=auto nginx:
Total nginx memory =  14107896
Test #5:  
2496 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
4679128 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
4679172 nginx: worker process
4679188 nginx: worker process
 4900 nginx: worker process
4679128 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 4900 nginx: worker process
 3488 nginx: cache manager process
  776 grep --color=auto nginx:
Total nginx memory =  18782176

I used same versions of my software and set lua_malloc_trim 1; in nginx.conf.

[agentzh]

@Zaynullin Some comments on your results:

1. It's completely normal for a single request not to return all the freed memory to the OS. This is because malloc_trim is called in request's log phase handler, which is before the nginx memory pool deallocation, so in the log phase, the request memory pool of the current request is not freed yet. But for the next request served by the same process, as long as the next request consumes less memory than the previous one, in the next request's log phase handler, malloc_trim() will return (most of) the memory difference back to the OS. And, if your next request is gonna use similar sized memory, then not returning the freed memory immediately to the OS has the advantage that your next request can allocate much faster due to less or zero system calls for its new allocations (this is all done by your glibc BTW).
2. It seems to me that it's just that different nginx worker processes handling your requests, giving the illusion that the memory keeps growing. When all your worker processes are hit, the memory will stop growing. I suggest you use a single nginx worker for such artificial testing since the multi-worker setting does not matter here and it just complicates your testing and even judgement.
3. We're more interested in memory characteristics with real traffic (or fake traffic emulating the real use cases). Memory management is complicated and should not optimize for an individual request. We always have to balancer between memory footprint and CPU cycles here. Having said that, we can do very little here for returning memory to the OS and what you're seeing are all your glibc behaviors anyway.

[Zaynullin]

Hi, @agentzh. Sorry for my late answer.
I have tested nginx build with lua_malloc_trim option for real users requests.
I had set lua_malloc_trim to default value.
Nginx have used big volume of memory and memory volume continued to grow.

Tell me, please, how much memory released when malloc_trim function called? Only memory used for last HTTP request or memory used for all requests from last malloc_trim function call?

[agentzh]

@Zaynullin Please read my previous comment more carefully. Your last comment suggests that either you haven't read my previous comment or you just haven't understood it. Sorry, I don't think I have a better way to explain things to you here.

[Zaynullin]

Hi, @agentzh.
I asked it because I detected extremely memory usage when used nginx build with new lua module version including lua_malloc_trim directive.
On simple tests lua_malloc_trim looks good, but when I tried to use new module version with real users requests, memory used by nginx has grown a lot.
The main feature of my configuration is using nginx slice module for proxying requests to origin server and saving answers to cache (you can find exemplary configuration above).
For example, when I turned on slice configuration with lua module, memory usage grew from 10 Gb to 30 Gb.
Do you have some ideas for debug memory usage?
Can I turn on extended logging for lua module?

[agentzh]

@Zaynullin How do you configure lua_malloc_trim in your nginx.conf? You can try smaller values there and also 0, to see if there's any difference.

[agentzh]

@Zaynullin It has nothing to do with the lua module itself, it is only related to how your glibc handles memory when it fails to move the program break to allocate the memory.

[agentzh]

@Zaynullin BTW, what exact memory metrics are you looking at? Maybe you are just looking at the wrong numbers. The correct way is to only look at the resident set size (RSS) of your nginx worker processes.