pcre

[Vittly]

Hi again! As I said earlier I am using pcre 8.2 with 'j' and 'u' flags in ngx.match. From time to time I got 'pcre_compile() failed to get memory...' is it bug of lua_nginx or nginx or pcre? What should I do? Do not use pcre?

[agentzh]

On Sun, Nov 27, 2011 at 12:43 AM, Vittly
reply@reply.github.com
wrote:

Hi again! As I said earlier I am using pcre 8.2 with 'j' and 'u' flags in ngx.match. From time to time I got 'pcre_compile() failed to get memory...' is it bug of lua_nginx or nginx or pcre? What should I do? Do not use pcre?

Could you please give me a minimized sample that can reproduce this
problem? It seems like a bug in ngx_lua.

Thanks!
-agentzh

[Vittly]

Seems that it is a memory leak because application could work properly all day and in the end of day I see that all regexs crashes. I think it is important that all regexs I use contains captures '(...)'. To reproduce the error try thomething like
location / {
ngx.re.match(ngx.req.get_headers()['User-Agent'],'(safari/(\d+).(\d+)'))
}
and call it in script 1000 times or more. I used few pcre versions and different match flags and all the time error was the same.

[agentzh]

On Sun, Nov 27, 2011 at 1:26 PM, Vittly
reply@reply.github.com
wrote:

Seems that it is a memory leak because application could work properly all day and in the end of day I see that all regexs crashes. I think it is important that all regexs I use contains captures '(...)'. To reproduce the error try thomething like
location / {
 ngx.re.match(ngx.req.get_headers()['User-Agent'],'(safari/(\d+).(\d+)'))
}
and call it in script 1000 times or more. I used few pcre versions and different match flags and all the time error was the same.

Thanks for the report! But I still need further information:

1. The output of the "nginx -V" command
2. The output of the "uname -a" command

Also, could you try adding the "o" regex option to your regex? As in

location / {

  content_by_lua "
ngx.re.match(ngx.req.get_headers()['User-Agent'],'(safari/(\d+).(\d+))',
'o')
";
}

Thanks!
-agentzh

[Vittly]

Ok.
nginx -V
nginx: nginx version: nginx/1.0.10
nginx: built by gcc 4.4.5 (Debian 4.4.5-8)
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/opt/nginx --add-module=../ngx_devel_kit --add-module=../chaoslawful --with-pcre=../pcre-8.2 --with-http_ssl_module

uname -a
Linux ip---- 2.6.32-5-xen-amd64 #1 SMP Mon Oct 3 07:53:54 UTC 2011 x86_64 GNU/Linux

would 'o' option work with 'j'? Or I need to replace 'j' by 'o' for a while?

[agentzh]

On Sun, Nov 27, 2011 at 1:49 PM, Vittly
reply@reply.github.com
wrote:

Ok.
nginx -V
nginx: nginx version: nginx/1.0.10
nginx: built by gcc 4.4.5 (Debian 4.4.5-8)
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/opt/nginx --add-module=../ngx_devel_kit --add-module=../chaoslawful --with-pcre=../pcre-8.2 --with-http_ssl_module

"pcre 8.2" is actually referring to pcre 8.20 here, right?

Which version of ngx_lua are you using? v0.3.1rc34?

uname -a
Linux ip---- 2.6.32-5-xen-amd64 #1 SMP Mon Oct 3 07:53:54 UTC 2011 x86_64 GNU/Linux

would 'o' option work with 'j'? Or I need to replace 'j' by 'o' for a while?

The "j" option should always be used with "o", or the regex will be
compiled and JIT'd at every run and the benefit of JITing is going
away. Maybe I should have made this clear in the docs of ngx_lua :)

Thanks!
-agentzh

[Vittly]

"pcre 8.2" is actually referring to pcre 8.20 here, right? Yes!
Which version of ngx_lua are you using? v0.3.1rc34? Yes!

ok I ll add 'o' to 'j'

[agentzh]

On Sun, Nov 27, 2011 at 1:26 PM, Vittly
reply@reply.github.com
wrote:

Seems that it is a memory leak because application could work properly all day and in the end of day I see that all regexs crashes. I think it is important that all regexs I use contains captures '(...)'. To reproduce the error try thomething like
location / {
 ngx.re.match(ngx.req.get_headers()['User-Agent'],'(safari/(\d+).(\d+)'))
}
and call it in script 1000 times or more. I used few pcre versions and different match flags and all the time error was the same.

I've tried a slightly modified version of your example which is buggy
in some minor places:

location /re {
    content_by_lua "
        ngx.req.set_header('User-Agent', 'safari')
        ngx.re.match(ngx.req.get_headers()['User-Agent'],[[(safari/(\\d+)\\.(\\d+))]])
        ngx.say('done')
    ";
}

with ngx_lua v0.3.1rc36 and nginx 1.0.10 and pcre 8.20 (with and
without --enable-jit for my pcre build) on Linux x86_64, similar to
yours. When hitting this location with ab for tens of thousands of
requests, there's no error messages at all in my error.log and no
memory leak observed in my "top" utility.

Could you please try out my example as is on your side?

It'll be easier for me to fix this issue if you can help me reproduce
it on my side ;)

BTW, pcre jit is disabled by default in pcre 8.20's build system. You
need to pass --enable-jit to it, maybe something like this for your
nginx ./configure command:

./configure --with-pcre=/path/to/your/pcre/source/tree

--with-pcre-opt="--enable-jit" ...

You should pass --enable-utf8 too if you want to enable the UTF-8
support in your PCRE build at the same time :)

Thanks!
-agentzh

[Vittly]

I know about this flags and it wasn't simple task. For example your method " ./configure --with-pcre=/path/to/your/pcre/source/tree
--with-pcre-opt="--enable-jit"" is not working becouse with-pcre-opt is list of options for make not configure. To put '--enable' right in nginx building process you should ./configure nginx and then modify objs/Makefile as shown at http://www.cslog.cn/Content/nginx-pcre-utf8-rewrite/ (there are utf8 and unicode, jit could be placed there too). Mey be you should write this technique in your documentation it helps people do not waste time.
Ok? I'll put your code in my application but how could it help you if error occurs? How would we locate it?

[Vittly]

You also could add to your documentation that r34 installs only if $CFLAGS = -Wno-error (in nginx makefile). Pleeeease:)

[agentzh]

On Sun, Nov 27, 2011 at 11:15 PM, Vittly
reply@reply.github.com
wrote:

You also could add to your documentation that r34 installs only id $CFLAGS = -Wno-error (in nginx makefile). Pleeeease:)

Can you try the latest rc36 release? I think I've fixed this issue in
rc34 already.

-agentzh

[agentzh]

On Sun, Nov 27, 2011 at 11:01 PM, Vittly
reply@reply.github.com
wrote:

I know about this flags and it wasn't simple task. For example your method " ./configure --with-pcre=/path/to/your/pcre/source/tree
--with-pcre-opt="--enable-jit"" is not working becouse with-pcre-opt is list of options for make not configure. To put '--enable' right in nginx building process you should ./configure nginx and then modify objs/Makefile as shown at http://www.cslog.cn/Content/nginx-pcre-utf8-rewrite/ (there are utf8 and unicode, jit could be placed there too). Mey be you should write this technique in your documentation it helps people do not waste time.

This is unfortunate. Then I'd recommend building pcre separately and
make nginx use it via the --with-cc-opt and --with-ld-opt configure
options. Here's an example:

./configure \
     --with-cc-opt=$'-I/opt/pcre820jit/include' \
     --with-ld-opt="-L/opt/pcre820jit/lib -Wl,-rpath,/opt/pcre820jit/lib \
     ...

assuming that your pcre is installed into /opt/pcre820jit as I am ;)
Here's the command I used to build and install pcre:

./configure --prefix=/opt/pcre820jit --enable-jit --enable-utf8
make -j2
sudo make install

Ok? I'll put your code in my application but how could it help you if error occurs? How would we locate it?

This is just a sample meant to reproduce the issue that you have. It's
not meant to be integrated into your application. I hope that you can
use a separate nginx.conf to include this location definition and run
it to try to reproduce the issue on your side.

Best,
-agentzh

[Vittly]

I have done 1 000 000 requests for ngx.re.match(ngx.req.get_headers()['User-Agent'],...) but it does not go down. Then I ve told my friends (which cause the errors before) to request this "regex" location but during the day there was not any errors at all. Main application logic now use 'jo' flags and similary do not crash.

[agentzh]

On Mon, Nov 28, 2011 at 12:15 PM, Vittly
reply@reply.github.com
wrote:

I have done 1 000 000 requests for ngx.re.match(ngx.req.get_headers()['User-Agent'],...) but it does not go down. Then I ve told my friends (which cause the errors before) to request this "regex" location but during the day there was not any errors at all. Main application logic now use 'jo' flags and similary do not crash.

Good to know :) But I still want to fix the case that "o" is not
specified. Your earlier standalone sample dos not lead to any errors
on my side. Can you paste the exact nginx.conf that you can produce
this issue. Your earlier snippet does have syntax errors:

location / {
    ngx.re.match(ngx.req.get_headers()['User-Agent'],'(safari/(\\d+)\\.(\\d+)'))
}

You definitely lost content_by_lua directive and the parentheses did
not match at all.

Please send me the exact config snippet that you actually run and that
can reproduce the "failed to allocate memory" error on your side.

Thanks!
-agentzh

[eoranged]

I've faced the issue during playing with shared dictionaries. I have only few regex'es (something about 3 or 4) in my code, all with and only with "o" option.
The error is happend just once and I've failed reproducing It, so I've decided that it was my fault.

[Vittly]

Yes, I also could not reproduce the error:(