nginx 1.6/modsecurity 2.8 http request hangs

[ebaron-instart]

Content update by @zimmerle on 2014-May-23, as it may contains sensitive information. The reported bug was not verified, it is under analysis. I will place it back the full report as soon as we verify.

@ebaron-instart whenever you think that something may be a security issue contact the developers directly, otherwise you will publish recipe for attackers. Thanks for the report.

[zimmerle]

Hi @ebaron-instart I am trying to reproduce the issue, without success. I am afraid i will need more details. Can you please tell me the OWASP CRS version that you are using, and if you did any customization on the rules? What is your modsecurity.conf? can you give me that details via e-mail? (security at modsecurity.org)? thanks.

[ebaron-instart]

Hi Felipe,

The rules file version is OWASP_CRS/2.2.9. The only customization I have
done is setting "SecRuleEngine On". Both nginx and modsecurity module were
built and run on Ubuntu 12.04 with gcc 4.6.3, if that helps.

Thanks,
Ella Baron

On Fri, May 23, 2014 at 1:43 PM, Felipe Zimmerle
notifications@github.comwrote:

Hi @ebaron-instart https://github.com/ebaron-instart I am trying to
reproduce the issue, without success. I am afraid i will need more details.
Can you please tell me the OWASP CRS version that you are using, and if you
did any customization on the rules? What is your modsecurity.conf? can you
give me that details via e-mail? (security at modsecurity.org)? thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44058095
.

[ebaron-instart]

Ok, I've just experimented with a very simple config setup. I used
modsecurity.conf-recommended from 2.8 and added the following rule to it:

SecRule REQUEST_HEADERS:User-Agent "WebZIP2" "id:'8976',phase:1,deny"

Of course with (SecRuleEngine On). The following command worked:

curl --user-agent "WebZIP2" -xlocalhost:8080 -v http://localhost/
http://localhost/ --data "test"

Then I changed phase to 2 and the same command made it to hang.

I hope this helps.

Ella Baron

On Fri, May 23, 2014 at 2:38 PM, Ella Baron ebaron@instartlogic.com wrote:

Hi Felipe,

The rules file version is OWASP_CRS/2.2.9. The only customization I have
done is setting "SecRuleEngine On". Both nginx and modsecurity module were
built and run on Ubuntu 12.04 with gcc 4.6.3, if that helps.

Thanks,
Ella Baron

On Fri, May 23, 2014 at 1:43 PM, Felipe Zimmerle <notifications@github.com

wrote:

Hi @ebaron-instart https://github.com/ebaron-instart I am trying to
reproduce the issue, without success. I am afraid i will need more details.
Can you please tell me the OWASP CRS version that you are using, and if you
did any customization on the rules? What is your modsecurity.conf? can you
give me that details via e-mail? (security at modsecurity.org)? thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44058095
.

[zimmerle]

Hi Ella,

Can you share the very first messages that ModSecurity is showing in your error log? It should be something like this:

[...] ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/) configured.
[...] ModSecurity: APR compiled version="1.5.1-dev"; loaded version="1.5.1-dev"
[...] ModSecurity: PCRE compiled version="8.31 "; loaded version="8.31 2012-07-06"
[...] ModSecurity: LUA compiled version="Lua 5.1"
[...] 26917#0: ModSecurity: LIBXML compiled version="2.9.1"
[...] 26917#0: Status engine is currently disabled, enable it by set SecStatusEngine to On.

There is any kind of segfault in your nginx process? Any information that may help us on your nginx logs? ModSecuirty debug log was enabled? If so, can you share it?

[ebaron-instart]

Here is what I see:

2014/05/23 15:02:22 [notice] 30197#0: ModSecurity for nginx (STABLE)/2.8.0 (
http://www.modsecurity.org/) configured.
2014/05/23 15:02:22 [notice] 30197#0: ModSecurity: APR compiled
version="1.4.6"; loaded version="1.4.6"
2014/05/23 15:02:22 [notice] 30197#0: ModSecurity: PCRE compiled
version="8.12 "; loaded version="8.12 2011-01-15"
2014/05/23 15:02:22 [notice] 30197#0: ModSecurity: LIBXML compiled
version="2.7.8"
2014/05/23 15:02:22 [notice] 30197#0: ModSecurity: StatusEngine call:
"2.8.0,ModSecurity Standalone,1.4.6/1.4.6,8.12/8.12
2011-01-15,(null),2.7.8,f653ca44b4fa6f8c05a3227ccfa6ad6ec49936e4"
2014/05/23 15:02:22 [notice] 30197#0: ModSecurity: StatusEngine call
successfully sent. For more information visit:
http://status.modsecurity.org/

Ella

On Fri, May 23, 2014 at 2:58 PM, Felipe Zimmerle
notifications@github.comwrote:

Hi Ella,

Can you share the very first messages that ModSecurity is showing in your
error log? It should be something like this:

[...] ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/)
configured.
[...] ModSecurity: APR compiled version="1.5.1-dev"; loaded
version="1.5.1-dev"
[...] ModSecurity: PCRE compiled version="8.31 "; loaded version="8.31
2012-07-06"
[...] ModSecurity: LUA compiled version="Lua 5.1"
[...] 26917#0: ModSecurity: LIBXML compiled version="2.9.1"
[...] 26917#0: Status engine is currently disabled, enable it by set
SecStatusEngine to On.

There is any kind of segfault in your nginx process? Any information that
may help us on your nginx logs? ModSecuirty debug log was enabled? If so,
can you share it?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44065038
.

[zimmerle]

There is no segfault? Nothing on the logs? There are other nginx modules working together with ModSec and nginx? If you disbale ModSecurity or set SecRuleEngine to "Off" or "DetectionOnly" it works normally?

[ebaron-instart]

I did not check the logs in detail, but when you do a single request, it
comes back, but connection is not finalized till 5s later. When you do two
requests (keep-alive) on, everything gets stuck after the first request.
Ngunx stays up and running, but consumes more and more connections as you
do more of those requests. It only happens with engine on.

Were you able to reproduce it?

Ella

On Friday, May 23, 2014, Felipe Zimmerle notifications@github.com wrote:

There is no segfault? Nothing on the logs? There are other nginx modules
working together with ModSec and nginx? If you disbale ModSecurity or set
SecRuleEngine to "Off" or "DetectionOnly" it works normally?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44065953
.

[zimmerle]

Tried the exactly configurations that you posted here, and apparently it is working as expected.

[ebaron-instart]

What OS and gcc compiler are you using?

Ella

On Friday, May 23, 2014, Felipe Zimmerle notifications@github.com wrote:

Tried the exactly configurations that you posted here, and apparently it
is working as expected.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44068420
.

[ebaron-instart]

Felipe,

I've also reproduced it on Ubuntu 14.04 with gcc compiler 4.8.2. Here is
the sequence of steps I went through:

1. Build modsecurity-2.8.0:

./configure --enable-standalone-module --disable-mlogc
make

1. Build nginx-1.6.0:

./configure --add-module=../modsecurity-2.8.0/nginx/modsecurity
--prefix=/home/ebaron/nginx/nginx-1.6.0-install
make
make install

1. Configure nginx:

cd /home/ebaron/nginx/nginx-1.6.0-install
vim conf/nginx.conf
change listening port (so that I do not have to run it as root)
listen 8080;
change root location to
location / {
root html;
index index.html index.htm;
ModSecurityEnabled on;
ModSecurityConfig modsecurity.conf;
}

cp ../modsecurity-2.8.0/modsecurity.conf-recommended conf/modsecurity.conf
vim conf/modsecurity.conf
turn the engine on
SecRuleEngine On
set correct path for audit log
SecAuditLog /home/ebaron/nginx/nginx-1.6.0-install/logs/modsec_audit.log
comment out the following line (unicode.mapping is not found)
SecUnicodeMapFile unicode.mapping 20127
at the end of the file add the following rule:
SecRule REQUEST_HEADERS:User-Agent "WebZIP2" "id:'8976',phase:2,deny"

1. Run nginx:

cd /home/ebaron/nginx/nginx-1.6.0-install
sbin/nginx

1. Run curl request:

curl --user-agent "WebZIP2" -xlocalhost:8080 -v http://localhost/
http://localhost/ --data "test"

1. Result:

Request hangs.

Output in /home/ebaron/nginx/nginx-1.6.0-install/logs/error.log:

2014/05/23 21:40:29 [notice] 25253#0: ModSecurity for nginx (STABLE)/2.8.0 (
http://www.modsecurity.org/) configured.
2014/05/23 21:40:29 [notice] 25253#0: ModSecurity: APR compiled
version="1.5.1-dev"; loaded version="1.5.1-dev"
2014/05/23 21:40:29 [notice] 25253#0: ModSecurity: PCRE compiled
version="8.31 "; loaded version="8.31 2012-07-06"
2014/05/23 21:40:29 [notice] 25253#0: ModSecurity: LIBXML compiled
version="2.9.1"
2014/05/23 21:40:29 [notice] 25253#0: ModSecurity: StatusEngine call:
"2.8.0,ModSecurity Standalone,1.5.1-dev/1.5.1-dev,8.31/8.31
2012-07-06,(null),2.9.1,cda0aa1d938d8285e7c40500c983cad722ebed07"
2014/05/23 21:40:29 [notice] 25253#0: ModSecurity: StatusEngine call
successfully sent. For more information visit:
http://status.modsecurity.org/
2014/05/23 21:40:29 [notice] 25253#0: signal process started
2014/05/23 21:40:34 [error] 25254#0: [client 127.0.0.1] ModSecurity: Access
denied with code 403 (phase 2). Pattern match "WebZIP2" at
REQUEST_HEADERS:User-Agent. [file
"/home/ebaron/nginx/nginx-1.6.0-install/conf/modsecurity.conf"] [line
"230"] [id "8976"] [hostname "minnesund"] [uri "/"] [unique_id
"qcAtAVfcAcxcAcAVAcrcAcic"]

Output in /home/ebaron/nginx/nginx-1.6.0-install/logs/modsec_audit.log: None

This is as far as I can go in terms of providing you with step by step
instructions to reproduce this problem. So far, I've reproduced in two
different Ubuntu environments (12.04 and 14.04). So, I believe this has
nothing to do with the environment, but is related to some strange
interaction of modsecurity with nginx. Also, "phase:2" in the rule is the
critical component here because using "phase:1" does not trigger this bug.

I hope this helps.

Thanks,
Ella Baron

On Fri, May 23, 2014 at 4:14 PM, Ella Baron ebaron@instartlogic.com wrote:

What OS and gcc compiler are you using?

Ella

On Friday, May 23, 2014, Felipe Zimmerle notifications@github.com wrote:

Tried the exactly configurations that you posted here, and apparently it
is working as expected.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44068420
.

[zimmerle]

Hi @ebaron-instart, your nginx is working like a proxy, right? did you tried to have it "proxying" another server? I am trying to understand what is going on...

Just in case, do you mind to test this version:
https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

[ebaron-instart]

Hi Felipe,

I've tried with the your refactored version of modsecurity and it still has
the same problem.

Thanks,
Ella

On Sat, May 24, 2014 at 4:35 AM, Felipe Zimmerle
notifications@github.comwrote:

Hi @ebaron-instart https://github.com/ebaron-instart, your nginx is
working like a proxy, right? did you tried to have it "proxying" another
server? I am trying to understand what is going on...

Just in case, do you mind to test this version:
https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/724#issuecomment-44084702
.

[ebaron-instart]

One more thing. When I send only a single request, it times out after 5 seconds and connection gets terminated. If I send two requests on the same connection (with keep-alive on), the connection gets stuck indefinitely.

[oddjobz]

Ok, I'm having a similar problem with 1.6/2.8, but I can supply some more detail. Firstly, the main cluprit seems to be POST, whereas GET seems Ok. (this is running with "no" rules, just the recommended modsecurity.conf)

POST produces the following once memory is exhausted;

kernel: senginx invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
kernel: senginx cpuset=/ mems_allowed=0
kernel: Pid: 7332, comm: senginx Not tainted 2.6.32-431.17.1.el6.x86_64 #1
kernel: Call Trace:
kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0
kernel: [] ? dump_header+0x90/0x1b0
kernel: [] ? security_real_capable_noaudit+0x3c/0x70
kernel: [] ? oom_kill_process+0x82/0x2a0
kernel: [] ? select_bad_process+0xe1/0x120
kernel: [] ? out_of_memory+0x220/0x3c0
kernel: [] ? __alloc_pages_nodemask+0x89f/0x8d0
kernel: [] ? alloc_pages_vma+0x9a/0x150
kernel: [] ? handle_pte_fault+0x73d/0xb00
kernel: [] ? alloc_pages_current+0xaa/0x110
kernel: [] ? pte_alloc_one+0x37/0x50
kernel: [] ? handle_mm_fault+0x22a/0x300
kernel: [] ? vma_adjust+0x556/0x5e0
kernel: [] ? __do_page_fault+0x138/0x480
kernel: [] ? vma_merge+0x29a/0x3e0
kernel: [] ? do_brk+0x26c/0x350
kernel: [] ? do_page_fault+0x3e/0xa0
kernel: [] ? page_fault+0x25/0x30
kernel: Mem-Info:
kernel: Node 0 DMA per-cpu:
kernel: CPU 0: hi: 0, btch: 1 usd: 0
kernel: Node 0 DMA32 per-cpu:
kernel: CPU 0: hi: 186, btch: 31 usd: 63
kernel: active_anon:52563 inactive_anon:52190 isolated_anon:0
kernel: active_file:66 inactive_file:65 isolated_file:0
kernel: unevictable:0 dirty:6 writeback:1 unstable:0
kernel: free:1204 slab_reclaimable:1865 slab_unreclaimable:13051
etc ...
kernel: Out of memory: Kill process 7332 (senginx) score 920 or sacrifice child
kernel: Killed process 7332, UID 99, (senginx) total-vm:1484228kB, anon-rss:408176kB, file-rss:48kB

[amarwah]

Hi

Any updates on this? To summarize:

• We see a 5 second delay (from http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_timeout) introduced whenever we detect an alert in engine on mode.
• This is for both GET and POST requests
• For two requests on the same connection triggering an alert, the second request just gets stuck indefinitely.

The above would manifest in a denial of service situation whenever we detect an alert. Why would the linger timeout get activated when an alert is triggered?

Thanks
Abhishek