Container crashing makes nginx crash

[cescoferraro]

If a container crashes, and it is the only one within a service. The next ingress the api happen to take will make nginx reload to fail because it wont be able to resolve this particular service ip at the time of daemon start.

[pleshakov]

I'm not sure I understand the problem. Could you provide an example?

[cescoferraro]

the default behaviour of the Nginx upstream and proxy modules is to resolve DNS at startup, but not to re-resolve DNS during the operation of the application.

https://www.jethrocarr.com/2013/11/02/nginx-reverse-proxies-and-dns-resolution/
http://stackoverflow.com/questions/26956979/error-with-ip-and-nginx-as-reverse-proxy
https://www.ruby-forum.com/topic/4407628

if all pods inside a service fail, the service becomes unresponsive, making nginx fail to reload on any subsequent ingress events. I think nginx+ can handle this better.

[pleshakov]

The NGINX controller doesn't rely on DNS to get endpoints. It talks to Kubernetes API to get IP addresses. There is no chance for such failure.

The NGINX Plus controller uses DNS to get endpoints. But it doesn't fail on reload when it can't resolve a name.

[cescoferraro]

Just try starting nginx using a ip address not reachable at start up time. It will fail

[pleshakov]

Could you provide an example of nginx config file or kubernetes resource files along with instruction that confirm the issue?

[cescoferraro]

FROM debian:jessie

MAINTAINER NGINX Docker Maintainers "docker-maint@nginx.com"

ENV NGINX_VERSION 1.8.1-1~jessie

RUN apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 \
    && echo "deb http://nginx.org/packages/debian/ jessie nginx" >> /etc/apt/sources.list \
    && apt-get update \
    && apt-get install -y ca-certificates nginx=${NGINX_VERSION} gettext-base \
    && rm -rf /var/lib/apt/lists/*

# forward request and error logs to docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log

EXPOSE 80 443

COPY nginx.conf /etc/nginx/nginx.conf

CMD ["nginx", "-g", "daemon off;"]

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    server {
        listen 80;
        server_name localhost;
        access_log /var/log/nginx/example.log;

        location / {
            proxy_pass http://333.99.333.22;
            proxy_redirect off;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
        }
}

cesco@desktop: ~/code/go/src/bitbucket.org/cescoferraro/cluster/nginx/example on HEAD [?]
$ docker run -p 80:80 cescoferraro/issue
2016/04/01 11:50:45 [emerg] 1#0: host not found in upstream "333.99.333.22" in /etc/nginx/nginx.conf:37
nginx: [emerg] host not found in upstream "333.99.333.22" in /etc/nginx/nginx.conf:37

In the kubernetes environment, the service waits trying to find an alive pod. If there are none alive, nginx will fail. I think there is a timeouts difference; the problem is that nginx fails instantly.