Skip to content

api '/user/events' always timeout #17205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
colben opened this issue Oct 1, 2021 · 4 comments
Closed

api '/user/events' always timeout #17205

colben opened this issue Oct 1, 2021 · 4 comments
Labels
type/docs This PR mainly updates/creates documentation type/question Issue needs no code to be fixed, only a description on how to fix it yourself.

Comments

@colben
Copy link

colben commented Oct 1, 2021

ENV

  • centos8 amd64
  • gitea 1.13.7 download from github release
  • nginx 1.20.1

REVERSE PROXY

  • gitea conf

    ...
    [server]
    PROTOCOL               = unix
    DOMAIN                 = {some pvivate domain}
    ROOT_URL               = http://{some private domain}
    HTTP_ADDR              = /socket/gitea
    UNIX_SOCKET_PERMISSION = 666
    ...
    
  • nginx conf

    ...
    server {
        listen      80;
        server_name X.X.X.X;
        #access_log  off;
        access_log  /var/log/nginx/access-gitea.log;
        error_log   /var/log/nginx/error-gitea.log;
        location / {
            proxy_pass              http://unix:/socket/gitea;
            client_max_body_size    1024m;
            client_body_buffer_size 1024m;
        }
    }
    ...
    

LOG

  • nginx error log

    ...
    2021/10/01 19:52:42 [error] 267059#267059: *1049742 upstream timed out (110: Operation timed out) while reading response header from upstream, client: X.X.X.X, server: X.X.X.X, request: "GET /user/events HTTP/1.1", upstream: "http://unix:/socket/gitea/user/events", host: "X.X.X.X", referrer: "http://X.X.X.X/js/eventsource.sharedworker.js"
    ...
    
  • nginx access log

    ...
    10.0.3.133 - - [01/Oct/2021:19:52:42 +0800] "GET /user/events HTTP/1.1" 504 562 "http://X.X.X.X/js/eventsource.sharedworker.js" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"
    ...
    
  • gitea output

    ...
    [Macaron] 2021-10-01 19:50:42: Started GET /user/events for @
    ...
    [Macaron] 2021-10-01 19:52:42: Completed GET /user/events 200 OK in 2m0.042893842s
    ...
    

WHAT I'VE DONE

  • i've tried to specify 'proxy_read_timeout 120;' in nginx conf, but the same error occurs again.

ISSUE

  • what is the api "/user/events", and is there any way to speed it up?
@noerw
Copy link
Member

noerw commented Oct 1, 2021

/user/events provides an event source of a server-sent events stream, to notify any frontend page about notifications for the user. It is supposed to be a longstanding connection lasting the entire time a logged in user has a browser tab with gitea open. There are ways to disable this, but there is rarely a reason to do so.
I'm not familiar with gitea+nginx, but didn't have timeout problems with other reverse proxies in the past.

@noerw noerw added type/question Issue needs no code to be fixed, only a description on how to fix it yourself. type/docs This PR mainly updates/creates documentation labels Oct 1, 2021
@techknowlogick
Copy link
Member

gitea 1.13.7 download from github release

please upgrade to latest stable (1.15.3) and try again, we cannot support older versions.

@colben
Copy link
Author

colben commented Oct 1, 2021

@noerw Thanks a lot for your reply! Awesome!
i just googled what is "server-sent events", and add a location block in my nginx conf:

    location = /user/events {
        proxy_pass                http://unix:/socket/gitea;
        proxy_cache               off;
        proxy_buffering           off;
        proxy_read_timeout        1d;
        proxy_http_version        1.1;
        proxy_set_header          Connection "";
        chunked_transfer_encoding off;
    }

i found nothing to disable nginx "proxy_read_timeout", so i just give it a long period like one day: "1d".

@colben colben closed this as completed Oct 1, 2021
@wxiaoguang
Copy link
Contributor

wxiaoguang commented Oct 2, 2021

I think the problem may be caused by chunked_transfer_encoding off;, the long-poll response need chunked transfer.

The error nginx reported timeout while reading response header from upstream shows that nginx thought that the response from gitea got stuck in http header, but the code of gitea 1.13.x and 1.15.x are similar, it's sure that gitea always complete http header in a very short time and there is a flush. So it's difficult to tell what's going wrong here.

Maybe you can try from a very simple configuration (eg: use all nginx default config, use TCP socket for proxy_pass) and maybe use tcpdump to see how the http protocol works, whether the response header is really incomplete.

@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type/docs This PR mainly updates/creates documentation type/question Issue needs no code to be fixed, only a description on how to fix it yourself.
Projects
None yet
Development

No branches or pull requests

4 participants