Skip to content

GitLab Oauth Internal Server Error #1134

@mcnesium

Description

@mcnesium

My GitLab authentication stopped working. CodiMD is running in Docker, here is the docker-compose.yml:

version: '3'
services:
    database:
        image: postgres:9.6-alpine
        container_name: hackmd_database
        environment:
        - POSTGRES_USER=hackmd
        - POSTGRES_PASSWORD=*****
        - POSTGRES_DB=hackmd
        volumes:
        - /data/hackmd/database:/var/lib/postgresql/data
        restart: always
    app:
        image: hackmdio/hackmd:alpine
        container_name: hackmd_app
        volumes:
        - /data/hackmd/uploads:/hackmd/public/uploads
        environment:
        - CMD_DB_URL=postgres://hackmd:*****@database:5432/hackmd
        - CMD_USECDN=false
        - CMD_DOMAIN=hackmd.mydomain.org
        - CMD_URL_ADDPORT=false
        - CMD_PROTOCOL_USESSL=true
        - CMD_ALLOW_FREEURL=true
        - CMD_EMAIL=false
        - CMD_ALLOW_ANONYMOUS=false
        - CMD_ALLOW_EMAIL_REGISTER=false
        - CMD_GITLAB_BASEURL=https://gitlab.mydomain.org/
        - CMD_GITLAB_CLIENTID=*****…
        - CMD_GITLAB_CLIENTSECRET=*****…
        - CMD_IMAGE_UPLOAD_TYPE=filesystem
        - CMD_ALLOW_GRAVATAR=false
        ports:
        - "127.0.0.1:44311:3000"
        restart: always
        depends_on:
        - database

The proxying is done by apache2 on the host:

  …
  RewriteEngine On
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://localhost:44311/$1 [P,L]
  ProxyPass           /  http://localhost:44311/
  ProxyPassReverse    /  http://localhost:44311/
  …

In the running hackmd_app container the environment variables are all set:

$ docker-compose exec app sh
/codimd # printenv
CMD_ALLOW_EMAIL_REGISTER=false
CMD_ALLOW_GRAVATAR=false
CMD_ALLOW_PDF_EXPORT=false
CMD_GITLAB_CLIENTID=*****…
CMD_GITLAB_CLIENTSECRET=*****…
NODE_VERSION=8.15.0
HOSTNAME=2548fff412f3
YARN_VERSION=1.12.3
CMD_PROTOCOL_USESSL=true
SHLVL=1
HOME=/root
CMD_IMAGE_UPLOAD_TYPE=filesystem
CMD_ALLOW_FREEURL=true
CMD_GITLAB_BASEURL=https://gitlab.mydomain.org/
TERM=xterm
CMD_DB_URL=postgres://hackmd:*****@database:5432/hackmd
CMD_URL_ADDPORT=false
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CMD_EMAIL=false
CMD_DOMAIN=hackmd.mydomain.org
DOCKERIZE_VERSION=v0.6.1
GOSU_VERSION=1.11
PWD=/codimd
CMD_USECDN=false
CMD_ALLOW_ANONYMOUS=false
NODE_ENV=production

The application is correctly registered in GitLab:

appcodimd

When visiting the site and hitting Login button, the modal with the GitLab button appears, and when this one is hit, the browser tab keeps waiting for an answer from the hackmd domain for about two minutes and then shows Internal Server Error on a blank page.

Here is the output of docker-compose logs -f starting with the boot process until the failed GitLab authentication:

Creating network "codimd_default" with the default driver
Creating hackmd_database ... done
Creating hackmd_app      ... done
Attaching to hackmd_app, hackmd_database
hackmd_app  | 2019/01/31 14:10:48 Waiting for: tcp://database:5432
hackmd_app  | 2019/01/31 14:10:48 Connected to tcp://database:5432
hackmd_app  |
hackmd_app  | Sequelize [Node: 8.15.0, CLI: 2.8.0, ORM: 3.30.4]
hackmd_app  |
hackmd_database | LOG:  database system was interrupted; last known up at 2019-01-31 14:07:38 UTC
hackmd_database | LOG:  database system was not properly shut down; automatic recovery in progress
hackmd_database | LOG:  redo starts at 0/912E4C8
hackmd_database | LOG:  invalid record length at 0/9132170: wanted 24, got 0
hackmd_database | LOG:  redo done at 0/9132148
hackmd_database | LOG:  last completed transaction was at log time 2019-01-31 14:10:13.807358+00
hackmd_database | LOG:  MultiXact member wraparound protections are now enabled
hackmd_database | LOG:  database system is ready to accept connections
hackmd_database | LOG:  autovacuum launcher started
hackmd_database | LOG:  incomplete startup packet
hackmd_app  | Parsed url postgres://hackmd:*****@database:5432/hackmd
hackmd_app  | (node:25) DeprecationWarning: Using the automatically created return value from client.query as an event emitter is deprecated and will be removed in [email protected]. Please see the upgrade guide at https://node-postgres.com/guides/upgrading
hackmd_app  | == 20180525153000-user-add-delete-token: migrating =======
hackmd_app  | == 20180525153000-user-add-delete-token: migrated (0.028s)
hackmd_app  |
hackmd_app  |         #################################################################
hackmd_app  |         ###                                                           ###
hackmd_app  |         ###                        !!!WARNING!!!                      ###
hackmd_app  |         ###                                                           ###
hackmd_app  |         ###        Using local uploads without persistence is         ###
hackmd_app  |         ###            dangerous. You'll loose your data on           ###
hackmd_app  |         ###              container removal. Check out:                ###
hackmd_app  |         ###  https://docs.docker.com/engine/tutorials/dockervolumes/  ###
hackmd_app  |         ###                                                           ###
hackmd_app  |         ###                       !!!WARNING!!!                       ###
hackmd_app  |         ###                                                           ###
hackmd_app  |         #################################################################
hackmd_app  |
hackmd_app  | 2019-01-31T14:10:53.558Z - warn: Session secret not set. Using random generated one. Please set `sessionSecret` in your config.js file. All users will be logged out.
hackmd_app  | (node:1) DeprecationWarning: Using the automatically created return value from client.query as an event emitter is deprecated and will be removed in [email protected]. Please see the upgrade guide at https://node-postgres.com/guides/upgrading
hackmd_app  | >> WARNING: PostgreSQL does not support TEXT with options. Plain `TEXT` will be used instead.
hackmd_app  | >> Check: http://www.postgresql.org/docs/9.4/static/datatype.html
hackmd_app  | 2019-01-31T14:10:54.647Z - info: HTTP Server listening at 0.0.0.0:3000
hackmd_app  | 2019-01-31T14:11:09.542Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:09 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:09.685Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:09 +0000] "GET /config HTTP/1.1" 200 235 "https://hackmd.mydomain.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:09.934Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:09 +0000] "GET /me HTTP/1.1" 304 - "https://hackmd.mydomain.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:10.389Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:10 +0000] "GET /build/bootstrap.min.css.map HTTP/1.1" 302 57 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:10.394Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:10 +0000] "GET /build HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | [repeats about 20 times]
hackmd_app  | 2019-01-31T14:11:10.755Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:10 +0000] "GET /build/ HTTP/1.1" 301 69 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:11:12.326Z - info: 192.168.224.1 - - [31/Jan/2019:14:11:12 +0000] "GET /auth/gitlab HTTP/1.1" 302 0 "https://hackmd.mydomain.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | 2019-01-31T14:13:12.513Z - info: 192.168.224.1 - - [31/Jan/2019:14:13:12 +0000] "GET /auth/gitlab/callback?code=127717b5898fbd0a07ea5d56f20d04… HTTP/1.1" - - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"
hackmd_app  | TokenError: The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.
hackmd_app  |     at Strategy.OAuth2Strategy.parseErrorResponse (/codimd/node_modules/passport-oauth2/lib/strategy.js:329:12)
hackmd_app  |     at Strategy.OAuth2Strategy._createOAuthError (/codimd/node_modules/passport-oauth2/lib/strategy.js:376:16)
hackmd_app  |     at /codimd/node_modules/passport-oauth2/lib/strategy.js:166:45
hackmd_app  |     at /codimd/node_modules/oauth/lib/oauth2.js:191:18
hackmd_app  |     at passBackControl (/codimd/node_modules/oauth/lib/oauth2.js:132:9)
hackmd_app  |     at IncomingMessage.<anonymous> (/codimd/node_modules/oauth/lib/oauth2.js:157:7)
hackmd_app  |     at emitNone (events.js:111:20)
hackmd_app  |     at IncomingMessage.emit (events.js:208:7)
hackmd_app  |     at endReadableNT (_stream_readable.js:1064:12)
hackmd_app  |     at _combinedTickCallback (internal/process/next_tick.js:139:11)
hackmd_app  |     at process._tickCallback (internal/process/next_tick.js:181:9)
hackmd_app  | 2019-01-31T14:13:12.668Z - info: 192.168.224.1 - - [31/Jan/2019:14:13:12 +0000] "GET /auth/gitlab/callback?code=127717b5898fbd0a07ea5d56f20d04… HTTP/1.1" 500 148 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"

The TokenError: The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client. is not true, as they have been refreshed right before that session.

The CodiMD setup has been running for quite some time, you can see this has been running under the name HackMD first. The only thing I can think of what I did was upgrading the underlying host system Ubuntu from 1604 to 1804.

There was a database issue that came up in the logs stating

PANIC:  could not locate a valid checkpoint record

which I solved by doing pg_resetxlog on the database:

docker run -ti --user postgres  -v /data/hackmd/database:/var/lib/postgresql/data postgres:9.6-alpine pg_resetxlog /var/lib/postgresql/data/

While I was puzzling about what and why it has happened to the db, I cheeringly saw that the site going back up, but then the stated GitLab authentication fails.

So whats going on with this? Any help?

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionYou asked us something. We try to find the answer

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions