[Self-Hosted] Support for own CA certificates

[stefanstoeckigt]

Describe the bug

Can't get Oauth working with self-hosted gitpod and gitlab instance.

GitPod (version: 0.6.0-beta1): self-hosted deployment in kubernetes
GitLab: self-hosted premium

Additional information

Helm values

# Copyright (c) 2020 TypeFox GmbH. All rights reserved.
# Licensed under the MIT License. See License-MIT.txt in the project root for license information.

# This field must be set to your domain name. Leaving it set to its default value will result in
# a non-functional installation.
hostname: {{ gitpod_hostname }}
ingressMode: hosts

# If you have a static IP that your domain resolves to, set it here.
# Leaving this field set to its default value is fine. Kubernetes will assign you an IP address
# during deployment.

components:
  proxy:
    loadBalancerIP: {{ gitpod_proxy_loadbalancer_ip }}

  workspace:
    template:
      default:
        spec:
            dnsConfig:
            nameservers:
            - 10.26.103.10
            - 10.26.103.11
            dnsPolicy: ClusterFirst

# Gitpod needs at least one auth provider to allow users to log in.
# The auth providers below are examples only. Please change/remove them to fit your installation.
authProviders:
- id: "{{ gitpod_authProviders_gitlab_id }}"
  host: "{{ gitpod_authProviders_gitlab_host }}"
  protocol: "https"
  type: "GitLab"
  oauth:
    clientId: "{{ gitpod_authProviders_gitlab_oauth_client_id }}"
    clientSecret: "{{ gitpod_authProviders_gitlab_oauth_client_secret }}"
    callBackUrl: "https://{{ gitpod_hostname }}/auth/{{ gitpod_authProviders_gitlab_host }}/callback"
    settingsUrl: "https://{{ gitpod_authProviders_gitlab_host }}/profile/applications"

# RBAC is enabled by default. If your cluster does not use RBAC, set this flag to false so that
# we do not attempt to install PodSecurityPolicies and the likes.
installPodSecurityPolicies: true

#certificatesSecret: {}
certificatesSecret:
  secretName: proxy-config-certificates

GitPod Server Logs

{
    "component": "server",
    "severity": "INFO",
    "time": "2020-12-22T09:15:13.772Z",
    "environment": "production",
    "region": "local",
    "message": "Request getFeaturedRepositories unsuccessful: 401/\"User is not authenticated. Please login.\"",
    "payload": {
        "method": "getFeaturedRepositories",
        "args": [
            null,
            {
                "_isCancelled": false
            }
        ]
    }
}
{
    "component": "server",
    "severity": "INFO",
    "time": "2020-12-22T09:15:15.300Z",
    "environment": "production",
    "region": "local",
    "context": {
        "sessionId": "c67e1fc1-339d-4fbc-b170-328dc0e0645f"
    },
    "message": "(Auth) User started the login process",
    "payload": {
        "login-flow": true,
        "clientInfo": {
            "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19041",
            "fingerprint": "36489166f56651e48e01d3a6c34ae05e7f770abd2329a1aed83832dd0530f86c"
        }
    }
}
{
    "component": "server",
    "severity": "INFO",
    "time": "2020-12-22T09:15:18.746Z",
    "environment": "production",
    "region": "local",
    "message": "Auth Provider Callback. Path: /auth/gitlab.garrison.local/callback"
}
{
    "component": "server",
    "severity": "INFO",
    "time": "2020-12-22T09:15:18.746Z",
    "environment": "production",
    "region": "local",
    "context": {},
    "message": "(Auth-With-gitlab.garrison.local) OAuth2 callback call. ",
    "payload": {
        "clientInfo": {
            "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19041",
            "fingerprint": "36489166f56651e48e01d3a6c34ae05e7f770abd2329a1aed83832dd0530f86c"
        },
        "authProviderId": "garrison-gitlab",
        "requestUrl": "/auth/gitlab.garrison.local/callback?code=1bf718b9ef0325287fd89ae7fdb3bf2b045942e2135d6caeaaf67be8ddfbb8fb"
    }
}
{
    "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
    "serviceContext": {
        "service": "server",
        "version": "0.6.0-beta1"
    },
    "stack_trace": "InternalOAuthError: Failed to obtain access token\n    at GenericOAuth2Strategy.OAuth2Strategy._createOAuthError (/app/node_modules/passport-oauth2/lib/strategy.js:379:17)\n    at /app/node_modules/passport-oauth2/lib/strategy.js:166:45\n    at patchedCallback (/app/node_modules/@gitpod/server/dist/src/auth/generic-auth-provider.js:837:28)\n    at /app/node_modules/oauth/lib/oauth2.js:191:18\n    at ClientRequest.<anonymous> (/app/node_modules/oauth/lib/oauth2.js:162:5)\n    at ClientRequest.emit (events.js:315:20)\n    at ClientRequest.EventEmitter.emit (domain.js:483:12)\n    at TLSSocket.socketErrorListener (_http_client.js:426:9)\n    at TLSSocket.emit (events.js:315:20)\n    at TLSSocket.EventEmitter.emit (domain.js:483:12)\n    at emitErrorNT (internal/streams/destroy.js:92:8)\n    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)",
    "component": "server",
    "severity": "ERROR",
    "time": "2020-12-22T09:15:18.787Z",
    "environment": "production",
    "region": "local",
    "context": {},
    "message": "(Auth-With-gitlab.garrison.local) Redirect to /sorry from verify callback",
    "error": "InternalOAuthError: Failed to obtain access token\n    at GenericOAuth2Strategy.OAuth2Strategy._createOAuthError (/app/node_modules/passport-oauth2/lib/strategy.js:379:17)\n    at /app/node_modules/passport-oauth2/lib/strategy.js:166:45\n    at patchedCallback (/app/node_modules/@gitpod/server/dist/src/auth/generic-auth-provider.js:837:28)\n    at /app/node_modules/oauth/lib/oauth2.js:191:18\n    at ClientRequest.<anonymous> (/app/node_modules/oauth/lib/oauth2.js:162:5)\n    at ClientRequest.emit (events.js:315:20)\n    at ClientRequest.EventEmitter.emit (domain.js:483:12)\n    at TLSSocket.socketErrorListener (_http_client.js:426:9)\n    at TLSSocket.emit (events.js:315:20)\n    at TLSSocket.EventEmitter.emit (domain.js:483:12)\n    at emitErrorNT (internal/streams/destroy.js:92:8)\n    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)",
    "payload": {
        "authFlow": {
            "host": "gitlab.garrison.local",
            "returnTo": "https://gitpod.garrison.local/workspaces/",
            "returnToAfterTos": "https://gitpod.garrison.local/api/login/?returnTo=https%3A%2F%2Fgitpod.garrison.local%2Fworkspaces%2F&host=gitlab.garrison.local"
        },
        "clientInfo": {
            "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19041",
            "fingerprint": "36489166f56651e48e01d3a6c34ae05e7f770abd2329a1aed83832dd0530f86c"
        },
        "authProviderId": "garrison-gitlab",
        "err": "InternalOAuthError: Failed to obtain access token\n    at GenericOAuth2Strategy.OAuth2Strategy._createOAuthError (/app/node_modules/passport-oauth2/lib/strategy.js:379:17)\n    at /app/node_modules/passport-oauth2/lib/strategy.js:166:45\n    at patchedCallback (/app/node_modules/@gitpod/server/dist/src/auth/generic-auth-provider.js:837:28)\n    at /app/node_modules/oauth/lib/oauth2.js:191:18\n    at ClientRequest.<anonymous> (/app/node_modules/oauth/lib/oauth2.js:162:5)\n    at ClientRequest.emit (events.js:315:20)\n    at ClientRequest.EventEmitter.emit (domain.js:483:12)\n    at TLSSocket.socketErrorListener (_http_client.js:426:9)\n    at TLSSocket.emit (events.js:315:20)\n    at TLSSocket.EventEmitter.emit (domain.js:483:12)\n    at emitErrorNT (internal/streams/destroy.js:92:8)\n    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
    }
}
{
    "component": "server",
    "severity": "INFO",
    "time": "2020-12-22T09:15:19.101Z",
    "environment": "production",
    "region": "local",
    "message": "Request getLoggedInUser unsuccessful: 401/\"User is not authenticated. Please login.\"",
    "payload": {
        "method": "getLoggedInUser",
        "args": [
            null,
            {
                "_isCancelled": false
            }
        ]
    }
}

GitPod Server Error

InternalOAuthError: Failed to obtain access token
    at GenericOAuth2Strategy.OAuth2Strategy._createOAuthError (/app/node_modules/passport-oauth2/lib/strategy.js:379:17)
    at /app/node_modules/passport-oauth2/lib/strategy.js:166:45
    at patchedCallback (/app/node_modules/@gitpod/server/dist/src/auth/generic-auth-provider.js:837:28)
    at /app/node_modules/oauth/lib/oauth2.js:191:18
    at ClientRequest.<anonymous> (/app/node_modules/oauth/lib/oauth2.js:162:5)
    at ClientRequest.emit (events.js:315:20)
    at ClientRequest.EventEmitter.emit (domain.js:483:12)
    at TLSSocket.socketErrorListener (_http_client.js:426:9)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket.EventEmitter.emit (domain.js:483:12)
    at emitErrorNT (internal/streams/destroy.js:92:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

[AlexTugarev]

Hi @stefanstoeckigt!

Here what happens is, that Gitpod already get the callback request (after being redirected from Gitlab) and the OAuth2 stack tries to exchange the provided code (URL param) for an access token. This POST request back to Gitlab fails. Unfortunately, either the service doesn't provide an error message to the callback request, or the OAuth2 library fails to parse it and falls back to the generic message.

Could you please double-check the client secret for the OAuth2 configuration?

Also the Gitlab installation has to be reachable from the server pod. So, if there is anything in the k8s/network settings to prevent this, this needs to be resolved.

[stefanstoeckigt]

Hi @AlexTugarev

I have double-checked the OAuth2 configuration, and also recreated the OAuth Application in GitLab and I am still getting the same issue.
My gitpod and gitlab instance is on the same subnet, so there shouldn't be any firewall issues. I have also attached to the gitpod server pod and everything looks correct in the /etc/resolv.conf. All DNS entries are resolved correctly in the kubernetes instance.

It seems that gitlab "OAuth access granted", so not sure why it isn't working.

I have got some logs from gitlab:

{
    "severity": "INFO",
    "time": "2020-12-23T09:26:44.169Z",
    "correlation_id": "01ET7ER459FAZ1AQZ9PQZNH2SA",
    "author_id": 2,
    "author_name": "stefan.stoeckigt",
    "entity_id": 2,
    "entity_type": "User",
    "ip_address": "10.26.0.102",
    "custom_message": "OAuth access granted",
    "target_id": 23,
    "target_type": "User",
    "target_details": "gitpod-garrison",
    "entity_path": "stefan.stoeckigt"
}
Started GET "/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Fgitpod.garrison.local%2Fauth%2Fgitlab.garrison.local%2Fcallback&scope=read_user%20api&client_id=29df3e480f8099cbe9208340c101611b671218407d7bc14ca78da2274eafe966" for 10.26.0.102 at 2020-12-23 10: 21: 59 +0000
  Parameters: {
    "response_type"=>"code",
    "redirect_uri"=>"https://gitpod.garrison.local/auth/gitlab.garrison.local/callback",
    "scope"=>"read_user api",
    "client_id"=>"29df3e480f8099cbe9208340c101611b671218407d7bc14ca78da2274eafe966"
}
Redirected to https: //gitpod.garrison.local/auth/gitlab.garrison.local/callback?code=5d373c9dbfb69614252d60ee34602bc5cfcb484e71aa3fc12b2d8cd56282819a {
    "method": "GET",
    "path": "/oauth/authorize",
    "format": "html",
    "controller": "Oauth::AuthorizationsController",
    "action": "new",
    "status": 302,
    "location": "https://gitpod.garrison.local/auth/gitlab.garrison.local/callback",
    "time": "2020-12-23T10:21:59.438Z",
    "params": [
        {
            "key": "response_type",
            "value": "code"
        },
        {
            "key": "redirect_uri",
            "value": "https://gitpod.garrison.local/auth/gitlab.garrison.local/callback"
        },
        {
            "key": "scope",
            "value": "read_user api"
        },
        {
            "key": "client_id",
            "value": "29df3e480f8099cbe9208340c101611b671218407d7bc14ca78da2274eafe966"
        }
    ],
    "remote_ip": null,
    "user_id": null,
    "username": null,
    "ua": null,
    "redis_calls": 1,
    "redis_duration_s": 0.000437,
    "redis_read_bytes": 375,
    "redis_write_bytes": 968,
    "redis_shared_state_calls": 1,
    "redis_shared_state_duration_s": 0.000437,
    "redis_shared_state_read_bytes": 375,
    "redis_shared_state_write_bytes": 968,
    "db_count": 5,
    "db_write_count": 1,
    "db_cached_count": 1,
    "correlation_id": "01ET7HX9QXJZJETD1QZDXP5WDJ",
    "cpu_s": 0.01,
    "db_duration_s": 0.00228,
    "view_duration_s": 0.0,
    "duration_s": 0.01036
}
{
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "01ET7HX9QXJZJETD1QZDXP5WDJ",
    "duration_ms": 18,
    "host": "gitlab.garrison.local",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "proto": "HTTP/1.1",
    "referrer": "https://gitpod.garrison.local/",
    "remote_addr": "127.0.0.1:0",
    "remote_ip": "127.0.0.1",
    "route": "",
    "status": 302,
    "system": "http",
    "time": "2020-12-23T10:21:59Z",
    "ttfb_ms": 18,
    "uri": "/oauth/authorize?response_type=code\u0026redirect_uri=https%3A%2F%2Fgitpod.garrison.local%2Fauth%2Fgitlab.garrison.local%2Fcallback\u0026scope=read_user%20api\u0026client_id=29df3e480f8099cbe9208340c101611b671218407d7bc14ca78da2274eafe966",
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
    "written_bytes": 201
}
10.26.0.102 - - [
    23/Dec/2020: 10: 21: 59 +0000
] "GET /oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Fgitpod.garrison.local%2Fauth%2Fgitlab.garrison.local%2Fcallback&scope=read_user%20api&client_id=29df3e480f8099cbe9208340c101611b671218407d7bc14ca78da2274eafe966 HTTP/2.0" 302 201 "https://gitpod.garrison.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" 

[stefanstoeckigt]

I have also looked at the network trace from the browser (refer to the below images).
Any assistance would be appreciated, as to why the OAuth isn't succeeding.

[AlexTugarev]

@stefanstoeckigt, the broken part in the process as described above is the exchange of the code (provided as URL param) for the access token. It still might be as simple as an issue with the setup where the server pod cannot perform the request to the Gitlab instance. Please, try to ssh into the server pod and see if you can to a (POST) request to the relevant endpoint of the SH Gitlab instance. The URL pattern is https://${host}/oauth/token where host is the hostname of Gitlab.

[stefanstoeckigt]

@AlexTugarev, many thank for the reply.
It doesn't look like you can SSH into the kubernetes server pod. However i can get access to the server pod via the kubectl exec command.
The issue is that the server pod doesn't have curl install, and i don't have access to the root user, only the unode user.
Do you know the root password on the server pod?

[AlexTugarev]

@stefanstoeckigt, indeed I meant to exec' into the pod. 🤦🏻 Also, while curl isn't available, node is.

[stefanstoeckigt]

@AlexTugarev , many thanks for the update.
I ran the POST command via node and i get a TLS cert error.
I wasn't sure if this was due to node not having access to the cert, so i also tried with NODE_TLS_REJECT_UNAUTHORIZED=0.

I check the certificatesSecret: {} supplied to the helm chart value, and the gitpod cert look correct.
Would i need to supply gitlab public crt to the server pod somehow?

Node POST code

const https = require('https')

const options = {
  hostname: 'gitlab.garrison.local',
  port: 443,
  path: '/oauth/token',
  method: 'POST'
}

const req = https.request(options, res => {
  console.log(`statusCode: ${res.statusCode}`)

  res.on('data', d => {
    process.stdout.write(d)
  })
})

req.on('error', error => {
  console.error(error)
})

req.end()

Output

> const https = require('https')
undefined
>
> const options = {
...   hostname: 'gitlab.garrison.local',
...   port: 443,
...   path: '/oauth/token',
...   method: 'POST'
... }
undefined
>
> const req = https.request(options, res => {
...   console.log(`statusCode: ${res.statusCode}`)
...
...   res.on('data', d => {
...     process.stdout.write(d)
...   })
... })
undefined
>
> req.on('error', error => {
...   console.error(error)
... })
ClientRequest {
  _events: [Object: null prototype] {
    response: [Function: bound onceWrapper] { listener: [Function] },
    socket: [Function: bound onceWrapper] { listener: [Function: onSocket] },
    error: [Function]
  },
  _eventsCount: 3,
  _maxListeners: undefined,
  outputData: [],
  outputSize: 0,
  writable: true,
  _last: true,
  chunkedEncoding: false,
  shouldKeepAlive: false,
  useChunkedEncodingByDefault: true,
  sendDate: false,
  _removedConnection: false,
  _removedContLen: false,
  _removedTE: false,
  _contentLength: null,
  _hasBody: true,
  _trailer: '',
  finished: false,
  _headerSent: false,
  socket: null,
  connection: null,
  _header: null,
  _onPendingData: [Function: noopPendingOutput],
  agent: Agent {
    _events: [Object: null prototype] {
      free: [Function],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: { path: null },
    requests: {},
    sockets: { 'gitlab.garrison.local:443::::::::::::::::::': [Array] },
    freeSockets: {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  socketPath: undefined,
  method: 'POST',
  insecureHTTPParser: undefined,
  path: '/oauth/token',
  _ended: false,
  res: null,
  aborted: false,
  timeoutCb: null,
  upgradeOrConnect: false,
  parser: null,
  maxHeadersCount: null,
  reusedSocket: false,
  [Symbol(kCapture)]: false,
  [Symbol(kNeedDrain)]: false,
  [Symbol(corked)]: 0,
  [Symbol(kOutHeaders)]: [Object: null prototype] {
    host: [ 'Host', 'gitlab.garrison.local' ]
  }
}
>
> req.end()Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1501:34)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket.EventEmitter.emit (domain.js:506:15)
    at TLSSocket._finishInit (_tls_wrap.js:936:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:710:12)
    at TLSWrap.callbackTrampoline (internal/async_hooks.js:120:14) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}

ClientRequest {
  _events: [Object: null prototype] {
    response: [Function: bound onceWrapper] { listener: [Function] },
    error: [Function]
  },
  _eventsCount: 2,
  _maxListeners: undefined,
  outputData: [],
  outputSize: 0,
  writable: true,
  _last: true,
  chunkedEncoding: false,
  shouldKeepAlive: false,
  useChunkedEncodingByDefault: true,
  sendDate: false,
  _removedConnection: false,
  _removedContLen: false,
  _removedTE: false,
  _contentLength: 0,
  _hasBody: true,
  _trailer: '',
  finished: true,
  _headerSent: true,
  socket: TLSSocket {
    _tlsOptions: {
      allowHalfOpen: undefined,
      pipe: false,
      secureContext: [SecureContext],
      isServer: false,
      requestCert: true,
      rejectUnauthorized: true,
      session: undefined,
      ALPNProtocols: undefined,
      requestOCSP: undefined,
      enableTrace: undefined,
      pskCallback: undefined
    },
    _secureEstablished: true,
    _securePending: false,
    _newSessionPending: false,
    _controlReleased: true,
    secureConnecting: true,
    _SNICallback: null,
    servername: 'gitlab.garrison.local',
    alpnProtocol: false,
    authorized: false,
    authorizationError: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
    encrypted: true,
    _events: [Object: null prototype] {
      close: [Array],
      end: [Array],
      newListener: [Function: keylogNewListener],
      secure: [Function: onConnectSecure],
      session: [Function],
      free: [Function: onFree],
      timeout: [Function: onTimeout],
      agentRemove: [Function: onRemove],
      error: [Function: socketErrorListener],
      drain: [Function: ondrain]
    },
    _eventsCount: 10,
    connecting: false,
    _hadError: true,
    _parent: null,
    _host: 'gitlab.garrison.local',
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 16384,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: null,
      pipesCount: 0,
      flowing: true,
      ended: false,
      endEmitted: false,
      reading: true,
      sync: false,
      needReadable: true,
      emittedReadable: false,
      readableListening: false,
      resumeScheduled: false,
      emitClose: false,
      autoDestroy: false,
      destroyed: true,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: false,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: false
    },
    readable: false,
    _maxListeners: undefined,
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: true,
      decodeStrings: false,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      bufferedRequest: null,
      lastBufferedRequest: null,
      pendingcb: 0,
      prefinished: false,
      errorEmitted: true,
      emitClose: false,
      autoDestroy: false,
      bufferedRequestCount: 0,
      corkedRequestsFree: [Object]
    },
    writable: false,
    allowHalfOpen: false,
    _sockname: null,
    _pendingData: null,
    _pendingEncoding: '',
    server: undefined,
    _server: null,
    ssl: null,
    _requestCert: true,
    _rejectUnauthorized: true,
    parser: null,
    _httpMessage: [Circular],
    [Symbol(res)]: null,
    [Symbol(verified)]: false,
    [Symbol(pendingSession)]: null,
    [Symbol(asyncId)]: 36,
    [Symbol(kHandle)]: null,
    [Symbol(kSetNoDelay)]: false,
    [Symbol(lastWriteQueueSize)]: 0,
    [Symbol(timeout)]: null,
    [Symbol(kBuffer)]: null,
    [Symbol(kBufferCb)]: null,
    [Symbol(kBufferGen)]: null,
    [Symbol(kCapture)]: false,
    [Symbol(kBytesRead)]: 0,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(connect-options)]: {
      rejectUnauthorized: true,
      ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA',
      checkServerIdentity: [Function: checkServerIdentity],
      minDHSize: 1024,
      hostname: 'gitlab.garrison.local',
      port: 443,
      path: null,
      method: 'POST',
      _defaultAgent: [Agent],
      host: 'gitlab.garrison.local',
      servername: 'gitlab.garrison.local',
      _agentKey: 'gitlab.garrison.local:443::::::::::::::::::',
      encoding: null,
      singleUse: true
    }
  },
  connection: TLSSocket {
    _tlsOptions: {
      allowHalfOpen: undefined,
      pipe: false,
      secureContext: [SecureContext],
      isServer: false,
      requestCert: true,
      rejectUnauthorized: true,
      session: undefined,
      ALPNProtocols: undefined,
      requestOCSP: undefined,
      enableTrace: undefined,
      pskCallback: undefined
    },
    _secureEstablished: true,
    _securePending: false,
    _newSessionPending: false,
    _controlReleased: true,
    secureConnecting: true,
    _SNICallback: null,
    servername: 'gitlab.garrison.local',
    alpnProtocol: false,
    authorized: false,
    authorizationError: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
    encrypted: true,
    _events: [Object: null prototype] {
      close: [Array],
      end: [Array],
      newListener: [Function: keylogNewListener],
      secure: [Function: onConnectSecure],
      session: [Function],
      free: [Function: onFree],
      timeout: [Function: onTimeout],
      agentRemove: [Function: onRemove],
      error: [Function: socketErrorListener],
      drain: [Function: ondrain]
    },
    _eventsCount: 10,
    connecting: false,
    _hadError: true,
    _parent: null,
    _host: 'gitlab.garrison.local',
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 16384,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: null,
      pipesCount: 0,
      flowing: true,
      ended: false,
      endEmitted: false,
      reading: true,
      sync: false,
      needReadable: true,
      emittedReadable: false,
      readableListening: false,
      resumeScheduled: false,
      emitClose: false,
      autoDestroy: false,
      destroyed: true,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: false,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: false
    },
    readable: false,
    _maxListeners: undefined,
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: true,
      decodeStrings: false,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      bufferedRequest: null,
      lastBufferedRequest: null,
      pendingcb: 0,
      prefinished: false,
      errorEmitted: true,
      emitClose: false,
      autoDestroy: false,
      bufferedRequestCount: 0,
      corkedRequestsFree: [Object]
    },
    writable: false,
    allowHalfOpen: false,
    _sockname: null,
    _pendingData: null,
    _pendingEncoding: '',
    server: undefined,
    _server: null,
    ssl: null,
    _requestCert: true,
    _rejectUnauthorized: true,
    parser: null,
    _httpMessage: [Circular],
    [Symbol(res)]: null,
    [Symbol(verified)]: false,
    [Symbol(pendingSession)]: null,
    [Symbol(asyncId)]: 36,
    [Symbol(kHandle)]: null,
    [Symbol(kSetNoDelay)]: false,
    [Symbol(lastWriteQueueSize)]: 0,
    [Symbol(timeout)]: null,
    [Symbol(kBuffer)]: null,
    [Symbol(kBufferCb)]: null,
    [Symbol(kBufferGen)]: null,
    [Symbol(kCapture)]: false,
    [Symbol(kBytesRead)]: 0,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(connect-options)]: {
      rejectUnauthorized: true,
      ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA',
      checkServerIdentity: [Function: checkServerIdentity],
      minDHSize: 1024,
      hostname: 'gitlab.garrison.local',
      port: 443,
      path: null,
      method: 'POST',
      _defaultAgent: [Agent],
      host: 'gitlab.garrison.local',
      servername: 'gitlab.garrison.local',
      _agentKey: 'gitlab.garrison.local:443::::::::::::::::::',
      encoding: null,
      singleUse: true
    }
  },
  _header: 'POST /oauth/token HTTP/1.1\r\n' +
    'Host: gitlab.garrison.local\r\n' +
    'Connection: close\r\n' +
    'Content-Length: 0\r\n' +
    '\r\n',
  _onPendingData: [Function: noopPendingOutput],
  agent: Agent {
    _events: [Object: null prototype] {
      free: [Function],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: { path: null },
    requests: {},
    sockets: {},
    freeSockets: {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  socketPath: undefined,
  method: 'POST',
  insecureHTTPParser: undefined,
  path: '/oauth/token',
  _ended: false,
  res: null,
  aborted: false,
  timeoutCb: null,
  upgradeOrConnect: false,
  parser: null,
  maxHeadersCount: null,
  reusedSocket: false,
  [Symbol(kCapture)]: false,
  [Symbol(kNeedDrain)]: false,
  [Symbol(corked)]: 0,
  [Symbol(kOutHeaders)]: [Object: null prototype] {
    host: [ 'Host', 'gitlab.garrison.local' ]
  }
}

[stefanstoeckigt]

@AlexTugarev, so i tried editing the deployment for the server pod and added the following:

      - env:
        - name: NODE_TLS_REJECT_UNAUTHORIZED
          value: "0"

so now i can login correctly, however when i try and open a project in gitpod from gitlab i get the following:

Another option would be to set NODE_EXTRA_CA_CERTS, and manually add my CA certificate. However, i am unsure if this would solve the issue for other pods.

Is there a global way of adding the ca certificated via the helm chart value?

[AlexTugarev]

@stefanstoeckigt, awesome that you drilled down that far!

Indeed it's an issue with certificates then. I forwarded the question to the team.

[geropl]

Hi @stefanstoeckigt ,

Gitpod requires to have HTTPS certificates from a valid, public CA (e.g., letsencrypt). Not because of the Gitpod services (be could work around those as you tried), but more because of the "external parts", like the container runtime (containerd in this case) running on the nodes.

Is that an option? In what scenario are you deploying Gitpod in?

PS: We just released v0.6.0-beta3, and by this moment I'm working on ironing out the last kinks for the 0.6.0 release 🙃

[stefanstoeckigt]

@geropl, many thanks for the reply.

Both gitpod and gitlab are self-hosted instances, and the CA is signed internally. So letsencrypt wouldn't be a option for our deployments.
All "external parts" have been configured with our private CA.

Is there a work around to allow private CA?