Hub console hang after upgrade nodes to 3.14.0-francium

[dlgiang]

Image(s):
node-chrome, hub, standalone-firefox
Docker-Selenium Image Version(s):
3.14.0-francium
Docker Version:
18.03.1-ce
OS:
Centos 7

Expected Behavior - Run probably

Actual Behavior - Hub Console hang and no response

Steps to reproduce -

Start below stack

version: '3.5'

services:

  hub:
    image: selenium/hub:latest
    ports:
      - 4444:4444
    deploy:
      mode: replicated
      replicas: 1
  chrome:
    image: selenium/node-chrome:latest
    volumes:
      - /dev/urandom:/dev/random
    depends_on:
      - hub
    environment:
      - HUB_PORT_4444_TCP_ADDR=hub
      - HUB_PORT_4444_TCP_PORT=4444
      - NODE_MAX_SESSION=1
    deploy:
      replicas: 1
  firefox:
    image: selenium/node-firefox:latest
    volumes:
      - /dev/urandom:/dev/random
    depends_on:
      - hub
    environment:
      - HUB_PORT_4444_TCP_ADDR=hub
      - HUB_PORT_4444_TCP_PORT=4444
      - NODE_MAX_SESSION=1
    deploy:
      replicas: 1

Go go hub console

[diemol]

Hi @dlgiang

How do you start the stack? I just used docker-compose up and it worked fine. I am using OSX. Does it only happen in Centos 7? It works well with the previous release?

[dlgiang]

Hi @dlgiang

How do you start the stack? I just used docker-compose up and it worked fine. I am using OSX. Does it only happen in Centos 7? It works well with the previous release?

I use Docker Swarm.

docker stack deploy --compose-file stack.yml selenium

[elemenophy]

Using Swarm you need to add to chrome and firefox the following:

entrypoint: bash -c 'SE_OPTS="-host $$HOSTNAME" /opt/bin/entry_point.sh'

If you don't do so, the Hub does not know where to ping for checking the status of the nodes, and therefore will not load the console.

Anyway, even if you add that line now, it will not work either with the images you are using due to bug #800.

[diemol]

@ay0o why do you need to use entrypoint?

Can't you just pass the SE_OPTS env var?

[elemenophy]

Because HOSTNAME is a random string generated on runtime.

That is, you need to edit SE_OPTS on runtime, and then call the entrypoint to execute the expected container function.

[diemol]

@dlgiang Did this work for you in previous versions or is it the first time you try this?

[diemol]

@ay0o just to see if I understand, is the HOSTNAME env var generated after the container is started? How does that work?

[elemenophy]

@diemol using Swarm mode, if you deploy the stack without entrypoint: bash -c 'SE_OPTS="-host $$HOSTNAME" /opt/bin/entry_point.sh', initially you will notice the Grid's console does not work (the point of this issue). After some time, the Hub will report the following errors:

grid_selenium-hub.1.mtwb5dj4pz7v@linuxkit-025000000001    | 15:11:47.041 INFO [DefaultRemoteProxy.onEvent] - Marking the node http://172.18.0.4:5555 as down: cannot reach the node for 2 tries
grid_selenium-hub.1.mtwb5dj4pz7v@linuxkit-025000000001    | 15:13:51.961 INFO [DefaultRemoteProxy.onEvent] - Unregistering the node http://172.18.0.4:5555 because it's been down for 124920 milliseconds
grid_selenium-hub.1.mtwb5dj4pz7v@linuxkit-025000000001    | 15:13:51.961 WARN [DefaultGridRegistry.removeIfPresent] - Cleaning up stale test sessions on the unregistered node http://172.18.0.4:5555

The issue here is that the Hub cannot reach the node by IP, only by its hostname. If you get into the Hub container and run curl http://172.18.0.4:5555 you will see how the connection timeout. However, if now you get into the Chrome container, get its hostname (e.g. 4407c0b74c64) and now you run curl http://4407c0b74c64:5555 in the Hub, you will see how it works.

So, what's going on here? The Hub tries to reach the node using 172.18.0.4, however, the node is actually on 10.0.1.6. Why? That's how the Swarm's overlay network works.

root@ef9e202727ac:~# nslookup 4407c0b74c64
Server:		127.0.0.11
Address:	127.0.0.11#53

Non-authoritative answer:
Name:	4407c0b74c64
Address: 10.0.1.6

So, to answer your question, yes, the HOSTNAME is generated after the container is started, and therefore, you need to pass it to SE_OPTS on runtime and not before.

You might wonder, why does it work with docker-compose then? That's because docker-compose uses a bridge network, whereas Swarm mode uses overlay.

[dlgiang]

@dlgiang Did this work for you in previous versions or is it the first time you try this?

Yes, I already used the SE_OPTS as @ay0o in previous versions and it worked.
Just with the latest version, if i use the SE_OPTS, the container cannot start
But if I don't use the SE_OPTS, the hub console will get in stuck

Right now, i reverted back to use 3.14.0-dubnium with SE_OPTS

[diemol]

Fixed in the referenced commit and will be released soon.