The hub cannot start because: "SEVERE: Port 4444 is busy, please choose a free port..." (the port is not in use)

[alinpa]

Meta -

Image(s):
selenium/node-chrome:3.12.0
hub: jenkins selenium hub 3.12.0
Docker-Selenium Image Version(s):
3.12.0
Docker Version:
18.03.1-ce
OS:
Linux 4.14.42-52.37.amzn1.x86_64 #1 SMP Tue May 22 00:41:10 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

-->

Expected Behavior -

The nodes have to connect to hub.

Actual Behavior - The nodes cannot connect to the hub because:

"14:33:16.014 INFO [SelfRegisteringRemote$1.run] - Couldn't register this node: The hub is down or not responding: Failed to connect to /xxx.xxx.xxx.xxx:4444"

On the hub logs I noticed the following error:

OFF: Grid Hub preparing to start on port 4444
2018-06-07 13:10:54.602:INFO::pool-1-thread-1 for channel id=0: Logging initialized @12793ms to org.seleniumhq.jetty9.util.log.StdErrLog
2018-06-07 13:10:54.780:INFO:osjs.Server:pool-1-thread-1 for channel id=0: jetty-9.4.7.v20170914, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
2018-06-07 13:10:54.842:INFO:osjs.session:pool-1-thread-1 for channel id=0: DefaultSessionIdManager workerName=node0
2018-06-07 13:10:54.843:INFO:osjs.session:pool-1-thread-1 for channel id=0: No SessionScavenger set, using defaults
2018-06-07 13:10:54.849:INFO:osjs.session:pool-1-thread-1 for channel id=0: Scavenging every 660000ms
2018-06-07 13:10:54.870:INFO:osjsh.ContextHandler:pool-1-thread-1 for channel id=0: Started o.s.j.s.ServletContextHandler@54620f3b{/,null,AVAILABLE}
Jun 07, 2018 1:10:54 PM org.openqa.grid.internal.DefaultGridRegistry assignRequestToProxy
INFO: Shutting down registry.
2018-06-07 13:10:54.880:INFO:osjs.AbstractConnector:pool-1-thread-1 for channel id=0: Stopped ServerConnector@21f99f64{HTTP/1.1,[http/1.1]}{xxx.xxx.xxx.xxx:4444}
2018-06-07 13:10:54.880:INFO:osjs.session:pool-1-thread-1 for channel id=0: Stopped scavenging
2018-06-07 13:10:54.882:INFO:osjsh.ContextHandler:pool-1-thread-1 for channel id=0: Stopped o.s.j.s.ServletContextHandler@54620f3b{/,null,UNAVAILABLE}
Jun 07, 2018 1:10:54 PM org.openqa.grid.web.Hub start
SEVERE: Port 4444 is busy, please choose a free port for the hub and specify it using -port option
Jun 07, 2018 1:10:54 PM hudson.plugins.selenium.HubLauncher call
OFF: Grid Hub started on port 4444 with args: -maxSession 5 -timeout 30 -browserTimeout 0 -host xxx.xxx.xxx.xxx

I have checked if the port is already used and is not in use.

[ddavison]

what command did you use to run?

also, do me a favor and paste the result of the following command (preferably as the root user):

$ netstat -tulpn | grep 4444

[alinpa]

Hi @ddavison,

I ran the previous command and the port is not in use.
The selenium grid is started by default by selenium jenkins plugin

[alinpa]

Hi @mach6

Can you take a look at this issue ?
And this: jenkinsci/selenium-plugin#128 (comment)

It seems to be an issue introduced by you.

Please let me know if you need more details.

[ddavison]

i'm tempted to close this as i can't reproduce this. this looks like it's a setup issue on your end @alinpa , not necessarily an issue with our images

[alinpa]

@ddavison : it is enough to start the selenium grid with 1 node on two ec2 machines. Start the docker images and you will see the host is not discovered properly. Please check the following issue I already documented there: jenkinsci/selenium-plugin#128 (comment)

[mtilev]

We can reporduce this issue, port 4444 is reported as in use while plugin starts, hweover, a netstat reports no process serving on that port. We are using a jenkins deployment within a kubernetes cluster.

[alinpa]

@mtilev Please check my comments here: jenkinsci/selenium-plugin#128 (comment)

The error message is wrong!

[mtilev]

@alinpa We implemented the workaround; nevertheless, this is an undocumented change in behaviour and thus either a bug or a documentation bug.

[alinpa]

@mtilev Did you implement my workaround or something new?

[mtilev]

@alinpa We followed your advice.

Kind regars,

m

[dober]

got same issue with node.

v3.12:

One can also add -ncache_cr for smooth 'copyrect' window motion.
More info: http://www.karlrunge.com/x11vnc/faq.html#faq-client-caching

2018-06-29 13:15:57.593:INFO::main: Logging initialized @636ms to org.seleniumhq.jetty9.util.log.StdErrLog
Failed to read: session.ignoreBorder
Setting default value
Failed to read: session.forcePseudoTransparency
Setting default value
Failed to read: session.colorsPerChannel
Setting default value
Failed to read: session.doubleClickInterval
Setting default value
Failed to read: session.tabPadding
Setting default value
Failed to read: session.styleOverlay
Setting default value
Failed to read: session.slitlistFile
Setting default value
Failed to read: session.appsFile
Settingdefault value
Failed to read: session.tabsAttachArea
Setting default value
Failed to read: session.cacheLife
Setting default value
Failed to read: session.cacheMax
Setting default value
Failed to read: session.autoRaiseDelay
Setting default value
Failed to read: session.ignoreBorder
Setting default value
Failed to read: session.forcePseudoTransparency
Setting default value
Failed to read: session.colorsPerChannel
Setting default value
Failed to read: session.doubleClickInterval
Setting default value
Failed to read: session.tabPadding
Setting default value
Failed to read: session.styleOverlay
Setting default value
Failed to read: session.slitlistFile
Setting default value
Failed to read: session.appsFile
Setting default value
Failed to read: session.tabsAttachArea
Setting default value
Failed to read: session.cacheLife
Setting default value
Failed to read: session.cacheMax
Setting default value
Failed to read: session.autoRaiseDelay
Setting default value
13:15:57.782 ERROR [SeleniumServer.boot] - Port 5559 is busy, please choose a free port and specify it using -port option

with 3.11 looks fine:

13:25:23.281 INFO [SeleniumServer.boot] - Welcome to Selenium for Workgroups....
13:25:23.282 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 5559
13:25:23.282 INFO [GridLauncherV3$3.launch] - Selenium Grid node is up and ready to register to the hub
13:25:23.299 INFO [SelfRegisteringRemote$1.run] - Starting auto registration thread. Will try to register every 5000 ms.
13:25:23.302 INFO [SelfRegisteringRemote.registerToHub] - Registering the node to the hub: http://jenkins.domain.tld/:4445/grid/register
Failed to read: session.screen0.slit.acceptKdeDockapps
Setting default value
Failed to read: session.screen0.slit.autoHide
Setting default value
Failed to read: session.screen0.slit.maxOver
Setting default value
Failed to read: session.screen0.slit.placement
Setting default value
Failed to read: session.screen0.slit.alpha
Setting default value
Failed to read: session.screen0.slit.onhead
Setting default value
Failed to read: session.screen0.slit.layer
Setting default value
Failed to read: session.screen0.toolbar.autoHide
Setting default value
Failed to read: session.screen0.toolbar.maxOver
Setting default value
Failed to read: session.screen0.toolbar.visible
Setting default value
Failed to read: session.screen0.toolbar.alpha
Setting default value
Failed to read: session.screen0.toolbar.layer
Setting default value
Failed to read: session.screen0.toolbar.onhead
Setting default value
Failed to read: session.screen0.toolbar.placement
Setting default value
Failed to read: session.screen0.toolbar.height
Setting default value
Failed to read: session.screen0.iconbar.mode
Setting default value
Failed to read: session.screen0.iconbar.alignment
Setting default value
Failed to read: session.screen0.iconbar.iconWidth
Setting default value
Failed to read: session.screen0.iconbar.iconTextPadding
Setting default value
Failed to read: session.screen0.iconbar.usePixmap
Setting default value
13:25:24.259 INFO [SelfRegisteringRemote.registerToHub] - Updating the node configuration from the hub

k8s deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-node-chrome
  namespace: qa
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: selenium-node-chrome
  template:
    metadata:
      labels:
        app: selenium-node-chrome
        pod: chrome
    spec:
      affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: environment
               operator: In
               values:
               - qa
      containers:
      - name: selenium-node-chrome
        image: selenium/node-chrome-debug:3.11
        ports:
          - containerPort: 5900
            name: vnc
          - containerPort: 5559
            name: node
env:
          - name: HUB_PORT_4444_TCP_ADDR
            value: jenkins.domain.tld
          - name: HUB_PORT_4444_TCP_PORT
            value: "4445"
          - name: NODE_PORT
            value: "5559"
          - name: SE_OPTS
            value: -host myhost.tld
        resources:
          limits:
            memory: "2048Mi"
        volumeMounts:
        - name: vncpasswd
          mountPath: /home/seluser/.vnc/passwd
          subPath: passwd
      volumes:
        - name: vncpasswd
          secret:
            secretName: vncpasswd

[ghost]

Seeing the same issue on Docker node (both Chrome & Firefox) as @dober

root@e8fcd13aa9ff:~# cat /opt/selenium/config.json
{
  "capabilities": [
    {
      "version": "61.0.1",
      "browserName": "firefox",
      "maxInstances": 1,
      "seleniumProtocol": "WebDriver",
      "applicationName": "",
      "moz:firefoxOptions":
        {
          "log":
            {
              "level": "info"
            }
        }
    }
  ],
  "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
  "maxSession": 1,
  "host": "10.10.10.3",
  "port": 12345,
  "register": true,
  "registerCycle": 5000,
  "nodePolling": 5000,
  "unregisterIfStillDownAfter": 60000,
  "downPollingLimit": 2
}

root@e8fcd13aa9ff:~# ss --tcp --numeric --listening                                                                                                                                                                                                                              
State       Recv-Q Send-Q                                                                                           Local Address:Port                                                                                                          Peer Address:Port
              
root@e8fcd13aa9ff:~# java -jar /opt/selenium/selenium-server-standalone.jar -role node -hub http://10.10.10.4:4444/grid/register -nodeConfig /opt/selenium/config.json                                                                                                         
23:50:17.142 INFO [GridLauncherV3.launch] - Selenium build info: version: '3.12.0', revision: '7c6e0b3'
23:50:17.144 INFO [GridLauncherV3$3.launch] - Launching a Selenium Grid node on port 12345
2018-08-06 23:50:17.215:INFO::main: Logging initialized @310ms to org.seleniumhq.jetty9.util.log.StdErrLog
23:50:17.361 ERROR [SeleniumServer.boot] - Port 12345 is busy, please choose a free port and specify it using -port option

Sadly the same issue is present on 3.13 and 3.14, however I can confirm it was working as expected on 3.11.

[dober]

any news?

[diemol]

Hi all,

Yes, this was a change in the upstream project where all the downstream projects had to adjust. The reasons why the change was done are out of my reach, but in the end it was done and here in docker-selenium we had to adjust the "host" parameter in the hub as well.

The original issue was the port being used when starting the hub, and the Jenkins project as a downstream project needs to adapt as well.

The other posts are related to nodes which are not related to the original issue. Would it be possible to have separate issues that mention only those ones with clear ways to reproduce them? For that, it will be ideal if you create a new issue with the template completely filled.

[diemol]

Closing as we didn't get replies to the last comment.

For the comments related to nodes, feel free to open a new issue with all the needed details to reproduce any potential problem.

[PriyaSoni164]

Can try in this way it worked for me :
java -jar selenium-server-standalone-3.3.1.jar -port 4455 -role hub