Jetty does not fail to start when its thread pool is misconfigured

[venkata6]

Dear Spring Boot maintainers,

I think I found a bug in the JettyWebSocketContainerCustomizer configuration

I wrongly configured jetty.threadPool.maxThreads to a value which is less than optimal which should throw an error and not started the server at all. But Spring Boot ignores the error from Jetty and starts the server.

Then when the request comes , TCP/IP connection succeeds but there is no response.

Initially I thought it is Jetty issue but when I run Jetty separately and configure the same values ./etc/jetty.xml , Jetty fails to start the server

So my conclusion is Spring Boot ignores the underlying Jetty exception and starts the server (which could cause serious consequences if one deploys the code where we don't have any control over number of cores)

More detailed in my original ticket here to Jetty team

jetty/jetty.project#1464 (comment)

[philwebb]

Here's the stack trace from the Jetty issue.

java.lang.IllegalStateException: Insufficient threads: max=8 < needed(acceptors=1 + selectors=8 + request=1)
at org.eclipse.jetty.server.Server.doStart(Server.java:414)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1511)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1438)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1437)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:222)
at org.eclipse.jetty.start.Main.start(Main.java:486)
at org.eclipse.jetty.start.Main.main(Main.java:79)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:222)
at org.eclipse.jetty.start.Main.start(Main.java:486)
at org.eclipse.jetty.start.Main.main(Main.java:79)
Caused by: java.lang.IllegalStateException: Insufficient threads: max=8 < needed(acceptors=1 + selectors=8 + request=1)
at org.eclipse.jetty.server.Server.doStart(Server.java:414)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1511)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1438)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1437)
... 7 more
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:222)
at org.eclipse.jetty.start.Main.start(Main.java:486)
at org.eclipse.jetty.start.Main.main(Main.java:79)
Caused by: java.lang.IllegalStateException: Insufficient threads: max=8 < needed(acceptors=1 + selectors=8 + request=1)
at org.eclipse.jetty.server.Server.doStart(Server.java:414)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1511)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1438)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1437)

[philwebb]

@venkata6 Please can you provide a sample application that shows the problem. I'm not totally following how JettyWebSocketContainerCustomizer relates to the issue and it's not clear to me how you are attempting to customize Jetty.

[venkata6]

@philwebb , my bad I was not clear.

We use SpringBoot for our REST APIs. In our project we ( wrongly) configured the jetty maxthreads as below code fragment. When we do that, Jetty server starts , accepts TCP/IP connection but there is no response OR error. Please note this creates a JAR file and we start the server like " java -jar "jar file name".

So initially I thought it is a Jetty issue. But it turned out it is not. How did I verify that ?

1. First I created a WAR file of the same project . 2. Downloaded Jetty ( same version ) and ran as standalone server. 3. Deploy the WAR file in that server 4. Modify the same maxthreads but now in the ./etc/jetty.xml . 5. But now the jetty server (CORRECTLY) fails to start with the above stack trace.

So why I am not getting that error when I use the below code fragment and use the embedded Jetty server. Am I missing something here ?

public class JettyConfiguration extends JettyWebSocketContainerCustomizer{
    private final Logger log = LogManager.getLogger(JettyConfiguration.class);
    @Value("${jetty.threadPool.maxThreads:8}")
    private int maxThreads;
    @Value("${jetty.threadPool.minThreads:8}")
    private int minThreads;
    @Value("${jetty.threadPool.idleTimeout:60000}")
    private int idleTimeout;
    @Override
    protected void doCustomize(JettyEmbeddedServletContainerFactory container) {
        log.info("Setting Jetty ThreadPool: max {}, min {}, idle timeout {}", maxThreads, minThreads, idleTimeout);
        super.doCustomize(container);
        container.addServerCustomizers((server) -> {
            // Tweak the connection pool used by Jetty to handle incoming HTTP connections
            QueuedThreadPool threadPool = server.getBean(QueuedThreadPool.class);
            threadPool.setMaxThreads(maxThreads); // THIS IS THE CULPRIT LINE , removing this LINE make the server respond normally 
            threadPool.setMinThreads(minThreads);
            threadPool.setIdleTimeout(idleTimeout);
        });
    }
}

This is serious issue as if we deploy in prod env like AWS where we don't have control over "number of cores", this code would work in local boxes and stop working in prod and it is very hard to find the root cause

[philwebb]

@venkata6 What are your jetty.threadPool.maxThreads, minThreads & idleTimeout properties set to?

[venkata6]

in the code fragment above these are the lines , in a 4 core which acts as 8 core in the hyperthreading , we wrongly set to 8

threadPool.setMaxThreads(maxThreads); // THIS IS THE CULPRIT LINE , removing this LINE make the server respond normally
threadPool.setMinThreads(minThreads);
threadPool.setIdleTimeout(idleTimeout);

[venkata6]

all values are same as the default values above

[philwebb]

I mean what is the value of minThreads that gets passed to that method. You're reading it using @Value("${jetty.threadPool.minThreads:8}") but I don't know what's in your application.properties

[philwebb]

OK, so 8/8/60000?

[venkata6]

yes that is correct