Make Jetty and Jetty compat do log rotations

[ludoch]

We do not want to fill the VM disk space with logs.
The environment providing the local logs to the console supports log rotation.
Just need to verify we do it.

[gregw]

The request log is setup in jetty9 by gae.xml and in jetty9-compat by a similar gae.xml

This indicates we are rolling the logs daily and that we keep the logs for 2 days.

Note that the pure jetty request log does not have a size based log rotation, so we could still fill up the disk. However, we have the ability to plugin something like logback which I believe does have this capability. @joakime can you confirm that and perhaps provide a PR to change the configuration? @ludoch please say if you want size based rotation

[ludoch]

I think 2 days is too long (there is another agent looking at the /var/log/appengine_logs area and it can catch up to 5 or 10 minutes with rotation..
Logback: can it be configure by time and size, i.e rotate every 30mins or when the size is bigger than xMb? That would be ideal.

[gregw]

@joakime also it would be good to filter out the routine health check requests

[joakime]

With logback, you can do this .

See http://logback.qos.ch/manual/appenders.html#RollingFileAppender

There's 2 ways to do this ...

Using separate rolling & triggering policies:

What you'll setup is a rollingPolicy which is based on time.
and a triggeringPolicy based on file size.

Something like this ...

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>gae.log</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <!-- daily rollover -->
      <fileNamePattern>gae.%d{yyyy-MM-dd}.log</fileNamePattern>

      <!-- keep 30 days' worth of history capped at 3GB total size -->
      <maxHistory>30</maxHistory>
      <totalSizeCap>3GB</totalSizeCap>
    </rollingPolicy>

    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>30MB</maxFileSize>
    </triggeringPolicy>

    <encoder>
      <pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
    </encoder>
  </appender>

  <root level="DEBUG">
    <appender-ref ref="FILE" />
  </root>
</configuration>

Using time & size based rolling policy only:

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>gae.log</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <!-- rollover daily -->
      <fileNamePattern>gae-%d{yyyy-MM-dd}.%i.txt</fileNamePattern>
      <!-- each file should be at most 30MB, keep 60 days worth of history, but at most 3GB -->
      <maxFileSize>30MB</maxFileSize>    
      <maxHistory>60</maxHistory>
      <totalSizeCap>3GB</totalSizeCap>
    </rollingPolicy>

    <encoder>
      <pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
    </encoder>
  </appender>

  <root level="DEBUG">
    <appender-ref ref="FILE" />
  </root>
</configuration>

I've personally never setup a log to roll as aggressively as every 30 minutes, so I'll need to dig some more to figure that aspect out. (Hourly is probably already supported by the built-in policies)

[aozarov]

@joakime also it would be good to filter out the routine health check requests

+1 however I don't think we should completely filter them out. Instead we should either filter out
only the ones that return 200 or better yet change the log level of the ones that return 200 to debug
and if possible change the failed ones to warning.

[janbartel]

@joakime could you also please document how to make logging work when running locally from jetty9-compat-base: doesn't seem to notice a logging.properties file in the webapp's WEB-INF dir.

[joakime]

Wait, the webapp's WEB-INF dir?

Do we want to have the webapp's logging punch a hole through the WebApp isolation layers? (ie: WebAppClassloader)

I would strongly discourage this behavior, as there are existing webapps that people deploy that do not like this (hudson, jenkins, and nexus are some obvious examples)

This issue has been about server side logging, and that's the way I've been progressing, throwing forced webapp logging into the mix should be a separate issue, with a separate PR (to make the eventual revert easier).

[gregw]

@joakime The GAE deployment unit is the war file, so any configuration uploaded for deployment is in the war file. So it's not necessarily punching a hole in the classloader, rather it is just like how WEB-INF/appengine-web.xml is used to configure the container. It is documented here.

So as this is established practise, I do not believe we can change that. It doesn't need a hole punched in the classloader and the VmRuntimeWebAppContext actively seeks the logging.properties file from the webapp and uses it to amend the configuration of the containers logger. So it really is a container configurations that is just deployed via the webapp.

The issue Jan is having is that it does not appear to work when a webapp is run locally in the jetty-9-compat-base. So when you review the logging, can you also investigate why it doesn't work locally (specially as that will be useful for you to test any changes that you make).