Skip to content

Commit c5b08f6

Browse files
zeripathtechknowlogicklafriks
authored
Pause, Resume, Release&Reopen, Add and Remove Logging from command line (#11777)
* Make LogDescriptions race safe * Add manager commands for pausing, resuming, adding and removing loggers Signed-off-by: Andrew Thornton <[email protected]> * Placate lint * Ensure that file logger is run! * Add support for smtp and conn Signed-off-by: Andrew Thornton <[email protected]> * Add release-and-reopen Signed-off-by: Andrew Thornton <[email protected]> Co-authored-by: techknowlogick <[email protected]> Co-authored-by: Lauris BH <[email protected]>
1 parent 38fb087 commit c5b08f6

File tree

17 files changed

+924
-17
lines changed

17 files changed

+924
-17
lines changed

cmd/manager.go

Lines changed: 371 additions & 9 deletions
Large diffs are not rendered by default.

docs/content/doc/advanced/logging-documentation.en-us.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,28 @@ COLORIZE = true # Or false if your windows terminal cannot color
316316

317317
This is equivalent to sending all logs to the console, with default go log being sent to the console log too.
318318

319+
## Releasing-and-Reopening, Pausing and Resuming logging
320+
321+
If you are running on Unix you may wish to release-and-reopen logs in order to use `logrotate` or other tools.
322+
It is possible force gitea to release and reopen it's logging files and connections by sending `SIGUSR1` to the
323+
running process, or running `gitea manager logging release-and-reopen`.
324+
325+
Alternatively, you may wish to pause and resume logging - this can be accomplished through the use of the
326+
`gitea manager logging pause` and `gitea manager logging resume` commands. Please note that whilst logging
327+
is paused log events below INFO level will not be stored and only a limited number of events will be stored.
328+
Logging may block, albeit temporarily, slowing gitea considerably whilst paused - therefore it is
329+
recommended that pausing only done for a very short period of time.
330+
331+
## Adding and removing logging whilst Gitea is running
332+
333+
It is possible to add and remove logging whilst Gitea is running using the `gitea manager logging add` and `remove` subcommands.
334+
This functionality can only adjust running log systems and cannot be used to start the access, macaron or router loggers if they
335+
were not already initialised. If you wish to start these systems you are advised to adjust the app.ini and (gracefully) restart
336+
the Gitea service.
337+
338+
The main intention of these commands is to easily add a temporary logger to investigate problems on running systems where a restart
339+
may cause the issue to disappear.
340+
319341
## Log colorization
320342

321343
Logs to the console will be colorized by default when not running on

docs/content/doc/usage/command-line.en-us.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,3 +318,85 @@ var checklist = []check{
318318
```
319319

320320
This function will receive a command line context and return a list of details about the problems or error.
321+
322+
#### manager
323+
324+
Manage running server operations:
325+
326+
- Commands:
327+
- `shutdown`: Gracefully shutdown the running process
328+
- `restart`: Gracefully restart the running process - (not implemented for windows servers)
329+
- `flush-queues`: Flush queues in the running process
330+
- Options:
331+
- `--timeout value`: Timeout for the flushing process (default: 1m0s)
332+
- `--non-blocking`: Set to true to not wait for flush to complete before returning
333+
- `logging`: Adjust logging commands
334+
- Commands:
335+
- `pause`: Pause logging
336+
- Notes:
337+
- The logging level will be raised to INFO temporarily if it is below this level.
338+
- Gitea will buffer logs up to a certain point and will drop them after that point.
339+
- `resume`: Resume logging
340+
- `release-and-reopen`: Cause Gitea to release and re-open files and connections used for logging (Equivalent to sending SIGUSR1 to Gitea.)
341+
- `remove name`: Remove the named logger
342+
- Options:
343+
- `--group group`, `-g group`: Set the group to remove the sublogger from. (defaults to `default`)
344+
- `add`: Add a logger
345+
- Commands:
346+
- `console`: Add a console logger
347+
- Options:
348+
- `--group value`, `-g value`: Group to add logger to - will default to "default"
349+
- `--name value`, `-n value`: Name of the new logger - will default to mode
350+
- `--level value`, `-l value`: Logging level for the new logger
351+
- `--stacktrace-level value`, `-L value`: Stacktrace logging level
352+
- `--flags value`, `-F value`: Flags for the logger
353+
- `--expression value`, `-e value`: Matching expression for the logger
354+
- `--prefix value`, `-p value`: Prefix for the logger
355+
- `--color`: Use color in the logs
356+
- `--stderr`: Output console logs to stderr - only relevant for console
357+
- `file`: Add a file logger
358+
- Options:
359+
- `--group value`, `-g value`: Group to add logger to - will default to "default"
360+
- `--name value`, `-n value`: Name of the new logger - will default to mode
361+
- `--level value`, `-l value`: Logging level for the new logger
362+
- `--stacktrace-level value`, `-L value`: Stacktrace logging level
363+
- `--flags value`, `-F value`: Flags for the logger
364+
- `--expression value`, `-e value`: Matching expression for the logger
365+
- `--prefix value`, `-p value`: Prefix for the logger
366+
- `--color`: Use color in the logs
367+
- `--filename value`, `-f value`: Filename for the logger -
368+
- `--rotate`, `-r`: Rotate logs
369+
- `--max-size value`, `-s value`: Maximum size in bytes before rotation
370+
- `--daily`, `-d`: Rotate logs daily
371+
- `--max-days value`, `-D value`: Maximum number of daily logs to keep
372+
- `--compress`, `-z`: Compress rotated logs
373+
- `--compression-level value`, `-Z value`: Compression level to use
374+
- `conn`: Add a network connection logger
375+
- Options:
376+
- `--group value`, `-g value`: Group to add logger to - will default to "default"
377+
- `--name value`, `-n value`: Name of the new logger - will default to mode
378+
- `--level value`, `-l value`: Logging level for the new logger
379+
- `--stacktrace-level value`, `-L value`: Stacktrace logging level
380+
- `--flags value`, `-F value`: Flags for the logger
381+
- `--expression value`, `-e value`: Matching expression for the logger
382+
- `--prefix value`, `-p value`: Prefix for the logger
383+
- `--color`: Use color in the logs
384+
- `--reconnect-on-message`, `-R`: Reconnect to host for every message
385+
- `--reconnect`, `-r`: Reconnect to host when connection is dropped
386+
- `--protocol value`, `-P value`: Set protocol to use: tcp, unix, or udp (defaults to tcp)
387+
- `--address value`, `-a value`: Host address and port to connect to (defaults to :7020)
388+
- `smtp`: Add an SMTP logger
389+
- Options:
390+
- `--group value`, `-g value`: Group to add logger to - will default to "default"
391+
- `--name value`, `-n value`: Name of the new logger - will default to mode
392+
- `--level value`, `-l value`: Logging level for the new logger
393+
- `--stacktrace-level value`, `-L value`: Stacktrace logging level
394+
- `--flags value`, `-F value`: Flags for the logger
395+
- `--expression value`, `-e value`: Matching expression for the logger
396+
- `--prefix value`, `-p value`: Prefix for the logger
397+
- `--color`: Use color in the logs
398+
- `--username value`, `-u value`: Mail server username
399+
- `--password value`, `-P value`: Mail server password
400+
- `--host value`, `-H value`: Mail server host (defaults to: 127.0.0.1:25)
401+
- `--send-to value`, `-s value`: Email address(es) to send to
402+
- `--subject value`, `-S value`: Subject header of sent emails

integrations/testlogger.go

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,11 @@ func (log *TestLogger) Init(config string) error {
170170
func (log *TestLogger) Flush() {
171171
}
172172

173+
//ReleaseReopen does nothing
174+
func (log *TestLogger) ReleaseReopen() error {
175+
return nil
176+
}
177+
173178
// GetName returns the default name for this implementation
174179
func (log *TestLogger) GetName() string {
175180
return "test"

modules/graceful/manager_unix.go

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,10 @@ func (g *Manager) handleSignals(ctx context.Context) {
113113
log.Info("PID: %d. Received SIGHUP. Attempting GracefulRestart...", pid)
114114
g.DoGracefulRestart()
115115
case syscall.SIGUSR1:
116-
log.Info("PID %d. Received SIGUSR1.", pid)
116+
log.Warn("PID %d. Received SIGUSR1. Releasing and reopening logs", pid)
117+
if err := log.ReleaseReopen(); err != nil {
118+
log.Error("Error whilst releasing and reopening logs: %v", err)
119+
}
117120
case syscall.SIGUSR2:
118121
log.Warn("PID %d. Received SIGUSR2. Hammering...", pid)
119122
g.DoImmediateHammer()

modules/log/conn.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,13 @@ func (i *connWriter) connect() error {
7777
return nil
7878
}
7979

80+
func (i *connWriter) releaseReopen() error {
81+
if i.innerWriter != nil {
82+
return i.connect()
83+
}
84+
return nil
85+
}
86+
8087
// ConnLogger implements LoggerProvider.
8188
// it writes messages in keep-live tcp connection.
8289
type ConnLogger struct {
@@ -119,6 +126,11 @@ func (log *ConnLogger) GetName() string {
119126
return "conn"
120127
}
121128

129+
// ReleaseReopen causes the ConnLogger to reconnect to the server
130+
func (log *ConnLogger) ReleaseReopen() error {
131+
return log.out.(*connWriter).releaseReopen()
132+
}
133+
122134
func init() {
123135
Register("conn", NewConn)
124136
}

modules/log/console.go

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,20 @@ func (log *ConsoleLogger) Init(config string) error {
6868
func (log *ConsoleLogger) Flush() {
6969
}
7070

71+
// ReleaseReopen causes the console logger to reconnect to os.Stdout
72+
func (log *ConsoleLogger) ReleaseReopen() error {
73+
if log.Stderr {
74+
log.NewWriterLogger(&nopWriteCloser{
75+
w: os.Stderr,
76+
})
77+
} else {
78+
log.NewWriterLogger(&nopWriteCloser{
79+
w: os.Stdout,
80+
})
81+
}
82+
return nil
83+
}
84+
7185
// GetName returns the default name for this implementation
7286
func (log *ConsoleLogger) GetName() string {
7387
return "console"

modules/log/event.go

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ type EventLogger interface {
2929
GetLevel() Level
3030
GetStacktraceLevel() Level
3131
GetName() string
32+
ReleaseReopen() error
3233
}
3334

3435
// ChannelledLog represents a cached channel to a LoggerProvider
@@ -117,6 +118,11 @@ func (l *ChannelledLog) Flush() {
117118
l.flush <- true
118119
}
119120

121+
// ReleaseReopen this ChannelledLog
122+
func (l *ChannelledLog) ReleaseReopen() error {
123+
return l.loggerProvider.ReleaseReopen()
124+
}
125+
120126
// GetLevel gets the level of this ChannelledLog
121127
func (l *ChannelledLog) GetLevel() Level {
122128
return l.loggerProvider.GetLevel()
@@ -145,6 +151,7 @@ type MultiChannelledLog struct {
145151
level Level
146152
stacktraceLevel Level
147153
closed chan bool
154+
paused chan bool
148155
}
149156

150157
// NewMultiChannelledLog a new logger instance with given logger provider and config.
@@ -159,6 +166,7 @@ func NewMultiChannelledLog(name string, bufferLength int64) *MultiChannelledLog
159166
stacktraceLevel: NONE,
160167
close: make(chan bool),
161168
closed: make(chan bool),
169+
paused: make(chan bool),
162170
}
163171
return m
164172
}
@@ -229,6 +237,33 @@ func (m *MultiChannelledLog) closeLoggers() {
229237
m.closed <- true
230238
}
231239

240+
// Pause pauses this Logger
241+
func (m *MultiChannelledLog) Pause() {
242+
m.paused <- true
243+
}
244+
245+
// Resume resumes this Logger
246+
func (m *MultiChannelledLog) Resume() {
247+
m.paused <- false
248+
}
249+
250+
// ReleaseReopen causes this logger to tell its subloggers to release and reopen
251+
func (m *MultiChannelledLog) ReleaseReopen() error {
252+
m.mutex.Lock()
253+
defer m.mutex.Unlock()
254+
var accumulatedErr error
255+
for _, logger := range m.loggers {
256+
if err := logger.ReleaseReopen(); err != nil {
257+
if accumulatedErr == nil {
258+
accumulatedErr = fmt.Errorf("Error whilst reopening: %s Error: %v", logger.GetName(), err)
259+
} else {
260+
accumulatedErr = fmt.Errorf("Error whilst reopening: %s Error: %v & %v", logger.GetName(), err, accumulatedErr)
261+
}
262+
}
263+
}
264+
return accumulatedErr
265+
}
266+
232267
// Start processing the MultiChannelledLog
233268
func (m *MultiChannelledLog) Start() {
234269
m.mutex.Lock()
@@ -238,8 +273,35 @@ func (m *MultiChannelledLog) Start() {
238273
}
239274
m.started = true
240275
m.mutex.Unlock()
276+
paused := false
241277
for {
278+
if paused {
279+
select {
280+
case paused = <-m.paused:
281+
if !paused {
282+
m.ResetLevel()
283+
}
284+
case _, ok := <-m.flush:
285+
if !ok {
286+
m.closeLoggers()
287+
return
288+
}
289+
m.mutex.Lock()
290+
for _, logger := range m.loggers {
291+
logger.Flush()
292+
}
293+
m.mutex.Unlock()
294+
case <-m.close:
295+
m.closeLoggers()
296+
return
297+
}
298+
continue
299+
}
242300
select {
301+
case paused = <-m.paused:
302+
if paused && m.level < INFO {
303+
m.level = INFO
304+
}
243305
case event, ok := <-m.queue:
244306
if !ok {
245307
m.closeLoggers()
@@ -275,7 +337,7 @@ func (m *MultiChannelledLog) LogEvent(event *Event) error {
275337
select {
276338
case m.queue <- event:
277339
return nil
278-
case <-time.After(60 * time.Second):
340+
case <-time.After(100 * time.Millisecond):
279341
// We're blocked!
280342
return ErrTimeout{
281343
Name: m.name,

modules/log/file.go

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,19 @@ func (log *FileLogger) Flush() {
249249
_ = log.mw.fd.Sync()
250250
}
251251

252+
// ReleaseReopen releases and reopens log files
253+
func (log *FileLogger) ReleaseReopen() error {
254+
closingErr := log.mw.fd.Close()
255+
startingErr := log.StartLogger()
256+
if startingErr != nil {
257+
if closingErr != nil {
258+
return fmt.Errorf("Error during closing: %v Error during starting: %v", closingErr, startingErr)
259+
}
260+
return startingErr
261+
}
262+
return closingErr
263+
}
264+
252265
// GetName returns the default name for this implementation
253266
func (log *FileLogger) GetName() string {
254267
return "file"

modules/log/log.go

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
package log
66

77
import (
8+
"fmt"
89
"os"
910
"runtime"
1011
"strings"
@@ -192,6 +193,42 @@ func IsFatal() bool {
192193
return GetLevel() <= FATAL
193194
}
194195

196+
// Pause pauses all the loggers
197+
func Pause() {
198+
NamedLoggers.Range(func(key, value interface{}) bool {
199+
logger := value.(*Logger)
200+
logger.Pause()
201+
logger.Flush()
202+
return true
203+
})
204+
}
205+
206+
// Resume resumes all the loggers
207+
func Resume() {
208+
NamedLoggers.Range(func(key, value interface{}) bool {
209+
logger := value.(*Logger)
210+
logger.Resume()
211+
return true
212+
})
213+
}
214+
215+
// ReleaseReopen releases and reopens logging files
216+
func ReleaseReopen() error {
217+
var accumulatedErr error
218+
NamedLoggers.Range(func(key, value interface{}) bool {
219+
logger := value.(*Logger)
220+
if err := logger.ReleaseReopen(); err != nil {
221+
if accumulatedErr == nil {
222+
accumulatedErr = fmt.Errorf("Error reopening %s: %v", key.(string), err)
223+
} else {
224+
accumulatedErr = fmt.Errorf("Error reopening %s: %v & %v", key.(string), err, accumulatedErr)
225+
}
226+
}
227+
return true
228+
})
229+
return accumulatedErr
230+
}
231+
195232
// Close closes all the loggers
196233
func Close() {
197234
l, ok := NamedLoggers.Load(DEFAULT)

modules/log/smtp.go

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,11 @@ func (log *SMTPLogger) sendMail(p []byte) (int, error) {
9797
func (log *SMTPLogger) Flush() {
9898
}
9999

100+
// ReleaseReopen does nothing
101+
func (log *SMTPLogger) ReleaseReopen() error {
102+
return nil
103+
}
104+
100105
// GetName returns the default name for this implementation
101106
func (log *SMTPLogger) GetName() string {
102107
return "smtp"

0 commit comments

Comments
 (0)