Skip to content

Enable defining a handler for buffer overflow when using the library as a python logging handler #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 30, 2017
Merged

Enable defining a handler for buffer overflow when using the library as a python logging handler #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3ab3aa2
Enable defining a handler for buffer overflow when using the library …
montaro Jan 26, 2017
dcfee05
logging-interface-handle-buffer-overflow Updated the README to includ…
montaro Jan 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
/.tox
/build
/dist
.idea/
15 changes: 14 additions & 1 deletion README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ module.

logging.basicConfig(level=logging.INFO)
l = logging.getLogger('fluent.test')
h = handler.FluentHandler('app.follow', host='host', port=24224)
h = handler.FluentHandler('app.follow', host='host', port=24224, buffer_overflow_handler=handler)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add handler code like Sender example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

Copy link
Contributor

@yoichi yoichi May 9, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example seems incorrect since handler is a module (from fluent import handler). buffer_overflow_handler should be a callable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sent pull request to avoid variable name confliction by changing variable name in example:
#87

formatter = handler.FluentRecordFormatter(custom_format)
h.setFormatter(formatter)
l.addHandler(h)
Expand All @@ -211,6 +211,18 @@ You can also customize formatter via logging.config.dictConfig

logging.config.dictConfig(conf['logging'])

You can inject your own custom proc to handle buffer overflow in the event of connection failure. This will mitigate the loss of data instead of simply throwing data away.

.. code:: python

import msgpack
from io import BytesIO

def handler(pendings):
unpacker = msgpack.Unpacker(BytesIO(pendings))
for unpacked in unpacker:
print(unpacked)

A sample configuration ``logging.yaml`` would be:

.. code:: python
Expand Down Expand Up @@ -242,6 +254,7 @@ A sample configuration ``logging.yaml`` would be:
host: localhost
port: 24224
tag: test.logging
buffer_overflow_handler: handler
formatter: fluent_fmt
level: DEBUG
none:
Expand Down
6 changes: 4 additions & 2 deletions fluent/handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,14 @@ def __init__(self,
host='localhost',
port=24224,
timeout=3.0,
verbose=False):
verbose=False,
buffer_overflow_handler=None):

self.tag = tag
self.sender = sender.FluentSender(tag,
host=host, port=port,
timeout=timeout, verbose=verbose)
timeout=timeout, verbose=verbose,
buffer_overflow_handler=buffer_overflow_handler)
logging.Handler.__init__(self)

def emit(self, record):
Expand Down