inefficiency caused by double event creation and extra field lookup

[colinsurprenant]

the current codec implementation leverages the lines codec for doing the line parsing. The problem is that the line codec decode method is called which produces and event and then a new event is created using the json deserialization of the previous event message field.

This means that for each input line, and extra and superfluous event is created and an extra event field lookup is done.

Since the line parsing logic is so simple, we just should directly use the buffered tokenizer here and not rely on the line codec.

[jordansissel]

+1 your proposal.

On Friday, January 29, 2016, Colin Surprenant notifications@github.com
wrote:

the current codec implementation leverages the lines codec for doing
the line parsing. The problem is that the line codec decode method is
called

logstash-codec-json_lines/lib/logstash/codecs/json_lines.rb

Line 40 in 1eb838d

@lines.decode(data) do |event|

which produces and event and then a new event is created using the json
deserialization of the previous event message field

logstash-codec-json_lines/lib/logstash/codecs/json_lines.rb

Line 55 in 1eb838d

LogStash::Event.new(LogStash::Json.load(event["message"]))

.

This means that for each input line, and extra and superfluous event is
created and an extra event field lookup is done.

Since the line parsing logic is so simple, we just should directly using
the buffered tokenizer here and not rely on the line codec.

—
Reply to this email directly or view it on GitHub
#17.

[guyboertje]

Are we handling remainder flushing on close at all - in the current impl?

[jsvd]

I made some arguments against this through another channel on the overall choice of buftok vs line codec:

implementation-wise I dont know if using the line codec makes the code more complex, which is my only concern on my suggestion (to use codec line instead of buftok)

That aside I see multiple benefits:
* reusing code
* one less place we have to maintain the contract of an external dependency
* using the codec this way helps us reinforce what one day might become the line tokenizer

Also, the performance gain is unknown and, for me, doesn't justify hardwiring buftok in json_lines (and multiline).

That said, for coherence sake and dismissing the performance argument, I'm +1 on going with this proposal if logstash-plugins/logstash-codec-multiline#26 is accepted

[colinsurprenant]

given the simplicity of this implementation (without the boilerplate, literally 1 line @buffer.extract(data).each do |line| ) I think DRYness is not really a concern and I'd rather opt for performance.

[guyboertje]

+1 to use buftok and to fix the "flush if remainder" problem.

[guyboertje]

+1 to improve performance too.

[guyboertje]

Here too is a flavour mismatch if json_lines is used with the file input, and because there is no flush one gets absolutely no events generated at all, not even incorrect ones.

[colinsurprenant]

@guyboertje codec flush is meant as a final flush, why events are not generated when used with file input?

[suyograo]

Fixed in #18

[guyboertje]

@colinsurprenant

codec flush is meant as a final flush, why events are not generated when used with file input?

Because json_lines does not implement a flush method! But on reflection, unless the remainder is a parseable JSON snippet there is not much point, but maybe an event with a parse_failure is better than nothing at all.