Ensure two or more plugin instances don't select the same sincedb, causing data loss

[jsvd]

Currently in the documentation we warn that, when using more than one file { } section, you need different sincedb paths (doc here).

However, we should enforce this somehow to prevent silent data loss, because the same sincedb file will be updated by different instances, causing each plugin to potentially backtrack or skip ahead.

To ensure no other plugin instance or event logstash instance uses the same sincedb file, a file lock could be used.
To guarantee the exclusivity just at the logstash instance level we could create some threadsafe class level metadata that would be checked at register time.

[nellicus]

thanks @jsvd !!

[jordansissel]

This problem should only exist if users are specifying the sincedb path themselves. The default (where no sincedb path is set) will likely avoid collision with other file inputs. I highly recommend using the default behavior of this setting (do not set it). This comment should not be taken to mean that I am against using file locks. But further, file locks are insufficient because two file inputs in the same process will not be prevented by file locks from using the same file.

…

On Fri, Dec 23, 2016 at 3:01 AM Antonio Bonuccelli ***@***.***> wrote: thanks @jsvd <https://github.com/jsvd> !! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#150 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6iP3tcQwIz8EV_mJcjtqucO1hO-Xks5rK6oJgaJpZM4LUvOy> .

[jsvd]

@jordansissel agreed this is not a very common scenario. I wonder if a simple "hey I see you set sincedb path, be careful because..."

And yes, good point wrt to flock. Then I guess the only "100% guarantee" against this problem is having BOTH flock and class level checking.

[jordansissel]

@jsvd +1 to having a warning and using flock (and later, maybe, some shared structure in Logstash to fail startup if conflicts are found)