Need for recursive watcher

[jbowtie]

I've had to implement this several times now, and I really think that a recursive option should be part of the core API because it really is needed in many scenarios.

My usual implementation is to use filepath.Walk on the root directory and call watcher.Watch on every directory; I think it's not obvious and I also have to remember to handle directories specially on a Create event so that new directories are also watched.

I'd be happy to create a pull request for this functionality if there's a chance that you'll accept it. If you don't want to add such an option, perhaps we could provide an example that shows how to do a recursive watch?

[pmonks]

+1
It appears that this kind of thing is possible natively for most OSes - see https://github.com/gorakhargosh/watchdog/ for example.

[nathany]

@jbowtie I also had to create a recursive watcher with the same characteristics for Looper. I am currently working to extract the code into a reusable package, along with adding options for Throttling, filtering by Pattern and logging to help in debugging. I would be grateful for some code review once it's further along.

[jbowtie]

@nathany I'd be happy to review it when ready.

Regarding API changes, I would imagine the only user-facing change needs to be a boolean option for recursion. With implementation we can either go the platform-agnostic approach or modify the per-platform API calls. Since @howeyc hasn't indicated a preferred approach we should probably just go ahead and prepare pull requests for review.

[nathany]

I'm going to aim for a platform agnostic approach first, given that's the code I have on hand, while keeping in mind that it should be possible override with platform-specific optimizations.

I'm spiking ideas for the public API here but the code itself is a mess. Working on cleaning it up tonight.

[howeyc]

Thanks for working on this, I appreciate it.

Send pull requests whenever you think it's ready.
On Sep 3, 2013 6:23 PM, "Nathan Youngman" notifications@github.com wrote:

I'm going to aim for a platform agnostic approach first, given that's the
code I have on hand, while keeping in mind that it should be possible
override with platform-specific optimizations.

I'm spiking ideas for the public API herehttps://github.com/gophertown/fsnotify_ext/issues/1but the code itself is a mess. Working on cleaning it up tonight.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/56#issuecomment-23759929
.

[nathany]

@howeyc Will do. Making use of the internal state in fsnotify will be better than wrapping it with another package and having to query things again (eg. IsDir()). I'm now considering fsnotify_ext a throw-away spike.

Here is the GoDoc for a proposed public API, including deprecations.

[nathany]

@pmonks I haven't had a chance to dig into Watchdog or Listen yet to find out which OSes support recursive watches. If anyone wants to do the research, that'd be great :-)

I imagine we could save a bunch of extra book keeping if all the supported OSes have native support. If not, I'll keep on working towards a cross-platform option that can be overridden after (with OS X being an exception, because as it turns out, FSEvents don't support non-recursive watchers!).

[htoothrot]

Windows can support this. ReadDirectoryChanges[1] has a "bWatchSubtree" argument. If this argument is true, you will receive events for all files/directories for the whole directory tree rooted at the given directory.

As watching a large/busy tree can lead to many events I thought this was worth bringing up: It may be helpful to have the buffer size configurable in some way. Right now it's a hard coded 4096 bytes. When many events fire quickly, events can be dropped when the buffer is full. There are caveats to simply increasing the size ( http://stackoverflow.com/a/3250454 explains some of the issues ) and it seems there's not one answer to fit all uses.

Thanks for your work on this.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365465%28v=vs.85%29.aspx

[nathany]

Thanks @htoothrot.

Does anyone know if inotify supports recursive watches? or kqueue?

[htoothrot]

http://lists.qt-project.org/pipermail/development/2012-July/005279.html

has a quick overview. It's from last year but some checking seems to indicate that nothing has changed...

[dubcanada]

Poco.DirectoryWatcher requires each directory to be watched separately. Most likely recursive watching is not crossplatform.

watchdog and listen both add each directory to monitor if recursive is turned on.

watchdog example...

for root, directories, files in path_walk(self._path, recursive):
    for directory_name in directories:
      try:
        directory_path = os.path.join(root, directory_name)
        stat_info = os.stat(directory_path)
        self._stat_snapshot[directory_path] = stat_info
        self._inode_to_path[stat_info.st_ino] = directory_path
        walker_callback(directory_path, stat_info)
      except OSError:
        continue

if recursive is turned on, it looks through all the directories and adds them to the snapshot, which is then all monitored (individually on the system level).

Moral of the story is this isn't possible 100% crossplatform. You would need to monitor each individually on Linux (Windows has recursive, inotify isn't recursive, and neither is kqueue).