Why there is a delay in detecting a file change using WatchService? (Java)

Question

I've an application in which when a file is added to the directory, WatchService detects the file and the file is added to a file list for further processing. This is my code

 public void run() {

    /*
     * Goes in an infinite loop
     */
     while(!finished) {

     /*
      *  Get a watch key, poll() returns a queued key 
      *  if no queued key, this method waits until the specified time.
      */
     WatchKey key;
     try {
             key = watcher.poll(eofDelay,TimeUnit.MILLISECONDS);
      } catch (InterruptedException x) {
          return;
      }

     Path dir = keys.get(key);

     if (dir == null) {
         continue;
      }

     Path child=null;

         /*
          * Fetching the list of watch events from
          * pollEvents(). There are four possible events
          */

         for (WatchEvent<?> event: key.pollEvents()) {
            WatchEvent.Kind kind = event.kind();

            /*
             * Overflow indicates that events 
             * might have been lost or discarded
             */
             if (kind == OVERFLOW) {
                 continue;
             }


             WatchEvent<Path> ev = cast(event);

             /*
              * Filename is the context of the event
              */
             Path name = ev.context();

             /*
              * Resolves the name of the file to a path
              */
              child = dir.resolve(name);

             /*
              *  If directory is created, and watching recursively, then
              * register it and its sub-directories
              */
             if (nested && (kind == ENTRY_CREATE)) {
                 try {
                     if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
                         registerAll(child);
                     }
                 } catch (IOException x) {

                 }
             }
         }

         File file = child.toFile();

         /*
          * Only add the file if there is no wild card 
          * or it matches the specified wild card 
          */
         if (matcher == null || matcher.matches(file.toPath().getFileName())) {
             fileList.add(file);
         }
     /*
      * Key is reset so that it can receive further
      * events 
      */

         boolean valid = key.reset();
         if (!valid) {
             keys.remove(key);

            /*
             * If key is no longer valid and empty,
             * exit the loop
             */
             if (keys.isEmpty()) {
                continue;
             }
         }

     }
 }

This code works as expected but I'm designing a high performance application, which processes data in the files at very high speed. So the problem here is inconsistency in time taken to detect a file. For instance initially there are some files in the directory and they're processed by the application, now when a new file is added it takes 4-5 sec to detect the file or sometimes it takes 2 sec or 20ms and so. My eofDelay value is 10ms. What is the reason for this inconsistency? Is there a way to enhance this implementation? Or any other efficient library available for directory changes? I want the time taken to detect a file to be minimal and consistent, taking more than a second is very expensive. Any help in this regard will be appreciated. :)

Answer 1

You may be able to get faster results by adding a sensitivity flag to the folder (see below).

// copied from http://stackoverflow.com/questions/9588737/is-java-7-watchservice-slow-for-anyone-else
folder.register(watcher, new WatchEvent.Kind[]{StandardWatchEventKinds.ENTRY_MODIFY}, SensitivityWatchEventModifier.HIGH); 

However, you will still be at the mercy of the underlying OS. Most file watching applications I've seen have a few second delay between when the file is added and when it is picked up. You're seeing normal lag times in your application.

If your application must respond to a few file being added in a few milliseconds, you should not use Java (NIO or otherwise), but C/C++. This will add significant complexity to your code.