How to do `tail -f logfile.txt`-like processing in node.js?

Question

tail -f logfile.txt outputs the last 10 lines of logfile.txt, and then continues to output appended data as the file grows.

What's the recommended way of doing the -f part in node.js?

The following outputs the entire file (ignoring the "show the last 10 lines") and then exits.

var fs = require('fs');
var rs = fs.createReadStream('logfile.txt', { flags: 'r', encoding: 'utf8'});
rs.on('data', function(data) {
  console.log(data);
});

I understand the event-loop is exiting because after the stream end & close event there are no more events -- I'm curious about the best way of continuing to monitor the stream.

http://stackoverflow.com/questions/8224333/scrolling-log-file-tail-f-animation-using-javascript — gavenkoa, Sep 06 '13 at 07:56

score 53 · Accepted Answer · edited Dec 01 '20 at 12:30

53

The canonical way to do this is with fs.watchFile.

Alternatively, you could just use the node-tail module, which uses fs.watchFile internally and has already done the work for you. Here is an example of using it straight from the documentation:

Tail = require('tail').Tail;

tail = new Tail("fileToTail");

tail.on("line", function(data) {
  console.log(data);
});

edited Dec 01 '20 at 12:30

Community

1
1

answered Jan 25 '12 at 01:18

Rohan Singh

18,750
1
39
48

1

Many thanks -- `fs.watchFile` is exactly what I was looking for. Also thanks for pointing out the node-tail module. – mike Jan 25 '12 at 03:32
As of now, `fs.watchFile()` uses stat polling. `fs.watch()` can be used instead, but `fs.stat()` will have to be used to find a file's current and previous size. – hexacyanide Mar 11 '13 at 20:46
The node-tail module has since been updated to use `fs.watch()`. – Rohan Singh Feb 28 '14 at 14:15
always-tail is better – AshBringer Dec 02 '15 at 07:51
Tail module works great! Thank you. – eerrzz Oct 31 '21 at 22:03

score 8 · Answer 2 · answered Jul 18 '12 at 17:52

node.js APi documentation on fs.watchFile states:

Stability: 2 - Unstable. Use fs.watch instead, if available.

Funny though that it says almost the exact same thing for fs.watch:

Stability: 2 - Unstable. Not available on all platforms.

In any case, I went ahead and did yet another small webapp, TailGate, that will tail your files using the fs.watch variant.

Feel free to check it out here: TailGate on github.

score 3 · Answer 3 · answered Oct 02 '14 at 12:15

Substack has a file slice module that behaves exactly like tail -f, slice-file can stream updates after the initial slice of 10 lines.

var sf = require('slice-file');

var xs = sf('/var/log/mylogfile.txt');
xs.follow(-10).pipe(process.stdout);

Source: https://github.com/substack/slice-file#follow

score 3 · Answer 4 · answered Jan 25 '12 at 01:15

you can try to use fs.read instead of ReadStream

var fs = require('fs')

var buf = new Buffer(16);
buf.fill(0);
function read(fd)
{
    fs.read(fd, buf, 0, buf.length, null, function(err, bytesRead, buf1) {
        console.log(buf1.toString());
        if (bytesRead != 0) {
            read(fd);
        } else {
            setTimeout(function() {
                read(fd);
            }, 1000);
        }
    }); 
}

fs.open('logfile', 'r', function(err, fd) {
    read(fd);
});

Note that read calls callback even if there is no data and it just reached end of file. Without timeout you'll get 100% cpu here. You could try to use fs.watchFile to get new data immediately.

this approach loses some data when log rotate or similar tools are involved — Bogdans, Nov 10 '20 at 08:29

score 2 · Answer 5 · answered Aug 14 '15 at 07:48

2

https://github.com/jandre/always-tail seems a great option if you have to worry about log rotating, example from the readme:

var Tail = require('always-tail');
var fs = require('fs');
var filename = "/tmp/testlog";

if (!fs.existsSync(filename)) fs.writeFileSync(filename, "");

var tail = new Tail(filename, '\n');

tail.on('line', function(data) {
  console.log("got line:", data);
});


tail.on('error', function(data) {
  console.log("error:", data);
});

tail.watch();

answered Aug 14 '15 at 07:48

user1278519

816
6
9

always-tail looks like it was abandoned; the last update was in 2015. See https://github.com/lucagrulla/node-tail for something more recent. – James Moore Oct 06 '21 at 16:50

How to do `tail -f logfile.txt`-like processing in node.js?

5 Answers5

Linked