1

I have a pretty big (20GB) CSV file, and I need to modify some of its columns. What is the MOST OPTIMIZED way of importing the data table line by line (or probably few thousands of line per read) ? I have tried the solution given below What is a good way to read line-by-line in R?

But it seems to be very slow. Is there any library which can read line by line, in the table structure itself -- also which has some kind of Buffer logic to make the read faster ?

Ajay
  • 45
  • 4

1 Answers1

2

You can use the fast fread() from data.table. By skip=, you're setting the beginning of the read segment and by nrow=, the number of rows to read.

NelsonGon
  • 12,469
  • 5
  • 25
  • 52
Yuriy Barvinchenko
  • 1,303
  • 1
  • 8
  • 15