2


I get a fast stream of data (objects) and I would like to write it to a file.
This is a stand alone process so it doesn't do anything but read the data from a socket parse it to csv and write all to a file.

What is the best way to write a lot of csv lines to a file?
Is a buffer writing my solution?
Is there a buffered File object in Java ?
Should I manage it myself and use writeLines()?

Bick
  • 16,965
  • 49
  • 140
  • 240
  • 1
    What, now? It needs to be very fast, but not very efficient? – Lukas Eder May 23 '11 at 15:51
  • It means that I dont do any computation on the data . So I only need it to be fast enough in order to write all. I will fix my question. – Bick May 23 '11 at 15:54

3 Answers3

9

Fastest way to write huge data in text file Java

Community
  • 1
  • 1
bpgergo
  • 15,165
  • 3
  • 39
  • 64
3

If you're dealing with a huge throughput of data then I suggest you use a set of in-memory buffers where you deposit the data arriving and then have a thread/threadpool which uses Java NIO to "consume" these buffers and write them onto disk. You will however be limited by the disk writing speed -- bear in mind that it's not unusual for the speed of network to be faster than the speed of your hard disk! so you might want to consider a threadpool which writes in different physical locations and only "pastes" these files after all the data has been received and written.

Liv
  • 5,926
  • 1
  • 21
  • 29
1

As mentioned above, chances are that its disk I/O that limits you, not Java abstractions.

But beyond using a good lib to deal with CSV, you might consider using other (even more) efficient formats like JSON; as well as compression. GZIP is good at compressing things, but relatively slow; but there are faster ones too. For example, LZF (like this Java implementation) is fast enough to compress at speeds higher than typical disk I/O (and uncompress even faster). So compressing output may well increase throughput as well as reduce disk usage.

StaxMan
  • 108,392
  • 32
  • 202
  • 235