Reading big matrix of integers fast

Question

I am trying to read a big table (around 500000 x 1000) in R.

read.table works, but is horribly slow.

scan works, too, and is OK speedwise, but I have been unable to change its format to a normal data.frame or matrix.

I do not know the number of rows in the table in advance (I can find the number of columns by template_line <- read.table(nrow=1,file=my_file)). It needs to be compatible with R 2.15 - so it seem fread is out of the question.

So the question is either: How do I convert the output from:

my_matrix <- scan(file=my_file,what=template_line);

to a data.frame or matrix (fast)?

Or: how do I read a table of integers fast in R if I do not know the size?

If `fread` is out, what about `sqldf`? Also, what is the problem you have with the output of `scan`? — A5C1D2H2I1M1N2O1R2T1, Oct 17 '13 at 15:08
@AnandaMahto I need to be able to access `my_matrix[1,2]`, and that does not work given the code above. — Ole Tange, Oct 17 '13 at 15:09
eh? Why is `fread` out? Current CRAN `data.table` is `1.8.10` which states: **`Depends: R (≥ 2.12.0)`** — Simon O'Hanlon, Oct 17 '13 at 15:11
try adding `colClasses = "integer"` to your `read.table` call. — flodel, Oct 17 '13 at 15:11
@SimonO101 You seem to be right, and that is much preferable. Can I have data.table::fread read a .gz file? — Ole Tange, Oct 17 '13 at 15:24
Sorry but why would `data.table::fread` be excluded because of `R.2.15.x` ?? I was using it every day when on `2.15.2` — statquant, Oct 17 '13 at 15:39

score 0 · Answer 1 · answered Oct 17 '13 at 15:17

0

What about this?

num_cols <- 5
my_matrix <- matrix(scan(file=my_file, what=template_line), ncol=num_cols, byrow=TRUE)

answered Oct 17 '13 at 15:17

user1578653

4,688
15
43
72

Reading big matrix of integers fast

1 Answers1