how can i use huge csv data in `n` number of chunks?

Asked Jul 31 '21 at 17:54

Active Jul 31 '21 at 17:54

Viewed 26 times

I huge csv dataset with 20 million entries, in this i have an id column too. i am asking how can i read this csv file in 10 separate pandas dataframes like pd1,pd2... and if values of id is 1 to 20 mil,i want pd1 should get 0 to 2000000, pd2 should get 4000000 to 6000000 and that pattern should follows till pd10. How can i do this ?

Note: The main problem with this i can load whole `csv file` at same time because of not enough `ram` or if i load it with `dask` process of dividing it would be so expensive, so how can i divide it without loading it ?

asked Jul 31 '21 at 17:54

1

When you read the csv with pandas, can't you just specify the rows you want to skip with skiprows? – Mady Daby Jul 31 '21 at 17:56

how can i use huge csv data in `n` number of chunks?

Note: The main problem with this i can load whole csv file at same time because of not enough ram or if i load it with dask process of dividing it would be so expensive, so how can i divide it without loading it ?

0 Answers0

Note: The main problem with this i can load whole `csv file` at same time because of not enough `ram` or if i load it with `dask` process of dividing it would be so expensive, so how can i divide it without loading it ?