Note that the answers here and here are not helpful because the dataset size should be known beforehand if I'm going to use the approaches mentioned there.
Before converting data to TFRecord, I have the data in a pandas DataFrame so I have 2 options:
A) Split the DataFrame into separate chunks(train and test or train, dev, test ...) and then save each chunk separately as a TFRecord.
B) Keep the DataFrame as is and convert it to TFRecord file but the downsides would be there won't be much flexibility(as I know of) during runtime to split the whole chunk of data into 2 separate entities, specially the TFRecord won't be keeping the dataset size as metadata.
The question is: which way should I choose? what other options/better options that you know of?
