Say, for example, I have 5 splits of my data. Can I randomly initialize the weights for my neural network at the start of each split? Or should I save the initial weights randomly initialized for the first fold, then on the second fold, load up the randomly initialized weight for the first fold before training?
The latter seems more sound, though would this have a big effect on my results?
The basic principle of cross-validation is that the hyperparameters are fixed, same for each fold, and the training is always run with different data. I think both of your scenarios match this: in the first one, the hyperparameters of the weight initialization algorithm are fixed between the runs; in the second, initial weights themselves are considered hyperparameters and are fixed. So, formally, both is okay.
Now the real question is: what is the difference between them?
Cross-validation gives you an estimate how well will the model perform on average given the fixed hyperparameters. The result is only valid for you if you then use the same hyperparameters for the final, output model. So, if you plan to always use the same saved weight initialization, use it also for each fold. If you later plan to initialize the model randomly, use also random initialization during the cross-validation. You can then tune the hyperparameters of the weight initialization by running multiple cross-validations.
Final remark is that with NN, cross-validation is often somewhat omitted for practical purposes (if training one fold takes two days, performing 10-fold CV takes nearly three weeks). Then, using the same initialization configuration is actually useful way to compare performance.