0

Context:

A paper I'm reading uses PDEs to characterise the effects of cancer treatments on the tumour microenvironment. The exact wording used in the paper is:

The predictive power of the [Quantitative Systems Pharmacology] model was assessed via an external cross validation: the model was used in a forward-simulation mode, by simulating new experimental scenarios for which tumor size data had been independently generated, to indeed determine whether we could predict such data – data which had not been used in the model development and evaluation steps described above. The following scenarios were simulated for this purpose, with a post-hoc verification against the existing data.

Though it isn't stated explicitly, the model has been fit to experimental data sets, then the authors have input the experimental initial conditions into the model and generated tumour growth curves to compare to the original experimental data. i.e. the model-generated data has been treated as an external data set to test the accuracy of the model.

Question:

Is what is described below a way of performing external cross-validation? I haven't found a good resource online to describe the use of model-generated data to test a model, so any additional resources would be appreciated.

kjetil b halvorsen
  • 77,844
Kell
  • 15

1 Answers1

2

Cross-validation typically checks how well a model trained on some real data makes predictions on other real data. But that's not happening here. Instead of "external cross-validation," I would call this a "posterior predictive check." That might be a better term to search for online if you'd like to understand how these kinds of checks work. See for instance this answer.

The terminology "posterior predictive checks" has been popularized in a Bayesian context by folks like Andrew Gelman, but you don't have to be Bayesian to use the general idea. It seems to be what's described in the paper you are reading: After you fit a model, you generate new data from that model and see if it broadly resembles your original data.

These are very useful as a sanity-check on your model-fitting process. For example, if it's important to account for the fact that your real data are discrete counts, but your initial model generates new data that can be fractional or negative, then this kind of check can help you notice that fact and correct it.

Think of your posterior predictive checks as a minimum bar that your models should pass. One benefit is that it's a "free" check: you're not "using up" or overfitting to your holdout/validation data when you make these checks. It's debugging, not inference.

But that means they are not so useful for making comparisons between well-developed models. Once all your "serious candidates" for models pass this minimum bar, you'll have to use held-out data (or other approaches) to actually compare models or to estimate their predictive performance on future data.

civilstat
  • 3,785
  • Thank you for your answer, that's incredibly helpful. Many of the papers I am reading are from researchers without backgrounds in modelling and the language is often difficult to interpret for someone new to the field. Is there a reference material that you recommend that details model testing options? – Kell May 19 '22 at 07:40
  • 1
    @Kell If you're specifically asking about this type of predictive model checking, I'm afraid I haven't found a favorite introductory reference. Chapter 6.3-6.4 of Bayesian Data Analysis (Gelman et al) discusses it and it's freely available online: http://www.stat.columbia.edu/~gelman/book/ but if this is all new to you, it may be hard to separate out the generic model-checking advice from digressions about Bayes vs frequentist p-values etc. (Also worth knowing eventually, just not necessarily worth starting with.) – civilstat May 20 '22 at 00:44
  • 1
    @Kell If you don't already have a strong Bayesian background, I'd actually recommend instead Chapter 8.3-8.4 of Data Analysis Using Regression and Multilevel/Hierarchical Models (Gelman and Hill): http://www.stat.columbia.edu/~gelman/arm/ Unfortunately, the authors have not made this one freely available online, but I think it's a better introduction to the core idea (simulate new data from the fitted model and see if it captures important features of the real data) without getting into the weeds on frequentist-vs-Bayesian differences. – civilstat May 22 '22 at 14:37
  • 1
    Thank you for the recommendations! The Data Analysis Using Regression... text is freely available through my university. I'll have a look at that first and progress to the next text. – Kell Jun 15 '22 at 05:29