Combining meta-analysis and machine learning (accounting for heterogeneity)

Question

What are the standard methods for training machine learning algorithms on data, issued from multiple studies?

In conventional meta-analysis one usually uses weighted average to combine estimates obtained in separate studies, $$ M=\frac{\sum_{i=1}^kW_iY_i}{\sum_{i=1}^kW_i}, $$ where the weights are determined by the number of samples in the study or its variance (fixed-effect model), or may incorporate possible uncontrolled variation between studies (random effects model.) See, e.g., A basic introduction to fixed-effect and random-effects models for meta-analysis or Introduction to Meta-Analysis.

Suppose now that I have data issued from different studies (e.g., the sequences of measurements obtained for each sample), and I want to train a ML algorithm on these data. The simplest option would be to pool all the sample data into a single dataset, which is analogous to the fixed effect model for the average. However, how would one proceed in order to take into account heterogeneity? (i.e., what would be the analogue to the random-effects model.)

Remark
An alternative approach is performing ML for each study separately and then carrying out meta-analysis of the results - e.g., in classification tasks one could use false discovery rate as the analysis variable (for which the weighted averages are calculated.)

Related:
Weighting an hypothetical biased dataset

Answer 1

One of the more common approaches is to frame this as a setting where you are trying to predict for a new study (or for some of the already seen studies, if that's what you're interested in), and you want to account for differences between studies (e.g. different methods/design/studied population) as well as for unexplained study-to-study variation.

There's of course different modeling techniques / types of models one can use to do that. E.g. just about any type of model could use predictors on the study level (just set to be identical for everyone in a study). On the other hand, having a study random effect that explicitly reflects variation between studies is most easily done in mixed models (aka "random effects model" aka "hierarchical models"), which tend to automatically reflect when you have more or less information about some of the studies. Other types of models have similar concepts such as embeddings in neural networks or forms of target encodings in gradient boosted decision trees (such as the handling of categories used in catboost - in fact, one way to create target encodings is with fitting a random effects regression model), but these are not so "aware" of how much you know about each study (but there are proposals or here or here for how one could make this work) and struggle to deal with completely unseen new studies/categories.

The last bit is pretty difficult to deal with. E.g. if you try ideas like simultaneously fitting trees to each dataset, with some hyperparameter controlling how much these models vary from each other, then you find that you can fit the observed data sensibly, but you struggle to generate a sensible tree for a completely new study (the model is not really generative). I assume someone is working on this and/or has already proposed solutions, because this "cold-start-problem" is a common major problem.