Comparing Methods for Estimating Marginal Likelihood in Symbolic Regression

Marginal likelihood has been proposed as a genetic programming-based symbolic regression (GPSR) fitness metric to prevent overly complex expressions and overfitting, particularly when data is limited and noisy. Here, two particular methods for estimating marginal likelihood - the Laplace approximation and sequential Monte Carlo - are studied with a focus on tradeoffs between accuracy and computational efficiency. The comparison focuses on practical challenges in the context of two sets of example problems. First, the methods are compared on handcrafted expressions exhibiting nonlinearity and multimodality in their respective posterior distributions. Next, the methods are compared on a real-world set of equations produced by GPSR using training data from a well-known symbolic regression benchmark. A key finding is that there are potentially significant differences between the methods that, for example, could lead to conflicting selection of expressions within a GPSR implementation. However, it is concluded that there are scenarios where either method could be preferred over the other based on accuracy or computational budget. Algorithmic improvements for both methods as well as future areas of study are discussed.