Question about AI-Feynman Performance Discrepancy in SR Benchmark Results

[yuzhTHU]

Hello guys,

I have a question regarding the performance of the AI-Feynman algorithm as shown in the SR Benchmark results, particularly in comparison with the original AI-Feynman paper. In the results presented on the SR Benchmark website (link to results), the AI-Feynman algorithm finds a model symbolically equivalent to the ground-truth process on the Feynman dataset with a frequency of around 55%.

However, in the original AI-Feynman paper (AI Feynman: A physics-inspired method for symbolic regression, link to arXiv), the authors report being able to discover 118 out of 120 equations, which is close to 100%.

The difference between these two figures—55% in the benchmark versus nearly 100% in the original paper—seems quite significant, and I am unsure what might be causing this discrepancy.

Potential Questions:

• Could there be differences in the dataset or pre-processing steps used in SR Benchmark compared to the original paper?
• Are there any specific details about the benchmarking process or configuration that could explain the variance in performance?
• Has anyone else encountered a similar gap between published results and the SR Benchmark performance for other algorithms?

[lacava]

There could be several reasons:

1. they use larger dataset sizes in training.
2. we used fixed evaluation budgets and time limits per method. if i recall they didn't report this or control it.
3. we used a fixed criteria for exact solutions (sympy quasi-equivalence). i think they used visual inspection.
4. a number of other factors may explain it, but since they didn't provide a reproducible version of their experiment/results, we don't know what they are. we reached out to the authors about the benchmark back when we were doing it but never received a response.

Has anyone else encountered a similar gap between published results and the SR Benchmark performance for other algorithms?

Yes, when we use a fully transparent benchmarking process like srbench, we often see differences from what authors report in their papers that introduce new methodologies. often it comes down to differences in experiment design that may favor one method or another. part of the motivation for this benchmark is to alleviate those issues.