“New CSAIL research highlights how LLMs excel in familiar scenarios but struggle in novel ones, questioning their true reasoning abilities versus reliance on memorization” https://news.mit.edu/2024/reasoning-skills-large-language-models-often-overestimated-0711