Exploring how two API settings—retained reasoning and compaction—significantly improved GPT-5.6’s performance on the ARC-AGI-3 benchmark, highlighting the impact of harness design on AI evaluation. By Ilan Bigio, Ted Sanders.
The performance of AI models is often influenced by more than just their inherent capabilities; the settings and design of the evaluation harness play a crucial role. This is exemplified in the case of GPT-5.6 Sol’s performance on the ARC-AGI-3 benchmark, where two specific API settings—retained reasoning and compaction—were pivotal in tripling the model’s scores. Initially, GPT-5.6 Sol struggled with the ARC-AGI-3 benchmark, scoring only 7.8%, due to the harness’s design which discarded private reasoning and used a rolling truncation window. This setup forced the model to re-interpret the game from scratch with each action, severely limiting its ability to learn and strategize over time.
By implementing the Responses API, which retains reasoning and employs compaction, the model’s performance improved dramatically. Retained reasoning allowed GPT-5.6 Sol to remember its past thoughts and actions, reducing the time spent on interpreting the game state and enabling more coherent strategies. Compaction further enhanced performance by preserving learned information across longer runs, allowing the model to achieve higher scores with fewer output tokens. This case study underscores the importance of harness design in AI evaluation, revealing that seemingly minor settings can have a substantial impact on model performance.
The trade-off here involves balancing the complexity and resource demands of maintaining detailed reasoning and compaction against the performance gains they provide. Additionally, a potential failure mode is the risk of overfitting to specific harness settings, which may not generalize well across different evaluation environments. Before adopting this approach, teams should consider whether the benefits of retained reasoning and compaction align with their specific use cases and evaluation goals. Nice one!
[Read More]