Скачать или смотреть The Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical Benchmarks

This paper investigates the reliability of large frontier models, such as GPT-5, on medical benchmarks. Despite achieving high scores, these models often exhibit vulnerabilities. They can guess correctly even without key inputs like images, change answers due to trivial prompt alterations, and fabricate flawed reasoning. The study reveals that current benchmarks reward test-taking strategies rather than genuine medical understanding, leading to an illusion of readiness. The research evaluates six models across six benchmarks, exposing brittleness and shortcut learning. Clinician-guided analysis demonstrates that benchmarks vary widely in what they measure. The authors caution against relying solely on benchmark scores and advocate for evaluating robustness, reasoning, and alignment with real-world medical needs. The findings suggest a need to reevaluate how progress in health AI is measured to ensure models are trustworthy for medical decision-making.
#AI #Healthcare #MedicalAI #Benchmarking #MachineLearning #GPT5 #Robustness #Evaluation

paper - http://arxiv.org/pdf/2509.18234v1
subscribe - https://t.me/arxivpaper
donations:
USDT: 0xAA7B976c6A9A7ccC97A3B55B7fb353b6Cc8D1ef7
BTC: bc1q8972egrt38f5ye5klv3yye0996k2jjsz2zthpr
ETH: 0xAA7B976c6A9A7ccC97A3B55B7fb353b6Cc8D1ef7
SOL: DXnz1nd6oVm7evDJk25Z2wFSstEH8mcA1dzWDCVjUj9e
created with NotebookLM