New paper: emerging trends in AI agent evaluation
The authors of AI Snake Oil have published a paper on how AI agents are evaluated. It surveys the methods in use, the open questions and the wider ecosystem around agent evaluation.
Related stories
Why teaching AI about symmetry could make prenatal diagnosis more reliable
Researchers at MIT and Boston Children's Hospital built a model that adjusts scans in real time to follow the movements of a fetus. The approach could reduce the need for families to return to hospital for repeat scans.
New AI tool can predict cancer survival from a single patient selfie, study suggests
Researchers at Mass General Brigham have built an AI system that estimates biological age from a photo of a patient's face. In a study, it was also used to predict survival outcomes for people diagnosed with cancer.
GenRecs tokenization: how Meta is transforming ads recommendation with a stable model vocabulary
Meta describes Generative Recommenders, which reason about ad content and user preferences together instead of scoring every ad, drawing on the world knowledge in LLMs. GenRecs tokenization is the data representation underpinning the approach.
The thin orchestrator: how Meta made its AI agents more reliable by giving the router less to do
Meta engineers report better efficiency and accuracy in their data pipeline agents by keeping the orchestration layer deliberately thin. The orchestrator only routes work and enforces reliability checks, while specialized agents handle the domain reasoning.
AI chatbots change 'personality' depending on the language, study finds
Chinese researchers studied whether chatbots answer differently depending on the language used. They report that prompts in Mandarin made the language models more open-minded.