Kartik Tiwari

Abstract

Vision-language models can now draft fluent chest-radiograph reports — but fluency doesn't guarantee the model was looking at the right part of the image when it made a claim. VERA is a training-free auditing layer that reuses a model's own decoder attention as a spatial grounding check. It breaks a generated report into individual clinical claims, tags each with its finding, location, negation, and severity, then verifies whether the visual attention behind that claim actually falls on the correct anatomical region. Evaluated against real radiologist reports, VERA flags the majority of ungrounded claims — including high-severity findings — and, unlike a single opaque confidence score, gives clinicians an interpretable trail showing exactly which sentence and which image region produced each flag.

Abstract

Knowledge-editing methods like ROME and MEMIT can rewrite a single fact inside a language model, but the edit sometimes damages neighbouring knowledge that should have been left untouched — the ripple effect. This paper runs the largest side-by-side comparison of editing safety to date: 7,800 rank-one edits across 13 open models spanning both dense and mixture-of-experts architectures. The central finding is that a model's architecture — specifically how routing and expert capacity are structured — predicts ripple severity far better than any signal available before the edit is made: coarse-routed, high-capacity MoE models suffer close to four times the ripple damage of typical dense models, while a handful of MoE designs resist rank-one editing almost entirely. Per-fact fragility scores that looked reliable inside a single model failed to transfer across models, suggesting editing-safety screening belongs at the level of architecture choice, not individual facts.