diegodoimo.bsky.social
👤 Joined on 08 December 2024
26
Followers
75
Following
6
Total Posts
📜 Kullanıcı Geçmişi
- 08 Ara 2024, 06:51
🎉 Bluesky'e katıldı - İlk kullanıcı adı:
@diegodoimo.bsky.social
🚫 Blokladığı Kullanıcılar
❤️ Beğendiği Gönderiler
Really excited to share our latest interpretability work on multimodal models! The communication between image and text is localised in a single token in multimodal-output vision-language models. Paper: arxiv.org/html/2412.0664… Happy to discuss it at #NeurIPS2024 More below 👇
Show PostNice start of @neuripsconf.bsky.social! Our work with @francescortu.bsky.social and @diegodoimo.bsky.social on the Competition of Mechanisms to understand counterfactuality in LLMs featured in the "Causality for LLMs" workshop :-) Check out our ACL2024 paper aclanthology.org/2024.acl-long.…
Show Post
🎯 Key finding: In these models the hidden representations of images and text form disjoint clusters and the communication between modalities is mediated by the special token <end-of-image>!
Show Post✅ This shows that, starting from the mid-layers, a single token effectively summarizes all 1024 image tokens! ❌ This does not occur in models fine-tuned for visual understanding (such as Pixtral).
Show PostAdditionally, blocking communication from this token significantly disrupts performance on standard benchmarks, while blocking image-text communication does not
Show Post