Introducing “An Evolved Universal Transformer Memory” 🧠 (Blog: https://sakana.ai/namm/) Neural Attention Memory Models (NAMMs) are a new kind of neural memory system for Transformers that not only boost their performance and efficiency but are also transferable to other foundation models, without any additional training! Memory is a crucial component of cognition, allowing humans to selectively store and extract important notions from our ceaseless exposure to information and noise. Our work learns an artificial memory to replicate these capabilities thanks to the power of evolution, leading to smarter, faster, and more adaptable foundation models. Full Paper: https://lnkd.in/gkDzMCG3 Code: https://lnkd.in/gmvFSma7
amazing, pushing the boundaries as always
はじめまして
Tech Subject Matter Expert | 🇸🇪🇺🇸-innovation and policy alignment
1wIt’s fascinating how NAMMs differs fundamentally from H2O/L2! While those use static rules to compress memory, NAMMs learns what to remember, similar to how military intelligence must adaptively filter crucial signals from noise. The zero-shot transfer capability is particularly relevant for some application areas that I’m looking into.