Joel Cerqueira Ponte

Hello All! It is great to be back again and this time I am really excited to discuss about the concept I have been long waiting to write about and the most transformational one(in literal sense ;)) in AI space for the past 6–7 years— the Transformers(I get that it is already too late to write about it, but better late than never :)) In this article in my series "Neural Networks Intuitions: 19. Transformers", I will be discussing about the fundamentals of Attention mechanism in the context of Image Classification(and touch upon Language Modeling in the end). Most importantly, I will be approaching Transformers from the perspective of looking at them as a message passing architecture, viewing Attention(Key and Query matrices) as learnable weights introducing dynamic connections and understand why a transformer can be thought of as a more general case of MLPs whereas convolutions and recurrences are special cases of MLPs. Link to my article: https://lnkd.in/ecDD3NRc Do read my article and provide your thoughts/comments. Thank you. #deeplearning #transformers #attention #visiontransformers #languagemodels

74

1 comentario

A paper published this week studies the ideological differences between LLMs trained in different regions: https://lnkd.in/dyEehnhH It finds that Chinese LLMs have a more favorable view of "Marxism", "Supply Side economics", and "PRC" (People's Republic of China) compared to Western-trained LLMs. Similar differences were found based on the language in which questions were asked. Conversing in English gave much more western-centric responses compared to speaking in Chinese. A few days ago, the chairman of Infosys, Nandan Nilekani explicitly said India shouldn't train LLMs: https://lnkd.in/duJ5MYNX . His point is understandable, since the capital required, in terms of the cost of buying GPUs and running data centers, is immense. Not to mention the difficulty of getting large amounts of data. However, I think this paper shows us why training an LLM in India is important. Now that Infosys is out, will someone else who has the resources step up to this challenge? Could it be the Ambanis? Or perhaps the Adanis? How about the Birlas? 😊 #LLMBias #LLMResearch

10

2 comentarios

Our new paper at Qantev!! In this paper we show how to generate a synthetic OCR dataset for the Visual Rich Documents problem and how to fine-tune TrOCR using this generated dataset in Spanish!

37

"𝗪𝗵𝗮𝘁 𝘁𝗵𝗲 𝗵𝗲𝗰𝗸 𝗮𝗿𝗲 𝗽𝗲𝗼𝗽𝗹𝗲 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗱𝗼𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲𝗶𝗿 𝗖𝗹𝗼𝘂𝗱 𝗗𝗮𝘁𝗮 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲?" is a question we asked ourselves when we joined the Redshift team. We have since found out, and publish as many findings as we can in our VLDB2024 paper "𝗪𝗵𝘆 𝗧𝗣𝗖 𝗶𝘀 𝗻𝗼𝘁 𝗲𝗻𝗼𝘂𝗴𝗵: 𝗔𝗻 𝗮𝗻𝗮𝗹𝘆𝘀𝗶𝘀 𝗼𝗳 𝘁𝗵𝗲 𝗔𝗺𝗮𝘇𝗼𝗻 𝗥𝗲𝗱𝘀𝗵𝗶𝗳𝘁 𝗳𝗹𝗲𝗲𝘁", together with a comparison to the industry-standard TPC benchmarks for analytical systems. https://lnkd.in/dAPys5H4 Next to our analysis of fleet data, we open source 𝙍𝙚𝙙𝙨𝙚𝙩, a dataset of query logs containing statistics over all queries that ran on a sample of Redshift instances over the span of three months. With Redset we want to specifically encourage drawing further conclusions about real world workloads, conducting research on ML-based techniques such as query prediction models and workload forecasting and creating novel benchmarks that closer mimic the observed real-world workloads.  https://lnkd.in/dZMdrkKF Thanks to my co-authors Alexander van Renen, Dominik Horn, Kapil Vaidya, Wenjian Dong, Balakrishnan(Murali) Narayanaswamy, Zhengchun Liu, Gaurav Saxena, Gaurav Saxena, and Tim Kraska, and members of the Redshift team Mohammed Al-Kateb, Matt Abrams, Naresh Chainani, George Erickson, Bruce McGaughy, Davide Pagano, Ippokratis Pandis, and Rahul Pathak for making this possible.

42

1 comentario

🚀 uv (Python) enabling new workflows! Last night I had an idea: what if we could open any Jupyter notebook from GitHub with all its dependencies installed instantly? The result 2 hours later: opennb (https://lnkd.in/gGZ6CDPN) Example: uvx --with "pipefunc[docs]" opennb pipefunc/pipefunc/example.ipynb This single command: ✨ Creates a temporary environment 📦 Installs dependencies (instant with uv's cache!) 📓 Downloads and opens the notebook What made this possible? The amazing uv tool (https://docs.astral.sh/uv/), which handles Python package management at insane speeds. The best ideas are often the simplest - combining uv's speed with Jupyter's accessibility to make trying out Python tutorials friction-free. #Python #JupyterNotebook #OpenSource #DeveloperTools #Programming

92

5 comentarios

Part 2 of our comprehensive guide on Machine Learning resources is out! In Part 1, with Josep, we laid the groundwork with essential theoretical foundations, including top-rated courses and must-read books/blogs. In Part 2, we shift from theory to hands-on learning with practical resources that will help you gain experience and build your portfolio. Here's what you can expect: • 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗖𝗼𝘂𝗿𝘀𝗲𝘀 𝗮𝗻𝗱 𝗧𝘂𝘁𝗼𝗿𝗶𝗮𝗹𝘀: Dive into hands-on learning with courses. • 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 NLP and Computer Vision Courses • 𝗖𝗹𝗮𝘀𝘀𝗶𝗰 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝘀: Get started with some classic projects that reinforce your theoretical understanding. • 𝗘𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹 𝗧𝗼𝗼𝗹𝘀 𝗮𝗻𝗱 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀 for your area of interests. • 𝗧𝗶𝗽𝘀 𝗳𝗼𝗿 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗬𝗼𝘂𝗿 𝗣𝗼𝗿𝘁𝗳𝗼𝗹𝗶𝗼: Find examples to understand how to showcase your projects. Stick to the end ! Here's the link to the article: https://lnkd.in/ghvjDKp4 Dive in and as always happy learning! 🌟 It was so much fun to work on this two-part series with Josep. Fun fact: I started following his articles on KDNuggets before I even knew him, and I reached out to him to get his feedback on my articles, not knowing at the time that it would eventually lead to this collaboration. :). -- 𝘐𝘧 𝘺𝘰𝘶 𝘭𝘪𝘬𝘦𝘥 𝘵𝘩𝘦 𝘢𝘳𝘵𝘪𝘤𝘭𝘦, 𝘢𝘭𝘴𝘰 𝘤𝘰𝘯𝘴𝘪𝘥𝘦𝘳 𝘴𝘶𝘣𝘴𝘤𝘳𝘪𝘣𝘪𝘯𝘨, 𝑙𝑖𝑛𝑘 𝑏𝑒𝑙𝑜𝑤 𝑚𝑦 𝑝𝑟𝑜𝑓𝑖𝑙𝑒 𝑝𝑖𝑐𝑡𝑢𝑟𝑒 ☝️ #MachineLearning #DataScience #HandsOnLearning #MLProjects

56

16 comentarios

Have you also noticed that language models sometimes start their responses alright, then make a mistake, and from that point on they hallucinate and drift off-topic? 🤔 Our paper "Know When to Stop: A Study of Semantic Drift in Text Generation" (which just got accepted at NAACL 2024! 🥳) looks into this phenomenon, now called semantic drift. Here's what you can expect from our paper: 1️⃣ Full definition and a new metric for measuring semantic drift 🚀 2️⃣ Results on LLaMa 2, GPT 4 and Falcon models 🔍 3️⃣ Comparison between early-stopping, resampling and knowledge retrieval methods 📊 Check it out and let's dive deeper into understanding hallucinations in AI text generation! ➡️ https://lnkd.in/ed4m7dCA #SemanticDrift #AIResearch #TextGeneration #LanguageModels #ArtificialIntelligence #MachineLearning #NLP #DeepLearning #AIInnovation #KnowledgeRetrieval #EarlyStopping #Resampling #LLaMa2 #NAACL2024

119

2 comentarios

🚀 Exciting Updates from the AI World! 🌐 I just delved into the latest paper on Llama 3.1, the new set of foundation models developed by the brilliant team at Meta AI. Here's a summary for my LinkedIn network: 🔍 Introducing Llama 3.1: Llama 3.1 is a suite of multilingual, coding, reasoning, and tool-usage language models. The highlight is a dense Transformer with an astounding 405 billion parameters and a context window of up to 128,000 tokens. The Llama 3.1 models are designed to perform on par with leading AI models, like GPT-4, across a wide range of tasks. 📊 Key Highlights: Scalability and Performance: The largest Llama 3.1 model boasts 405 billion parameters, offering exceptional performance and quality. Extensive tests show Llama 3.1 matches the quality of top models on various benchmarks. Multilingual Capabilities: Supports a wide range of languages, enhancing global communication and interaction. Innovative Architecture: Built with a standard dense Transformer model, optimized for stability and scalability. Uses 15 trillion multilingual tokens for pre-training, significantly more than its predecessors. Advanced Multimodal Integration: Incorporates image, video, and speech capabilities via a compositional approach. These multimodal models are still under development but show competitive performance in initial tests. Public Release and Safety: Llama 3.1 models, including the 405B parameter version, are publicly available under the updated Llama 3 Community License. Enhanced safety features with Llama Guard 3 for input and output safety. 🔬 Technical Insights: Pre-Training: Leveraged a vast corpus of about 15 trillion tokens, ensuring a robust and comprehensive understanding of language. Post-Training: Focused on fine-tuning the models to align with human preferences and improve specific capabilities like coding and reasoning. Managing Complexity: Opted for simpler, more stable training processes to ensure maximum scalability and efficiency. 🌟 Impact and Future Prospects: Llama 3.1 is expected to spur innovation in the AI research community and accelerate the development of responsible AGI. The integration of multimodal capabilities is set to revolutionize AI’s interaction with various data types, paving the way for more versatile and intelligent systems. For those interested in the detailed paper, you can check it out https://lnkd.in/gZ6j3gnK. Excited to see how Llama 3.1 will shape the future of AI! 🚀🔮 #AI #MachineLearning #Llama3.1 #MetaAI #Innovation Feel free to engage and share your thoughts on this exciting development!

3

💡 𝗘𝘃𝗲𝗿 𝘄𝗼𝗻𝗱𝗲𝗿𝗲𝗱 𝗵𝗼𝘄 𝗔𝗜 𝗺𝗼𝗱𝗲𝗹𝘀 𝗹𝗶𝗸𝗲 𝗚𝗣𝗧 𝗮𝗻𝗱 𝗕𝗘𝗥𝗧 "𝗴𝗲𝘁" 𝗵𝘂𝗺𝗮𝗻 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲? It all comes down to 𝗦𝗲𝗹𝗳-𝗦𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗲𝗱 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (𝗦𝗦𝗟)! I just published a blog that breaks down this fascinating topic into easy-to-understand sections. 𝗪𝗵𝗮𝘁'𝘀 𝗜𝗻𝘀𝗶𝗱𝗲: 𝗦𝗶𝗺𝗽𝗹𝗲 𝗧𝗲𝗿𝗺𝘀: What SSL is (no jargon, I promise!). 𝗛𝗼𝘄 𝗜𝘁 𝗪𝗼𝗿𝗸𝘀: How models like BERT and GPT use SSL to learn from massive text data. 𝗖𝗼𝗿𝗲 𝗖𝗼𝗻𝗰𝗲𝗽𝘁𝘀: The nuts and bolts of SSL, like masking tokens and next-word prediction. 𝗣𝗿𝗼𝘀 & 𝗖𝗼𝗻𝘀: The benefits of SSL and the challenges it brings (like data noise). 𝗥𝗲𝗮𝗹-𝗪𝗼𝗿𝗹𝗱 𝗜𝗺𝗽𝗮𝗰𝘁: How SSL is powering chatbots, translations, and content creation. 𝗛𝗮𝗻𝗱𝘀-𝗢𝗻 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: A quick look at how to use SSL with PyTorch and Hugging Face. If you're curious about AI and how it's mastering language, this is for you! 📖 Give it a read and let me know what you think. 👇 #GenerativeAI #SelfSupervisedLearning #AI #NLP #30DaysOfAI #TechLearning

8

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

Here's my notes and video for the keynote I gave at PyCon Italia ❤️ 🐍 🇮🇹 yesterday, "Closer to the Metal." The post itself is titled "don't worry about LLMs" and that's really what I was trying to get at with this talk. LLMs are a big topic and there is a lot of demand to build them quickly, but the space is very confusing and inflated with hype. How do we remove the hype and use LLMs to build things we need? We focus our attention on one thing at a time and use the practical engineering and machine learning skills we already have.

217

5 comentarios

In the last post: https://lnkd.in/dgKCkZUB, we discussed fine-tuning of LLMs. Today we will be looking at one new technique RAFT. 🔎 In the context of working with LLMs, two common approaches are often used: RAG Fine-tuning Fine-tuning LLMs involves adapting the model to specific domains whereas RAG is more flexible and can continuously query external sources for up-to-date information. However, despite these approaches, certain limitations hinder LLMs from achieving optimal results. 😔 Retrieval Augmented Fine Tuning (RAFT) is a technique that optimizes LLMs for RAG on domain-specific knowledge by improving its ability to extract information from in-context documents using simple yet efficient prompts and instructions. 😮 It can fine-tune a LLM in such a way that helps it better understand and respond to queries outside its training domain, such as enterprise private documents, time-sensitive news, or recently updated software packages. 💪 Studies show that RAFT outperforms baseline models on various benchmarks, including medical document question-answering. 🚀 For even more better results, Fine-tuning LLMs for longer context and better RAG systems can be achieved by creating a custom fine-tuning dataset for long context and benchmarking popular options. 👍 P.S: What do you think is better, fine-tuning or RAG? ----------------------------------------------------------------------------------------------------- At Antematter, we are building Maximum Performance Blockchain & AI solutions by leveraging our efforts from R&D Lab. #llms #ai #genai #rag

31

7 comentarios

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 comentarios

I'm sure is common knowledge for most of us that LLMs and deep learning are kinda like black boxes, right? But did you know there's a whole subfield of interpretability research trying to crack them open, like the AI version of neuroscience? 🤓 I just wrapped up a fun little project on mechanistic interpretability, digging into bilingual features inside one of GemmaScope’s sparse autoencoders (trained on Gemma 2, Google's latest open-source LLM). If you're into interpretability and like to have a deeper understanding of current LLMs, check it out: https://lnkd.in/eRHd3cdJ

24

"Can LLMs introspect? We define introspection as acquiring knowledge that is not contained in or derived from training data but instead originates from internal states. Such a capability could enhance model interpretability. Instead of painstakingly analyzing a model's internal workings, we could simply ask the model about its beliefs, world models, and goals. More speculatively, an introspective model might self-report on whether it possesses certain internal states such as subjective feelings or desires and this could inform us about the moral status of these states. Such self-reports would not be entirely dictated by the model's training data." https://lnkd.in/e2Cs3WA2

3

After the exciting eureka moment, it's essential to consider how we can integrate Large Language Models (LLMs) into our existing products. We should also keep an eye on potential risks, like the chance that all the proprietary LLMs might start behaving differently or become less intelligent one day. For example, research has shown that models like GPT-4 have seen performance drops on various benchmarks over time. It's an excellent reminder to stay adaptable and evaluate our tools! Ref: https://lnkd.in/g-9Zqi_K

21

2 comentarios

After reading Anthropic and OpenAI success stories with sparse autoencoders for interpretability of text, I was curious about its application to images... After a quick search, I found out that Hugo Fry wrote a blogpost in LessWrong: https://lnkd.in/dA33fWqE which also provides some playground! Of course part of the success is due to the ImageNet structure itself (as many papers out there) but still is an interesting exercise. It enables lots of experiments, and I already can think about practical usage for improving image search in marketplaces. #interpretability #image #CLIP

7

3 comentarios