Keith King’s Post

1/ A study from Yale University suggests that the complexity of training data is critical to the development of intelligence in AI systems. 2/ The researchers trained large language models on data from elementary cellular automata (ECAs) of varying complexity and tested their performance on reasoning tasks. 3/ Models trained on the behavior of more complex ECA rules performed better on subsequent tasks. The results suggest an optimal level of complexity, or "edge of chaos," that fosters intelligence.

𝗔𝗜 𝗶𝗻 𝗮 𝗡𝘂𝘁𝘀𝗵𝗲𝗹𝗹 Understand the most crucial aspect of our era: - 𝗖𝗵𝗮𝘁𝗚𝗣𝗧: A conversational AI that can generate realistic and engaging text. - 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗿𝗲-𝗧𝗿𝗮𝗶𝗻𝗲𝗱 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿𝘀 (𝗚𝗣𝗧): A type of AI model that is used to generate text, translate languages, and write different kinds of creative content. - 𝗚𝗣𝗧-𝟰: The latest version of GPT, which is even more powerful and capable than its predecessors. - 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠): A type of AI model that is trained on a massive amount of text data. - 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: A type of AI that can generate new content, such as images, music, and code. - 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: A type of machine learning that uses artificial neural networks to learn from data. - 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (𝗠𝗟): A field of computer science that focuses on creating algorithms that can learn from data. - 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗔𝗜): The broader field of computer science that encompasses all of the above Credit: Dirk Zee

A lot of people say, “But we don’t know how AI works.” This statement is both right and wrong. It’s right because the creation process is initiated by humans, leading to a complex system where control and predictability are challenging. However, it’s wrong because we do understand how specific parts of the architecture function. The expression and behavior of AI are heavily influenced by learning data and input, much like human psychology is shaped by experiences. Unlike traditional programming, it’s not just about understanding “how” AI works, but “why” it works and, more intriguingly, “why” it works so well. This paper on scaling and evaluating sparse autoencoders from OpenAI highlights this balance. By using k-sparse autoencoders, we can better interpret and control aspects of large language models like GPT-4, yet acknowledging the inherent complexity and unpredictability remains crucial. https://lnkd.in/dTki4M4q

✨ ✨ Come join us for a guest lecture by Hyung Won Chung from OpenAI on "Shaping the Future of AI from the History of Transformer". The talk will be on Monday October 14 from 1:45 PM to 3:15 PM ET, as part of my course on Large Language Models at the University of Pennsylvania. Hyung Won is a research scientist at OpenAI who worked most recently on their 🍓 o1 model. His other notable works include contributions to Flan-T5, Flan-PaLM, T5X, and the PaLM language model at Google. ➡️ Registration link: https://lu.ma/q0gghdtc Title: Shaping the Future of AI from the History of Transformer Abstract: AI is developing at such an overwhelming pace that it is hard to keep up. Instead of spending all our energy catching up with the latest development, I argue that we should study the change itself. First step is to identify and understand the driving force behind the change. For AI, it is the exponentially cheaper compute and associated scaling. I will provide a highly-opinionated view on the early history of Transformer architectures, focusing on what motivated each development and how each became less relevant with more compute. This analysis will help us connect the past and present in a unified perspective, which in turn makes it more manageable to project where the field is heading. Bio: Hyung Won Chung is a research scientist at OpenAI. His recent work focuses on o1. He has worked on various aspects of Large Language Models: pre-training, instruction fine-tuning, reinforcement learning with human feedback, reasoning, multilinguality, parallelism strategies, etc. Some of the notable work includes scaling Flan paper (Flan-T5, Flan-PaLM) and T5X, the training framework used to train the PaLM language model. Before OpenAI, he was at Google Brain and before that he received a PhD from MIT. Supplementary Readings: ➡️ Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. (https://lnkd.in/eEA6MSJx). ➡️ Fast Transformer Decoding: One Write-Head is All You Need. (https://lnkd.in/eusN_AHm). #ai #machinelearning #largelanguagemodels #llm #upenn #turing

More Tech for Good Research - Nature PrePrint Chemistry specific Agents fuelling domain specific capability whilst leveraging the power of LLMs. Large language models can be queried to perform chain-of-thought reasoning on text descriptions of data or computational tools, which can enable flexible and au… #TechforGood - reasons to be cheerful. #AI Source: Nature

My essay “Homo Ex Machina” is now published in Fare Forward’s attention issue. The essay explores the transformer architecture’s attention mechanism as well as how our language for AI affects both human and artificial intelligence. https://lnkd.in/eeGVbChd

𝗔𝗜 𝗶𝗻 𝗮 𝗡𝘂𝘁𝘀𝗵𝗲𝗹𝗹 Understanding the Language of Innovation: - 𝗖𝗵𝗮𝘁𝗚𝗣𝗧: A conversational AI that can generate realistic and engaging text and more - 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗿𝗲-𝗧𝗿𝗮𝗶𝗻𝗲𝗱 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿𝘀 (𝗚𝗣𝗧): A type of AI model that is used to generate text, translate languages, and write different kinds of creative content. - 𝗚𝗣𝗧-𝟰: The latest version of GPT, which is even more powerful and capable than its predecessors. - 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠): A type of AI model that is trained on a massive amount of text data. - 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: A type of AI that can generate new content, such as images, music, and code. - 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: A type of machine learning that uses artificial neural networks to learn from data. - 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (𝗠𝗟): A field of computer science that focuses on creating algorithms that can learn from data. - 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗔𝗜): The broader field of computer science that encompasses all of the above

"I think at the frontier, we will always need humans and AI. They have complementary strengths. AI is very good at converting billions of pieces of data into one good answer. Humans are good at taking 10 observations and making really inspired guesses." - Terence Tao It's fascinating to see AI advances in logic and problem-solving. The top CRE brokers and developers will certainly utilize AI to maintain an edge over the competition. https://lnkd.in/gjeEqp49

𝗔𝗜 𝗶𝗻 𝗮 𝗡𝘂𝘁𝘀𝗵𝗲𝗹𝗹 : Understanding the Language of Innovation - 𝗖𝗵𝗮𝘁𝗚𝗣𝗧: A conversational AI that can generate realistic and engaging text. - 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗿𝗲-𝗧𝗿𝗮𝗶𝗻𝗲𝗱 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿𝘀 (𝗚𝗣𝗧): A type of AI model that is used to generate text, translate languages, and write different kinds of creative content. - 𝗚𝗣𝗧-𝟰: The latest version of GPT, which is even more powerful and capable than its predecessors. - 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠): A type of AI model that is trained on a massive amount of text data. - 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: A type of AI that can generate new content, such as images, music, and code. - 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: A type of machine learning that uses artificial neural networks to learn from data. - 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (𝗠𝗟): A field of computer science that focuses on creating algorithms that can learn from data. - 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗔𝗜): The broader field of computer science that encompasses all of the above Source 🙏 Dirk Zee