Jack F.

We set up infrastructure for managing Metaflow at Nutrien and the AWS integration is so profoundly sweet. The batch and secretsmanager integration not withstanding, the S3 module, which I believe is just a wrapper around the odd cantrips that exist within boto3, is so seamless. The `get_many` and `put_files` functions are smooth and quick. I did not even know there was such a thing as a S3 Transient Error from slamming S3 with put requests, but Metaflow does and it gracefully handles the retries out of the box. Truly S-tier python package for working with AWS. https://lnkd.in/gTe3ApAc #AWS #Metaflow #MLops #boto3

17

Ten years ago, having a Feature Store in my pocket was unimaginable 🤔. Everything was stored in CSV files, scattered across folders, and crammed into a single database. Back then, the concept of MLOps was unfamiliar to me, so everything was done manually. 🤣 Do you remember when you wanted to train a new model, you'd be waiting for the latest CSV file from the data science team? Or if you wanted to run inference, you'd be on hold until the ML team provided the updated .pt file? 𝐵𝑢𝑡 𝑡ℎ𝑜𝑠𝑒 𝑑𝑎𝑦𝑠 𝑎𝑟𝑒 𝑔𝑜𝑛𝑒... Today, 𝐅𝐞𝐚𝐭𝐮𝐫𝐞 𝐒𝐭𝐨𝐫𝐞𝐬 are a cornerstone of 𝐌𝐋𝐎𝐩𝐬, streamlining data management and model deployment. They provide a centralized repository for features, ensuring consistency, reusability, and faster model iteration. A feature store plays a critical role in machine learning workflows by: 𝐂𝐞𝐧𝐭𝐫𝐚𝐥𝐢𝐳𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐌𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭: It centralizes feature data, making it accessible and reusable across multiple machine learning models and projects. 𝐄𝐧𝐬𝐮𝐫𝐢𝐧𝐠 𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲: Consistent feature calculation ensures that the same data preprocessing steps are applied in both training and prediction phases, reducing errors. 𝐈𝐦𝐩𝐫𝐨𝐯𝐢𝐧𝐠 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲: By storing pre-computed features, it significantly speeds up the experimentation process, allowing for rapid testing of different models. 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐄𝐚𝐬𝐞: As projects grow, a feature store can manage the scaling of data operations efficiently, supporting larger datasets and more complex feature engineering tasks. If you want to learn more about feature stores: You can read more here ↓↓↓ Features Store: https://lnkd.in/d9DV7UaJ 🔗 𝑇ℎ𝑒 𝑅𝑜𝑙𝑒 𝑜𝑓 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝑆𝑡𝑜𝑟𝑒𝑠 𝑖𝑛 𝐹𝑖𝑛𝑒-𝑇𝑢𝑛𝑖𝑛𝑔 𝐿𝐿𝑀𝑠: https://lnkd.in/d6SfMtpc

22

5 Comments

https://lnkd.in/d2RJB9J9 Pump and Dumps, Quantum Computing, Hype Cycles and more -this is relevant for anyone working in a field that is high on the Gartner Hype Cycle... but what about #AI? (see my take below) Sabine hits it right on the mark with this video. #QuantumComputing is mostly just #Hype. I say this with a PhD in Physics/Engineering, and as someone who was told by many to "go to that field". However, I've also lived and been burnt by many other "hype" fields in my career (nanotech, ARVR, LIDARs) to recognize one [See: "The Life of Brian" for a reference 🙄 ] Quantum computing is extremely interesting. It's extremely hard. There are many things that need to be solved before this industry actually becomes an industry. By jumping onto #SPACs, some of these companies may have burnt the bridges on this technology for the future, as investors will be turned off by it. But it needs to be researched - the fundamental usage of it makes sense (as opposed to, say, NFTs). Well, what about AI?!? Is it another hyped technology: I get this question all the time! While my personal preference is to work on complex systems (eg quantum stuff), my Product sense recognizes massive Opportunities in the AI space. Not just in the development of new AI models (and I mean truly new models, not cherries or strawberries on top), but in the Application of what already exists. If you think that AI hasn't already disrupted the world, then clearly you don't have children. The next generation of workers are more AI-savvy, and will outperform those who aren't. We need to ensure, though, that they use it effectively and responsibly.

3

Welcome FalconMamba 7B - permissively licensed, State Space Model trained on 5.5T tokens, near infinite sequential pre-fill! 🔥 > Base model only - scores higher than Llama 3.1 8B on ARC, GSM8K & TruthfulQA > Also, beats L3.1 8B on MUSR and GPQA > Trained on 256 8x H100s on AWS using 3D parallelism w/ ZeRO (took 2 months) > Trained on 5.5 T tokens (4096 -> 8192 ctx len) > Uses the same tokenizer as Falcon 7B & 11B > Released under Apache 2.0 licensed w/ acceptable use policy > Integrated w/ transformers 🤗 Kudos to TII for a brilliant base model! Now go fine-tune it y'all! 🫡

312

8 Comments

BRIEFING REPORT: Drut’s DX3.0 Maximizes GPU Hardware Utilization For AI Workloads The folks at Drut Technologies Inc (pronounced “droot”) have created an AI compute environment for the rest of us. Who’s the rest of us? Anyone that’s doing AI-related work–training models for generative AI applications, for instance–but isn’t doing so with hundreds of billions or trillions of tokens. This matters because most of the noise around the challenges of AI data centers is related to problems that hyperscalers have…but probably not your company. For AI computation, hyperscalers are running massive networks with many thousands of nodes feeding data from a storage network on the frontend to a GPU network on the backend. The GPU network is working on distributed math problems–problems that are broken into interdependent chunks, each chunk fed to a GPU in a cluster. If a chunk of the distributed math problem doesn’t make it to its assigned GPU due to network latency or packet loss, the entire GPU cluster is idled. And no one wants that. - GPUs are expensive resources that shouldn’t sit around idle. - The training process in the case of very large models takes weeks and even months, and infrastructure efficiencies compound over that span of time. For most AI compute networks, the preceding problem isn’t the same scale of concern as it is to hyperscalers. The average enterprise isn’t working with models that large or GPU clusters that big. In fact, the opposite is a more likely situation. An enterprise might have big iron servers loaded with GPUs, but doesn’t have AI compute jobs big enough to merit tying up an entire server while the job runs. Or maybe they do have meaningfully large AI compute jobs, but only from time to time. In that case, maybe they wish they could easily spread the work over more than one server. The problem is less about maximizing GPU utilization, and more about flexibly allocating GPUs and other hardware. This is where Drut comes in. With their DX3.0 software release, Drut allows you to bring together hardware from different chassis’ into what they call a vPOD. A vPOD is defined as “some number of CPU slices, some number of memory slices, some number of OS slices, some number of GPUs, and some number of NICs.”   With this architecture, you’re free to create an AI compute node out of whatever elements are available. Drut takes care of all of this with the DX 3.0 system, a combination of Linux, KVM, Drut’s Software Platform and Fabric Manager, and AI/ML Workbench. DX3.0 is virtual disaggregation that allows for flexible allocation of expensive hardware. This LI post is an abbreviated version of a more detailed post at PacketPushers.net. Read the rest here: 👉 https://lnkd.in/gJwY8JZu Thanks for briefing Packet Pushers, Simon McCormack & William Koss.

14

1 Comment

Large language models are wonderful at ingesting large amounts of content & summarizing. Uploading an academic paper and I can pester it with an infinite list of questions & it will respond with equally infinite patience. In comparing the two most recent Microsoft earnings calls, Claude highlighted: - faster than expected Azure growth (29% vs 27%) - AI contributing 8% of Azure revenue up from 7% - higher CapEx spending & greater capacity constraints for data centers - much better commercial bookings growth : 17% vs 13% Excellent analysis. Benn Stancil described LLMs as great averagers of information. But what if I don’t want the average? If I seek the data point two standard deviations out? If I’m channeling my inner Anthony Bourdain & I’m seeking the fermented shark or cobra heart within my query? At Google, we ranked web pages & ads with many signals. An engineer taught me that of them found rarity of a word across a set of documents. For example, if there’s a document containing 10 instances of the word ghotiacross a collection of documents, where the average document has 0 ghotis, that document is likely the best to answer a search about fish. I haven’t found a way to goad an LLM to produce the rare result. Why did one user decide to use a particular piece of software when 10 others did not? Maybe I haven’t learned how to prompt an LLM well. Ideogram launched a feature called Magic Prompt that expands a basic prompt into richer instructions that elicit better results, marrying the language of a user & a computer. A user might write : A boy with a dog in a park. Magic Prompt replies : A heartwarming scene of a young boy playing with his happy, wagging dog in a lush, green park. The boy has a playful smile on his face while he tosses a bright yellow ball for his furry friend to fetch. The park is filled with beautiful flowers and trees, creating a serene atmosphere for this bonding moment between boy and dog. Magic Prompt surprised me in three ways. First, unlike Google search where terse queries tend to work very well, LLMs relish verbosity. Less Hemingway & more Faulkner. Second, maybe there’s a prompt for rarity. If you know what it is, please share with the rarity shibboleth. Third, I wonder what other prompt techniques are out there that have yet to be discovered to wrest the best from an LLM. This technique is called TF-IDF. It means Term Frequency Inverse Document Frequency.

21

5 Comments

Been working on an evaluation method for benchmarking JSON-LD entity extraction with LLMs. Somewhat surprising how poor all the big-name LLMs are for validating a pre-extracted entity. It’s kind of a needle in a haystack eval. So far, GPT-4o Mini does the best, even over GPT-4o and 4 Turbo. Sonnet 3.5 works ok but not yet reliable. Here’s my method for constructing the prompt, and suggestions are welcome. - Advise LLM of the job to be done, ie evaluate previously extracted entity in JSON-LD format, against the original text it came from. - Insert JSON-LD to be evaluated. - Provide CoT steps to follow, for reading over text and entity, finding entity in text, eval of entity properties, and how to respond with classification enum and reason string - Insert document text (~7k tokens, 5pg PDF) - Wrap up prompt and say to return tab-delimited string With GPT-4o Mini, this now seems to work reliably but other models - even GPT-4o - will say entity isn’t found, or entity is incorrect in comparison to the text. (For exact same formatted prompt) I tried duplicating the instructions above and below the text, and that didn’t hurt but still was seeing issues on some models. Kind of surprising how poor this kind of round-trip analysis is with LLMs. Using LLM-as-a-judge when comparing to longer format source text isn’t as reliable as I expected. Or I’m screwing something up. 🤔😄

3

Do LLMs really have "Emergent Properties"? That one single feature that is used to market and over-hype LLMs is actually a big debate topic! In this length medium article “A Sanity Check on Emergent Properties in Large Language Models” the author, Anna Roger, critically examines the often-cited phenomenon of emergent properties in LLMs. She highlights the lack of clarity surrounding what constitutes “emergence” and presents empirical data that challenges its validity. One of the most striking findings is that many NLP researchers are skeptical of emergent properties themselves—only 39% believe they exist, yet 70% think that most of their peers believe in them, reflecting a sociological misconception in the field. Furthermore, the article argues that what is often called emergence could be an artifact of in-context learning or data contamination, suggesting that these properties are not as “unexpected” as believed. Perosnally, I think that while LLMs undoubtedly show impressive capabilities, we should remain cautious in ascribing phenomena like “emergence” to them without more rigorous definitions and investigations, especially with what this promise brings to business stakeholders and how it sets the expectations of what LLMs can do.. if overdone, it may put us again into another AI winter. Article Link : https://lnkd.in/eKQCdcN2

6

Brendan Gregg has been busy. He is now trying to build full stack profiling for GPU kernels, i.e., flame graphs which show all the code right from the CPU down to the last bit of kernel code running on the GPU. He calls it AI flame graphs. It seems he has it ready for the Intel GPUs. He shows it in his latest blog post. He is so right that while we take the tooling available in the CPU world for granted, the GPU world is still very primitive with no visibility into what's running on the GPU, and no tooling to profile or control tasks. Hopefully things will improve soon. Link to his blog post in the comments

172

2 Comments

NVIDIA's very own proprietary LLM is here. But how does NVLM 1.0 stack up against competitors like LLaMA? 👁️‍🗨️ NVLM 1.0 is probably still best suited for an enterprise setting. It excels in multimodal tasks, delivering top performance on complex datasets and achieving rapid inference speeds with NVIDIA's H100 GPUs. This makes it ideal for high-throughput, real-time applications. LLaMA 3.2, on the other hand, is all about efficiency and flexibility. It shines in mobile and edge environments with a lightweight architecture that balances power consumption and speed. That means it's perfect for developers working on constrained devices or looking for open-source flexibility. So, at the end of the day… which is best for you? It comes down to whether you need enterprise-level performance or scalable, low-power efficiency. Have you gotten your hands on either of these models? I'm curious to hear what you think. #AI #NVIDIA #Meta #GenerativeAI #MachineLearning

19

Data contamination may be the biggest problem in LLM evaluation benchmarks today. Because LLMs have used so much training data, their evaluation sets often include identical or similar data to something they've already seen (i.e. testing on the train set). This completely undermines evaluation results and trust in the model quality. Consider how much GPT-4o performance plummets in the LiveCode code generation benchmark on problems after the cutoff date for its training data (over 20%!) One partial fix is to only ever evaluate LLMs on unseen data collected after the training data cutoff.

12

Building the next agentive architecture is where a lot of Value unlock will happen. Building the Agent infra is now accessible by using the following Agent-Dev tools. LLMs like Llama, ChatGPT etc are for general use case. They will not give you the best outcome for your specific usecase. That's where Agents comes in, its just opposite of LLMs, they are trained, custom-built and used for a very specific task or set of task for a very definite desired outcome. I have been speaking to a few VCs, Founders brainstorming to see what gaps are there and the opportunity that still lies in certain sectors. If you are a founder or VCs and would love to chat on "Agent Infra/Agent Economy", I am just a DM away. We have been cooking something amazing. Image credit - Felicis #AI #Agent #AIAgent #llms

9

5 Comments

Transformer-based LLMs have an inherent performance bottleneck. SSMs like Mamba provide an alternative LLM architecture that could work well when transformer-based LLMs struggle to scale with very large context. I've been intrigued by Mamba since it was launched back in March. Finally getting around to reading a bit more about it in this article from The Gradient: https://lnkd.in/ebVtWQPv

12

How does online iterative RLHF improve LLMs? A team at Salesforce released a paper with a reproducible recipe for online iterative RLHF, showing online RLHF methods, such as online iterative DPO, outperform offline methods.😍 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 0️⃣ Train or select a Reward Model that is used to rate responses 1️⃣ Select SFT LLM as “Policy” and a set of training inputs (prompts), e.g. 60k, and split them into y iterations, e.g. 3x 20k 2️⃣ Use the Policy to generate k samples for all y prompts 3️⃣ Rate all k samples per y prompt and build preference pairs of the best and worst response 4️⃣ Apply a DPO to train Policy on these pairs. ⇒ Repeat 2️⃣ to 4️⃣, where for 2️⃣, the previous DPO LLM is used. 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 🔄 Used an iteration size of 20k prompts and generated 8 responses per prompt 🌈 Diverse prompts and filtering of low-quality data are important for effectiveness 🏆 Iterative DPO outperforms offline DPO on 7/9 tested benchmarks 🥇 Iterative DPO Llama 3 8b outperforms Meta Llama 3 8B instruct on MT Bench, Chat-Arena-Hard 📐 Adding a length penalty into the reward calculation helped to reduce length bias 🤗 Code, Models, Dataset & training details open source Paper: https://lnkd.in/eV3_av5F Code: https://lnkd.in/eDMMABiR Reward Model: https://lnkd.in/e3sqdXej DPO Model: https://lnkd.in/eNuSWEw9

257

2 Comments

🩺 Quantum-classical Bayesian neural networks are not only long enough to be a title all their own, but taking on a challenging medical dataset. Talk about ambitious. Today's dispatches from quantum: ⚆ Zaiku Group Ltd, National Quantum Computing Centre (NQCC), and Oxford Quantum Circuits (OQC) are using hybrid classical-quantum federated learning to securely handle massive biomedical datasets, showing promise in genetic data analysis ⚆ MITRE advises the next U.S. administration to prepare for quantum computing's potential to crack encryption, emphasizing advancements in infrastructure protection and a zero trust framework ⚆ Berkeley Lab uses femtosecond lasers to create precise telecom-band optical qubits in silicon ⚆ Riverlane joins Tech Nation’s Future Fifty to develop a quantum error correction stack for error-free operations ⚆ Institute for Quantum Computing team completes a quantum source for the QEYSSat, Canada’s first quantum satellite ⚆ IonQ and SC Quantum kick off a Faculty & Researcher Learning Series to promote quantum tech adoption in academia and business ⚆ Quantum phase estimation and quantum algorithms for the discovery of homogeneous catalysts, improve design and efficiency in chemical manufacturing ⚆ A proposed quantum-classical Bayesian neural network improves reliability in medical image classification by combining classical CNNs with quantum circuits for better uncertainty estimation ...and more. 📰 Stay quantum-savvy. The Daily Qubit offers daily insights into the rapidly evolving world of quantum computing. Join leaders in academia, founders, and industry experts in staying informed. Sign up at www.thedailyqubit.com and never miss a qubit. #quantumcomputing #quantumphysics #innovation

3

The amount of human effort that went into RLHF methods at OpenAI for alignment on top of Pretraining + SFT , are surely reducing after gpt3.5 . Meaning you have now LLMs doing most of the RLHF methods such as ranking of responses, given they are already aligned. Of course this process would also be human controlled at top level but the overall human effort surely decreases with every release. Its almost like already aligned AIs now aligning the next generation AI. Just like how humans have created Governments , and Governments once established, can align individual at each level as they pass on the hierarchy of policies from top to down.

1

This paper brings back fond memories of my own programming days: C++ design patterns for low-latency applications including high-frequency trading This work aims to bridge the existing knowledge gap in the optimisation of latency-critical code, specifically focusing on high-frequency trading #HFT systems. The research culminates in three main contributions: 🔹 the creation of a Low-Latency Programming Repository, 🔹 the optimisation of a market-neutral statistical arbitrage pairs trading strategy, and 🔹 the implementation of the Disruptor pattern in C++. The repository serves as a practical guide and is enriched with rigorous statistical benchmarking, while the trading strategy optimisation led to substantial improvements in speed and profitability. The Disruptor pattern showcased significant performance enhancement over traditional queuing methods. Evaluation metrics include speed, cache utilisation, and statistical significance, among others. Techniques like Cache Warming and Constexpr showed the most significant gains in latency reduction. Future directions involve expanding the repository, testing the optimised trading algorithm in a live trading environment, and integrating the Disruptor pattern with the trading algorithm for comprehensive system benchmarking. The work is oriented towards academics and industry practitioners seeking to improve performance in latency- sensitive applications.

15

3 Comments

When I talk about BaldrAI, many people ask if we're using LLMs (Large Language Models). If I ask them where they think LLMs would be useful it appears that they believe the hype and assume that LLMs are AGI (Artificial General Intelligence). There is only one way to say this: LLMs are not and can never be AGI. The long and short of this is that Kaspar Von Grünberg is right! LLMs are massively over-rated, and I think this mostly comes from the way that OpenAI talks about GPT and how the next version will have "PhD-level intelligence". What OpenAI claims is fundamentally impossible as the critical reasoning skills of LLMs are literally non-existent. It doesn't matter how much data you train it on, it cannot think, it can only guess the next word. This gives the illusion of thought, but it is not actually thinking, it's like a child who has studied for a test by memorizing the answer sheet - it has no understanding, only the answers to the questions. We must not hand the world to these models or treat them like gods. If we do, Azimov will be right and the robots will lead to the fall of humanity.

2

1 Comment