Mihailo Backovic’s Post

Apple's recent announcement at WWDC about Apple Intelligence has created an excitement in the tech community. The utilization of On-Device LLM inference is truly impressive and in sync with Apple's privacy and responsible AI policies. What stands out is the handful of remarkable optimizations applied to the foundation model and adapters, enabling the model to operate with an impressive latency of just 0.6 milliseconds per token. The incorporation of speculative decoding further elevates the results. Here are some key highlights from this innovative development: - Shared input and output vocab embedding tables - Implementation of low-bit palletization for on-device inference - Use of grouped-query-attention in both on-device and server models - Development of a new framework using LoRA adapters with a mixed 2-bit and 4-bit configuration strategy (3.5 bit per weight average) - Introduction of Talaria, an interactive model latency and power analysis tool, for bit rate selection guidance - Adoption of activation quantization and embedding quantization - Innovation in efficient Key-Value (KV) cache update on neural engines, which must be a custom kernel level optimization These optimisations signify a remarkable step forward in machine learning and artificial intelligence, setting new standards for efficiency and performance. Learn more about Apple's Foundation Models at https://lnkd.in/enwB3GJS. #Apple #WWDC #MachineLearning #ArtificialIntelligence #optimization #LLM

If you are interested in AI and Machine Learning, I really recommend you head to machinelearning.apple.com to read up on how Apple developed their "Apple Intelligence" coming in upcoming iOS, iPadOS and MacOS. The local model is a 3B parameter SLM (Small Language Model) that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. This is in line with my predictions last week that I posted in Tech Insights Week 23. Anything running locally or in Apple's Secure Cloud is an Apple model, not OpenAI. https://lnkd.in/dCr7UqTC

Introduction of Apple Intelligence (AI wink wink) marks a groundbreaking advancement in AI, enhancing Apple OSes. These powerful on-device and server-based models transform user experiences by refining text, summarizing notifications, creating images, simplifying app interactions and much more. With Apple’s overall focus on privacy and efficiency, Apple's new models are set to revolutionize how their vast user base interacts with technology. Learn more about it here: https://lnkd.in/g6qdcXmz #apple #wwdc #ai #ios18

Are LoRA adapters adept ? Recently at #WWDC2024, Apple announced "Apple Intelligence", a suite of AI tools which seamlessly integrates into everyday workflow (the best kind of AI tools). With Apple's strong focus on privacy, one of the key highlights is their on-device handling of AI models. Particularly (https://lnkd.in/eJv2X5ym), they present how they use LoRA adapters to handle different tasks using a single foundational model.  If you have been in the LLM space, you might also know about NVIDIA's NIM, which provides LoRA adapter swarm (https://lnkd.in/evSZQrSw), which allows you to host a single model with multiple adapters fine-tuned for specific tasks. The benefits are obvious: 1) One model deployment can handle multiple specialized tasks. 2) Less resource requirements for deployment and fine-tuning. 3) Lower model management overhead. Seems like a great idea and something you should leverage, right ? Well, not necessarily... From a technical perspective, (read https://lnkd.in/ehFbsXQm): 1) LoRA in most cases, is just a style transfer, it shifts the token probability distribution. The tasks that we are fine-tuning using LoRA, already need to be in the knowledge base of the foundational model.    2) The fine-tuning does not scale with dataset size.  Adding more data samples to fine-tuning will have diminishing results on model performance. And in the case of out-of-distribution tasks (tasks not seen by base model), we might not even get any improvements. From a use-case perspective: 1) If our use-case has fundamentally different tasks (language translation vs code translation), then it does not make sense to use LoRA adapters. 2) In case we want to have efficient resource utilization and want to provide more functionality, provided we won't have concurrent tasks, this arrangement fits perfectly. Not to say standard fine-tuning is better, far from it. In most cases, full fine-tuning of LLMs actually has more issues. But that is a discussion for another time. For now, let's make lemonade. #artificialintelligence #llm #machinelearning #data

On June 10th, Apple, via their “Machine Learning Research” site, released a paper titled “Introducing Apple’s On-Device and Server Foundation Models, Our Focus on Responsible AI Development.” Link: https://lnkd.in/gVNQdpWJ This is an in-depth paper on Apple’s AI advancements, specifically focusing on their new on-device and server foundation models, which were introduced at the 2024 Worldwide Developers Conference. The paper details how these models are integrated into iOS 18, iPadOS 18, and macOS Sequoia (still no mention of the Apple Vision Pro), and how they are designed to perform specialized tasks efficiently, accurately, and responsibly. I’m still only halfway through, but there are a few things that have already stood out to me: 🍎 1. In the paper, they outline their set of “Responsible AI principles” to guide how they develop AI tools. The first principle states: “Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.” This emphasizes Apple's strategic and very verticalized approach to personalized AI and what I believe will further develop their Apple moat. They are not building a generalized world model; they left that to the OpenAI experts. Instead, they’re building very narrow and specific models for specific use cases, and I think that is the trend that we will continue to see take off: Niche AI, where AI is developed and mastered for very specific use cases because that is how maximum value will be recognized. When you can have individual, highly efficient, low-cost models running for specific use cases, coupled with an orchestration layer that can choose the right model for the right use case – that’s when I believe the magic will really happen. 🍎 2. In the Apple research paper, they mention the use of Tensor Processing Units (TPUs), which are essentially specialized hardware accelerators designed specifically for machine learning and artificial intelligence tasks. Specifically, Apple states, “Our foundation models are trained on Apple's AXLearn framework, an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs.” This had me scratching my head, as TPUs are widely known to be developed by Google. Johnathan Ross, the founder of Groq and a pioneer in cloud inference technology, is credited with inventing the TPU, so it’s very interesting to see Apple leveraging Google’s technology. Given this surprising integration, it will be fascinating to see what unexpected collaborations and innovations emerge as these tech giants strive to maintain their lead in the AI race. What collabs do you foresee?

🚀 Yesterday at WWDC, Apple showcased the future of AI inference: 📉 Smaller LLMs: These smaller language models pack a punch, offering robust AI capabilities without sacrificing performance through fine-tuning. 🔧 Model Evaluations with Talaria: Ensures high performance and resource efficiency. 🌟 LoRA Adapter and Low-bit Palletization: For highly efficient optimization. ⚡ Local Inference: Achieving an impressive 0.6 ms latency per prompt token and 30 tokens per second on the iPhone 15 Pro 🔒 Custom Foundation Model: built-in opt-out feature for web publishers enhances privacy and control in AI applications. 🔄 Adaptive Model Specialization: Models can now dynamically specialize for specific tasks without compromising core pre-trained parameters References : https://lnkd.in/e2HhSxcW Also, the security and privacy handled end to end check out this https://lnkd.in/ey62DQrb #WWDC #Apple #AI #MachineLearning #AIInference #FoundationModels #MLOPS

What happens when the sleek design of Apple meets the world of AI? The answer might surprise you. When it comes to AI, Apple has made significant strides both on-device and on their servers. Let’s break it down. Apple’s on-device AI is comparable to other tiny models. This means it’s great for quick, efficient tasks right on your device, offering supercharged capabilities for Siri and other features. However, their server-based AI, roughly equivalent to GPT-3.5, is where more complex processing happens. While these models are not at the frontier level, they still provide robust performance for various tasks, including decent writing assistance. Despite these advancements, it’s important to note that Apple’s AI offerings aren’t aimed at the highest-end use cases. They are more about enhancing user experience through improved interaction and support, rather than pushing the boundaries of AI research and application. For many users, this balance between functionality and efficiency will be more than sufficient. Check out more details here: https://buff.ly/4eaUJA9 #AI #Apple #Technology #MachineLearning #Siri

Great to see Apple Intelligence's focus on privacy, making it clear to people how and when data might be shared. But did Apple underplay the strength of the model behind their offering? The charts in this release seem to indicate a greater level of human satisfaction with the outputs from Apple's models compared to others, "We find that our models are preferred by human graders over most comparable competitor models". Read more at https://lnkd.in/g4svAR9q #Apple #AppleIntelligence #AI

Apple Intelligence 🍎 🧠 - "AI for the rest of us" 🤔 As a long-time Apple customer, I’ve been willing to smash open the piggy bank every once in a while to get my hands on a device that I am confident will work securely and without year-on-year deterioration. For me, that is Apple's signature, and has been achieved largely by virtue of Apple's closed ecosystem, abhorred by developers worldwide and championed by Steve Jobs himself. It was always going to be difficult to square Apple’s DNA with the #GenAI gold rush, the latter being characterised by premature releases, unreliable products, and disregard for legal rights. So, after yesterday’s announcement, have Apple pulled it off, or is the late Mr Jobs spinning in his grave? 👓 In my opinion, they've done ok. By placing a heavy emphasis on privacy, Apple is leaning into a key differentiator over its competitors. Your data will remain on device or in a "private cloud", and most tasks will be handled by a number of relatively small, specialised, on-device models, which compare favourably on various metrics to similar-sized models from Microsoft, Meta, and Google. I expect this has been achieved by meticulous filtering of training data. Similarly strong performance is observed in IBM's #Granite models, for which the training datasets were decimated to remove profane and low quality content (as well as clear cases of piracy). Similar filters are applied to Apple's training data, as set out in the following technical page: https://lnkd.in/eW5ynywQ. It's not all good. I would have much preferred to see Apple train on legally clean datasets. The technical page states that they used licensed data and "publicly available" scraped data. The latter amounts to a fair use defence to copyright infringement, which (whatever anyone says) is legally uncertain. As far as I am aware, the #KL3M models by 273 Ventures are still the only #LLMs that don't rely on this. These models achieve particularly low scores on toxicity and perplexity (a proxy for uncertainty - i.e. hallucination rate), which are metrics I care about a lot, particularly if we are giving the models agency to interact with apps on my device. I could also have done without the "Image Playground", but the biggest disappointment for me is Apple outsourcing more complex queries to OpenAI. On one hand I get it - they have the most capable models and I expect Apple has the leverage to force Sam to play nice with your data. But the company is a hotbed of malpractice and their association is bad for the Apple brand. Another concern is that we seem to be heading for a global single point of failure and single source of bias. I expect Apple to liberate themselves from OpenAI as soon as they viably can. In summary, I don’t love everything about Apple's entry to the #AI arms race, but in the absence of a better alternative, they've done just enough that I’ll probably raid the piggy bank once more 🐷

🚀 Very proud to have contributed to Responsible AI efforts for Apple Intelligence 🍎! Looking forward to seeing the new horizons of creativity and productivity that will be unlocked by these innovative GenAI features, and grateful to be a part of such a talented and dedicated team that not only pushes the boundaries of technology, but is constantly redefining them, with privacy and safety top of mind. Learn more about Apple's responsible AI principles here: #AppleIntelligence #TechInnovation #TeamApple #ResponsibleAI #WWDC24