Sanjay Basu PhD’s Post

Apple's recent announcement at WWDC about Apple Intelligence has created an excitement in the tech community. The utilization of On-Device LLM inference is truly impressive and in sync with Apple's privacy and responsible AI policies. What stands out is the handful of remarkable optimizations applied to the foundation model and adapters, enabling the model to operate with an impressive latency of just 0.6 milliseconds per token. The incorporation of speculative decoding further elevates the results. Here are some key highlights from this innovative development: - Shared input and output vocab embedding tables - Implementation of low-bit palletization for on-device inference - Use of grouped-query-attention in both on-device and server models - Development of a new framework using LoRA adapters with a mixed 2-bit and 4-bit configuration strategy (3.5 bit per weight average) - Introduction of Talaria, an interactive model latency and power analysis tool, for bit rate selection guidance - Adoption of activation quantization and embedding quantization - Innovation in efficient Key-Value (KV) cache update on neural engines, which must be a custom kernel level optimization These optimisations signify a remarkable step forward in machine learning and artificial intelligence, setting new standards for efficiency and performance. Learn more about Apple's Foundation Models at https://lnkd.in/enwB3GJS. #Apple #WWDC #MachineLearning #ArtificialIntelligence #optimization #LLM

https://lnkd.in/grAibE9m 🚀 **Apple Unveils Next-Gen Foundation Models for On-Device and Server AI** Apple has launched its Apple Intelligence system, introducing powerful generative models for iOS 18, iPadOS 18, and macOS Sequoia. These include a 3 billion parameter on-device model and a larger server model, enhancing user experiences with text refinement, notification summarization, image creation, and more. Prioritizing privacy and responsible AI, these models aim to streamline daily tasks effectively and securely. Some of the benchmarks published are top of the rung 🙌 Learn more: [Apple Foundation Models](https://lnkd.in/grAibE9m) #AI #MachineLearning #Apple #Privacy #Innovation #Tech 🚀📱💡

Apple's WWDC 2024 Highlights: Glimpse into Next-Gen AI Capabilities This year's Worldwide Developers Conference showcased innovation, particularly w/ the intro of Apple Intelligence. As we dive into the details here’s my take on the groundbreaking developments and how they might transform our interaction w/ tech Apple Intelligence: A New Era? Apple’s announcement of integrating Apple Intelligence into iOS 18, iPadOS 18 & macOS Sequoia is a new chapter in personalized tech Comprising various generative models this system adapts to user activities enhances tasks from writing texts to managing notifications, and more Four Things Stand Out: 1️⃣ Dual Models for Diverse Needs: The integration of a ~3 billion parameter on-device model and a more extensive server-based model is intriguing. The balance between on-device processing and cloud-based capabilities could set new standards for speed and privacy. 2️⃣ Commitment to Responsible AI: Apple’s outlined AI principles resonate deeply, especially their focus on user empowerment and privacy protection. The proactive approach to AI misuse and bias prevention is commendable. 3️⃣ Optimization Excellence: The reported improvements in model latency and efficiency on the iPhone 15 Pro are impressive. Apple’s use of low-bit palletization and quantization techniques could be game-changers. 4️⃣ Evaluating the Impact: The benchmarks shared show that Apple’s models not only perform well but are preferred by users over competing models. This human-centered evaluation approach is crucial for understanding real-world utility. Questions: How will this impact developers and creators? What are potential privacy concerns for end-users? How will these changes impact daily tech interactions? I’d love to hear your thoughts on these developments Here's the link to read more: https://lnkd.in/eGnWdimE #Ai #Future #Innovation #Technology #SocialNetworking

If you are interested in AI and Machine Learning, I really recommend you head to machinelearning.apple.com to read up on how Apple developed their "Apple Intelligence" coming in upcoming iOS, iPadOS and MacOS. The local model is a 3B parameter SLM (Small Language Model) that uses adapters trained for each specific feature. Diffusion model does the same thing, adapter for each style. This is in line with my predictions last week that I posted in Tech Insights Week 23. Anything running locally or in Apple's Secure Cloud is an Apple model, not OpenAI. https://lnkd.in/dCr7UqTC

“Apple Intelligence is designed with our core values at every step and built on a foundation of groundbreaking privacy innovations. Additionally, we have created a set of Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them: Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals. Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models. Design with care: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback. Protect privacy: We protect our users' privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users' private personal data or user interactions when training our foundation models.” #aiml #appleintelligence #wwdc2024 #apple #wwdc https://lnkd.in/g8bTreeP

🚀 Yesterday at WWDC, Apple showcased the future of AI inference: 📉 Smaller LLMs: These smaller language models pack a punch, offering robust AI capabilities without sacrificing performance through fine-tuning. 🔧 Model Evaluations with Talaria: Ensures high performance and resource efficiency. 🌟 LoRA Adapter and Low-bit Palletization: For highly efficient optimization. ⚡ Local Inference: Achieving an impressive 0.6 ms latency per prompt token and 30 tokens per second on the iPhone 15 Pro 🔒 Custom Foundation Model: built-in opt-out feature for web publishers enhances privacy and control in AI applications. 🔄 Adaptive Model Specialization: Models can now dynamically specialize for specific tasks without compromising core pre-trained parameters References : https://lnkd.in/e2HhSxcW Also, the security and privacy handled end to end check out this https://lnkd.in/ey62DQrb #WWDC #Apple #AI #MachineLearning #AIInference #FoundationModels #MLOPS

Last year, Apple released open-source tools for training and inference on Apple silicon. Yesterday, they provided more details about their models running on Apple silicon. While Apple's on-device models currently lag behind GPT-4 in performance, their strategy of leveraging the computing power in personal devices offers numerous advantages, potentially leading to low latency and high energy efficiency. Utilizing the processing capabilities of personal devices incurs no additional cost for Apple and ensures data remains on the device. Even the relatively low costs of using OpenAI models can accumulate over time and block some use cases. It remains to be seen whether Apple's on-device models will become sufficiently advanced for widespread applications.

Introduction of Apple Intelligence (AI wink wink) marks a groundbreaking advancement in AI, enhancing Apple OSes. These powerful on-device and server-based models transform user experiences by refining text, summarizing notifications, creating images, simplifying app interactions and much more. With Apple’s overall focus on privacy and efficiency, Apple's new models are set to revolutionize how their vast user base interacts with technology. Learn more about it here: https://lnkd.in/g6qdcXmz #apple #wwdc #ai #ios18

Apple's annual developers' conference kicked off two days ago. As usual, the internet had a field day, with a tsunami of posts about how Apple just revolutionised AI and that nothing will ever be the same… The boring news of day 1 - partnership with OpenAI, ended being the media focus. There are already hundreds of apps which do this 😴 . I also think it will be tricky to convince regulators and customers that letting Siri decide whether your entire phonebook should be sent to OpenAI is “safe". Then again, this is just my speculation so make of it what you wish 😉 The stock market didn't like the news. Apple stock dropped about 2% (but rebounded yesterday… we'll get to that later) The most interesting part of the announcement was actually about Apple's own line of foundational models. You can find the blog post here: https://lnkd.in/eC3r_JMH. I'll try to break down a few important points: - Apple is aware that LLM benchmarks are getting saturated. There is not much to gain here, but they have to show that they can reach SOTA (and they do). I wouldn't expect the new models to perform better than any of the open source models out there. - Instead, Apple focuses on two important aspects: minituarisation and UI/UX. What we saw yesterday is to me a proof of a trend (first with Microsoft's Phi series of models, now with Apple) to get more performance out of smaller LLMs which can run on-device. This is significant because it breaks away from the dogma of "one LLM to rule them all" and it solves a whole plethora of issues regarding data privacy etc. - Apple is focusing on Human experience. A lot about AI is in UX and whoever gets this right will succeed in the AI race! This was the first time I'm seeing emphasis on evals. like "Human Evaluation of Output Harmfulness" or "Human Satisfaction Score" in a large tech company press release, suggesting this is central to Apple's foundational model program. - Apple has been preparing for the "big AI move" for quite some time. There's been a lot of criticism of the company lagging behind competition, but I think this is because they took their time to prepare the ground (e.g. M series chips), giving Apple an edge over the competition. - Apple invested a lot on technical details of minituarisation (e.g. custom LoRA, model quantisation...). The most interesting detail is the Talaria toolkit, which allows to simulate effects of various optimisations on model performance and speed up the optimisation process. To my best knowledge, there is no open source equivalent of Talaria. Apple is in a good place to pull ahead. They seem to be the only company that prepared the ground well before introducing AI into their products, they are doing impressive things with minituarisation and they are aware that the one who solves the "UI of AI" is the ones who wins. The stock market yesterday seemed to agree (AAPL +6.5% at closing)

🚀 **Introducing Apple’s On-Device and Server Foundation Models** 🚀 At the 2024 Worldwide Developers Conference, Apple unveiled **Apple Intelligence**, a personal intelligence system integrated into **iOS 18, iPadOS 18, and macOS Sequoia**. This innovation leverages multiple advanced generative models designed to enhance user experience through specialized tasks like text refinement, notification summarization, and creating playful images for personal conversations. ### Key Highlights: 🔹 **Diverse and Specialized Generative Models:** Tailored for specific user activities, adaptable in real-time. 🔹 **Dual Model Approach:** A ~3 billion parameter on-device language model and a larger server-based model for efficiency and accuracy. 🔹 **Comprehensive Support:** Includes a coding model for Xcode and a diffusion model for visual expression in Messages. 🔹 **Responsible AI Development:** Upholds principles to empower users, represent diverse communities, design with care, and protect privacy. ### Responsible AI Principles: 1. **Empower Users with Intelligent Tools:** Respecting user autonomy. 2. **Authentic Representation:** Avoiding biases and authentically representing global users. 3. **Careful Design:** Considering potential misuse and harm, improving based on feedback. 4. **Privacy Protection:** Utilizing on-device processing and Private Cloud Compute. ### Technical Overview: 🔹 **Pre-Training with AXLearn:** Leveraging advanced parallelism techniques and rigorous data filtering. 🔹 **Post-Training Innovations:** Enhancing instruction-following with novel algorithms like rejection sampling fine-tuning. 🔹 **Optimization for Performance:** Advanced techniques ensure speed and efficiency. ### Performance and Evaluation: 🔹 **User-Centric Evaluation:** Benchmarking through human evaluations and adversarial testing. 🔹 **Superior Instruction-Following:** Outperforming comparable open-source and commercial models. 🔹 **Enhanced Writing Ability:** Leading in summarization and composition tasks. #Apple #WWDC2024 #AI #MachineLearning #iOS18 #Innovation #Technology #Privacy #ResponsibleAI