Yunfei Cheng

Huge release by Daekeun Kim and Hyo (HK) Choi from our AI GBB team in Korea, sharing Korean language benchmarks for our latest Azure OpenAI model, gpt-4o-mini - like gpt-4o, the model excels on Korean language tasks, in this case mini completely eclipses gpt-35-turbo and nearly achieves gpt-4-turbo performance. The benchmarks and code are open source, extensible beyond our Azure models, so we encourage our customers and the larger community to try it out and even fork/share your own findings - and please let me in the comments what other Asia languages beyond Korean you may be looking to evaluate with your language models.

52

7 Comments

I am thrilled to share my recent project - an implementation of the cutting-edge MLP-Mixer architecture using PyTorch! MLP-Mixer is a groundbreaking neural network architecture that has demonstrated exceptional performance in a variety of computer vision tasks. My implementation is now publicly available on GitHub: https://lnkd.in/gfgcXJKW In this repository, you will discover a comprehensive explanation of the MLP-Mixer architecture, along with clear guidelines on how to utilize my implementation to train your own MLP-Mixer models effectively. Additionally, I have included a Jupyter notebook that showcases the power of the MLP-Mixer by training it on the well-known CIFAR-10 dataset for image classification. I would like to extend my gratitude to the original authors of the MLP-Mixer paper for their innovative contribution and for openly sharing their code. It is my hope that my implementation will serve as a valuable resource for fellow researchers and practitioners in the deep learning community, fostering further exploration and advancement in the field of computer vision. Feel free to explore my repository and provide your valuable feedback! I am always open to collaborations and engaging discussions. #MLP #MLPMixer #PyTorch #DeepLearning #ComputerVision #AI #MachineLearning #Innovation

5

𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝗡𝗲𝘄𝘀 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠𝘀) 𝗦𝗽𝗮𝗰𝗲! PyTorch has officially released TorchTune, a new tool designed to fine-tune Large Language Models (LLMs) with ease and efficiency. This groundbreaking tool promises to streamline the fine-tuning process, making it more accessible to researchers and practitioners alike. https://lnkd.in/eMf6dNZJ 𝗧𝗼𝗿𝗰𝗵𝘁𝘂𝗻𝗲 𝗢𝗳𝗳𝗲𝗿𝘀: ▪ 𝗣𝘆𝗧𝗼𝗿𝗰𝗵-𝗻𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀: Build on familiar ground with composable building blocks for popular LLMs. ️ ▪ 𝗠𝗲𝗺𝗼𝗿𝘆-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗿𝗲𝗰𝗶𝗽𝗲𝘀: Train on consumer-grade GPUs with 24GB VRAM – no beefy machines needed! ▪ 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗮𝗻𝗱 𝗲𝗮𝘀𝗲 𝗼𝗳 𝘂𝘀𝗲:  YAML configs and support for common dataset formats streamline your workflow. 𝗙𝗼𝗿 𝘁𝗵𝗼𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗱𝗶𝘃𝗶𝗻𝗴 𝗱𝗲𝗲𝗽𝗲𝗿, 𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Code: https://lnkd.in/gEgETe5t Blog: https://lnkd.in/gP3mctA6 Tutorials: https://lnkd.in/gvh2eEsU This is a fantastic opportunity to explore and innovate, I'd be very interested in hearing your thoughts. ▪ 𝗪𝗵𝗮𝘁’𝘀 𝘆𝗼𝘂𝗿 𝘁𝗮𝗸𝗲 𝗼𝗻 𝘁𝗵𝗶𝘀?  ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝗶𝗺𝗽𝗮𝗰𝘁𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗼𝗿 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵? ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝘄𝗶𝗹𝗹 𝗶𝗺𝗽𝗮𝗰𝘁 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗜 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁? #AI #PyTorch #TorchTune #LLMs #MachineLearning

24

1 Comment

I am excited to write this blog together with (Kevin) Huan-Ping Su from Union. - Anyscale is the best place to run Ray workloads, and Ray is the go-to solution for ML and platform engineers to manage the end-to-end lifecycle of ML workloads as a unified computing engine. - Union.ai is the best place to orchestrate Flyte pipelines, and Flyte is a great solution to address gaps within ML pipelines, such as managing task lifecycles for infra engineers and accessing data from data warehouses. This blog will dive into a RAG example to show the perfect marriage between Anyscale and Union. Sign up for Ray Summit (raysummit.anyscale.com) to learn more about Ray!

61

https://lnkd.in/gRenYMXW I really enjoy this distributed training guide from Lambda Labs. It starts from a single gpu to train a moderate GPT2 model, and progressively evolve code and settings to utilize multi-node multi-gpu, all the way up to training Llama 405B. With my learning budget, I likely will stay under 7B models for a while. But these distributed training recipes surely help me run experiments faster.

16

4 Comments

I recently saw an amazing video on 𝐆𝐏𝐔 𝐨𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐰𝐨𝐫𝐤𝐬𝐡𝐨𝐩 organized by Chip Huyen. Here is the video link: https://lnkd.in/gVm-KQtt If you are an ML Engineer my 2 cents to improving LLM deployments on GPUs is to use 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 and 𝐨𝐩𝐭𝐢𝐦𝐢𝐳𝐞 𝐊𝐕-𝐜𝐚𝐜𝐡𝐢𝐧𝐠 as mentioned in the video. So how can we go about doing it? Here I provide a few open source projects which you can use today to heavily optimize your LLM deployments across both servers and on device. I am pretty sure (depending on your server load and number of users) you are looking at saving potentially millions of dollars! 🎁 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 If you are deploying models on servers such as GCP/AWS/Azure where you are generally batching the requests then you are better off with using SmoothQuant(https://lnkd.in/gDxt4F_Z) to quantize the models to 8𝐛𝐢𝐭𝐬 𝐰𝐞𝐢𝐠𝐡𝐭𝐬 𝐚𝐧𝐝 8𝐛𝐢𝐭𝐬 𝐚𝐜𝐭𝐢𝐯𝐚𝐭𝐢𝐨𝐧𝐬. If you are deploying models on device (Mobile, Laptop) etc. Then you should use AWQ(https://lnkd.in/gGNhPcPR) to quantize the models to 4𝐛𝐢𝐭𝐬 𝐰𝐞𝐢𝐠𝐡𝐭𝐬 𝐚𝐧𝐝 16𝐛𝐢𝐭𝐬 𝐚𝐜𝐭𝐢𝐯𝐚𝐭𝐢𝐨𝐧𝐬. 𝐈𝐧 𝐛𝐨𝐭𝐡 𝐨𝐟 𝐭𝐡𝐞𝐬𝐞 𝐜𝐚𝐬𝐞𝐬 𝐲𝐨𝐮 𝐰𝐢𝐥𝐥 𝐞𝐧𝐣𝐨𝐲 𝐚𝐜𝐜𝐮𝐫𝐚𝐜𝐲 𝐰𝐢𝐭𝐡𝐢𝐧 1% 𝐨𝐟 𝐭𝐡𝐞 𝐨𝐫𝐢𝐠𝐢𝐧𝐚𝐥 𝐁𝐅𝐥𝐨𝐚𝐭16 model. 🎁 𝐏𝐚𝐠𝐞𝐝 𝐊𝐕 𝐂𝐚𝐜𝐡𝐢𝐧𝐠 𝐚𝐧𝐝 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐫𝐞𝐮𝐬𝐞. Paged KV caching is a way to use non-contiguous blocks of memory to store KV caches so that we aren't wasting precious GPU memory. KV cache reuse is: if two sets of queries are similar, we can reuse the same KV caches instead of recaching them. Both of these might sound intimidating to implement but, the great news is that, this is all part of vLLM(https://lnkd.in/g7uyGPQA) which is based on PagedAttention(https://lnkd.in/gRExM3c5). So in short just use the vLLM as your deployment server! #machinelearning #deployment #inference #llm #chatgpt #inference #costsavings #artificialintelligence

147

2 Comments

🎉I will be at CVPR in Seattle next week to present our accepted paper: "BoQ: A Place is Worth a Bag of Learnable Queries" 📄Summary: We introduce Bag-of-Queries (BoQ), a novel neural network architecture for Visual Place Recognition that employs learnable global queries to capture universal place-specific attributes through cross-attention. BoQ offers a consistent and interpretable attention mechanism and achieves SOTA results on multiple large-scale benchmarks. 📑Paper on ArXiv: https://lnkd.in/ePquujQU  💻Code: https://lnkd.in/eQKsKjX7 Many thanks to Philippe Giguère and Brahim Chaib-draa for their invaluable contributions. I am looking forward to discussing our work with the community and learning from other brilliant minds. If you’re attending CVPR in Seattle, let’s connect! #CVPR2024 #MachineLearning #ComputerVision #Research #AI #Localization #PlaceRecognition

37

4 Comments

Excited to be at #NeurIPS2024! 🎉 Join us for the poster presentation on "OTTER: Effortless Label Distribution Adaptation of Zero-shot Models" 📍 Location: West Ballroom A-D, Poster #6700 🗓️ Date & Time: Friday, Dec 13, 4:30 PM – 7:30 PM PST Short thread ⬇ 🤔 Imagine you’re taking a standardized exam like the SAT and realize you’ve chosen option (d) on 15 out of 20 questions. What would you do? 🤔 💡 Knowing that standardized exams often have evenly distributed answer choices, you’d likely revisit uncertain questions and modify your answers to the next most plausible options. We translate this intuition into an optimal assignment problem: selecting the most plausible answer set to match the known answer (label) distribution. Our method (OTTER) effectively mitigating label bias and boosting performance by: 🔹 +4.8% for CLIP image classification 🔹 +15.9% for text classification tasks. 📖 Our theoretical results show OTTER addresses label shift efficiently, decomposing errors into label distribution specification and calibration errors. Curious to learn more? Check out our paper: https://lnkd.in/gChtxQ5M Swing by the poster session for a chat—I’d love to hear your thoughts! 🚀

55

This week we are presenting our paper "LiNR: Model Based Neural Retrieval on GPUs at LinkedIn" accepted at CIKM 2024 (https://lnkd.in/gUrWqRcD). Please stop by and say hi to Aman Gupta, who will be there in person :) We discuss our experiences and challenges in creating scalable, differentiable search indexes using TensorFlow and PyTorch at production scale. In LiNR, both items and model weights are integrated into the model binary. Viewing index construction as a form of model training, we describe scaling our system for large indexes, incorporating full scans and efficient filtering. We believe LiNR represents one of the industry's first Live-updated model-based retrieval indexes at production scale. Talented co-authors include Fedor Borisyuk, Qingquan Song, Mingzhou Zhou, Ganesh Parameswaran, Madhulekha Arun, Siva P., Tugrul Bingol, Zhoutao Pei, Stanley(Kuang) Lee, Lu Z., Hugh Shao, Syed Ali Naqvi, Sen Zhou, Aman Gupta

171

4 Comments

This is my second anniversary working at RisingWave. I started there as an intern working with the mentorship of ZHI LIU, where I got to deliver cutting-edge work such as implementing SqlSmith for RisingWave in Rust, and streaming query optimization. From there I started deep-diving into the RisingWave Kernel under the leadership of Eric Fu, working on features such as Backfilling, and new deployment modes, closely collaborating with fellow engineers, and having strong cross-collaboration with other teams (cloud, docs, product) as well. In these two years, I'm always amazed by my fellow engineers working on the Database Kernel. They are extremely strong engineers with both breadth and depth, have high ownership and accountability, making them a delight to work with. The same qualities are consistent across HR/Ops, Cloud, Product, Performance and Reliability, and Documentation teams. In my opinion, the #streamprocessing market is still nascent, where many are still using batch OLAP / OLTP systems to do adhoc computation, instead of leveraging streaming databases to do incremental computation. We are beginning to see a bigger shift in the market, where many move from just doing stream ingestion, and start to explore streamlined systems such as RisingWave to do stream processing, in view of cost-savings and greater performance. Moving forward, I'm doubling down on user-experience and developer-experience in RisingWave to continue driving adoption and improve our engineering efficiency. Looking forward to the rest of 2024 and beyond! #streamprocessing #database #sql #streaming #kafka #rust #risingwave

107

8 Comments

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

179

2 Comments

I believe that feedback is one of the most powerful tools to improve at a rapid pace, and I always endeavor to obtain as much feedback and as detailed feedback as I can to help accelerate my growth! I saw a recent interview piece where Jensen Huang from NVIDIA was commenting about feedback, which made me reflect. 🛠️ 𝐖𝐡𝐲 𝐅𝐞𝐞𝐝𝐛𝐚𝐜𝐤 𝐌𝐚𝐭𝐭𝐞𝐫𝐬: 𝐂𝐥𝐚𝐫𝐢𝐭𝐲 𝐚𝐧𝐝 𝐃𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧: Feedback provides clear insights into our performance, helping us understand what we're doing well and where we can improve. It acts as a compass, guiding us on the path to mastery. 𝐒𝐤𝐢𝐥𝐥 𝐄𝐧𝐡𝐚𝐧𝐜𝐞𝐦𝐞𝐧𝐭: Constructive feedback identifies areas for skill development, enabling us to focus our efforts on specific aspects whether that's a deeper understanding of a technology or language or the way an algorithm is structured. 𝐁𝐨𝐨𝐬𝐭𝐬 𝐌𝐨𝐭𝐢𝐯𝐚𝐭𝐢𝐨𝐧: Positive feedback acknowledges our hard work and achievements, fueling our motivation to keep pushing forward. It reinforces our confidence and encourages a growth mindset. 𝐅𝐨𝐬𝐭𝐞𝐫𝐬 𝐂𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧: Regular feedback fosters a culture of open communication and collaboration. It creates an environment where ideas can be freely exchanged, leading to innovative solutions and collective growth. 𝐀𝐝𝐚𝐩𝐭𝐚𝐛𝐢𝐥𝐢𝐭𝐲: The tech landscape is ever-evolving. Feedback helps us stay adaptable by highlighting new trends, tools, and best practices. It ensures we're not just keeping up, but staying ahead of the curve. 🗣️ 𝐄𝐦𝐛𝐫𝐚𝐜𝐞 𝐅𝐞𝐞𝐝𝐛𝐚𝐜𝐤: 𝐒𝐞𝐞𝐤 𝐈𝐭: Don’t wait for feedback to come to you. Actively seek it from peers, mentors, and leaders. The more perspectives you gather, the more well-rounded your growth will be. 𝐆𝐢𝐯𝐞 𝐈𝐭: Providing constructive feedback to others is equally important. It not only helps your colleagues but also reinforces your own understanding and expertise. 𝐀𝐜𝐭 𝐨𝐧 𝐈𝐭: Feedback is only valuable if we act on it. Reflect on the insights and implement the suggestions. And then importantly get feedback again. #SoftwareEngineering #ContinuousLearning #GrowthMindset #ProfessionalDevelopment

1

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

`llm deployment` was one of the most interesting topic from the GPU optimization workshop by Chip Huyen Sharan Chetlur talks about inference optimization for llms 1. majority of the apps are real-time(online systems) and they require acceptable latencies, should be highly accurate(helpful) and should be cost effective as deploying these large models can be expensive 2. these models will keep getting bigger and better(chinchilla scaling laws) 3. so to optimize for these challenges we need a high-performant, robust and customizable solution 4. quantization can be an all-round win w/ just a bit of effort as long as you maintain accuracy 5. post-training quantization - so you can train llms in the BF16 dtype but inference can be done in lower precision dtypes like int8, int4 or even lower 6. post-training quantization can make compute faster, communications between GPUs can happen with high throughput 7. there are other optimization techniques as well like quantization aware training, sparsity 8. at inference an llm request has 2 phases prefill - processes the prompt, generate 1st token and initialize the KV cache generate - generate next token using last generated token and the KV cache and update the cache 9. compute in the attention block depends on the kind of implementation (can be compute or memory bound) 10. these are compute and memory bounded phases and serving solution should have custom CUDA kernels for high perf (e.g. flash attention). remember we need to keep our CUDA cores busy. 11. static batching - traditionally, requests are accumulated over some time window, and executed as a batch until completion. this can work if we have to do fixed amount of work but in the space of llms outputs differs massively in length(e.g. "the tallest ferry wheel" vs "explain the multiverse theory") 12. in-flight batching - in llm inference theres multiple forward passes per request which is kinda unknown and unbounded(we don't know how many tokens need to be generated). so in-flight batching treats an llm iteration as a single request 13. sometimes in a single iteration some requests are in generation phase whereas others are in prefill phase. these tokens in different phases can then be concatenated for higher throughput. 14. paged KV cache - traditionally these caches were contiguous in memory and this leads to wastage of memory since its based on max sequence length. paged KV cache however are partitioned into blocks and these do not need to be contiguous. this way memory waste only happens in the last block. 15. KV cache reuse - now this representation allows memory sharing. for example the system prompt which might be common for 2 requests can now be cached and reused for the 2nd request 16. speculative decoding - makes educated guesses about future tokens while generating the current token, all within a single forward pass which engine do you use for llm serving? trt-llm, vllm, tgi or rayllm?

129

6 Comments

📖 In the last couple of months, I’ve spent quite a lot of time working at the intersection of LLMs and offline reinforcement learning. This topic becomes now more and more exciting since a lot of companies are doing research in test-time compute scaling and offline RL offers one of the alternatives to look at that. So I decided to make up a short list of papers that I found interesting for agent applications: ✅ GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements. A great overview of methods to improve LLM reasoning. It looks at critic models that can steer the generator towards better solutions and suggests training a critic via optimal value function approximation. Experiments are supported by evaluations on GSM8K. ✅ Stop Regressing: Training Value Functions via Classification for Scalable Deep RL. Most of the papers in reinforcement learning treat Q/Value function predictions as a regression task, i.e., using classic L2 loss. The authors suggest switching from regression to classification and present a series of elegant methods to do that: from two-hot encoding to distributional RL. Such methods allow for estimating distributions instead of scalars, leading to more flexibility during inference. ✅ Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. This paper tries to answer which model (prover) we should use to get data to train our critics. Should we use the most capable prover possible (spoiler: no), or how should we find a balance? They suggest an answer to this and also other questions like reward assignment during inference. ✅ IQL (Implicit Q-Learning) & CQL (Conservative Q-Learning). These papers come from classic offline reinforcement learning. Both offer ways to tackle out-of-distribution actions. IQL does that by estimating the upper expectile of the value function, while CQL directly penalizes unseen actions through additional regularization during training. Nevertheless, it can be hard to incorporate into LLM pipelines due to computational restrictions. ✅ Offline RL for Natural Language Generation with Implicit Language Q Learning. An approach with Q/V function predictions at the token level. In this scenario, the authors suggest adjusting final log probabilities to perturb tokens in the direction of maximum utility. This is a rather interesting method applying offline RL to LLMs. ✅ VerifierQ: Enhancing LLM Test Time Compute with Q-Learning-based Verifiers. This paper applies the concepts described in IQL and CQL directly to LLMs, trying to estimate the whole actions in a multi-step scenario. Due to computational constraints, the authors conduct experiments only with TinyLlama-1.1B, so it would be very interesting to look at more powerful models here. If you’d like to see a more detailed overview of this or any other paper related to this area, let me know in the comments. All links are attached in the comments. #llm #agents #reinforcement_learning #transformers

49

1 Comment

🚀 I'm excited to share that our paper, "Enhancing E-commerce Product Title Translation with Retrieval-Augmented Generation and Large Language Models," has been accepted at CIKM 24' - Data-centric AI. Thanks for the co-authors Stephan Walter and Taichi Nakatani for the collaboration! 😀 Our study explores how we can improve the machine translation of e-commerce product titles using LLMs, particularly languages that the LLMs do not primarily support. The results of our experiments show that our proposed approach leverages large volumes of bilingual product data, and utilizes a probabilistic retrieval model to retrieve similar bilingual product information for few-shot learning, can improve overall product title translation quality by up to +15.3% in chrF scores. Notably, we observe that language pairs where the LLM has limited proficiency can benefit the most from our method, addressing a common challenge in this domain. As new and more capable LLMs continue to emerge, our RAG-based approach allows for the quick integration of these models (without extensive fine-tuning). By using the large volume of bilingual product data, including titles, bullet points, and descriptions, as few-shot learning examples, our method provides a flexible and adaptable solution that is particularly suitable for industrial applications, where the ability to keep pace with changing requirements is crucial. #amazonscience #LLM #CIKM #RAG #machinetranslation #searchengine #ecommerce #translation #GenerativeAI #search #MultilingualSearch #mixtral #localization #retail

66

2 Comments

FP8 training with FSDP2 is here, with 1.5X speed up compared to BF16 - 80% from FP8 compute -  20% of the speedup from FP8 all-gather. Great post with all the details, https://lnkd.in/gsdjVR4J #pytorch #Distributed #FP8

65