Pengfei Xing

Check out our Amazon Science blog post on flow-adhering planning with LLMs. Here we discuss how we can enable LLMs to make reliable API calls by preserving API dependencies and following prescribed sequence of operations (e.g., pre-defined workflows, business policies, and so on). The study was presented at NAACL this year. Paper link: https://lnkd.in/g8gccxpp

48

1 Comment

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

The BayLearn 2024 abstract submission deadline has been extended to Aug 5, 2024. If you are working on cutting-edge ML research, please consider submitting an abstract. BayLearn 2024 will be an in-person event, held on Oct 10, 2024: https://baylearn.org This is a great opportunity for student researchers to showcase their latest research work and get good feedback. Submissions are non-archival and the review process is quite informal. #machinelearning #research

68

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

Super interesting study about removing attention layers in LLMs with little performance degradation. I think as models become more and more commodified, it becomes less about having the biggest model and more about computational efficiency. #LLM #GenAI

6

🚀 𝗥𝗲𝗰𝗲𝗻𝘁 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀 𝗶𝗻 𝗔𝗜 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 🚀 𝗚𝗿𝗼𝗸𝗙𝗼𝗿𝗺𝗲𝗿: Graph Fourier Kolmogorov-Arnold Transformers Introducing a novel Graph Transformer network that surpasses conventional self-attention mechanisms. 1. Tackles limitations of existing GTs in modeling complex node label patterns. 2. Incorporates learnable activation functions in the graph spectrum. 3. Demonstrates superior performance on multiple node and graph classification datasets. Paper: https://lnkd.in/gDKKk-2x 𝗦𝗧𝗔𝗥 𝗔𝗧𝗧𝗘𝗡𝗧𝗜𝗢𝗡: Efficient LLM Inference Over Long Sequences Recent advancements in Large Language Models (LLMs) now handle contexts extending to millions of tokens. 1. New attention mechanism designed to address computational challenges during inference while preserving accuracy. 2. Opens up new applications, including repository-level code analysis, multi-document summarization, and large corpus retrieval. Paper: https://lnkd.in/gtj29ghz 𝗚𝗹𝗶𝘁𝗰𝗵 𝗧𝗼𝗸𝗲𝗻𝘀 𝗶𝗻 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀: 1. Categorization Taxonomy and Effective Detection. 2. Addressing the issue of glitch tokens that lead to hallucinations and erratic behavior in LLMs. 3.Introduction of GlitchHunter, a novel clustering-based detection method. 4.Constructs a Token Embedding Graph (TEG) and uses clustering for efficient detection. 5.High precision and recall across multiple LLMs. Paper: https://lnkd.in/gTVr8jgK #MachineLearning #AIResearch #GraphTransformers #LLM #DeepLearning

4

Just did a keynote in PyTorch conference, on LLM deployment. Thanks for the great work of PyTorch team, our partners, and the interests from the audience! Please try out our deployment ecosystem of torchchat, torch.compile, torch.export, AOT Inductor, torchao and ExecuTorch, and checkout all the amazing talks, posters and demos in the conference!

50

3 Comments

Check out this course from deeplearning.ai, explaining the advanced capability of function calling in LLMs. #LLMs #GAI #NexusAI

4

A platform to learn and gain insights as per interest. Look into the GitHub Repositories to gain knowledge on Machine Learning and Artificial Intelligence.

2

Large language model on a microcontroller? That won't be happening any time soon—but what we CAN do is build a pipeline that distils a subset of knowledge from an LLM (or other foundation models) and squeezes it into a tiny model that can run on any device. And you can do the same with Edge Impulse! This post shows how easy it is to integrate tools like OpenAI GPT-4o with Edge Impulse. We use GPT-4o to rapidly label data and use it to train an NVIDIA TAO object detection model—which we've squeezed down to fit on an Arduino! https://lnkd.in/eSSHpq_i Follow Edge Impulse for more features like this that you can use right now!

87

6 Comments

It may be easier than you think to use your skills for server-based deep learning on Apple devices. Yunfei Cheng and I attempted to evaluate the learning curve by comparing MLX kernels working on Metal GPUs in Apple Silicon chips to PyTorch kernels on CUDA GPUs. The image below depicts the scalability of self-attention and linear projection on M1 Max, M2 Ultra, A100, and H100. The x-y plane represents the beam shape size used in our Recurrent Drafting work (https://lnkd.in/dvrvUwbU). All of these kernels show a similar scalability trend as the beam shape grows. It is interesting to reveal that the performance difference between CUDA and Metal in SDPA is considerably lesser than in linear projection. For example, linear projection indicated a roughly 100x performance difference between the M1 Max and the H100, whereas SDPA showed just a 25x difference on the same hardware.

28

#LLM sizes will be a significant consideration in the next generation of products that we will be witnessing in the coming years. Router models cannot be the go-to solution eventually. As a founder in this domain, one needs to think beyond constrained boxes and democratize these powerful systems as much as possible for small and medium business. We are planning to set things straight in the coming months. #Angel #angelfunding #VC #investments #VCfunding #AI #AIStartup #Quantization #LLMs #LLM

4

Andrew Ling from Groq is hosting an insightful session on "Extending PyTorch for Custom Compiler Targets." Groq has pioneered the world’s first LPUs (Language Processing Unit), designed for LLM inference and deep learning acceleration. In this session, Andrew will delve into: 🔹 Maximizing inference efficiency for custom hardware targets by annotating PyTorch code with arbitrary information. 🔹 How to use PyTorch model annotations to adjust workload mapping to accelerators while preserving the semantics of the PyTorch inference graph. 🔹 A generalizable technique to inject and recover metadata in PyTorch graphs at different granularities, enabling custom data types and precision information through PyTorch into the Groq compiler. This talk is a must-attend for anyone pushing the boundaries of PyTorch and exploring custom hardware acceleration. See the full abstract here: https://lnkd.in/gwSRjPjz #MachineLearning #PyTorch #groq #hardware Toronto Machine Learning Society (TMLS)

39

Tried to install NVIDIA Isaac Sim to run some local reinforcement learning experiments, but it turns out my (expensive at the time) TITAN V GPU is not compatible. Isaac Sim requires an RTX-branded GPU (which offer hardware-accelerated ray tracing). Fun fact: Even thee latest NVIDIA H100 data center GPUs don't support ray tracing - their flagship GPUs are optimized for AI training + inference. For simulation workloads, you need something like an an L4, or one of the desktop-class GeForce (gaming) or A6000 (pro workstation) GPUs. NVIDIA refer to this as the "3 GPU architecture" (simulation, training/inference, and on-robot). So I've purchased a GeForce RTX 4090, and damn this thing is huge. "Titan" V for scale. Barely fits in my desktop case, and turns out I need to order some more power cables off Amazon before I can even plug it in. More of my RL journey to come.

325

44 Comments

Instruction back-and-forth translation: technique to generate high-quality synthetic data for better LLM Alignment In this paper, authors propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). In this approach, given documents from a web corpus, authors generate and curate synthetic instructions using the backtranslation approach and rewrite the responses to improve their quality further based on the initial documents. 𝗠𝗲𝘁𝗵𝗼𝗱 𝗼𝘃𝗲𝗿𝘃𝗶𝗲𝘄 Proposed pipeline consist of following steps: i) Backtranslation - first fine-tune a base model, e.g. Llama-2, on some seed examples from Open Assistant, for the task of instruction generation.  - then extract initial candidate responses from a web corpus, e.g. Dolma, and use the fine-tuned model to obtain synthetic instructions that would go with the corresponding responses ii) Filtering - separately obtain an instruction-following model by fine-tuning the same base model on the seed examples, and use it to score the quality of the (synthetic instruction, web-scraped response) pairs iii) Rewriting -  With the highest scoring pairs, we ask an existing aligned model (e.g. Llama-2-chat) to improve the responses further, conditioned on the generated instructions and the initial web texts.  - Steps (i) and (ii) follow Backtranslation approach proposed in (Self-alignment with instruction backtranslation. by Li et al.) with some modifications (i.e. using preprocessed documents from Dolma instead of parsing raw HTMLs from ClueWeb). 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - Fine-tuning with the resulting (backtranslated instruction, rewritten response) pairs yields higher win rates on AlpacaEval than using other common instruction datasets such as Humpback, ShareGPT, Open Orca, Alpaca-GPT4 and Self-instruct - demonstrate that rewriting the responses with an LLM outperforms direct distillation, and the two generated text distributions exhibit significant distinction in embedding space - results shows that backtranslated instructions are of higher quality than other sources of synthetic instructions, while our responses are more diverse and complex than those obtained from distillation - find that instruction back-and-forth translation combines the best of both worlds—making use of the information diversity and quantity found on the web, while ensuring the quality of the responses which is necessary for effective alignment. 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 - proposed pipeline revolves around general-purpose English instructions, with limited coding or science-related tasks 𝗕𝗹𝗼𝗴: https://lnkd.in/ei7pUGP9 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/e26geGgC

30

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

6

1 Comment

Excited to have presented our paper on Bayesian Prompt Ensembles at ACL 2024. In this paper, we explore the use of multiple prompt re-formulations to capture model uncertainty in LLMs. We derive the connection between prompt re-formulation and model uncertainty/Bayesian NNs and propose a method to learn the relative importance of different prompts for best performance. As part of this work, we open-sourced a code package and tutorial with which you can a) easily use LLMs to perform classification tasks and b) apply our method, BayesPE, to improve uncertainty estimation through the use of multiple prompts. Hope it is useful to folks out there! Paper: https://lnkd.in/etpRRwJC Code: https://lnkd.in/eDHdQ6k4

26