Ivan Kobyzev

In our latest paper with Jean-Baptiste Waring, B.Eng., we generated a dataset on `ibm_quebec` the 127-qubit IBM Quantum System One located in Canada. The dataset contains more than 4000 experiments of long-range entanglement path qubits from 0 to 100+ qubits. The dataset and the code can be found here: https://lnkd.in/eussBW9G Paper: https://lnkd.in/ekb9Vqja A special congratulations to Jean-Baptiste Waring, B.Eng. for this repo and the quality of the experiments. #quantumcomputing

47

1 Comment

One good news: one of my research papers gets accepted in ICSE-2025 (research track). The title of the paper is “Ranking Relevant Tests for Order-Dependent Flaky Tests”. #ICSE2025, #LLM, #testing, #ottawa - #Canada

89

20 Comments

Recently, the TIGER Lab at the University of Waterloo released MMLU-Pro, a more challenging version of MMLU that includes ~10 options per question. It is a more robust and challenging massive multi-task understanding dataset, tailored to rigorously benchmark large language models' capabilities. The dataset contains 12k complex questions across various disciplines. Since the topics are mostly language-agnostic, I translated it into Italian (the translation should be of high quality) to evaluate Italian LLMs. 📂 You can access the dataset here: https://lnkd.in/ed626hgV

35

1 Comment

I just got my own copy of “Building LLMs for Production” by Louis-François Bouchard and Louie Peters. As I am getting more into applications of GenAI I expect this book to become one of my go-to resources in the upcoming months. https://amzn.to/3Yv5FmL #AI #GenAI #LLMs

2,250

51 Comments

🌟 Saeid Abolfazli and I are thrilled to start new batch for Applied Natural Language Processing course in collaboration with the University of Toronto School of Continuing Studies. Over 12 weeks, we’ll dive into the world of NLP and large language models (LLMs), learning how to process and analyze textual data to extract actionable insights using cutting-edge algorithms and techniques. Whether you're looking to upskill or explore NLP from scratch, this course will equip you with practical knowledge and hands-on experience. Here’s a sneak peek at what we’ll cover: 1️⃣ Week 1: Introduction to Language Data Processing 2️⃣ Week 2: Text Corpora & Preprocessing 3️⃣ Week 3: Fundamentals of Machine Learning 4️⃣ Week 4: Text Vectorization & Feature Engineering 5️⃣ Week 5: Supervised & Unsupervised Learning on Text 6️⃣ Week 6: Context-Aware Language Modeling 7️⃣ Week 7: Text Visualization & Graph Analysis 8️⃣ Week 8: Scaling with Multiprocessing & Spark 9️⃣ Week 9: Deep Learning on Text 🔟Week 10: Transformer Learning for NLP 1️⃣1️⃣ Week 11: Generative AI for NLP 1️⃣2️⃣ Week 12: Capstone Project Join us for this exciting journey into the heart of NLP! Register at http://ow.ly/drbj50B8tzp 🚀 #NLP #GenAI #UofT

39

1 Comment

If you didn't have a chance to attend my Getting Started with Generative Text & Fine-tuning LLMs in Hugging Face workshop at Toronto Machine Learning Society (TMLS) Toronto Summit last month, all of the content is now available online! 🚀 Highlights of the workshop include: 💡 An intro to #largelanguagemodels and generative text models in #HuggingFace 🤗 🌡️An explanation of different decoding strategies for #LLMs (temperature, top-p, top-k, beam & greedy search) ⚙️Fine-tuning GPT-2 medium to talk like a Jedi Master 😱Using LoRA to fine-tune GPT-XL using only a Google Colab free instance 🙊Throwing shade at AGI, in-context learning, and George Lucas Check out the full 90 minute video of the workshop here 📽️: https://lnkd.in/gh5aXy4Q View the workshop materials on Github🧑‍💻: https://lnkd.in/gb7aKSCU Give the code a spin for yourself in Google Colab 🧪: https://lnkd.in/gvnpZWSu

11

Such an interesting read.😍🤯 This builds on anthropic's earlier research on mechanistic interpretability (Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.) Some things that stood out to me: - Social biases can be mapped to feature spaces - By steering these feature spaces we can actually change the social biases in an llm - Steering can sometimes lead to loss in model capabilities. - They find what's called a steering sweet spot which -5 to 5 in steering feature scale where one can change the bias of the model without much change in model capabilities. - Steering a certrain feature are sometimes unpredictable and can extend to other contexts. such as one bias steering can actually effect another bias. - They even found what's called a "neutrality feature" which consistently reduced bias in multiple categories. - This study used a 34M Sparse auto encoder but they suggest that scaling this can lead to better feature sensitivity which makes sense. This is such nuanced research but i'm grateful to the team at Anthropic for pursuing this. I believe this might be one of the most impactful works in the long run. https://lnkd.in/eXeruJkq

8

Quality in, Garbage out. Briefly, this recent work on Nature ( among others ) claims and gives evidence that LLMs seriously degrade their behavior if trained from LLM generated data. To the point of wreaking havoc if the process is repeated 4-5 times. This says a lot about how lossy and noisy is LLM generated content wrt the human produced training data. Not a surprise. Nice read. "We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet."

65

9 Comments

This paper has an interesting concept called PlanRAG, a novel method designed to enhance decision-making capabilities of large language models (LLMs) by structuring the process into two distinct steps: planning and retrieval. Initially, the model generates a decision-making plan, which is then followed by retrieving the necessary data to support those decisions. This structured approach helps the model handle complex tasks more effectively than the traditional agentic approach, where decisions are made directly without a clear plan. By separating planning and retrieval, PlanRAG ensures a more comprehensive and accurate decision-making process. The researchers tested PlanRAG using scenarios from video games and found it outperformed existing methods. They also introduced a new benchmark, Decision QA (DQA), to evaluate the decision-making abilities of LLMs in complex scenarios. The structured plan-then-retrieval method allowed the model to understand what decisions need to be made first and then gather the relevant information, leading to better performance. The study's results demonstrate that PlanRAG is a significant improvement over the agentic approach, making it a promising advancement for using LLMs in complex decision-making tasks. https://lnkd.in/gCbG9ie7

10

I recently read “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning” by Bricken et al 2023. Published at transformer-circuits.pub In this paper the authors discuss the use of autoencoders to extract interpretable features from a one-layer transformer model. The approach aims to improve the understanding and transparency of artificial neural networks. To do this they use the concept of dictionary learning which is a method used to decompose the activations of the transformer model into a set of basis features that are sparse and interpretable. This involves training sparse autoencoders to identify a dictionary of monosemantic features (or features with a single meaning). These monosemantic features in single units (artificial neurons) reminds me of the “grandmother neuron” theory in neuroscience which proposes that specific neurons respond to specific highly complex stimuli. However, in real brains, it is widely believed that complex representations are encoded by the collective activity of groups of neurons (and not monosemantic). But going back to the paper, I really like the visualization tools and illustration to make such complex networks more transparent (as opposed to black boxes). Paper: https://lnkd.in/gR87pJcs

3

Vancouver AI people: I'm hosting an event! I'm founding the Vancouver chapter of The Ai Salon and will be hosting our inaugural session on Thursday, November 28. The Ai Salon is a community founded in San Francisco that brings together scientists, founders, builders, and the generally curious to hold small-sized group conversations on the sociological, economic, cultural, and philosophical impacts of AI. Our inaugural event: Autonomous Scientific Discovery Will we ever be able to build an AI scientist? What would that look like? Is the genius and creativity of scientific breakthroughs something AI could ever replicate? What is the significance of the recent Nobel Prizes in physics and chemistry awarded for work on AI? Let's discuss. This will be an in-person event. Details and registration information at the link below. https://lu.ma/u1px71f0

38

3 Comments

I just looked at this paper that claims to have removed the multiplication of matrices from deep learning. The claim is that one can reduce the electricity demands by a huge margin. I needed to know how they got rid of the multiplication of matrices. The trick is simple to understand. Normally. all the multiplications are between inputs and weights. If they give weights only three possible values: 0, 1, or -1, they can "multiply" any number with those values without actually multiplying. To multiply with zero you return zero. To multiply with 1, you just return the input value. And to multiply with -1, you just change the sign on the input (which is a one-bit operation) and return the changed value. This saves a lot of electricity because the normal floating-point multiplications are expensive -- a lot of work for the processor. This confirms that there is still a lot to optimize in the current DNN implementations. We probably do not need a 16-bit addition either. This can be reduced too, probably again without much loss in model performance. I may have missed something as I ran through the paper quickly.

16

Now available on edX, CS50's new Introduction to Programming with R, a popular language for statistical computing and graphics in data science and other domains, with CS50's own Carter Zenke. Overview at https://lnkd.in/eY-Rw4hk. Enroll for free at https://cs50.edx.org/r. #education #community #r #statistics 

1,584

34 Comments

Happy to share our latest paper, "𝐅𝐢𝐧𝐝𝐢𝐧𝐠 𝐚 𝐍𝐞𝐞𝐝𝐥𝐞 𝐢𝐧 𝐚 𝐇𝐚𝐲𝐬𝐭𝐚𝐜𝐤: A Machine Learning Framework for Anomaly Detection in Payment Systems", co-authored by Anneke Kosse and Jacob Sharples, now available as both the Bank of Canada (https://lnkd.in/e7ch4zeW) and the Bank for International Settlements – BIS (https://lnkd.in/eNdGE2iP) working paper. It employs a layered ML framework designed to effectively handle large volumes and address the scarcity of pre-labeled examples of anomalous transactions, providing new tools for safeguarding financial systems. #AI #machinelearning #anomalydetection #paymentsystems #transactionmonitoring

82

7 Comments

We are organizing a time series workshop in Neurips 2024

15

2 Comments