Pavel Velikhov

In computational linguistics, researchers are focused on enhancing language models' ability to extract precise information from extensive textual data. Recent studies introduce "retrieval heads," specialized attention mechanisms that selectively focus on critical segments of large texts, improving targeted data retrieval. These heads, integrated into transformer-based models like LLaMA and Yi, significantly enhance accuracy and efficiency, particularly in handling long-context scenarios. The methodology involves thorough experiments across various models, including the Needle-in-a-Haystack test, which embeds specific information to measure precision. Results show models with active retrieval heads outperform those without, with accuracy dropping notably when these heads are masked. The empirical evidence underscores the effectiveness of retrieval heads in improving information retrieval precision and reliability. This research deepens our understanding of attention mechanisms in large-scale text processing, paving the way for more efficient and accurate language models. https://lnkd.in/dNNPvMXw https://lnkd.in/ddbHX6EA Innovation Hacks AI #aiinnovation #generativeai

1

How to optimize Ollama for CPU-only inference? I am wondering if anyone has documentation, blog posts, ideas, ... really anything(!) about optimizing inference for CPU-only. The reason is that most people do not have nice GPU(s) in their notebooks and rely on API providers or remote hardware. Would be great to give those people a good hint on which model they could use locally. We are currently on a good way of doing a massive benchmark about CPU-only inference so i appreciate your input greatly!

5.2 greenfield (act 2) (See https://lnkd.in/eP3SxRxm for act 1) Most moving parts were relatively small and isolated components running on autoscaling groups of EC2 instances in different regions. We had most challenges with 1. scaling Elasticsearch on relatively small instances, and two reliably firing thousands of events per second. At that time there was no reliable distributed way to create timed events. The closest in functionality was the good old fashioned cron. That wasn't really an option. Our first working implementation was threaded Python. But we ended with some Scala magic no one was eager to touch. It wasn't distributed, but it was extremely reliable. There are now several established serverless frameworks like Knative, Kubeless and Fission. These frameworks offer triggers with cron-like semantics. They also come with failover features due to their distributed (cloud native) nature. The second challenge is Elasticsearch. Elastic was one of the earlier software producers to distribute their software as a Kubernetes operator. The Elasticsearch operator still requires (public) cloud specific configuration, but it is a step closer to cloud independence. The alternative to Elasticsearch for this particular challenge is Prometheus. Both come with excellent graphical UIs in the form of Kibana and Grafana. These are indispensable for developing, debugging and prototyping. The third major component we used was SQS, AWS' queueing service. This has been so important to us it is almost unimaginable to build what we did without it. At this moment I would look at two different solutions: Kafka and KubeMQ. The first looks to be more heavily supported, but the most important criteria would be developer convenience. We would probably have experimented with both before making a choice. And that leaves us with a Amazon DynamoDB alternative, as the rest is straightforward. We have always been looking at Apache Cassandra, even as very satisfied DynamoDB developers. So that would be the first alternative to experiment with. Perhaps this would be a good point to look at Postgresql, as their Kubernetes distributions look very solid. Or perhaps ScyllaDB… We started developing our gateway with the excellent Raspberry Pi platform. Later we moved to Beaglebone. We would for sure start with K3s on these devices. This simplifies development and deployment. Writing this makes me anxious to try to rebuild this system, if only for the monitoring functionality we have used with great pleasure until the end. Another day... With a small-ish team like we had in the first 6 years we would definitely use EKS (AWS' Kubernetes) and several other AWS services like @Route53. The reliability of the AWS cloud, our experience and the presence (regions everywhere) make this the most reasonable choice. If you have more experience with Google Cloud or Microsoft Azure that would make more sense to you. Act 3: https://lnkd.in/eHREXYQd https://lnkd.in/eamFwzB5

7

Reposting Sohrab Rahimi's review of research direction of #AgenticAI". His conclusion "gaps remain" is true if not an understatement. The reality is that most of the early approaches, even stateful ones, are intellectual dead-ends, based on applying primitive methods (scripts, FSMs, python code, static workflows, etc.) in naive implementations. No matter how much is spent on training, tuning and tokens, #LLMs are inaccurate, inconsistent and in-explainable blackboxes (#ChainofThought itself is not a cure and adds cost/complexity). You can improve pattern matching around the periphery, but an #LLM never becomes #deterministic. As he notes, the early #AgenticAutomation prototypes offer limited capability and aren't secure, scalable, performant or energy efficient. They won't extrapolate into complex, enterprise-grade, production deployments. The reason for that is you need more than just state to improve reasoning. State by itself is just data, without domain knowledge you don't have context beyond basic parameterization. To truly succeed agents need #ontologies with domain #semantics and real-time operational state/telemetry, which is connected to real-world elements (systems, devices, services, databases, applications, etc.). An ontology supports the #reasoning an LLM can't and won't ever do on its own; it supports highly #reflective and deeply #recursive Agentic automation. Simply put, domain knowledge and fact improve inferences (i.e., "informed guesses"). We call this agent-based, ontology-driven, intelligent #GraphRAG #orchestration. Our agents can use the ontology to interpret events, automate decisions and optimize actions - dynamically assembling complex, multi-step workflows and long-running processes. Agent processing is stateless, event-driven, asynchronous and massively parallel. The scope is not limited to #GenerativeAI, an right agent technology with the right ontology can coordinate #hybridAI (LLMs, algorithms and analytics) to optimize workflows and orchestrate business objects. The future is here - EnterpriseWeb. Demo 1: "Talk to the Network" to design, deploy and manage complex services - https://lnkd.in/egJgXYx4. Demo 2: "Talk to Business Systems" to generate and execute ETL workflows - https://lnkd.in/eAYGD4xS.

17

2 комментария

https://lnkd.in/dBkJGXwN A really detailed paper comparing latest results in linear mat-mul free progress. Worth reading and if you are a Ph.D. Researcher something to consider as a great investment for your carreer

6

🔍 Fascinating breakthrough in AI transparency! Just discovered a project that helps us understand how Llama 3 (Meta's latest AI model) processes information internally. Think of it like a CT scan for AI! The Research Team: • Captured 25 million sentences from the web • Fed them through Llama 3 • Built a tool to analyze which "neurons" activate for different concepts Real Example They Found: They discovered a specific neuron (#896) that consistently activates when processing text about the United Nations. 🔬 Why This Matters: • First step towards truly understanding how AI models think • Could lead to more reliable and transparent AI systems • Opens new possibilities for AI development The project is still early-stage research, but it's exciting to see the AI "black box" becoming more transparent! Learn more are at https://lnkd.in/dyz-cEgM What would you want to understand about how AI thinks? 👇 #AIResearch #MachineLearning #TechInnovation #OpenSource

4

How about Spark-like, distributed computing engine, written in Rust? Balista - https://lnkd.in/dzxcWNtx There are number of frameworks allowing you to crunch the data on multiple machines - from father of High Performance Computing - OpenMPI (https://www.open-mpi.org/) towards widespread domination of Apache Spark with some lightweight alternatives like Dask (https://lnkd.in/dqpgUuXB) Balista is comparably young project - but it is already offer: ▪  high performance Rust based engine (up to 10x times ) ▪  support of CSV, json, parquet and arrow data formats from HDFS and S3 ▪  SQL layer on top of Apache Arrow     ▪  taking advantages of columnar format     ▪  lower or even diminish serialization overhead via Arrow Flight (https://lnkd.in/dntViEXF) ▪  that can be used for data transformation - ETLs - beware dbt(!) or OLAP ▪  run (and scale) on top of Kubernetes using keda (https://keda.sh/) auto-scaler ▪  no (yet) dataframes API, Adaptive Query Execution (AQE) or GPU support :( There are other notable projects, that aim to overcome complexity of running (and scaling) Spark cluster as well as JVM issues (mem overhead and sometimes capricious behavior of garbage collector) Ray https://lnkd.in/dZhEcWC6 - more tailored for ML related tasks (both train and inference) or Daft (https://lnkd.in/dH-qMFfY) - that provide DataFrame API on top of Ray's distributed compute capabilities or even Chorovod (https://horovod.ai/) - that mainly focus on distributed DL training. It is really fascinating read and skim through architecture and design docs of project at this stage to learn more of what is happening under the hood, when you submit query - from building AST towards physical execution plan and how compute nodes coordinating to perform parallel execution. TIL;  Substrait (https://substrait.io/) - engine agnostic representation of logical execution plan as alternative for engine specific Catalyst or DataFusion #tech #engineering #distributedcomputing #highload #bigdata #datafusion #spark #balista

4

Sonnet 3.5 has been makingn waves. Personally I have found that perhaps it lacks breadth of knowledge ( not able to comprehensively answer many questions as well as openai) but its definitely faster , mostly gives better code and provides projects and artifacts features which are reason enough for switching to it for techies doing using it for programming tasks.

1

💥💥💥 Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I. Jordan, Song Mei Abstract Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. We elucidate the mechanisms behind extreme-token phenomena. First, we show that these phenomena arise in very simple architectures -- transformers with one to three layers -- trained on a toy model, the Bigram-Backcopy (BB) task. In this setting, we identify an active-dormant mechanism, where attention heads become sinks for specific input domains while remaining non-sinks for others. Our theoretical analysis of the training dynamics reveals that these phenomena are driven by a mutual reinforcement mechanism. Building on these insights, we propose strategies to mitigate extreme-token phenomena during pretraining, including replacing softmax with ReLU and Adam with SGD. Next, we extend our analysis to pretrained LLMs, including Llama and OLMo, showing that many attention heads exhibit a similar active-dormant mechanism as in the BB task, and that the mutual reinforcement mechanism also governs the emergence of extreme-token phenomena during LLM pretraining. Our results reveal that many of the static and dynamic properties of extreme-token phenomena predicted by the BB task align with observations in pretrained LLMs. 👉 https://lnkd.in/d-3mWsh9 #machinelearning

6

Hypothesis: when we talk about "vibes" for a generative model, what we usually mean is a high-dimensional characteristic. If your evaluation outputs a scalar value (eg accuracy or F1), then you're using a one-dimensional metric to evaluate a 1000+ dimensional process and output. Humans can process high-dimensional characteristics easily -- we can "smell" whether writing is good, or interesting, or funny, but they are very hard to reduce to a formalized definition that is programmatically testable, with human-interpretable results. RLHF or RLAIF or expert-as-judge or LLM-as-judge dodge this issue by just having the "opinion" of a set of humans or the raw output of another high-dimensional vector process be the feedback loop for evaluation. This feels a little dissatisfying to me. If it works, then you just know it works because it feels right. If it doesn't, you don't have a principled way of interrogating why. Curious if there is some middle ground to evaluate high-dimensional characteristics in a way that is complex enough to encompass broader types of behavior in models, but simple enough to break down mechanistically to understand why something is or is not working. Let me know if you encounter anything in this vein!

9

2 комментария

I did a quick test of the new function calling in #ollama using llama3.1 8B model. The good news is that the API works! The bad news is that llama3.1 does a poor job at formatting function args according to my tool spec. A comparison w/ gpt-35-turbo: https://lnkd.in/g_R3ExQn Anyone had success with local SLMs and function calling?

17

We've turbocharged nannyML open source 💨 Nikolaos Perrakis has been able to optimize some of our calculations. Amongst them, we have: The estimate method for performance estimation with CBPE is 30% faster. Fitting a Multivariate data drift with PCA is now 3x faster than before. All univariate data drift methods are, on average, 30% faster. Read more about this update and how it impacts you in this blog post by Taliya Weinstein: https://lnkd.in/dziz_EF7

13

🚀 How long will it take for AI to code independently? Let's see it at work on one of the most crucial and difficult problems in Operating Systems design: the scheduler! 🖥️ Before we dive into how AI generates and optimizes scheduler code, let's have a look at the exciting new Linux developments that allowed such a test. ✨ Andrea Righi is doing amazing and fun things with sched_ext, sched_scx, and scx_rustland. With this new Linux kernel framework, developers can experiment with custom scheduling policies without requiring a system reboot or risking kernel crashes. 💡 This is possible because sched_scx uses BPF (Berkeley Packet Filter), an isolated virtual machine inside the kernel that safely executes custom code. Even if a BPF-based scheduler misbehaves, it won’t crash the system, allowing developers to make dynamic, real-time adjustments to scheduling policies without any downtime! 🔄 And here’s where AI comes in. Andrea ran an experiment where he used ChatGPT-4 to help generate and improve scheduling policies written in Rust. 🧠 The AI worked on a simple FIFO scheduler, and through iterations, it optimized the code for specific workloads, reducing execution times and enhancing performance. Though AI isn’t ready to fully replace human developers just yet, it’s a powerful tool for lowering the barrier to entry in kernel development and helping with code refinement! 🛠️✨ Check out more details from Andrea’s experiment in his blog! #AI #LinuxKernel #Scheduling

10

4 комментария

Coolest tool I've found lately. It also has a Neovim plugin if you want to do it live in a buffer. Text UIs are simple useful if your development flow resolves around terminal editors and multiplexers. https://k9scli.io/

4

2 комментария

💥💥💥 ᴏᴘᴇɴꜱᴄʜᴏʟᴀʀ: a retrieval-augmented LM to help scientists synthesize knowledge. With open models & 45M-paper datastores, it outperforms proprietary systems & match human experts. It's a retrieval-augmented LM with 1️⃣ a datastore of 45M+ open-access papers 2️⃣ a specialized retriever and reranker to search the datastore 3️⃣ an 8B Llama fine-tuned LM trained on high-quality synthetic data 4️⃣ a self-feedback generation pipeline Demo 👉 https://lnkd.in/dMG7e_ct Blog post 👉 https://lnkd.in/da24E6uV GitHub👉 https://lnkd.in/dSY5suDm Paper 👉 https://lnkd.in/dTwwFTcF 💥💥💥 ꜱᴄʜᴏʟᴀʀQᴀʙᴇɴᴄʜ, a new large-scale multi-domain benchmark for scientific research synthesis, covering CS, Bio and Physics. 1️⃣ 7 datasets across four scientific disciplines 2️⃣ 2,000+ expert-annotated question and 200 answers 3️⃣ Automated metrics for citation accuracy, coverage, & quality #machinelearning

4

Biological models need much longer context than textual models. Custom models and techniques are required to solve long context biological problems. adding here a great medium post about long context models by Krzysztof K. Zdeb https://lnkd.in/dD9bz_nS

35

4 комментария

The DataFusion SIGMOD paper is officially out. Congrats to Andrew Lamb and the other authors! The paper is great, accessible read covering the architecture of a modern, state-of-the-art query engine. Also exciting to see Arroyo (YC W23) get a mention! The coolest thing about the paper (and about DataFusion itself) is that it demonstrates how building the engine in an open, extensible way doesn't preclude great performance. Systems built on top of DF (like Arroyo) can easily add new functions, operators, tree-rewrites, optimizations, data types, and more while using the the same tools and infrastructure that the built-in functions/operators/etc. are built with. This is incredibly powerful, and has already led to a blossoming of powerful data systems including Comet (a Spark accelerator), LanceDB (a vector DB and query engine), and ParadeDB (which embeds an OLAP DB in postgres). Read the paper here: https://lnkd.in/dpknuPy4

33

2 комментария