Shobhit Saxena

With Sarath Shekkizhar, we are exploring the geometric structures behind LLM reasoning capabilities. So far, our findings suggest: 1. Higher density in self-attention graphs implies higher intrinsic dimension. 2. Higher intrinsic dimension induces more expressive power/reasoning with MLPs. 3. Longer context and more attention heads can boost intrinsic dimension. This geometric lens offers new insights into LLM performance and potential improvement strategies. Early days, but we believe it is a promising direction! Stay tuned for more as we dive deeper into the geometry of AI reasoning. Read our pre-print: https://lnkd.in/gcekSWYD #AI #MachineLearning #LLM

25

Intuit leveraged K8sGPT and GenAI in their on-call process, resulting in improved issue detection, debugging, and resolution. 🚀 On-call engineers often face frequent alerts, tedious log scanning, and multiple team handoffs to resolve issues. Often, few teams have high operations load and engineers face fatigue resulting in low productivity. 😌 Intuit was facing similar issues with their complex Kubernetes deployment comprising 325+ clusters and 7000 services. They came up with a methodical approach to solve the problem through GenAI. What were the goals of the project ? 👉 Reduce the Mean Time To Detect (MTTD) an issue 👉 Reduce the Mean Time To Remediate (MTTR) an issue How did they solve the problem ? 1️⃣ Golden cluster signals - Single view of cluster's health and availability. It helped engineers to detect the bad clusters quickly reducing MTTD. 2️⃣ Debugging with K8s GPT - Adopted open-source K8s GPT to analyze root cause of failures. K8s GPT has integration with Open AI and other models. 3️⃣ Remediation with GenAI - Since public LLMs don't have Intuit specific data, they hosted local models and augmented them with Intuit specific data (RAG) that suggested remediation Intuit is planning to extend the tool and use it for other use cases, such as traffic and Java virtual machine debugging. They have published a detailed blog on how they adopted K8s GPT and the challenges they faced. You can find the link in the first comment. 👇 On-call weeks are challenging, but investing in GenAI can streamline the process and improve work-life balance. Share in the comments if you've used GenAI to improve your on-call process. Also, explore how K8sGPT could boost your team's operational efficiency. Also, folks who are interested to know more about K8s GPT can go to the github link, understand the code and start contributing. 🔥 #chatGPT #GenAI #softwareengineering

56

9 Kommentare

https://lnkd.in/gPFuAYBV Real execution case of knowing what and how of profiling can help analyse real time workloads and bottlenecks. This is good reminder of how every code segment adds to PPA.

6

🚀 Making LLMs work for you - A Guide to Become 10x Programmer/MLE/DS I am sure many of you use LLMs for your day-to-day work. Mostly, I have seen people use LLMs while writing code (Copilot), or as an alternative to Copilot to catch bugs, but they have become much more useful than that, provided you know how to use them well. Here are a few ways I have been using LLMs in my daily work that are helping me increase my productivity: 📚 1. Quick Paper Assessment with ELI5 Summaries - Paper abstracts are called abstracts for a reason. Asking an LLM for an ELI5 summary makes it much easier to know if I want to read a paper. 💻 2. Implementing Research Papers in Code - Try giving an LLM like Claude/ChatGPT a research paper and ask to code it. You would be surprised. Tip: Claude performs much better for this task. 🔄 3. Upgrading Code Structure and Readability - Do you want to add typing hints when writing your Python code, or do you need comments in your code, or do you have a big function that you want to be converted to classes in your code? LLMs can do it all, and you would be amazed at the code-conversion power they have. ⚡ 4. Optimizing Code Performance - The thing with LLMs is that if you tell them what to do, they can do it well. For example, if you give them a non-optimized piece of code that runs correctly, they will be able to write it in a much-optimized way. For example, I just used an LLM to convert a matrix multiplication in PyTorch to sparse MM as I was getting an OOM memory error because one of the matrices became too large. I am not even joking, but that would have taken at least 1-2 hours for me to come up with. 📝 5. Automated Documentation - Got bored of writing documentation for what you did but have the code? LLMs also work the other way around. Want to write a summary of what you did in a week, you can use an LLM for that. Saves so much time that gets wasted in formatting. 🧪 6. Streamlined Test Framework Creation - Let LLMs handle the boilerplate for Unit tests & Integration tests while you focus on core logic. 🌐 7. Language-Agnostic Development - You can paste a function in Python and ask for its equivalent in Rust, complete with language-specific best practices and error handling. This dramatically reduces the learning curve when working with new languages and helps maintain productivity during language transitions. 🤝 Apart from all of the above, LLMs work well as mentors as well as interns. LLMs are great when you just need someone to bounce ideas with. I sometimes just talk to an LLM as I would to a colleague talking about the problems I have and how I can solve them. And they come up with the right ideas or send me in the right direction a lot of times. They also act as a personal assistant for free, who can deal with writing messages, creating charts, and doing all sorts of uninteresting work for you. 💭 What's your favorite way to use LLMs? Let me know in the comments! #MachineLearning #AI #Programming

47

Is Protobuf the future default for everything? Timothy Chen & I chatted with Buf CTO Akshay Shah about the future of Protobuf, their new Kafka-compatible streaming layer, and everything he learned scaling Uber that led him to build Buf. This episode covered a pain point I've repeatedly experienced while working at Snyk & see at many other companies: managing cross-team migrations and building a productive internal API ecosystem. Here's a great example: You have two services (A and B), and A depends on B. How do you modify an API or Event in A without requiring B to do a complicated lock-step migration? These are the challenges in any at-scale organization where a project that should take 30-60 days turns into months as you try to navigate alignment and delivery. They're mundane papercuts that turn into hundreds of hours of lost productivity. I love what Buf are doing to help make this a problem of the past.

51

3 Kommentare

🍕 Kafka Partitioning: Slicing Topics for Scale! 🚀 Ever wondered how Kafka handles massive data streams? Let's dive into partitioning! 🏊‍♂️ 🔹 Why Partition? 🤔 Without partitioning, a single topic on one machine would limit Kafka's scalability. Partitioning breaks this barrier! 💪 🔹 How It Works 🛠️ - Splits a topic log into multiple logs - Each partition lives on a separate cluster node - Distributes storage, writing, and processing across nodes 🔹 Message Distribution 📨 1. No Key: Round-robin across partitions   - Even data distribution 📊   - No message ordering preserved 🔀 2. With Key: Computed from key hash   - Same key = Same partition 🔒   - Preserves message order for each key 📅 🔹 Real-world Example 🌍 Customer events with customer ID as key: - All events for one customer land in the same partition - Guarantees order for each customer's events 🔹 Trade-offs ⚖️ - Possibility of "hot" partitions with active keys - Often worth it for key-based ordering Partitioning is Kafka's secret sauce for handling big data at scale! 🌟 #ApacheKafka #BigData #DataEngineering #DistributedSystems #StreamProcessing #ScalableArchitecture #BasuKafka

3

"revolutionized" is a strong word 😅 I am skeptical here. Anyway, one major thing they fail to highlight properly in the blog is: They changed the TTL strategy dramatically enough to impact user experience. How? Hear me out. They are now setting TTL on "UserId" and not on "userId-productId". Earlier each individual userId-productId was tracked for its "own" 24 hour period. Now, for every new productId inserted, we update TTL of the overall "userId" key. They do not say if they are using any special flags while setting TTL. So I am assuming default behaviour. Imagine this use case: Only 2 productIds. P1 and P2. Let's take 1hr TTL value. At 12:01, insert P1 : U1 -> (P1 -> 1) : TTL = Lives till 13:01 At 13:00, insert P2: U1 -> (P1 -> 1, P2 -> 1) : TTL = Lives till 14:00. Now P1 stays till 14:00. In earlier implementation, it would have gone away at 13:01. What it means is that, ALL the "userId-productId" keys stay in memory way way longer than they were meant to -> User experience dramatically changes. The counter value for each individual productId is higher now. Since the new data structure has efficient storage, they are able to get away with it + get good savings on memory. If they had kept the earlier scheme of TTL on the new Redis Hash data structure, it would require way more complex TTL management strategy and infra. But they are not. So all good. Let me know if I am thinking in the right direction in the comments.

7

GitHub faced an interesting bug when they moved their Rate Limiter implementation from Memcached to Redis ⚡ I spent some time yesterday dissecting their engineering blog to understand the nuances and compiled my learnings. By the way, after a very long time, I recorded a longer video to slowly cook and explain every little detail. give it a watch - youtu.be/LAdUhenKDJ8 watching this video will make you understand why you cannot assume latency to be zero and why we have to always think about clocks whenever we build distributed systems. ⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff. #AsliEngineering #SystemDesign #GitHub

489

I am so jealous of the current generation, mostly Gen Z !! This generation can enjoy their whole life digitally anytime and can show it to their future generations. As a Millennial ( whatever they call us 90s kids) I get so happy when I find any old, buried photo with stained marks and try to make it pixel-perfect using AI photo enhancers!! However, this isn’t true for software !! The GitHub Arctic Code Vault project is designed “to preserve open-source software for future generations” for the next 1,000 years !! Quite fascinating isn't it ?? https://lnkd.in/gciEe3A4

5

Before we dive into the intricacies of CPU scheduling algorithms, it’s crucial to get a solid understanding of fundamental process timings. These concepts play a major role in how an OS handles multiple processes efficiently. Here’s a quick breakdown: • Arrival Time: The moment a process enters the ready queue and signals its presence to the OS. • Burst Time: The total time a process needs the CPU to complete its execution. • Completion Time: The time when the process finishes execution and leaves the system. • Turnaround Time: Total time taken by a process from arrival to completion, including waiting and execution time. • Waiting Time: Time a process spends in the ready queue, waiting for its turn to execute. These timings form the backbone of how we approach process scheduling and are essential to grasp before exploring scheduling algorithms. ⚡️ I keep on writing and sharing my learning across my socials, so if you resonate do follow along. I keep it no fluff; just engineering. #50daysofOS #DrGViswanathanChallenge

16

LRU is a popular but memory-intensive algorithm, hence Redis tweaked it ⚡ The standard implementation of LRU requires a doubly-linked list; wherein each node will have an overhead of 3 pointers = 24 bytes on a 64-bit machine. To keep eviction memory efficient, Redis has tweaked the LRU and made it approximate. The implementation just requires an overhead of 4 bytes per object. By trading off accuracy and correctness, Redis optimized to reduce memory overhead. Redis also leverages the same set of bits to implement LFU with Sampling. Pretty interesting. So, whenever possible, trade accuracy for performance. Most fancy data structures like Bloom Filters, Hyperloglogs, Count-min sketches, etc, do exactly this. ⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff. youtube.com/c/ArpitBhayani #AsliEngineering #RedisInternals

534

7 Kommentare

The "cost frontier" keeps shifting: With this speedup on Google TPUs, cost per million tokens of this e.g. 70B param model, is $1.77. The cost for 70B param models on Groq looks like it's at about half that, at ~$0.80/megaoutputtoken (in comparison, GPT4o is $10/M ($5 for the batch api) and Claude Sonnet is $15 (and as of today: also half-off [$7.50/M] for batch)

3

🚀 No Change Is A Small Change! Read how LinkedIn slashed latency by 60%! LinkedIn uses the Rest.li framework for microservices, which relies on JSON for data transfer. However, JSON can be bulky, consuming significant network bandwidth and introducing latency. To tackle this, LinkedIn switched from JSON to Protobufs. Protobufs streamline data serialization and deserialization, while also compressing data to reduce bandwidth usage. This seemingly small change led to a remarkable 60% reduction in latency. Keep learning and keep growing! 🌟 #linkedin #learning #engineering #latency

26

1 Kommentar

Fanning out queries across multiple shards makes it expensive - this is a common understanding but it is not entirely true ⚡ While going through the internals of Rockset, a real-time analytics database, I found they are fanning out queries across nodes, but still offer very low latency response times. My dissection video on how they execute the query in their distributed setup while being able to scale horizontally, along with some performance optimization nuances is dropping tomorrow. hit the notify button - youtu.be/HWeTVy5i9vE By the way, this will be the 3rd video in the Rockset internals series and I dissected and researched through 18 of their blog posts and 2 whitepapers. Exploring the details was fun, and I am sure you will love it too. Here are the first two videos of the series 1. Architecture deep-dive - youtu.be/I4eEEAgoCig 2. Converged Indexes - youtu.be/xGaUJTHuehQ ⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff. youtube.com/c/ArpitBhayani #AsliEngineering #SystemDesign #DatabaseInternals

170

2 Kommentare

When Frontend engineer makes a critical mistake generally it leads to an app looking bad. When a backend engineer makes a critical mistake some functionality is affected. When a platform engineer makes a critical mistake the company looses revenue as core product ceases to function. When a low level middleware/kernel development engineer makes a mistake, all the critical systems that makes the universe work, malfunctions causing a catastrophic global outage. Thats the power of C programming (besides getting the best runtime on leetcode LOL). #CrowdStrike

6

2 Kommentare

Why Kafka outperforms other message Queues in terms of speed? (Disclaimer : A small Note of my personal understanding only, not any company's views :)) Checkout: #TBEB Consider two key factors: the cursor model and compute work. **Cursor Model: Ingestors like Kafka manage cursors differently from queues. Queues maintain a shared cursor for message consumption, managing locks and ensuring each message is delivered to only one consumer at a time. In contrast, ingestors provide messages based on an offset, with clients managing cursors independently and no locks (stateful). This difference becomes a bottleneck for message brokers around cursor and lock management, impacting throughput. **Compute Work: Message brokers offer various features like filtering, deduplication, and scheduling, which require indexing and immediate or ongoing computational work. Ingestors like Kafka don't perform these additional tasks, resulting in a more streamlined process and higher throughput. Let me know the other pointers!! Think Big Execute Bigger (#TBEB) 🙂 #Kafka #RabbitMQ #MessageBrokers #SystemDesign

1

A postgres data dump of over 3 million news articles from Indian news providers I aggregated these to make a semantic search website with different analytics like title tone analysis and be able to track how news evolved around a given context. Sadly Life got busier; I will come back to this later on, but feel free to use this dataset to make something of your own. A lot more can be done with this dataset if you are more data-savvy :) dataset link: https://lnkd.in/gxAjRbXi I do have a slightly chonkier dump of 3.3 mil records from more news providers in my VPS - do send a DM requesting it. #datadump #datascience #data

9

Interesting to read actual evidence of what one could expect as an inevitability of scale!

21

3 Kommentare

Last Sunday we did a marathon session on performance engineering. It was a lot of work and was received quite well. Some of the comments towards the end were "this is blowing my mind". We started with a background on the CPU cache, branch prediction and instruction level parallelism and how to take advantage of them. Then we discussed 6 key optimizations used in 1BRC, that were based on these three hardware fundamentals. Recording link in the comments.

40

4 Kommentare