Ezra Hoch

Alex Komoroske's writing is so fun to read and so easy to draw lines into our reality: "Sarumans can’t see the value of systems; it will seem too fuzzy and abstract; [...] For similar reasons, a Saruman is deeply skeptical of other people’s perspectives on what’s a constraint, because the only constraints that are real are physical constraints, and everything else is just a fiction in people’s minds that can be overcome with boldness. Sarumans are the kinds of people who build impressive, powerful cathedrals. When you’re building a cathedral, you can’t have naysayers pointing out imagined constraints. You need decisiveness. As they get more powerful, accumulating followers, capital, and respect, it becomes easier and easier to cast the naysayers away. [...] Radagasts intuitively understand the power of the swarm: the system, in its buzzing, blooming, emergent glory. Radagasts know how to let go and dance madly with the system. They don’t try to control the system and its components. They love the system and its components: a love borne out of respect." https://lnkd.in/e9TNgNXb

2

So I became fascinated by C++ coroutines late last week, and have spent a good portion of the weekend figuring out how they work. The TL;DR of my discoveries so far is that C++20 coroutines: 1) follow the best and the worst of C++ practices, and 2) are a really clever way to decouple stackless functions from their execution model. For (1), well, it's C++. No crisp & clean syntax, lots of weird compilation errors, and even weirder runtime errors. I hated them with passion for the first several hours. But then they grew on me, because the flip side is also true. The flip side is backwards compatibility and "only pay for what you use". And introducing a totally new concept — the stackless execution model, effectively, [proto-] green threads — into a mature and complex language is one hell of a pain! For (2), I do believe it's a genius solution for the language to separate stackless functions from their execution model. And C++20 did exactly that. There is literally no event loop in C++20! The claim (as I understand it) is that a tiny microcontroller may want to use C++20 coroutines, and the authors of the Standard are very explicit about not constraining possible runtimes. All that C++20 coroutines do is introduce stackless functions into the language as first-class citizens. And yes, stackless immediately implies resumable. So it is one step about the most-clever-possible syntactic sugar over `setjmp()`. Full one step. But not a millimeter more than this one step. Whether you "execute" your coroutines in a dedicated single-threaded event loop, or a thread-pool-based event loop, or by individual steps via some timer event or even a network packet — that is entirely your call. The stackless functions are there, `co_await` (as well as `co_yield` and `co_return`) is there, no callback hell is needed, and all the RAII and scoped initialization/ownership model is fully intact. Now, I'm still elbows deep in my experiments, but here's the first big LinkedIn-worthy question. Why don't we write our C++20 [unit] tests as corourines? Say, you want to limit some test to five seconds. And fail it as "timed out" within these five seconds. Before coroutines, doing this without instrumenting the very code of the test involved `fork()`-ing the process, effectively relying on the operating system's primitives to ensure proper resource deallocation. With C++20 coroutines, as long as the slow test is slow on non-blocking I/O (i.e. not in an infinite loop, but on something that `co_await`-s stuff), it is possible to time-box each individual test to some five seconds very much organically. Because a coroutine frame, unlike a stack frame, can be `.destroy()`-ed safely and cleanly. So, are we doing it yet? If no, why not? If yes, what would be some next-gen `gtest` to look into? Also, I'll be writing some code and blog posts on C++20 coroutines soon. If you are reading this text and are an expert, DM me pls — I'd love to pick your brain!

19

4 Comments

Although some #Kakfa vendors are considering S3 Express One Zone for write caching, a model developed by my colleague Jack Vanlightly suggests this works only in a few cases. And with cloud vendors like Microsoft reducing or eliminating cross availability zone networking charges, traditional Kafka replicated logs still work well for many workloads. https://lnkd.in/gHrNPfSZ How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out.

4

#LLM sizes will be a significant consideration in the next generation of products that we will be witnessing in the coming years. Router models cannot be the go-to solution eventually. As a founder in this domain, one needs to think beyond constrained boxes and democratize these powerful systems as much as possible for small and medium business. We are planning to set things straight in the coming months. #Angel #angelfunding #VC #investments #VCfunding #AI #AIStartup #Quantization #LLMs #LLM

4

I love looking back to old projects I've done! So nostalgic... One of them is my Kinect floor mapping algorithm from 2017 (I was about 13 back then). I did it in C++ with no external dependencies (apart from the Kinect SDK), and turned the raw depth values returned from the SDK into this beautiful map you can see below! Camera position is the red square. It automatically calibrates to the floor height and then creates a birds-eye-view map like you can see below. Github repository is here: https://lnkd.in/gSiQwcAG (I actually didn't use Git yet & added the repo in 2022 to share with others. This was WAY back in the old days :), before I learned Git)

3

Just did a keynote in PyTorch conference, on LLM deployment. Thanks for the great work of PyTorch team, our partners, and the interests from the audience! Please try out our deployment ecosystem of torchchat, torch.compile, torch.export, AOT Inductor, torchao and ExecuTorch, and checkout all the amazing talks, posters and demos in the conference!

50

3 Comments

When you need to play with your data in Gen Ai, you need RAG and Chunking is one of the main component in the RAG framework while consuming or processing your data. Excellent post below describing how in different ways chunking in RAG can be implemented..

23

1 Comment

💡 Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? Christopher highlights some crucial findings from a recent study that caught my attention. While LLMs are powerful, integrating new knowledge through fine-tuning presents some very real challenges. The balance between learning efficiency and avoiding hallucinations is more critical than ever. How do we manage this trade-off at scale in practical AI deployment? 🤔 For #AI professionals, understanding the risks of overtraining and hallucination is key to building reliable, future-proof models. Curious to hear others' thoughts on strategies you've found effective in tackling this challenge!

1

I've been saying this for 2 years now and we are finally here. Deep learning models went through this pruning phase as well. If I recall correctly, they were able to do up to 70% with hardly any loss. SOOOO, what that means is, inference is going to be even cheaper. Open source on-prem models are going to be even more prevalent. And, while pruning has nothing to do with this, it does usually follow, people will start experimenting with new techniques and architecture. Now is the time to keep your eyes peeled. This is where the cool shit starts happening. I give it 6 months before we start hearing about a flurry of new techniques. They are probably already working on the right now, to be honest. Some quiet hidey hole with researchers seeing if they can tease out new designs and new techniques or resurrect old ideas to strengthen the model or give it new capabilities. Maybe it will come from a big company, but they are too focused on products right now, so I don't know how that will fit into the calculus of how things used to work. Definitely keep your ear to ground for the universities or random private research. Updates come from unusual places. Anything that is being worked on right now will start getting finished up here mid next year. Maybe even sooner. My guess, is either completely new architecture but still transformer, or graph based enhancements, mixture of experts improvements, or something using older but solid techniques like q learning, k-means, etc. I suspect a huge reevaluation of ethics training as well. Because that is a cluster fuck that only makes the model worse and encourages it to be non-compliant with instructions. Anyway, long story short, the king is dying, long live the next one.

11

2 Comments

Interesting thread on Twitter from Derya Unutmaz and Nick Dobos who think that medicine is going to crumble much faster to AI than any other industry. And they have good points: 1. Huge amount of gatekeeping and refusal to empower patients to make their own decisions. 2. Not an optional thing in many cases, even when you can’t afford it. So you are forced to use the next best thing. 3. Overpaid admin nightmare to get simple things diagnosed suddenly vanishes. What % of patients is that? -- "This is the final warning for those considering careers as physicians: AI is becoming so advanced that the demand for human doctors will significantly decrease, especially in roles involving standard diagnostics and routine treatments, which will be increasingly replaced by AI. This is underscored by the massive performance leap of OpenAI’s o-1 model, also known as the “Strawberry” model, which was released as a preview yesterday. The model performs exceptionally well on a specialized medical dataset (AgentClinic-MedQA), greatly outperforming GPT-4o. The rapid advancements in AI’s ability to process complex medical information, deliver accurate diagnoses, provide medical advice, and recommend treatments will only accelerate. Medical tasks like diagnosing illnesses, interpreting medical imaging, and formulating treatment plans will soon be handled by AI systems with greater speed and consistency than human practitioners. As the healthcare landscape evolves in the coming years, the number of doctors needed will drastically shrink, with more reliance on AI-assisted healthcare systems. While human empathy, critical thinking, and decision-making will still play an important role in certain areas of medicine, even these may eventually be supplanted by future iterations of models like o-1. Consequently, medicine is becoming a less appealing career path for the next generation of doctors—unless they specialize in intervention-focused areas (such as surgery, emergency medicine, and other interventional specialties), though these, too, may eventually be overtaken by robotic systems…maybe within a decade or so." Link to the thread in comments #healthcare #ai #innovation #technology #diagnostics #medical

1

2 Comments

🚀 Introducing 'Claude François' 🕺—The AI Code Reviewer with Style! A few weeks ago, we were invited to the first Menlo Ventures/Anthropic Builder Day to craft something remarkable with Anthropic’s flagship Claude 3.5 Sonnet LLM. After some post-hackathon polish, we're thrilled to share 'Claude François'—the AI Code Reviewer with style! It’s just a quick 4-hour hack demonstrating how easy it is to leverage Crossing Minds' RAGSys in-context learning technology with an LLM like Claude. Just give it any Github Pull Request (PR), and it generates an expert code review for you! Try it now: https://lnkd.in/ezfRpbMc 🔍 How Does It Work? We leveraged our patent-pending RAGSys In-Context Learning (ICL) technology to build a database of Pull Requests (PRs) from the Keras GitHub repository. The primary maintainer of Keras is none other than François Chollet, the creator of Keras and an AI luminary known for his succinct, expert code reviews focusing on real issues rather than superficial code hygiene. Here's the magic: Active Learning & Fine-Tuned Embeddings: RAGSys pairs code diffs with François's reviews, using active learning to train custom embeddings optimized for generating "fchollet-level" code reviews. Intelligent Retrieval: When you input a GitHub PR, RAGSys taps into our industry-leading recommendation engine to retrieve the most relevant examples—not just the most similar ones, but those optimized for maximum mutual information. Self-Reflection: Finally, we employ sophisticated self-reflection loops to unlock the LLM's reasoning capabilities, and extract more information from the in-context examples, delivering the best possible code review. 🌟 Why Is This Exciting? This demo showcases the power of our RAGSys technique for In-Context Learning and Domain Adaptation. Imagine replacing Keras and François Chollet with your own in-house code reviews and standards! Or apply RAGSys to any domain-specific task with input-output pairs—classification, translation, summarization, labeling, search, recommendations, and more. 💡 The Future of AI with Crossing Minds RAGSys in-context learning is faster, cheaper, and more flexible than traditional domain adaptation approaches like fine-tuning, while producing more accurate and consistent outputs. At Crossing Minds, we're leading the way in AI and LLM innovation. 👉 Try It Out! Take 'Claude François' for a spin on your own codebase and experience the difference. We'd love to hear your feedback! If you’re curious to learn more about RAGSys, check out our recent webinar (https://lnkd.in/eF7BkvqZ), or reach out to me directly. Let's explore how this technology can transform your workflow. p.s. in case you're wondering, the name and 🕺 logo are a playful nod to a certain iconic French musician. https://lnkd.in/eEVzgTnx #GenAI #AnthropicBuilderDay #LLM #Anthropic #Claude #Hackathon

42

5 Comments

It seems that with C/C++ -> Rust, specifically, you could do this by comparing the output compiled code as one of your verifications of correctness for ML learning This is something that e.g. copilot doesn't do - but it'd be straightforward to do so when doing training The expectation would be that the compiled rust code should be equivalent to the compiled C code, i.e. the assembly should be semantically no different As the ML learns, it may end up doing some funky rust code simply to get that to match - but that's probably a good thing, as this is the part that would require some manual intervention & you'd at least be guaranteed equivalent behavior

1

Beauty of Imaginary numbers https://lnkd.in/drAycZ5Y #imaginarynumbers I think this is the best source to understand imaginary numbers and why they exists in depth.

3

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

In thinking about what to post here about Pod Engine's new developments, I realized there's a lot I've taken for granted and not mentioned! Something I find particularly cool is the sheer scale of our architecture and ML pipelines. Pod Engine now has tens of thousands of _fully tracked_ podcasts. For each podcast, this means: 1. At least the last 6 months of episodes transcribed and analyzed (for many we've processed every episode in their back-catalog too!) 2. Every transcript fully searchable semantically and by keyword 3. Their own podcast analytics page (attached, will improve a lot this week too!) We actually track millions of podcasts (specifically every active podcast, and a fair few more), and the number of fully tracked podcasts is steadily growing. Essentially our machines have "listened to" many, many hundreds of thousands of episodes and extracted insights extracted from every single one. Super powerful stuff when harnessed appropriately! I'll be adding some extra data to the podcast pages this week and would love some feedback. They'll have: 1. Podcast contact details 2. Overall reviews (Itunes + Spotify) 3. Detailed (individual) reviews for Itunes 4. Various proxies for listenership data (Castbox subscribers etc.) 5. All social links You can access all of this for free today just by logging in to a free account. Our pilot customers will soon have access to lots more though, which I'm extra excited about!

17

Insightful how multi-agent AI can provide a better solution to many problems.

2

2 Comments

Agreed Srini Pagidyala. This is where we need to to understand the limitations of LLM or Generative AI. Accurately representing the real world requires interaction with it. I feel that it requires more brain functions added to the generative part. The generative part is like our brain creating abstract thoughts based on experiences (data). Now some entity should course correct those thoughts to match with the real world. Our human brain does interact with the real world. Point here is that LLM should have the capability to collect relevant world data (from the context it is generating for) and course correct it’s content. Well a far fetch for today’s LLMs

5

Replacing Redis with SQLite? Wild idea, but it turns out to be way faster! In this episode, I talk to my longtime friend Michael Buckbee about some of their peculiar needs and why SQLite is such a good solution for them. Go check it out! https://lnkd.in/gUh2xFcg

55

2 Comments