Ian Brown

A new table-format comparison post is out. How do the table formats support the streaming ingestion of row-level operations (insert/update/delete), such as ingesting CDC streams? https://lnkd.in/dFDs4Ah3

36

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

Missed one aspect of writing the TLA+ spec yesterday. How to gain confidence it is correct. https://lnkd.in/dQn2VUUq

8

I’m skeptical about measuring open-source package quality with metrics alone.    •   Low commit/update cadence: Doesn’t indicate low quality. It might signal a stable, well-functioning package that needs minimal maintenance.    •   High number of historical CVEs: Not a sign of low quality. It could show that the package has undergone thorough security scrutiny.    •   Many open issues: Not a reliable indicator. Many issues are more like support tickets, and there are often many duplicates. Quality assessment requires more than just metrics!

3

tldr: I'm hiring! As many of you know, I recently joined Lattice as their SVP of Engineering and I'm excited to announce that I'm hiring for all Engineering levels and more broadly, Lattice has open roles across EPD. If you're wondering, why Lattice? The last few years have seen a step change in how we think about the meaning of work and the employer <> employee relationship - add on the fuel of AI taking our jobs and you could be forgiven for thinking the future of work looks bleak for employees. It doesn't need to be that way - here at Lattice we're building the next generation of talent tools to give employers insights and employees purpose because people matter at work. Among other things, we're using generative AI to securely analyze enormous volumes of data to unlock career insights faster and investing in building equitable feedback, compensation and employer analytics tools - not just for one company but for all companies. If you're interested in making a difference to the future of work then let's chat - all roles are remote eligible and based in the US. We have physical offices in San Francisco and NYC if you prefer the in-office environment.

324

49 Comments

Does anyone else remember sitting in class thinking, "Am I ever going to actually use this?" 🤦‍♂️ I still ask myself this weekly when reading blog posts, listening to podcasts, catching up on massive Slack threads, or diving into documentation and feature listings. I don't always use the info, but it feels great when a memory from six months ago suddenly becomes relevant. I always think of that original Jurassic Park Unix scene. This often happens when discussing DevOps and Serverless, where boundaries blur already. Trying to keep all of the AWS documentation in my head while discussing the design of new features and optimizations feels like a superhuman task. What are some things you wish you had remembered when solving an issue or problem? For me, we designed a Lambda function to fail purposefully instead of attempting to recover for efficiency. Unfortunately, the Lambda control plane self-throttles a function once a failure threshold is reached... slowing our processing to a crawl and forcing some quick code changes. 😅 #AWS #lambda #serverless #learninginpublic #hugops

13

1 Comment

An eye-opening piece on the "Observability Cost Crisis" by bitdrifts Matt Klein. It suggests a major shift in how we handle logging, metrics, and traces to curb the spiraling costs. The idea? . . . . . . . . . . . . Default to storing no telemetry data unless specifically identified as needed on the fly. While this sounds radical, the argument is that recent advancements in infrastructure AI/ML can make it a reality. #CloudEngineering #Infrastructure #Observability #SaaS #CostEfficiency

1

I just updated my first table format comparison blog post to fix some missing details of Apache Hudi. https://lnkd.in/dwSUS6qS

5

We now have auto-generated documentation for our #protobuf and #grpc API's at LittleHorse! For most day-to-day activities, such as building a WfSpec or executing TaskRun's, you won't actually need to use our protobuf API directly. However, at LittleHorse we believe in empowering engineers to use either high-level abstractions or low-level nuts and bolts as they please, so we expose a GRPC API which is intimately related to the internals of LittleHorse. If you're curious about it, check out the documentation! You can find it here: https://lnkd.in/gBqBqKKr And side note, below is the most tricky line of code(?) I've ever written. It's the part that generates the pretty docs for you: https://lnkd.in/gZwDMfVp

14

Markdown to... command line app - wat? That's the topic for today's unsolicited face time. A single Pipedown pipeline can be used in numerous contexts thanks to the way it consistently accepts inputs and produces outputs. I waffled about my love of pipelines a while back: https://lnkd.in/gERrAsvA Though it's much handier to see in action! https://lnkd.in/giR-KVDT Let me know your thoughts 🚀

4

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

20

So true! In my opinion, every effort, whether it's new features or services, should include the creation of proper data and analytics to measure its effectiveness and adoption. The necessity of a feedback loop through telemetry and analytics cannot be overstated. It turns data into actionable insights, enabling iteration and refinement of the offerings. Without this, we risk repeating the same errors or just offering the wrong solutions. understanding the customers and having a constant feedback look is a must!

14

Our customers can now create Route53 Alias records for their Amazon OpenSearch custom endpoints and gain faster DNS resolutions while saving on additional CNAME costs. Good job on the launch Sachet Alva and team - https://lnkd.in/g_aaUp7S. #aws #opensearch #route53

48

The latest advances in artificial intelligence (particularly large language models) continue to reverberate. Even for an “old school” AI person like myself (who cut his teeth with Prolog) it is clear that there has been a step change in our ability to create computer systems that can interact with humans using natural language. GPT-4 et al are exhibiting early signs of “common sense” and have encoded useful conceptual representations of the world. The debate rages on as to whether this is “intelligence”, but to an engineer like me, it sure seems useful! #LLM #AI #Intelligence #Programming https://lnkd.in/ebAgNapK

15

4 Comments

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

2 Comments

Confession: I am lazy. A while ago I showed a Prescient Designer/Node-RED subflow with multiple "virtual" inputs. Not too hard, but not simple enough for me. So I made a couple of nodes to make this super fast and dead simple. Watch how in my latest video! https://lnkd.in/e7WxhxpH #designer #nodered #lazy

13

2 Comments

We need to move past the idea that knowing everything is the best; asking good questions is a much more critical and underrated skill that we need to embrace. I’ll demonstrate with a case study on how the exact problem to solve can dramatically shift based on answers to basic questions. But first, why does this matter? This is because good questions are the cornerstone to truly understanding the underlying requirements. So many issues would simply not exist if the right questions were asked at the right time - Many products with little to no adoption would not be built - Numerous designs would not be scrapped very late in the game - Countless incidents would be caught before they make it to production Let’s use my favorite leveling question shared previously https://lnkd.in/gxCHTWSu. "Imagine you join our team/company X and discover that our SLAs (Service Level Agreements) are consistently in the red. What steps would you take to address this issue?" Fundamental questions to ask: Q1: Are the customers actually unhappy? If there are no customer complaints, the problem may boil down to a simple SLA update. If there are complaints, changes to the system are likely necessary Q2: What specific SLAs are in red? How long is “consistent”? If multiple SLAs (latency, availability, consistency) have been red for 12 months, it's a deeper systemic problem. If a single SLA is red for 3 months, it’s a simpler problem. Q3: What is the SLA and what is the system performance? If the SLA and system performance don’t diverge that much, it can be solved with incremental steps. However, if the SLA is 99.99% and performance is 99%, then deeper issues must be addressed. To recap… Asking the right questions at the right time can entirely transform the approach and solution, saving a lot of time, effort and grief. It is a critical skill for senior engineers. I’ll leave you with this iconic clip from Margin Call… “Explain, as you might to a young child. Or a golden retriever” ~~~ ✍️ Do you agree? Share if you have examples where the right question changed project’s direction. ♻️ Repost if this is helpful.

30

2 Comments

Want a pandas-like Python dataframe library that uses DuckDB as its default backend while supporting 20+ others? Try Ibis project! It allows you to efficiently and reliably process datasets much larger than RAM. https://lnkd.in/eErAffqK

14

Our customers are scaling their ThoughtSpot deployments managing millions of metadata objects and relationships. 📈 Atlas, our graph database, is the secret sauce. It enables lightning-fast querying and efficient management of analytical objects. ⚡️ But we didn’t stop there. We've slashed Atlas initialization times from nearly an hour to under 2 minutes! 🤯 Want to know how? Check out our latest blog by Indresh Gupta to learn about our optimization techniques: Eliminating string interning Turbocharging Postgres queries Optimizing in-memory processing #ThoughtSpot #Atlas #GraphDatabase #DataEngineering #PerformanceOptimization #Technology #Innovation

38

1 Comment