Owein Reese

Great article by Roi Yarden Building Data pipelines comes with its set of challenges. One of the most important decisions is to understand what patterns suit the business evolving needs and allow for a system to scale. This can prevent very costly mistakes.

8

The subject of support for incremental reads of mutable tables and generating CDC streams in the four table formats I'm covering is far more nuanced than I expected. What was going to be one post is going to have to be a 5 parter to keep it manageable: 1. Introduction to the concepts 2-5. Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon

15

2 Comments

Markdown to... command line app - wat? That's the topic for today's unsolicited face time. A single Pipedown pipeline can be used in numerous contexts thanks to the way it consistently accepts inputs and produces outputs. I waffled about my love of pipelines a while back: https://lnkd.in/gERrAsvA Though it's much handier to see in action! https://lnkd.in/giR-KVDT Let me know your thoughts 🚀

4

Practical explanation of API’s the next time you are in a buzzword salad meeting… https://lnkd.in/e-HSTwC3

3

We have been hiring a lot at ODIN AI lately and I've noticed a concerning trend during job interviews. It seems like some candidates are using AI assistants to cheat, which is frustrating and counterproductive. We've started to combat this by looking out for delays in responses and the inability to answer specific questions. Often, if you start a detailed train of thought, the candidate struggles to follow because the AI takes time to generate a response. This has become a reliable way to identify if someone is cheating. It's crucial to ensure that candidates are genuinely capable and not relying on AI to get through the interview process.

32

5 Comments

Excited to share that I was interviewed by FSR magazine to talk about Fishbowl GRM and how we’re helping restaurants better understand their guests and connect all their data in one place! 🎉 In today’s competitive landscape, knowing your guests and creating personalized experiences is more important than ever. Fishbowl GRM empowers restaurants to not only gather insights but also turn them into meaningful actions that drive loyalty and revenue. Huge thanks to FSR Magazine for the opportunity to share how we’re making this happen. Check out the article to learn more about how Fishbowl is transforming guest engagement for restaurants! #RestaurantTech #GuestEngagement #DataDrivenDecisions #RestaurantMarketing #HospitalityInnovation #GuestLoyalty #RestaurantSuccess #PersonalizedMarketing #FishbowlGRM

8

Super interesting (and technical) breakdown on how the folks at ClickHouse are transforming the simple, flexible JSON format into something that can be leveraged at hyperscale. An excellent example of how sometimes scale can make you do crazy things.

1

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

70

3 Comments

tldr: I'm hiring! As many of you know, I recently joined Lattice as their SVP of Engineering and I'm excited to announce that I'm hiring for all Engineering levels and more broadly, Lattice has open roles across EPD. If you're wondering, why Lattice? The last few years have seen a step change in how we think about the meaning of work and the employer <> employee relationship - add on the fuel of AI taking our jobs and you could be forgiven for thinking the future of work looks bleak for employees. It doesn't need to be that way - here at Lattice we're building the next generation of talent tools to give employers insights and employees purpose because people matter at work. Among other things, we're using generative AI to securely analyze enormous volumes of data to unlock career insights faster and investing in building equitable feedback, compensation and employer analytics tools - not just for one company but for all companies. If you're interested in making a difference to the future of work then let's chat - all roles are remote eligible and based in the US. We have physical offices in San Francisco and NYC if you prefer the in-office environment.

324

48 Comments

You don’t need to build complex tests to build performance comparisons. This is by far the most practical tests I ve come across that explains Databricks seemingly cost effective and performant in practical use cases.

3

“Being able to choose at the pipeline and task level, as opposed to making everything use the same execution mode, I think really opens up a whole new level of flexibility and efficiency for Airflow users,” LaNeve said. Before Airflow 2.10, there were some limitations on data lineage tracking. LaNeve said that with the new lineage features, Airflow will be able to better capture the dependencies and data flow within pipelines, even for custom Python code. This improved lineage tracking is crucial for AI and machine learning workflows, where the quality and provenance of data is paramount. “A key component to any gen AI application that people build today is trust,” LaNeve said. The goal for Airflow 3.0 according to LaNeve is to modernize the technology for the age of gen AI. Key priorities for Airflow 3.0 include making the platform more language-agnostic, allowing users to write tasks in any language, as well as making Airflow more data-aware, shifting the focus from orchestrating processes to managing data flows. “We want to make sure that Airflow is the standard for orchestration for the next 10 to 15 years,” he said. https://lnkd.in/g5uERWYS

20

1 Comment

If you're the kind of person who's excited about all the recent S3 feature announcements (bucket policy unlocks via temporary root sessions, up to millions of buckets per account, pagination and region filtering for ListBuckets, conditional writes, ListCallerAccessGrants API) and you're attending re:Invent, then you should come to our storage track session, STG306 Advanced security patterns on Amazon S3. Learn how to use all the exciting new stuff securely in the most interactive format available at re:Invent!

94

2 Comments

Are you running production workloads on Amazon EC2 Spot Instances to cut costs? You likely know that these discounted instances come with the risk of sudden interruptions when EC2 needs the capacity back. This can cause pod evictions and downtime if the interruptions aren't handled properly. I’ve seen many teams struggle with this, dealing with frequent outages and scrambling to replace terminated nodes manually. It’s a constant battle to maintain availability and optimize costs. But what if I told you there's a better way? https://hubs.la/Q02-HMYk0 #karpenter #devops #k8s #sre

6

Ready to revolutionize your approach to platform engineering? Join us at SKILup Day: SREs and the Rise of Platform Engineering on April 30, 2024, to explore the evolution of SREs in platform engineering. https://bit.ly/4az7z8R Discover how SREs are reshaping DevOps workflows at scale, playing a crucial role in driving efficiency and resilience. At SKILup Day: SREs and the Rise of Platform Engineering, we’ll discuss: ➡️ Resilience Redefined: SREs leverage platform engineering to fortify systems against disruptions, ensuring uninterrupted service delivery even amidst unpredictable challenges. ➡️ Scalability at the Core: Through Platform Engineering, SREs architect solutions that effortlessly scale with evolving demands, enabling businesses to adapt and thrive in rapidly changing environments. ➡️ Efficiency Amplified: By embracing automation and data-driven decision-making, SRE-led platform engineering maximizes operational efficiency, empowering teams to focus on innovation and strategic initiatives. Don’t miss out! Register now to gain valuable insights and actionable strategies on how to empower your journey to SRE and platform engineering. https://bit.ly/4az7z8R #SKILupDay #SRE #PlatformEngineering #DevOps

4

My Apache Hudi CDC first draft is sitting in my blog ready to publish. I'll reread it with fresh eyes on Monday and publish.

18

I'll be at #PlatformCon24 in a few weeks to moderate a panel discussion on Architecting the Future: Platform Strategies for API Adoption & Impact. I am eagerly looking forward to learning more about how API leaders from Wayfair, The New York Times and Cox Automotive have led digital transformation with Apollo GraphOS to yield tremendous impact for their users and their business! RSVP if you'd like to join in person! #ApolloFederation #Platformengineering #APIOps #GraphQL

24

New blog post! Many key trends in the data space that I'm seeing right now are actually countering a set of people problems that can be traced back to Conway's Law. I hope that these trends lead to real change, not only technologically but how we organize as humans to achieve shared goals. https://lnkd.in/defwWFuw

26

Speed up semantic search performance with token pruning for our ELSER model: https://lnkd.in/eXD6rHU7 #ElasticSearchLabs #SemanticSearch

2