Marc Lamberti

🚨 dbt Labs made some hugeee new announcements today at Coalesce 🤯: -Cross-platform dbt Mesh 🕸  -dbt Cloud now supports the Apache Iceberg table format 🧊  -Visual drag-and-drop interface for less technical users to develop in dbt without needing to know SQL 👨‍💻  -dbt Copilot, an embedded AI engine with various user experiences across the dbt workflow 📈  -dbt Cloud integrates with AWS Athena (GA) ✅ & Teradata (Preview) ✅  - AND More (Advanced CI, Auto-exposures with Tableau, a new Power BI integration for the semantic layer, Data health tiles, and more!) 🔥 This all brings together our ethos, 'One dbt', the idea that dbt provides a common unifying framework bringing together people, platforms, and workflows in support of the analytics development lifecycle. https://lnkd.in/gb_hEU9G

28

2 commentaires

What do you think of this new streaming data pipeline idea (without Dataflow in the middle) ? * Extract & Load: Pub/Sub -> BigQuery subscription -> BigQuery * Transform (Enrich) & Reverse: BigQuery Continuous SQL + Gemini -> Pub/Sub Watch BQ Continuous SQL demo: https://lnkd.in/dJpEtnCB Congrats on the launch Nick Orlove , Pavan Edara and team 🎉 #GoogleCloud #Gemini #BigQuery

20

2 commentaires

🎯 You shouldn't miss this last chat about #PostgreSQL and #Compression with Nikolay Samokhvalov and Michael Christofides from Postgres.FM brought. 🔥 My #suggestion? While listening to the #podcast #chat, access the video's #information for a #collection of #blogs and #publications about #compression on the #Postgres #World. 🐘 #databases #postgres #data #compression #cpucycles #toast

6

Streaming Tables and Materialized Views now supports Row filters and Column Masks. If you want to get started quickly with RW and CM : https://lnkd.in/eaNNJpci #Databricks

60

3 commentaires

Keeping in line with the last post, this is how we reduced our AWS costs (2021-2024). Points are color-coded as per the attached image. 1. Large to small 🟦🟥 Everyone believes they are going to get 50 million users the moment their app hits prod. But I implore you to memorize the following code. If company.isFunded('Series A') { instance.set('Large') } else { instance.set('Small') } Cost me a few lakhs to learn this. Totally worth it. 2. On-demand to reserved 🟦 I have heard serverless scales infinitely. Well, it does scale your costs infinitely. Reserve your instance. Then ask AWS for a discount. 3. Filestore to CloudWatch 🟧 Our system logs were stored for years. Our server's file store was bloated. Now we send them to Cloudwatch, which retains them for three days. Then it's asta-la-vista, baby. 4. ElasticSearch to Postgres 🟪 Why did I even spin up these instances? I thought our discussion forums would need "text search capabilities". Well, since I don't need to do DNA matching on the comments, the cost of ElasticSearch was prohibitively high. Instead, we moved our text search to the browser. Using javascript :D BLASPHEMY! -------------------------------------------- These experiences have helped us appreciate the practical aspect of designing systems (which often comes up during live classes at InterviewReady). If you are a startup founder, I wish you all the best! Cheers! #AWS #CostManagement

689

56 commentaires

Apache Fury : 4x faster serialization than protobuf I did not tested it yet, but it seems very promising and well thought. More details on this release note with a comparison to protobuf : https://lnkd.in/eY3gcdhj

6

I wrote a new blog post that answer some of the questions around embedding models, e.g. what are different embedding models? How are they pre-trained? What is MTEB leaderboard? How are these models benchmarked? How to choose a baseline embedding model for your RAG and where to go from there. Check it out 👇

38

1 commentaire

Very thorough and elegant :)

5

Big thanks to Stéphane Maarek for his excellent Udemy course on Apache NiFi! Gained valuable insights and excited to apply this knowledge in real-world projects! #AlwaysLearning #DataIntegration

77

8 commentaires

Will Bilal Aslam kill your job ? Of course not he is going to make you more productive with Databricks Lakeflow. Simplicity is the keyword here. Quentin Ambard Watch the full session here: https://lnkd.in/eMN7t6XT #Databricks

28

20 years ago, EDB bet on Postgres and won. Today, we’re doubling down on a new bet: EDB Postgres AI. Why? Because the data landscape is fragmented, complex, and overflowing with untapped potential. EDB Postgres AI—the intelligent platform for the unified management of transactional, analytical and AI workloads—is here to change that. In this piece, EDB's Chief Product Engineering Officer Jozef de Vries shows you exactly how we’re tackling those data challenges and raising the bar for what's possible with Postgres. Read the full article here ➡️ https://tcrn.ch/3xITjMK #PostgreSQL #EDBPostgresAI #JustSolveItWithPostgres #AI #DBMS #DatabaseRevolution

55

The arrival of Llama 3 was another win for #opensource AI ...and another step towards the commodification of large language models. What makes AI truly useful isn't the models, but your data. ➡ Read on in our latest blog post: https://lnkd.in/gnxi4fy9

5

I have mixed feelings about the proliferation of certifications in the data engineering space, but the training itself can be a nice opportunity to learn more about a product or domain. Databricks is currently is running an event where you can get a discount on certification exams & labs if you complete training before end of October. Looks like some of the courses aren't doable on the community edition, but you can sign up for a free trial of Databricks on AWS, Azure, or GCP https://lnkd.in/gKFexdck I'm curious if there are any training programs people have found particularly useful. If you've taken a training in data engineering that you would recommend let me know in the comments!

5

1 commentaire

There is a big pricing problem on cloud storage ! Why ? 12€/month for 1 To for an S3 in a cloud provider. 9.99€/month for 2 To in Google One. Nobody can do better than this without losing money, and this kill newcomer. This look like there is an antitrust problem here to me, CNIL - Commission Nationale de l'Informatique et des Libertés could you take a look ? or someone else ? (don't know if it's Google that kills the market by setting a not fair price or other that are too expensive) If I want to provide a service like that, I'm at least twice more expensive than Google without margin... The only possibility is with credit from the cloud provider which seems wrong to me, you can't do this because it's too expensive so we are going to help you financially and so all your infrastructure is at almost no cost🤦 why ? Why overpricing everyone to give credit to some instead of just ask for what it cost plus a margin ? Or maybe we need big tech to stop screwing the market ? Is this something normal ? cc Scaleway

3

4 commentaires

I wrote a new post on the Sympathetic Ink blog to sum up The Deconstructed Database and what makes it composable. Learn more about the role of #Parquet, #Arrow, #DataFusion, Apache Iceberg, #Calcite and OpenLineage

64

4 commentaires

New blog post "Updating legacy Apache Airflow DAGs with Amazon Q Developer" I share my experiences on using Amazon Q Developer workspace context awareness to update some old Airflow workflows to get them running on the latest versions of Apache Airflow. Read about it at https://lnkd.in/ec6Q2VuN

12

3 commentaires