Quentin Ambard

I'm very happy to share a technical article I co-created with Deeksha Gunde, that walks through enabling a federated Airflow environment using AWS SNS/SQS. https://lnkd.in/e4RwPirp #aws #dataengineering #analytics #bigdata

33

Good energy here, really appreciated this meetup! 🤩 Look at the recording and recognize the common mistakes with Kafka (the audience definitely did!). How to handle invalid messages, manage schema evolution, and ensure data quality. It’s a tough task, whether you’re using CI/CD, a custom SDK, or a Kafka proxy. #meetup #apachekafka

5

💸 Revenue Team Wants AI Costs, But Your MVP's Still Loading... 🤯 A Founding MLE Guide to Pre-MVP Cost Estimation 🔮 Initial Cost Estimation 1. Select Representative Models 📊:    Choose models at various scales. For NLP, consider BERT, GPT-J, (FLAN-)T5-XXL, and Falcon-40B. 2. Match Models to Hardware 🖥️:   Pair each model with appropriate GPU hardware. Example: GPT-J on A10G, Falcon-40B on A100 40GB. 3. Estimate Request Completion Time ⏱️:   Approximate time for each model to complete a request. Example:   - GPT-J: 1 second on A10G (made up number!)   - Falcon-40B: 10 seconds on A100 40GB (made up number!) 4. Calculate Hourly Costs 💸:   Research current GPU pricing. Example:   - A10G: ~$2 per hour (Modal Labs pricing)   - A100 40GB: ~$5 per hour (Modal Labs pricing) 5. Compute Cost per 1000 Requests 🧮:   Use the formula: (Seconds per request * Cost per hour) / (3600 seconds) * 1000   Examples:   - GPT-J: (1 * $2) / 3600 * 1000 ≈ $0.60 to serve 1000 requests   - Falcon-40B: (10 * $5) / 3600 * 1000 ≈ $3.00 to serve 1000 requests 6. Provide Order of Magnitude (OOM) Estimates 📏:   Present a range of costs based on different models. In this case, $0.60 to $3.00 per 1000 requests. 7. Factor in SLAs and Latency Requirements ⚡:   SLAs affect costs and can help constriant the solution space. For example, achieving a p99 latency of Xms might be 10x more expensive due to keeping a machine warm 🔧 Ongoing Cost Optimization (Thanks for the beautiful post Outerbounds) - Analyze Top-line Costs 📈:    Regularly review cloud bills to focus optimization efforts. - Identify Cost-driving Instances and Workloads 🔍:    Use tools to pinpoint expensive instances and tasks. - Monitor Resource Utilization 📊:    Avoid over-provisioning; pay attention to actual usage. - Optimize Workloads 🎛️:    Right-size resource requests based on real usage patterns. - Choose Optimal Execution Environments 🌐:    Leverage multi-cloud strategies for cost advantages. - Refine Based on Specific Needs 🎯:   Narrow estimates by understanding customer problems and required model scales. - Explore Serverless Options ☁️: - Stay Informed on Pricing 📚: Valuable resources: https://lnkd.in/gwragJBw https://lnkd.in/gVkGenhi https://lnkd.in/gq8pi4qd This framework is inspired by countless interactions with the ML/AI community #MachineLearning #CostEstimation #DataScience #AI #CloudOptimization

8

1 commentaire

When I read this first, I was like, Tools updated for MySQL 10.1 ? That has not been released yet... https://lnkd.in/e9EDWHWq #mysql

1

AutoMQ: Advancing Event Streaming Beyond Apache Kafka I've been exploring AutoMQ, a new event streaming platform that's fully compatible with Apache Kafka but addresses many of Kafka's limitations. Here's what you need to know: 𝗞𝗲𝘆 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀 𝗼𝗳 𝗔𝘂𝘁𝗼𝗠𝗤: ✅ 100% Apache Kafka compatible ✅  10x more cost-effective ✅  Instant auto-scaling capability ✅  Single-digit millisecond latency ✅  S3-based storage layer 𝗔𝘂𝘁𝗼𝗠𝗤 𝘀𝗼𝗹𝘃𝗲𝘀 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗞𝗮𝗳𝗸𝗮 𝗽𝗮𝗶𝗻 𝗽𝗼𝗶𝗻𝘁𝘀: 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀: Kafka's scaling involves time-consuming partition reassignment. AutoMQ uses stateless brokers for instant scaling. 𝗗𝗮𝘁𝗮 𝗦𝗸𝗲𝘄: Kafka's static partition distribution leads to skew. AutoMQ implements built-in self-balancing. 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗖𝗼𝘀𝘁𝘀: Kafka relies on expensive local SSDs. AutoMQ leverages cost-effective S3 storage. 𝗛𝗼𝘁/𝗖𝗼𝗹𝗱 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗲𝗻𝘁𝗶𝗼𝗻: AutoMQ separates hot and cold data access for optimized performance. 𝗟𝗶𝗺𝗶𝘁𝗲𝗱 𝗖𝗹𝗼𝘂𝗱-𝗡𝗮𝘁𝗶𝘃𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁: AutoMQ's architecture is designed for cloud environments and K8s. 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗜𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻𝘀: 𝗦𝘁𝗮𝘁𝗲𝗹𝗲𝘀𝘀 𝗕𝗿𝗼𝗸𝗲𝗿𝘀: Enable rapid scaling without data replication. 𝗦𝗲𝗹𝗳-𝗕𝗮𝗹𝗮𝗻𝗰𝗶𝗻𝗴: Automatically migrates partitions based on throughput. 𝗦𝗵𝗮𝗿𝗲𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: Utilizes cloud storage services for durability and performance. 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗔𝗰𝗰𝗲𝘀𝘀: Employs pre-reading, concurrency, and Direct I/O for superior performance. AutoMQ offers a compelling solution for teams looking to enhance their event streaming infrastructure without disrupting existing Kafka setups. Its architecture addresses scalability, cost-efficiency, and performance challenges inherent in traditional Kafka deployments. Interested in a deeper technical dive ? Here is the official website - https://www.automq.com/ Github Repository: https://lnkd.in/e5f2u7BF

326

19 commentaires

Recently langchain4j-memory-store-redis was added to the langchain4j Quarkusio extension. Basically it uses Redis instead of the RAM to keep the chat context during a conversation. But you might think : What could be the use case ? Here is one : Imagine a serverless chatbot pod (so scaling back to 0) but that needs to keep the context ! Also in case it's needed it can scale up and keep track of multiple sessions/contexts You can see in this short screencast that even after that the pod scaled back to 0, it was able to remember my name at the second request. Some cool stuff about this demo (blog will follow) : - I'm using of course Quarkus/langchain4j compiled natively with GraalVM - I'm using the openshift sandbox because it's free and it has the operators I need - It's using my favourite project around the k8s ecosystem : Knative - For the Redis part, I'm using the blazing fast DragonflyDB running on Aiven

44

4 commentaires

Here's a secret Snowflake tip 🤫: with the ability to host Notebooks on the new container runtime, we can now install arbitrary packages using !pip. 🔥 Yes, we can also install packages not available on the Anaconda Snowflake repo (with all the risks that come with it 😅). For instance, now you can install the Cinemagoer package and search the IMDb for all the movies containing the word "matrix" in the title 👇

93

3 commentaires

So many insightful informations through Databricks' system tables : https://lnkd.in/e3pP3Pre. Even if you enabled the preview feature on Unity, don't forget to enable each schema through API: curl -v -X PUT -H "Authorization: Bearer <PAT Token>" "https://<workspace>.https://lnkd.in/eeZPjAX3<​metastore-id>/systemschemas/<SCHEMA_NAME>" #unity #governance #databricks

5

Data Engineers ! This one's for you: 🧠 The Economics of Generative AI Stephanie Kirmer 🧠 Connecting ML Models in Sagemaker to Snowflake and dbt Hugo Lu 🧠 Introduction to Interpretable Clustering NAKUL UPADHYAY 🧠 Snowflake automatic clustering cost estimation Tim Sander 🧠 Death to Tech Debt? Ari Joury, PhD Check it here; https://lnkd.in/ebdFKNZ5 #dataengineering #dataroundup #startup

5

What's AIBI genie ? How does it work ? Who should use it ? Chao Cai Sr Director of Engineering at Databricks answers all this questions and share some best practices as well. If you have not tried it yet after this session you will definitely do it. If you're a Data Analyst, you shouldn't consider genie as threat it's more a companion in the data democratization journey. #Databricks

20

Will Bilal Aslam kill your job ? Of course not he is going to make you more productive with Databricks Lakeflow. Simplicity is the keyword here. Quentin Ambard Watch the full session here: https://lnkd.in/eMN7t6XT #Databricks

29

📢 New Video Alert: Keynote by Julien Le Dem on Apache Iceberg and the Deconstructed Database 🎥 Watch Julien's keynote from the #ChillDataSummit in #SanFrancisco. Julien Le Dem is one of the key innovators in the open-source community and a driving force behind projects like Apache Parquet. In this talk, Julien explores how Apache Iceberg is revolutionizing data lakes, breaking down the concept of the deconstructed database and its impact on large-scale data management. If you're passionate about data architecture or curious about the future of data processing, don’t miss Julien’s insights on how Iceberg is pushing the boundaries of what's possible in data lakes. 📽️ Watch the full video here: https://lnkd.in/d9eg2exq #ApacheIceberg #OpenSource #DataEngineering #DataLakes #BigData

26

1 commentaire

Small doubt, Why are Databricks not much interested in this area : Integrating Databricks Secrets with AWS secrets manager ? In the Azure cloud, Azure Key vault integration with Databricks Secrets made lives easier for secret management. Note : I know that we can use boto3 directly to retrieve the secrets from AWS secrets manager using IAM permissions. But it has its own limitations in case of shared access mode.

74

8 commentaires

🚀 New Blog Post Alert! 🚀 I just published my first post on Medium: "PySpark Hacks 1: How Repartitioning with Persist Can Transform Performance of Big Data Joins". In this article, I share insights on how leveraging repartition with persist can optimize join performance in PySpark, reducing data shuffling and improving cluster efficiency. 📖 Read the full article here : https://lnkd.in/gQNr6nX2 I’d love to hear your thoughts and feedback! Feel free to leave your comments or connect with me to discuss best practices for optimizing large-scale data processing in Spark. Let’s keep the conversation going! #BigData #PySpark #DataEngineering #DataProcessing #SparkOptimizations

15

Apache Hop is the reincarnation of #Pentaho that started as a #PDI fork but is now so much more: ⚙️ life cycle integration, container and cloud support, unit testing, ... Even more important than the technology: 🤝 As an Apache project, Hop is and will always be free and open source 🌎 Apache Hop is developed and supported by a thriving and growing global community 🏗️ A lot has changed under the hood, but the shared history allows Apache Hop to import your Pentaho job and transformations into Apache Hop projects. 🔥 The end of the Pentaho Community Edition can be the start of your Apache Hop journey. Reach out if you'd like to learn more about upgrading your project. #apachehop #asf #opensource #dataengineering #community

45

1 commentaire

Craig Kerstiens: pg_parquet: An Extension to Connect Postgres and Parquet 📢 Big News for Data Enthusiasts! 🚀 Craig Kerstiens has dropped a bombshell on the PostgreSQL community with the release of **pg_parquet**, an open-source extension that’s here to make your data life easier. Gone are the days of scratching your head when faced with Parquet files and PostgreSQL. This extension allows you to read and write Parquet files directly from Postgres, either locally or on S3. What’s the big deal with Parquet, you ask? Think efficient compression and column-oriented heaven for your SQL queries. Ideal for data lakes or when you just need to ditch that extra data pipeline. 🏞️ The GitHub repo has the goodies. Check it out: [pg_parquet](https://lnkd.in/ekpn37vH). And if you haven't set up your `.aws/config` file, it’s time to get those S3 credentials in order. ⛅ Not convinced yet? You can even inspect file structures and metadata with simple SELECT statements. Yes, it’s as cool as it sounds! So, whether you're archiving, analyzing, or sliding data around, pg_parquet is your new best buddy. Kudos to Craig and the team for pushing Postgres one step closer to analytical dominance. 🎉 #PostgreSQL #DataGeeksUnite #ParquetLove You can find the full article here: https://postgr.es/p/6F1

1

🔍Better Data from Large Language Models : The Power of Structured Outputs🔍 When we build applications that use LLMs, we often need machine-readable formats like XML or JSON from Large Language Models. Structured outputs help organize the information generated by these models into specific formats, making it easier to process and integrate the output into our applications. 👉Why Use Structured Outputs? ➡Simpler Prompting: No need for strongly worded prompts to achieve consistent LLM output formatting. ➡Deterministic Results: Ensure outputs adhere to specific schemas, with some edge cases. OpenAI recently announced support for Structured Outputs in its latest gpt-4o-2024–08–06 models. Additionally, open-source packages like Outlines offer structured outputs for both self-hosted open-weight models (e.g., Mistral and LLaMA) and proprietary APIs. Check out the sample code in the comments! ⚡ Building on LLMs? Let's connect and discuss! Follow along for more posts about AI/LLMs⚡

14

6 commentaires

New post! I talk with Philippe Noël about ParadeDB, the experience of building as a PostgreSQL extension, pg_duckdb, pg_lakehouse, and more...

25

2 commentaires

🚨This week marks as one of the historical acquisitions in data engineering community. A billion dollar 💰 acquisition of the data platform created out of Apache Iceberg, Tabular gets acquired by Databricks ! 📈🔍Value Prop: This acquisition can bring more openness and compatibility in the data Lakehouse to enable enterprise data by bringing creators of both the popular formats Delta Lake and Apache Iceberg together. It’s also interesting that yesterday, Snowflake announced their Iceberg catalog Polaris. 🌐🆘There will be some disruptions and also meaningful innovations coming to the data engineering systems. 👩🏻‍💻How do you feel about this acquisition? 🔗Full announcement link in the comments! #DataEngineering #Acquisition #Technology #LITrendingNews #TechNews #ArtificialIntelligence #Data #OpenSource #WallStreet

51

2 commentaires