Pedro Nieto Peláez

W𝗵𝗮𝘁 𝗲𝗱𝗶𝘁𝗶𝗼𝗻 𝗼𝗳 𝗺𝗼𝗻𝗴𝗼𝗱𝗯 𝘀𝗵𝗼𝘂𝗹𝗱 𝗜 𝘂𝘀𝗲 𝗶𝗻 𝗺𝘆 𝗽𝗿𝗼𝗷𝗲𝗰𝘁? Our team member Ernesto Lluesma compares each edition. Perfect if you work with NoSQL databases or are considering migrating to MongoDB. 📊 In this article, you will find: • Performance and feature comparisons • Security and scalability considerations • Maintenance options and costs 🎯 This information will help you: • Optimize your data infrastructure • Reduce operational costs • Improve the efficiency of your applications 📖 Read the full article and make informed decisions: https://lnkd.in/dBYrTVUF 💡 Looking for expert advice on MongoDB for your project? Contact us. #MongoDB #NoSQL #TechBlog

4

Take your first steps with #ApacheKafka and Flink on Confluent! In this video of the #DataAICommunity, Òscar del Álamo introduces how to build #DataEngineering stream pipelines and queues with Apache Kafka and how to query and process this data in real-time using #ApacheFlink. Plus, you'll learn how to easily deploy an instance on the cloud with the Confluent platform. Discover how to do it and start experimenting with Confluent's free entry-level plans! Watch the video now 🎥👇🏻 #data #betterjoinERNI #betteraskERNI

10

Play Tetris-sql, the game built on #Postgresql! GitHub - nuno-faria/tetris-sql: Using SQL's Turing Completeness to Build Tetris

3

Are you ready to boost your Code Efficiency! 🚀 In our latest blog post, Azure Data expert, Ines Olmos Alonso, shares practical tips to improve code structure, readability, and efficiency. From moving from notebooks to well-organized scripts to advice on naming variables and functions, this article offers essential tips to help you write cleaner, more collaborative code. 📈 Enhance your productivity and that of your team with best coding practices. 👉 Read the article here: https://lnkd.in/e53gQfEz

21

Big shout for #PyDay #Barcelona 2024. I'm presenting about some of my favorite #Open #Source #Databases. My 1.5 hour workshop focuses on my personal experiences having running Oracle, Postgres and MySQL at very high scale over the last 15 years. Apologies for my #python I mostly only need do the basics (regex, json, sql, matching, sorting... ) but I do love a good #jupyter #notebook too ❤️ 🧡 💛 💚 💙 💜 More details here: https://lnkd.in/dKBcJSTX

1

How can you handle long pauses of your data in #ApacheFlink? Since your stream is stopped, setting an idle timeout won’t work. In the latest video in his Apache Flink in Action series, Dan Weston explores one possible solution. Give it a watch and then share your solution in the comments! Watch it here ➡️ https://lnkd.in/gjmAv46B

50

1 comentario

🤔 What are the differences between Data Fabric & Data Mesh? How does Confluent enable these architectures? 🎙️ We answer these questions and more in Episode #4 of our Confluent Capabilities 'Bite Size' podcast series with Italo Nesi and John Dee! Discover how Confluent enhances both of these models through its cloud platform, showcasing it as a versatile solution and demonstrating its crucial role in modern data strategies. 🖥️ Watch the latest episode here: https://lnkd.in/emG9jRJ2 Paul Archer Joanna Grimmond Chris Gallagher Bob Feeney #HybridCloud #MultiCloud #Confluent #DataFabric #DataMesh

10

Building trust in your data is crucial, but dbt offers a powerful toolkit to address this challenge. ✅ Join us for a talk by Juan Manuel Perafan, Analytics Engineer Consultant at Xebia, who is coming to Budapest to share his knowledge of dbt testing. Take the opportunity to ask him your questions! In this session, you’ll learn about: 🟢 Essential tips and tricks for setting up effective testing in your dbt projects. 🟢 Various dbt testing methods to ensure data quality at every stage. 🟢 How to avoid common mistakes and build robust, reliable data pipelines. Come and level up your dbt testing skills! As always, we'll have snacks and beverages. 🗓️Space is limited, don’t forget to RSVP here: https://lnkd.in/dMuUBxZ4

1

We're sharing our journey implementing StringView in Rust for Apache Arrow and integrating it into Apache DataFusion. The result? String-intensive queries are now 20%-200% faster in the ClickBench benchmark! ⚡ https://bit.ly/4cYNOI9 #ApacheArrow #DataFusion #Rustlang #InfluxDB

9

We're all about keeping the simple things *simple*. With #Wasm (#WebAssembly) transformations directly in Redpanda, developers can read data, prepare messages, and make API calls without the dreaded “data ping pong.” The result? Simpler, less expensive data transformation 🤘 If you're curious how it all works under the hood, we wrote a blog post just for you👇

10

Azure AI Studio is the VS Code for AI engineers and developers. Through AI Studio, you can deploy and run multiple LLM models from OpenAI, Facebook Llama, Mistral, Microsoft Phi-3 and more. But it also have a very nice and easy feature to work with RAG. With just few steps and clicks, you can add your data into a RAG flow. Eager to learn more ? I created this video for you :) And if you are looking for a demo, I also created one for you. https://lnkd.in/efaqtDy3 Disclaimer: This video is part of my upcoming course on Udemy about AI services in Azure. https://lnkd.in/dcQJK-H

115

5 comentarios

Have you run into issues with balancing schema flexibility and data consistency as your app grows? With Fauna’s adaptive schema enforcement, you can start with a schemaless collection, progressively add strict schema over time, and evolve it with zero-downtime migrations to meet growing data requirements. This means you gain all the benefits of strict data consistency when you need it—without compromising flexibility in the early stages of development. If you’re curious about how it works, check out our step-by-step tutorial on progressively defining and enforcing document types in Fauna. 👉 Dive into the tutorial here: https://lnkd.in/d78g4i6G #databaseschema #serverless

3

𝘛𝘰𝘱 𝘦𝘮𝘦𝘳𝘨𝘪𝘯𝘨 𝘰𝘱𝘦𝘯-𝘴𝘰𝘶𝘳𝘤𝘦 𝘱𝘳𝘰𝘫𝘦𝘤𝘵𝘴 𝘴𝘩𝘢𝘱𝘪𝘯𝘨 𝘵𝘩𝘦 𝘧𝘶𝘵𝘶𝘳𝘦 𝘰𝘧 𝘥𝘢𝘵𝘢 𝘦𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨 🚀 • 🧊 Apache Iceberg: A data lakehouse table format which provides ACID guarantees, schema evolution, time travel, and other key features. • 🧡 dbt Labs: A SQL-based tool for performing data transformations directly within your data warehouse/lakehouse for efficient ELT processes. • ↔️ Airbyte: A data integration platform for moving data across various sources and destinations. It supports both ELT and ETL processes, and provides +300 connectors. • 🦆 DuckDB: A SQL OLAP database designed for fast analytical queries. It runs directly within your application and it’s really easy to install and deploy. • 🐙 Prefect: A workflow orchestration framework for building data pipelines in Python with an emphasis on simplicity and observability. • 🐻‍❄️ Polars: A data manipulation library that leverages Rust's speed and memory efficiency for processing data on a single machine. • ✅ Great Expectations: A data quality tool for ensuring data reliability through data validation, profiling… It generates documentation for visualizing data and its expectations. What are your thoughts on these open-source tools? Would you add any more to this list? Tell me in the comments 👇🏽 #data #dataengineering #datamanagement #etl #dataquality #dataintegration #dataarchitecture #lakehouse #datatransformation #dataprocessing #datapipelines #dataorchestration #dataingestion

16

5 comentarios

📷 New on YouTube: In his 2023 #SAGconf session, Matthias Niehoff delves into the latest developments in modern data architectures, including Data Mesh, Data Lakehouse, and the integration of analytics with operational systems, offering insights into their practical implementation. 👀 Watch now: https://lnkd.in/eHs3EkZa #SoftwareArchitecture #SoftwareArchitectureGathering #SoftwareArchitectureConference #DataArchitecture #DataMesh #DataLakehouse #ELTPattern #CloudDataWarehouse #ModernDataStack #DataStrategy #SoftwareEngineering

8

#ApacheFlink 1.20 is here! What's New? 1⃣ Deprecation of the #SinkFunction - ▶️ On the eve of Flink 2.0, the #SinkFunction type has finally been deprecated and users are recommended to move towards sinks that are implementations of the #UnifiedSink type. 2⃣ Introduction of Materialized Tables - ▶️ A new feature of Flink SQL, aimed to bridge the gap between streaming and batch pipelines, allows to create a table source that has a given data freshness tied to it, so that Flink can keep the table data as up-to-date as possible. 3⃣ Unified File Merging Mechanism for Checkpoints - ▶️ An update to the checkpointing mechanism which combines multiple small checkpoint files into fewer larger files, this reduces the number of file creation and file deletion operations and alleviates the pressure of file system metadata management during checkpointing operations. 4⃣ DataSet API has been fully deprecated - ▶️ DataSet will be fully removed in Flink version 2.0, users are recommended to either implement their pipelines using the DataStream API, the Table API, or Flink SQL for their data processing needs. 5⃣ Support Full Partition Processing on DataStream API - ▶️ A more technical update to the DataStream API, this allows users to implement aggregations upon non-keyed streams by using the new #MapPartitionFunction type. 6⃣ Massive Cleanup of Flink Configuration in preparation of Flink 2.0 - ▶️ Overhaul of Configuration for State & Checkpointing Options, moved to 'execution.checkpointing.' and 'execution.state-recovery.', and 'state backends' have been moved to 'state.' with backend settings going to 'state.backend.' ▶️ Smaller more technical changes for Batch Execution, Table Optimizer, & Aggregate Optimizer options. #apacheflink #dataobservability

1

Using Airflow? Log metadata generated from different tasks of Airflow DAG runs to Neptune. Thanks to the Neptune-Airflow integration, you can more easily: → Compare model results → Track the workflow/DAG config → Have metadata from different tasks in one place Check the docs and an example project here: https://buff.ly/48eX74E #ml #machinelearning #mlops

1

Learn How to Run Hugging Face Models Programmatically Using Ollama  and #Testcontainers 🤖💻 Dive into the future of AI/ML with easy model deployment 👋👋 Say goodbye to complex setups and hello to seamless integration 👉👉 Docker, Inc blog https://lnkd.in/g_F89xCk by Ignacio López Luna

8

💥 We’re back and better than ever with an all new issue of Confluent Chronicles! And this time, we’re exploring the fundamentals of stream processing, #ApacheFlink, and Confluent. Get ready to join our data heroes as they embark on a quest to stop multiple teams from processing the same data streams, resulting in some serious headaches in data quality and consistency. Will Ada and Jax deploy Apache Flink in time to ensure the success of a new shoe launch? Or will they get lost in the data lake because they forgot to check the watermark? Check out the latest comic and perhaps you’ll find learning Flink can be as entertaining as it is enlightening. See the preview below and download the full comic here ➡ https://cnfl.io/3WukvJ7

121

10 comentarios

Understand how to use Apache Iceberg within Databricks on Azure. Explore why this approach is ideal for organizations looking to avoid vendor lock-in and ensure their data infrastructure remains agile. https://lnkd.in/da_e8DGn

78