Jairo Neder Monassa Moreira

💡 Faster B-tree index scans in Postgres 17 for queries that involve IN lists or other cases where multiple array values are being passed to Postgres... - https://lnkd.in/dGzDNnSm

3

Forms can be customized in various different ways and here is a nice example using the new Layout Container 🙂 Data Capture using Inphinity ∞ from your #Qlik dashboards

14

Serverless Agentic Workflows with Amazon Bedrock by DeepLearning.AI and Amazon Web Services (AWS).

21

1 comentário

I just completed the dbt Fundamentals course. This badge demonstrates my fundamental understanding of models, sources, tests, docs, and deployment in dbt. #dbt #DataEngineering

10

Completion of a short assessment on fundamental concepts related to the Databricks Lakehouse Platform!

13

🚀 Over time and through hands-on experience, I've learned that adopting a Metadata-Driven approach in PySpark development is a more dynamic approach for data pipelines. How It Works: 1️⃣ Dynamic Data Discovery: Load metadata to understand schema and transformation rules. 2️⃣ Schema Management: Handle varying data types without manual intervention. 3️⃣ Custom Pipelines: Abstract transformation logic using metadata. Error Handling: Intelligent error checks based on metadata. Key Benefits: ✅ Scalability: Adapt easily to changing data structures without rewriting code. ✅ Flexibility: Build reusable transformations that work across datasets. ✅ Maintainability: Self-documenting code makes it easier to maintain and extend. Why It Matters: ➡️ Faster Development: Reusable components save time. ➡️ Consistency: Apply standardized logic across data sources. ➡️ Future-Proof: Easily adapt to new data or business requirements. 💡 Do share your view on this approach. #PySpark #DataEngineering #BigData #MetadataDriven #DataPipelines #ETL #ScalableSolutions #ADF #Databricks

18

There is a NEW Airflow book in town! 🔥 Look at that 👇 You will: ✅ Explore the new features and improvements in Apache Airflow 2.0 ✅ Design and build data pipelines using DAGs ✅ Implement ETL pipelines, ML workflows, and other advanced use cases ✅ Develop and deploy custom plugins and UI extensions ✅ Deploy and manage Apache Airflow in cloud environments such as AWS, GCP, and Azure ✅ Apply best practices for monitoring and maintaining Airflow And more! Not affiliated link whatsoever. Just a well written book by Dylan Storey and Kendrick van Doorn where yoj will find plenty of helpful actionable insights on Airflow. I miss you guys 🫶🫶 You will find it on Amazon Enjoy the book ❤️ #dataengineering #airflow #apacheairflow #dataengineer

572

20 comentários

Databricks Certified Data Engineer Associate. Here are the main concepts I've learned when preparing for the Databricks Certified Data Engineer Associate certification: 1.Apache Spark Architecture 2.Transformations and Actions in Spark 3. Data Management with Delta Lake 4. Optimization and Performance Tuning 5. Structured Streaming 6. Security and Best Practices

8

EN Hey guys, post I wrote about how to use PySpark with Airflow, I hope you like it! PT Hey guys, post que escrevi sobre como usar PySpark com Airflow, espero que gostem! https://lnkd.in/du9T7kGQ #dataengineering #datascience

6

Hello everyone! This is my friend Matthew, Global Developer Advocate on Databricks, responsible to produce and share great content about Apache Spark, Delta Lake, Unity Catalog e MLflow 🔥 I met him through LinkedIn because of the quality content about Delta and Spark and we started talking via message, in San Francisco I decided to meet him in person. Today I want to talk about something important: ENGLISH! English is one of the most important things you need to learn if you want to have a experience in a global company or experience going to a global conference and connecting with people! Few days ago I received SAD news, someone canceled their enrollment in an MBA that I teach simply because my class slides are in english 🤯 I was very sad, not for me, but for opportunity that this person will lose while running away from this. Other example, yesterday I had the opportunity to meet Bill Zhang, Product Director from Cloudera and if I didn't have English, this would NEVER have been possible. Do you want to have experience working in a global data team? English for yesterday!!! Reflect and think about this! Like, comment and share this post if you want Matthew Powers, CFA in BRAZIL next year! Lets make noise!

68

6 comentários

Data quality in ETL with Pandera and Python. - Repeated bugs break trust. - Data contracts to ensure a clear Pipeline. Thanks to Luciano Vasconcelos Filho and Renan Heckert.

7

1 comentário

https://lnkd.in/d7Vh8UBN - Building Data Vault on Databricks with PySpark and Delta Live Tables using streaming/batch AutoLoader - a very performant and scalable way of building enterprise DWH in the Lakehouse ecosystem

7

1 comentário

New VIDEO 🥳🥳 Apache Airflow with Spark 👇 In this video, you will learn: ✅ How to set up Airflow and Spark locally with Docker ✅ How to run a PySpark script with the SparkSubmitOperator ✅ How to run code on Spark with the new Spark decorator ✅ How to fix the most common setup issues with Spark and Airflow and more! Everything you need with the code 🤗 Link in the comment below 👇 Enjoy ❤️ #airflow #apacheairflow #dataengineering #dataengineer

705

25 comentários

What are user-defined functions in PySpark, and how should you use them? Tom Reid's latest tutorial covers all you need to know to get started with UDFs.

14

NEW VIDEO 🎉 The NEW way for transferring data with Airflow 👇 Stop wasting time finding the right operator to transfer data. Rule them all in a single task with Airflow and PyAirbyte! In this video, you will: ✅ Transfer data from S3 to BigQuery efficiently with PyAirbyte ✅ Create Python Virtual Environments to save running time ✅ Use the ExternalPythonOperator to avoid dependency conflicts ✅ Check data in BigQuery with the BigQueryHook and more! No need to juggle between different operators or use custom scripts anymore! Link in the first comment below 👇 Enjoy ❤️ #dataengineering #dataengineer #airflow #apacheairflow

301

9 comentários