Douglas Nunes de Melo

This fantastic course allows you to explore Databricks in greater depth, covering topics such as streaming, pipelines, governance, optimisation, and more.

11

Another Step into Snowflake: Hands-On Essentials: Collaboration, Marketplace & Cost Estimation Workshop

3

I recently completed a Udemy course to enhance my Snowflake skills. Snowflake is a powerful tool for data engineers, offering an intuitive UI and SQL-friendly interface. It's a game-changer for data management and analytics. Read more about its benefits in the post below! Curious to know more about how Snowflake can transform your data strategy? Let's connect and explore the possibilities! #DataAnalytics #CloudComputing #Snowflake #BigData #DataWarehouse #RealTimeAnalytics #DataStrategy #CostEfficiency

2

Finally, my certificate has arrived! I am thrilled to have passed the #Databricks Certified Data Engineer Professional exam. On this journey, I gained knowledge in Structured Streaming, CDC, Data Modeling, Security, and Testing. As professionals, I believe that earning certifications is an excellent way to challenge ourselves, improve our knowledge, and validate what we have learned. I am grateful to Derar Alhussein for his fantastic courses. I recommend complementing them with some documentation and practice. If you have any questions about the exam, feel free to contact me. #Databricks #Data #Engineering #Learning #Streaming #DataEngineering #DataModeling #DataManagement #Spark #AutoLoader #StructuredStreaming #Certification

212

19 Kommentare

I've been working with Dataiku for several years now, and I am incredibly satisfied with the possibilities it offers and the projects I've been able to lead, both within my team and in training other teams. Proud to share my certifications with you, and more are on the way! #Dataiku #MarchineLearning #ML #Designer Dataiku

29

2 Kommentare

I recently announced our collaboration with Databricks at CFC. I believe it's an excellent product and we are excited about what lies ahead. Apart from the technical features, there's another aspect that I'm personally very impressed about: Customer Support. I am dealing with a number of vendors at the moment, and working with the whole Databricks support team behind Tom Youngman, Tomasz Kurzydym, Matthew Tainton, Isabelle Pandeya and Ali Azzouz, is a really refreshing experience and I feel really valued as a customer. A few highlights of the support experience so far: - We raised a support ticket that affected our production environment, and within an hour, one of the support technicians contacted me to join a video call for him to see first hand what the issue is. Within 3-4 hours he provided a temporary workaround while the engineering team was further looking into the problem - We raised a support ticket for "spark support" (i.e. a Databricks engineer to review and help us optimise part of our code). We had a call the same day where we explained the challenge in detail. The support engineer took the challenge away, spoke to the internal engineering team and came back to us with a number of different options Databricks seems to understand that good customer experience is key to increase usage and therefore spending. In contrast, some of the other vendors I'm currently dealing with (who I shall not name): - One vendor hasn't acknowledged a support ticket after 3 days since it was submitted despite the follow-ups. I guess we fell in the black hole 🕳 - Another vendor who we contacted regarding documentation about their APIs, thought it was appropriate to forward our request to their professional services team (who can help us if we pay 🙄) Well done Databricks.

139

6 Kommentare

An idempotent pipeline is a very important concept in data engineering. It refers to a data processing workflow where executing the same operation multiple times produces the same result as executing it once. In simpler terms, it means that no matter how many times you run the pipeline, you end up with the same outcome, without causing any unintended side effects or duplications. In a simple example, let's say you have a pipeline that imports data from a CSV file into a database table. In an idempotent pipeline: • The pipeline checks if the data from the CSV file already exists in the database table. • If the data doesn't exist, it inserts the data into the table. • If the data does exist, it skips the insertion step to avoid duplicating the data. #dataengineering #zachwilson

10

This is a great (very short) online book for understanding fundamental data engineering concepts such as ETL, ELT, Data Modelling, Data Warehousing etc. https://lnkd.in/d2cPCe7H

2

Data Modeling Data modeling is 'the first thing' (yeah, maybe not first, some will say second, but that is not the point) from Data architect's role. So, what is it? In a nutshell, conceptual/logical/physical - all aspects of data control, access and flow defining action. If I draw a very simple diagram like below (This may look like hand drawing , meaningless picture - sorry), but when this is displaying some meaningful inter-staff recognizable concepts, then it's not a child's play, but a conceptual diagram. The core point of conceptual diagram is, making some meaningful structure, and if it is something visualizable, good. My personal preference is expressing core concepts/ideas, not worrying about formality of symbols (like MS Visio does) ** Well, it seems I cannot link multiple somethings in a single post, so in this case, I will yield to the short video below. Before I end this simple post, I want to link a short(very short) video from MS. It's about Azure Cosmos DB, but really I was impressed in the middle of video, stating 'Data modeling is the process by which you establish patterns and rules for storing data across documents and containers, determining how data is accessed' - very concise and well stated. (yes, Cosmos DB was previously called Document DB and really it's Document DB, so, in this context, this is correct, but not in general Data modeling concept, of course) Again, I am not advertising MS. Just this one short video I came across was good - whoever wrote 'the script' behind this video.

2

Check out my blog post https://lnkd.in/eb2ynwxh #newblogpost #datastewardship #datagovernance #dataresponsibility #dataethics Data is among a company's most powerful assets, with the potential to drive innovation, inform decisions, and create competitive advantages. However, this power brings responsibility. Mishandling data—whether through lack of governance, context, or ethical considerations—can have harmful effects, impacting individuals, communities, and even society at large. Without careful management, data can shift from a beneficial asset to a potential threat, underscoring the critical need for responsible data governance.

5

Data Engineering: Mastering Apache Spark Executor Memory Configuration! 🔧 Key Configurations and Their Impact: ➡️ spark.executor.memory (8GB): Allocates memory to the JVM heap for each executor. Finding the right balance here is crucial for optimal performance. ➡️ spark.memory.fraction (0.6): Determines the split between execution and storage memory. Adjusting this significantly affects shuffles, joins, and aggregations. ➡️ spark.memory.storageFraction (0.5): Divides memory for caching and execution within the allocated fraction. Fine-tuning this improved data caching efficiency and reduced disk spills. ➡️ spark.executor.memoryOverhead (800MB): Provides extra memory for Spark's internal operations and non-heap memory usage, preventing out-of-memory errors during intensive processing. 🔍 Insights and Learnings: ➡️ Proper configuration can drastically enhance Spark's performance, reduce job execution time, and improve resource efficiency. ➡️ Balancing storage and execution memory is crucial for caching RDDs, DataFrames, and performing complex transformations.

16

🚀 Exciting News for Data Enthusiasts! 🚀 Google Cloud has announced the preview of Conversational Analytics in Looker! 🎉 This innovative feature is set to transform how we interact with data by allowing users to engage with their data using natural language. 💡 Why Does This Matter? 𝐔𝐬𝐞𝐫-𝐅𝐫𝐢𝐞𝐧𝐝𝐥𝐲: No need for deep technical knowledge or complex query-building. Just ask questions naturally and get the answers you need. 𝐅𝐚𝐬𝐭𝐞𝐫 𝐃𝐞𝐜𝐢𝐬𝐢𝐨𝐧-Making: Teams can quickly access insights without waiting for specialized data analysts, speeding up decision-making processes. 𝐃𝐞𝐦𝐨𝐜𝐫𝐚𝐭𝐢𝐳𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐀𝐜𝐜𝐞𝐬𝐬: More team members can engage with data directly, fostering a data-driven culture within organizations. 📊 What’s Next? If you’re already a Looker user, keep an eye out for this feature in the preview. If not, it might be the perfect time to explore how Looker can revolutionize your data strategy! For more details, check out the full announcement from Google Cloud : https://lnkd.in/gn7BYNtC #ConversationalAnalytics #Looker #BusinessIntelligence #AI #GoogleCloud

12

1 Kommentar

If you are interested in integrating dbt transformations with PostgreSQL directly within Airflow using the Cosmos package then this post is for you. With Cosmos, profile mapping is a breeze, making it easier than ever to leverage dbt in orchestrated pipelines. This setup streamlines data transformation tasks by managing dbt dependencies and profiles through Airflow. If you're looking to make your dbt workflows even more efficient, the Cosmos package is worth exploring! 👉 Link is in the first comment #dbt #PostgreSQL #Airflow #DataEngineering #Cosmos

25

3 Kommentare

🔍 Determine Desired Partition Size: Choosing the correct partition size is crucial for maximizing performance and resource utilization in your Spark jobs. The general guideline suggests targeting partition sizes between 128 MB and 1 GB. This range is known to balance the trade-offs between performance and resource consumption effectively. 👇Key Points to Consider: 👉Default Setting: By default, Spark sets spark.sql.files.maxPartitionBytes to 128 MB. 👉Customization: You can adjust this setting based on your workload using the configuration: spark.conf.set("spark.sql.files.maxPartitionBytes", "1G") This allows flexibility to increase or decrease partition sizes, optimizing performance for different table sizes. No alternative text description for this image

7

Unlock the potential to apply filters across unrelated tables effortlessly with the TREATAS function. This powerful tool bypasses traditional relational constraints, offering flexibility and precision in your data analysis processes. 📈 Elevate your analytical strategies by incorporating TREATAS for more dynamic and customized insights. Learn how to apply it effectively within your projects with our practical use cases and optimization tips. #BusinessIntelligence #DataScience #PowerBI #DAXtips #Analytics

2

#azure #sql #databricks #spark #pyspark #dataengineer #dataanalsyt Hi Connections, Let's solve another SQL problem. Task: Find the max and min populated cities in each state.

18

1 Kommentar