Nevenka Lukic

Excellent article from Iván Palomares Carrascosa on Statology! For professionals without a profound statistics background, choosing the right statistical test to perform on their data might sometimes not be a straightforward task. This article provides a decision tree-based guide aimed at helping them navigate the problem of choosing the right test depending on the data and problem they are facing, and the hypothesis to be tested.

2

I just passed the Databricks Machine Learning Professional certification 🤓 This certification is great if you are interested in open-source tools: most of the exam is about MlFlow, SparkML, Delta, and - naturally - how it all fits together in the Databricks platform 📊 Studying for it is also quite a smooth experience: the videos and labs at the Databricks academy are super well elaborated and cover all the topics of the exam, and by complementing that with the documentation and a few books you get all you need to pass. My only criticism of Databricks is that the free version of the platform (Databricks Community Edition) is very limited, especially when it comes to machine learning. That may be fine if you are working on a project and have access to a premium/enterprise environment, but even then we can't just "play around" a customer's tenant 😢 If you want more tips about Databricks, feel free to ask or check my study annotations: https://lnkd.in/dpjPQ6Q2 #machinelearning #cloudcomputing

11

🚀 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝘄𝗶𝘁𝗵 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝘂𝘀𝗶𝗻𝗴 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗙𝗮𝗰𝘁𝗼𝗿𝘆 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀🚀 In the dynamic world of data engineering, efficiency is key. One effective strategy for improving performance and resource utilization is the 𝙄𝙣𝙘𝙧𝙚𝙢𝙚𝙣𝙩𝙖𝙡 𝘼𝙥𝙥𝙧𝙤𝙖𝙘𝙝. By processing only the changes since the last run, you can significantly reduce the load on your systems and accelerate data workflows. Here’s how you can leverage Azure Data Factory (ADF) and Databricks to implement an incremental approach with Spark: 1. 𝘿𝙖𝙩𝙖 𝙎𝙤𝙪𝙧𝙘𝙚 𝙄𝙣𝙩𝙚𝙜𝙧𝙖𝙩𝙞𝙤𝙣 𝙬𝙞𝙩𝙝 𝘼𝙯𝙪𝙧𝙚 𝘿𝙖𝙩𝙖 𝙁𝙖𝙘𝙩𝙤𝙧𝙮 Start by setting up an Azure Data Factory pipeline to orchestrate your data workflow. Use ADF’s built-in connectors to pull data from various sources like SQL databases, APIs, or cloud storage. Configure a watermark column (e.g., `LastModifiedDate`) to track changes. 2.𝙄𝙣𝙘𝙧𝙚𝙢𝙚𝙣𝙩𝙖𝙡 𝘿𝙖𝙩𝙖 𝙀𝙭𝙩𝙧𝙖𝙘𝙩𝙞𝙤𝙣 In ADF, use the 𝘾𝙤𝙥𝙮 𝘼𝙘𝙩𝙞𝙫𝙞𝙩𝙮 to pull only the data that has changed since the last extraction. This can be done by comparing the current timestamp with the timestamp stored from the previous run. The filtered dataset minimizes the data volume, making the process more efficient. 3. 𝘿𝙖𝙩𝙖 𝙏𝙧𝙖𝙣𝙨𝙛𝙤𝙧𝙢𝙖𝙩𝙞𝙤𝙣 𝙬𝙞𝙩𝙝 𝘿𝙖𝙩𝙖𝙗𝙧𝙞𝙘𝙠𝙨 Pass the incrementally extracted data to Databricks for transformation. Use Spark to process and analyze the data. With Spark’s 𝘿𝙚𝙡𝙩𝙖 𝙇𝙖𝙠𝙚, you can handle updates and merges seamlessly. Delta Lake’s support for ACID transactions ensures data consistency and reliability. 4. 𝙈𝙚𝙧𝙜𝙚 𝙖𝙣𝙙 𝙐𝙥𝙙𝙖𝙩𝙚 Leverage Spark’s powerful data manipulation capabilities in Databricks to merge the incremental data with existing datasets. This might involve updating existing records or inserting new ones based on your business logic. 𝙀𝙭𝙖𝙢𝙥𝙡𝙚 𝙒𝙤𝙧𝙠𝙛𝙡𝙤𝙬: - 𝘼𝘿𝙁 𝙋𝙞𝙥𝙚𝙡𝙞𝙣𝙚 extracts records from an SQL database where `LastModifiedDate` is greater than the last run timestamp. - The incremental data is then transferred to a Databricks notebook. - 𝘿𝙖𝙩𝙖𝙗𝙧𝙞𝙘𝙠𝙨 performs transformations using Spark, applying complex business rules and aggregations. - Finally, the processed data is loaded back to a data warehouse or lake, ensuring it’s updated and ready for analytics. 𝘽𝙚𝙣𝙚𝙛𝙞𝙩𝙨: - 𝙍𝙚𝙙𝙪𝙘𝙚𝙙 𝙇𝙤𝙖𝙙 on source systems by processing only changed data. - 𝙁𝙖𝙨𝙩𝙚𝙧 𝙋𝙧𝙤𝙘𝙚𝙨𝙨𝙞𝙣𝙜 times with smaller datasets. - 𝘾𝙤𝙨𝙩 𝙀𝙛𝙛𝙞𝙘𝙞𝙚𝙣𝙘𝙮 due to lower compute and storage requirements. Implementing an incremental approach with Azure Data Factory and Databricks can revolutionize your data operations, leading to faster insights and more efficient processes. Are you ready to take your data pipeline to the next level? 🌟 #DataEngineering #AzureDataFactory #Databricks #Spark #DataTransformation #IncrementalProcessing #DataPipelines

5

1 comentario

In this article we will focus on, 1. Unity Catalog Overview 2. Unity Catalog object model 3. Working with database objects in Unity Catalog 4. Other securable objects 5. Granting and revoking access to database objects and other securable objects in Unity Catalog 6. Managed versus external tables and volumes 7. Data isolation using managed storage 8. Auditing data access 9. Tracking data lineage 10. Lakehouse Federation and Unity Catalog 11. Delta Sharing, Databricks Marketplace, and Unity Catalog 12. How do I set up Unity Catalog for my organization? 13. Migrating an existing workspace to Unity Catalog 14. Unity Catalog requirements and restrictions https://lnkd.in/dU_PmJZ4

4

🎉 Just completed my Python course on Kaggle! 🚀 Curious about Python's impact on Machine Learning? Let's discuss! Python's pivotal role in Machine Learning cannot be overstated. As a versatile and powerful language, it is the backbone of numerous ML projects. Here are a few reasons why Python is indispensable for Machine Learning: · Ease of learning: Python's simple syntax makes it accessible to beginners and experts alike. · Rich libraries: Extensive libraries like TensorFlow, PyTorch, and scikit-learn accelerate development. · Community support: A robust community means ample resources and support. · Integration: Seamless integration with other technologies and tools. Do you agree with me? 👍🏼 #Python #MachineLearning #AI #Programming #DataScience #Tech #Coding #DeepLearning #ArtificialIntelligence #LinkedInLearning

1

A guide to choosing the correct classification model 🔷

3

motoring machine learning in python #python

10

#accelerates #BI with #AI in a single analytics #platform #Intelligence everywhere however you need it. https://lnkd.in/dXBkH6gS

1

✨In this new data engineering episode, I demonstrated how to append and convert multiple CSV files stored in ADLS Gen2 into a single columnar storage Parquet file format to enhance query performance and reduce storage costs leveraging Azure Data Factory . Watch Here: https://lnkd.in/eQ7ME4tu #azuredatafactory #parquet #bigdata #dataengineering #dataanalytics

10

A video of me showing you how to set up a CDC flow for #PostgreSQL in a minute with Estuary Flow :^) https://lnkd.in/d4im5uMg

12

3 comentarios