Useful scripts ahead
Data Platform Architect at World Bank Group | Co-Founder of Databricks Comunity BG | Microsoft Certified Trainer
Today I'm sharing some utility scripts to keep your Lakehouse in order. This is part 1 of my series of posts to help you standardize your Databricks Unity Catalog objects and make sure you are following best practices on object governance.
Managing Delivery Architect at Capgemini with expertise in Azure Databricks and Data Engineering. I teach Azure Data Engineering and Databricks!
What are some benefits of using managed Databricks Delta tables? Hands-off Data Management: Features like automatic vacuuming, clustering by AUTO (efficiently selecting partition keys based on query patterns), row-level concurrency, auto compactions, deletion vectors, etc., are all enabled by default. Predictive Optimized Table Layout: Automated statistics collection ensures optimal performance. Automatic Table Properties Upgrade: Adopts the latest Databricks features seamlessly. Future Compatibility: Managed Delta tables will be compatible with any Databricks runtime, eliminating issues with table upgrades or runtime incompatibility. Please note that some of these features are still in preview or on the roadmap. I am yet to use managed tables in the real world but it is very enticing based on these details. Reference: https://lnkd.in/gz5cQEEh
Are you ready to migrate to #UnityCatalog? Take our Unity Catalog Readiness Quiz to find out. Our Databricks experts have created a guide on what you need to know for a successful migration. ▶️ https://bit.ly/3AeAcev #unitycatalog #databricks #datamigration #dataengineering
Data Engineer || PySpark || Azure Databricks || Azure Data factory || Apache Spark || AWS-Redshift || Hadoop
- Unity Catalog Unity Catalog is a Fine-Grained Data Governance Solution for data in a Data Lake. - Why is Unity Catalog Used Primarily? Imagine a Lake House Project with a database in the Hive Metastore of a Databricks Workspace containing twenty Delta Tables. If the requirement is to provide a specific set of permissions, like Read Only or Write Only, to a specific group of users on one or some particular Delta Tables, or even at the row level or column level containing Personally Identifiable Information (PII), Unity Catalog can simplify the solution by implementing Unified Data Access Control. - Primary Reason for Using Unity Catalog: Unity Catalog helps to simplify the security and governance of data by providing a centralized place to administer data access and audit data access. #DataEngineering #Databricks #UnityCatalog #BigData #DataGovernance
🌟 Week 7 Recap: Migrating to Unity Catalog 🌟 This week, we delved into migrating to Databricks Unity Catalog from Hive Metastore. Highlights include: 1️⃣ Combining Hive Metastore & Unity Catalog: Centralize metadata for better management and security. 2️⃣ Sync Command: Automate migration, ensuring data integrity and reducing manual effort. 3️⃣ Data Replication: Utilize CTAS or Deep Clone for consistent data transition. 4️⃣ Automating with UCS: Leverage UCX for streamlined upgrades with new features. 5️⃣ Reading Open Delta Shared Data: Facilitate secure, efficient data sharing. Check out the full blog for in-depth insights! 📖 ⬇️ #DataEngineering #Databricks #WhatsTheData
In this article, the scenario shows how to integrate Azure Databricks Unity Catalog external Delta tables to OneLake using shortcuts. After completing this tutorial, you’ll be able to automatically sync your Unity Catalog external Delta tables to a Microsoft Fabric lakehouse.
✨ Excited to share a new article and code sample on automating the shortcut creation of #Databricks Unity Catalog external Delta tables in #OneLake. This leverages the new shortcut API. ⚡ Follow these three steps: 1. Import sync notebook to your Fabric workspace. Notebook available here: https://lnkd.in/dj-7u8H7 2. Configure the parameters in the first cell of the notebook to integrate Unity Catalog tables. 3. Run all cells of the notebook to start synchronizing Unity Catalog external Delta tables to OneLake using shortcuts. 👉 Read the full article here: https://lnkd.in/dejEA4gK Venkatesh Titte Amanjeet Singh Trevor Olson Josh Caplan #microsoftfabric #onelake #shortcuts #deltalake #databricks #unitycatalog
Head of Data & AI Engineer, Partners at @DataGalaxy, startup founder, Investor at @Formance ☁️ Delta Lake & openLineage lover
So many insightful informations through Databricks' system tables : https://lnkd.in/e3pP3Pre. Even if you enabled the preview feature on Unity, don't forget to enable each schema through API: curl -v -X PUT -H "Authorization: Bearer <PAT Token>" "https://<workspace>.https://lnkd.in/eeZPjAX3<metastore-id>/systemschemas/<SCHEMA_NAME>" #unity #governance #databricks
Do you have burning questions about #UnityCatalog that you'd like to ask a Databricks Champion live? Here's your chance! Join our 30 minute "Unity Catalog Ask me Anything" session on May 15th at 11 AM PT: #databricks #webinar #askmeanything
Tried my hands on rust for the first time. 🦀 Built a distributed hash setup utilizing consistent hash over a couple of weeks. The demo mentioned below is 1000 keys being inserted while nodes are added to the allocator live. Making virtual nodes would be easy enough so skipped it for now. You can check out the project *chrrrrrr* & the docs here: https://lnkd.in/g6ciqzrb #rust #distributedsystems #consistenthashing
Principal Customer Advisory for Data & AI Strategy, Architecture and Products at Microsoft
Are you using #Databricks for data engineering and looking to combine it with #Fabric self-serve capabilities to boost all business users in your company? Wondering if you can get the best of both worlds without duplicating data? The answer is a big YES! Find out more with Aitor Murguzur post below! #microsoftadvocate #dataculture #AI #Analytics #automation #microsoftfabric #unitycatalog
✨ Excited to share a new article and code sample on automating the shortcut creation of #Databricks Unity Catalog external Delta tables in #OneLake. This leverages the new shortcut API. ⚡ Follow these three steps: 1. Import sync notebook to your Fabric workspace. Notebook available here: https://lnkd.in/dj-7u8H7 2. Configure the parameters in the first cell of the notebook to integrate Unity Catalog tables. 3. Run all cells of the notebook to start synchronizing Unity Catalog external Delta tables to OneLake using shortcuts. 👉 Read the full article here: https://lnkd.in/dejEA4gK Venkatesh Titte Amanjeet Singh Trevor Olson Josh Caplan #microsoftfabric #onelake #shortcuts #deltalake #databricks #unitycatalog
