Alexandre BERGERE’s Post

What are some benefits of using managed Databricks Delta tables? Hands-off Data Management: Features like automatic vacuuming, clustering by AUTO (efficiently selecting partition keys based on query patterns), row-level concurrency, auto compactions, deletion vectors, etc., are all enabled by default. Predictive Optimized Table Layout: Automated statistics collection ensures optimal performance. Automatic Table Properties Upgrade: Adopts the latest Databricks features seamlessly. Future Compatibility: Managed Delta tables will be compatible with any Databricks runtime, eliminating issues with table upgrades or runtime incompatibility. Please note that some of these features are still in preview or on the roadmap. I am yet to use managed tables in the real world but it is very enticing based on these details. Reference: https://lnkd.in/gz5cQEEh

🚀 Exciting News from Databricks! 🚀 Introducing Unity Catalog: Your One-Stop Solution for Data Governance! Are you ready to take control of your data access policies, streamline auditing, and enhance data discovery across all your Databricks workspaces? Look no further than Unity Catalog! 🔍 With Unity Catalog, you can now: Define your data access policies in one centralized location and ensure security across all workspaces. Rest assured with our standards-compliant security model based on ANSI SQL, granting permissions with ease. Automatically capture user-level audit logs and track data lineage effortlessly. Tag and document your data assets for efficient data discovery. Gain easy access to operational data with our system tables feature (currently in Public Preview). Ready to revolutionize your data governance strategy? Learn more about Unity Catalog today: https://lnkd.in/gBpmGZjs #AzureDatabricks #UnityCatalog #DataGovernance #DataManagement #TechInnovation

🌟 Week 7 Recap: Migrating to Unity Catalog 🌟 This week, we delved into migrating to Databricks Unity Catalog from Hive Metastore. Highlights include: 1️⃣ Combining Hive Metastore & Unity Catalog: Centralize metadata for better management and security. 2️⃣ Sync Command: Automate migration, ensuring data integrity and reducing manual effort. 3️⃣ Data Replication: Utilize CTAS or Deep Clone for consistent data transition. 4️⃣ Automating with UCS: Leverage UCX for streamlined upgrades with new features. 5️⃣ Reading Open Delta Shared Data: Facilitate secure, efficient data sharing. Check out the full blog for in-depth insights! 📖 ⬇️ #DataEngineering #Databricks #WhatsTheData

Exploring the Microsoft DEV Proxy: https://lnkd.in/dkd-7zgF Enhancing #API Testing with Realistic Simulations #Microsoft365Dev #DevProxy #MicrosoftGraph

Useful scripts ahead

Have you encountered this issue recently? You're trying to access a Delta table that's "materialized" in Unity Catalog, but you're facing an error stating "Unity Catalog is not enabled on this cluster." Fortunately, it's a straightforward fix. This problem arises because you're using the "no isolation shared cluster" access mode, which isn't compatible with Unity catalog Delta tables. To resolve this, simply edit your cluster and change the access mode to "shared." Please note that Unity catalog compute clusters may not be compatible with the hive metastore file system file paths that start with "dbfs:/". You might need to standardize your file paths to an acceptable Unity catalog format to avoid issues. For further information, please refer to the official documentation: https://lnkd.in/gQyyHCtu PS. I host an Azure Data Engineering bootcamp focused on Databricks once or twice a year. If you're interested, you can sign up or join the waiting list in the first comment of this post by visiting my coaching website. #dataengineering #databricks

Are you ready to migrate to #UnityCatalog? Take our Unity Catalog Readiness Quiz to find out. Our Databricks experts have created a guide on what you need to know for a successful migration. ▶️ https://bit.ly/3AeAcev #unitycatalog #databricks #datamigration #dataengineering

I was recently asked about Polaris versus Unity Catalog. Last week, I mentioned that Polaris was still being conceived. However, after testing Polaris (Snowflake managed), I updated my statement to my client: it’s now jogging (not yet running!). Here’s what I’ve found: . Easy-to-use UI. . Ready-to-use internal catalog with Spark and Snowflake object integration. However, there are a few things you still need to wait for: . Snowflake MOR table integration. . External catalog sync with Snowflake tables (though many use cases might not require this). I will be conducting a workshop soon. comments 😎 Snowflake Dremio Check Alex Merced blog. Good one.

Microsoft introduces Magentic-One, a new generalist multi-agent system designed to handle complex web and file-based tasks. It uses an Orchestrator agent that directs four specialized agents: WebSurfer for browser operations, FileSurfer for file management, Coder for programming tasks, and ComputerTerminal for console operations. Magentic-One achieves competitive performance on multiple benchmarks including GAIA, AssistantBench, and WebArena, without requiring modifications to its core architecture. Built on Microsoft's AutoGen framework, Magentic-One employs a unique dual-loop architecture where the Orchestrator manages both task and progress ledgers. The system is open-source, along with AutoGenBench, a new evaluation tool for testing agent-based systems. It's very early, but this new movement of building generalist agentic systems is something to keep an eye out for. In addition, other current LLM-based applications like RAG will also benefit from this type of system that builds on top of multiple specialized agents.

Microsoft introduces Magentic-One, a new generalist multi-agent system designed to handle complex web and file-based tasks. It uses an Orchestrator agent that directs four specialized agents: WebSurfer for browser operations, FileSurfer for file management, Coder for programming tasks, and ComputerTerminal for console operations. Magentic-One achieves competitive performance on multiple benchmarks including GAIA, AssistantBench, and WebArena, without requiring modifications to its core architecture. Built on Microsoft's AutoGen framework, Magentic-One employs a unique dual-loop architecture where the Orchestrator manages both task and progress ledgers. The system is open-source, along with AutoGenBench, a new evaluation tool for testing agent-based systems. It's very early, but this new movement of building generalist agentic systems is something to keep an eye out for. In addition, other current LLM-based applications like RAG will also benefit from this type of system that builds on top of multiple specialized agents.