Data Engineer, Lead Consultant | Databricks Certified Professional | Expertise: Spark, Pyspark, SQL, Python | Proficient in Azure, ADF, dbt, Hive & Hadoop | Transforms raw data to insights
Small doubt, Why are Databricks not much interested in this area : Integrating Databricks Secrets with AWS secrets manager ? In the Azure cloud, Azure Key vault integration with Databricks Secrets made lives easier for secret management. Note : I know that we can use boto3 directly to retrieve the secrets from AWS secrets manager using IAM permissions. But it has its own limitations in case of shared access mode.
Vineeth Jain
3mo
Why Databricks will invest their time and effort on AWS integration when they already have set up their own secret scopes (Databricks backed scopes)
Ziyauddin Shaikh
3mo
Databricks is a key partner for Microsoft and it comes as first party service with azure and supports unified billing where as Aws & GCP get it as third-party service.
Shad Saleem
3mo
It might be because Databricks is very well integrated with Azure and the same is not the case with AWS
𝐃𝐚𝐭𝐚 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭 | 𝐆𝐞𝐧 𝐀𝐈 | 𝐆𝐥𝐨𝐛𝐚𝐥 𝐄𝐧𝐚𝐛𝐥𝐞𝐦𝐞𝐧𝐭 𝐋𝐞𝐚𝐝 (𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬) | Big Data & Cloud Data Architect | Experienced mentor to aspiring data engineers
3moAs we all know, Azure Databricks was co-developed by Microsoft and Databricks, leading to a more seamless integration with Azure services like Azure Key Vault. In contrast, while Databricks is available on AWS, it is not as tightly integrated with AWS services. But It’s possible that if there’s enough demands, then Databricks and AWS might consider adding this integration in the future same as Azure.