Gopi Kudaravalli

Curious how to use Apache Hudi & Iceberg tables in Databricks environment? Check out this hands-on blog by Dipankar Mazumdar, M.Sc 🥑 that shows how to use Apache XTable to interoperate between the 3 table formats in a lakehouse architecture. 🔗 Link: https://lnkd.in/dwTdBinz #dataengineering #softwareengineering

17

1 Comment

Excited to share that Vijay K. and I collaborated on a guide for Active-Active Replication on Amazon RDS for MySQL! This comprehensive resource is now available on AWS Solutions https://lnkd.in/d2nH2NH3 #aws #amazonrds #mysql #highavailability #disasterrecovery #rds #mysql

25

1 Comment

Databricks has a new feature that allows developers to create AI applications in just five minutes - 𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 𝐀𝐩𝐩𝐬. The new feature drastically reduces the time required to develop AI applications, empowering developers to focus on creativity and problem-solving rather than technical complexities. They can build scalable AI solutions that can be easily adapted to various business needs and environments. 🔗 https://lnkd.in/d2WiJzgc #AI #Databricks #Innovation #TechTrends #Scalability 🚀💡

6

🚨dbt for Snowflake on Snowflake Native Apps! I found this at Data Cloud Summit 2024！ 1. Transform data end-to-end, entirely within Snowflake. 2. "Ask dbt" generates SQL & tests from natural language queries using the Semantic Layer. #dbt #Snowflake #DataTransformation #Analytics​#DataCloudSummit

156

Building a generative AI platform requires strategic decisions to ensure flexibility, scalability, and efficiency. Key challenges include optimizing infrastructure, managing distributed systems, and handling multi-tenancy for diverse use cases. The trade-offs between using pre-built solutions versus custom frameworks impact deployment speed, resource allocation, and model performance. Lessons from industry trends reveal the importance of tailoring the approach for scalability and long-term maintainability. #GenerativeAI #MLOps #AIInfrastructure #Scalability #AIPlatform #DistributedSystems #MachineLearning #TechInnovation #AIModelDeployment https://lnkd.in/eGasgtiu

4

1 Comment

It gives us great pleasure to announce that Apache Spark in BigQuery is now officially available in Private Preview. This feature expands BigQuery’s processing capabilities by offering Apache Spark as an OSS processing engine under a unified control plane (admin, monitoring, billing, security, governance). It is the first step in delivering on the Unified Data Platform vision and was a key announcement at Next ‘24 #googlecloud #bigquery #spark

38

2 Comments

I am excited to share that I have earned a new Generative AI certification by Databricks. This certification allowed me to learn key concepts with hands on notebooks on following area •Architecture and design of AI applications, RAG Applications, Chains, Agentic Workflow, Multi-stage reasoning and so on. •Data Engineering, data preparation, chunking strategy, delta lake embedding, Vector search index. •Data Model deployment, MLFlow, Access Control with Unity Catalog, Model serving endpoints. •Qualitative and Quantitative evaluation of the Retrieval and LLM Model, LLM as a judge for model evaluation, Guardrails and responsible AI measures, LLMOps, Cost evaluation and Governance. I recommend this certification to those who want to learn and be part of the inevitable future. I’m eager to leverage these skills and connect with fellow tech enthusiasts. #GenerativeAI #TechInnovation #Certification #LLMs #AIApplications #MachineLearning #GenAI #RAG #Learning #Databricks #happylearning

61

14 Comments

Check out how Kurve automates schema inference between raw files and integrates with open catalogs and standards like Apache Iceberg for end to end ML workloads:

10

A comprehensive guide on constructing LLMs using Databricks. Guide showcases the incorporation of LLM-specific tools and resources within the Databricks framework. It covers aspects such as Vector Indexes, open-source LLM models, and hosted endpoints. Find the guide here: https://lnkd.in/ehtTQhhc Seattle Data Guy

12

Wanna learn something & challenge yourself with Databricks & #GenAI this fall? Join the challenge - https://lnkd.in/g2zFFve2 🌟🌟🌟 🌍 🏆🏆🏆 #Databricks #GenAI #CloudDataPlatform #AWS #Canada #USA

18

1 Comment

Thousands of companies have embraced Databricks to enhance their next-generation analytical capabilities. With the introduction of Databricks Serverless SQL, the experience has become better and more efficient. As a Senior Data Architect, I'd like to share my experiences using Databricks SQL in both classic and serverless compute modes, and explain why it's wise to consider serverless SQL along with the benefits it offers. Thank you Jason Y. and Jeffrey Lipkowitz for the support and guidance. Tredence Inc. #Tredence #databricks #DatabricksServelessSQL #Serverless #DAIS2024

27

2 Comments

"Excitingly, Databricks has introduced serverless job compute! In addition to Serverless SQL, users now have two options for serverless compute within Workflows. Serverless jobs offer reduced configuration options, providing Databricks users with a simplified way to schedule jobs without feeling overwhelmed by settings and allowing Databricks to manage the infrastructure and scaling." https://lnkd.in/exHrtKRr

2

Time series transformer models in Databricks to get trained on your data : Chronos TimesFM Moirai TimeGPT https://lnkd.in/gh_vwWE2

7

Databricks SQL Serverless is now generally available on Google Cloud! Unlock instant, elastic compute and elevate your data analytics with faster query performance and reduced costs. Focus on delivering insights that drive business value, rather than managing infrastructure. Available across 7 GCP regions and 40+ regions globally, SQL Serverless brings you the best of both worlds—unmatched performance and simplified analytics. Learn more here: https://lnkd.in/dbsE87KT

2

Let's examine a sample architecture in GCP to implement a Real-time Dashboard with an AI chatbot ✅Pub/ Sub ingests data from streams ✅DataFlow runs the ETL pipeleine ✅Big Query Analyze the data ✅Looker displays the dashboard ✅Vertex AI use the Language models to create a Chatbot 🛫Real-World use case: Gojek - A delivery app in Indonesia which started as a call center to book taxis, now utilize all of these features in GCP for their users to book Bikes and delivery services. The AI chatbot named Dira is also used with the help of Vertex AI #GCP #RealtimeDashboard #StreamingData

8

4 Comments

Meta has launched Llama 3.2, a powerful suite of large language models now accessible via Snowflake Cortex AI! With an impressive range of models from 1B to 90B parameters, organizations can seamlessly weave AI into their operations to drive real-time insights and foster innovation. Whether utilizing AI on-device for edge computing or leveraging secure, serverless inference on Snowflake, businesses have the freedom to select the ideal model tailored to their unique requirements—all within a secure and governed data landscape. 🌐💡 Featuring enhanced safety measures, extensive multilingual capabilities, and effortless scalability, Llama 3.2 is poised to empower sectors such as retail, healthcare, and manufacturing to discover exciting new horizons with generative AI.

16

My colleague Lars Skaar makes a good point on how to improve flexibility in analytical development across data platforms. Have a look at his latest blog post!

3

As our applications evolve, so must our approach to data storage. Let's examine key factors in selecting the right database for advanced use cases. Also, Join me for an upcoming Free webinar - RSVP here : https://brij.guru/ai 𝗪𝗲'𝗹𝗹 𝗰𝗼𝘃𝗲𝗿: • Fundamentals of database branching • Practical implementation strategies • How to leverage this technique for AI applications 1. Object-Oriented DBs: Consider for complex object models where relational mapping becomes inefficient. Useful in CAD/CAM systems or scientific applications. 2. Time-Series DBs: Crucial for IoT and monitoring at scale. Look into options like InfluxDB or TimescaleDB for high-write, time-indexed data. 3. Graph DBs: Essential for relationship-heavy data. Neo4j and Amazon Neptune excel in scenarios like fraud detection or knowledge graphs. 4. Document DBs: MongoDB and Couchbase offer flexibility for rapidly changing schemas and horizontal scaling. Ideal for content management and real-time big data. 5. Columnar DBs: Vertica and ClickHouse provide superior performance for analytical workloads, especially in data warehousing scenarios. 6. In-Memory DBs: Redis and Memcached shine in caching layers and real-time bidding systems where sub-millisecond response times are critical. 7. NoSQL DBs: Beyond basic key-value stores, consider specialized options like Cassandra for write-heavy, distributed systems. 8. Relational DBs: Postgres and MySQL continue to evolve, now offering JSON support and improved scalability. Still the go-to for transactional systems. 9. Vector DBs: Milvus and Pinecone are pushing boundaries in AI applications, offering efficient similarity search for large-scale machine learning models. Key considerations: - Data model complexity - Read/write patterns - Scalability requirements - Consistency vs. availability tradeoffs - Operational costs and expertise Have I overlooked anything? Please share your thoughts—your insights are priceless to me.

2