Ajith R.

If you ingest data from Kafka/Kinesis into Databricks, you should watch Charles Tan and Shawn Gordon's short vide on how to shift left your data pipelines to reduce cost and latency! #ApacheFlink #Kafka #StreamProcessing #RealTimeData https://lnkd.in/ggMsi9KU

14

1 Comment

Learn more about how Gemini in BigQuery helps bring the power of AI to your Data Warehouse! https://lnkd.in/d9digzES

13

1 Comment

Really interesting read on the evolution of Data Warehouses, Data Lakes and Lake Houses and how Databricks and Snowflake are battling for the top spot. https://lnkd.in/egg4P5VG This is one of the more complete analysis I've seen on the evolution of the analytics space and it's big gorilla's. In the end, it gets into the AI discussion of big LLM's vs. small models and how SLM's will have a place in the analytics world, which I completely agree with.

9

1 Comment

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

9

Curious how to use Apache Hudi & Iceberg tables in Databricks environment? Check out this hands-on blog by Dipankar Mazumdar, M.Sc 🥑 that shows how to use Apache XTable to interoperate between the 3 table formats in a lakehouse architecture. 🔗 Link: https://lnkd.in/dwTdBinz #dataengineering #softwareengineering

17

1 Comment

Take that Azure Cosmos DB powered app to prod with the help from Sandeep and Saranya! Watch the three part video series: "Mastering #AzureCosmosDB: A Comprehensive Guide from Prototype to Production." https://lnkd.in/eEMt5TFG

1

In this tutorial we’ll present a CDC pipeline with Kafka to stream changes to a Snowflake Lakehouse in Real Time

5

It gives us great pleasure to announce that Apache Spark in BigQuery is now officially available in Private Preview. This feature expands BigQuery’s processing capabilities by offering Apache Spark as an OSS processing engine under a unified control plane (admin, monitoring, billing, security, governance). It is the first step in delivering on the Unified Data Platform vision and was a key announcement at Next ‘24 #googlecloud #bigquery #spark

38

2 Comments

After speaking to 100+ data teams recently a clear pattern has emerged in the modern data stack:   Databricks is leading with its optimised Apache Spark runtime, accelerating data processing by up to 50%. Its collaborative workspace is becoming the go-to for data engineering and ML tasks. Snowflake's decoupled compute and storage architecture is reducing warehouse costs by 80%, making data migrations a 2 day job rather than a month-long ordeal.    Amazon Glue is filling a critical gap with its serverless ETL service. Its visual interface is democratising data preparation, though some find it limiting for complex, custom workflows. Microsoft's Azure Synapse is tying it all together, offering a unified platform for big data and warehousing with on-demand SQL pools.    The key insight? Each platform has its strengths and limitations, necessitating a combination of different technologies for optimal results.   This multi-platform approach may be transient – the real challenge lies in simplifying this complexity for the next wave of data innovation.   What's your experience with these platforms? Are you seeing similar trends in your data architecture? Let's talk. :)   #datalake #datawarehouse #databricks #amazon #microsoft #snowflake #etl #dataingestion #snowflake

21

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

4

How to fine-tune #LLMs on custom #datasets at Scale using Qwak and #CometML Alexandru Razvant https://lnkd.in/gP4aWM69

3

✨ If you have Databricks, do you need Snorkel AI? If you plan to customize #LLMs with your own enterprise data the answer is an easy yes. You need high quality data to build production-quality AI, but most unstructured enterprise data is inaccurate, incomplete, and inconsistent. Often you have a lot of data for common scenarios but very little for high-value edge cases (obviously!). Good news: Snorkel Flow integrates seamlessly with Databricks at every stage of the AI development workflow, from accessing data & models to deploying production-quality applications via Unity Catalog. For a guided walk-through check out this 5min demo and learn more at https://lnkd.in/gTiQ4wD8 #AIInnovation #ArtificialIntelligence #dataintelligence

31

7 Comments

A great read in iTnews today highlighting how Seven West Media leveraged Databricks to forecast audience view patterns on both a seven-day and 28-day horizon. The company is now scaling its use of Databricks to support additional analytics use cases. Andrew Brain, Audience Intelligence and Growth Director, emphasized the significance of this advancement, stating, "We can now understand what our audience is going to be from a value perspective on volume and also what they’re going to be watching." This insight is pivotal for their business. #databricks #ai Read more about Seven West Media's predictive analytics success here: https://lnkd.in/e4rb3_mp

174

4 Comments

Good article on practical tips to improve the #RAG pipeline - across indexing, retrieval and generation. • Indexing - level up with simple, semantic or #LLM based chunking • Retrieval - implement hybrid search, combining vector and keyword searches • Query Rewriting - optimize queries for better retrieval • Metadata - boost retrieval accuracy with rich contextual data • Autocut - filter out irrelevant information from retrieved content • Reranking - prioritize the most relevant content, enhancing retrieval precision https://lnkd.in/eJMvu3XJ

7

Great new article by Li (Luke) Yu on how to use #ray to perform feature extraction for #llms on #databricks: https://lnkd.in/d5qv3Xmk Key insights include: 🔧 How to setup and configure a Ray cluster on Databricks with #GPUs 📘 How to overcome tight context windows with various summarization approaches

50

4 Comments

This is a BIG DEAL: open variant data type. Makes JSON and semi-structured data processing 10x faster without losing the flexibility. We have heard from many customers that they really want this data type that’s available in some proprietary data warehouses, but don’t like the lock-in. So here you go. An open Variant type in open source Apache Spark and Delta Lake. https://lnkd.in/gJwD6mgh

398

10 Comments

With Genie, users can self-serve and obtain answers to questions not addressed in their Dashboards without having to learn complicated tools or rely on expert practitioners. No more needing to ask the data experts to construct dashboards and queries.

2

Customers are using the Apache Iceberg format because they can read and write data concurrently using different frameworks. Iceberg format also allows for time travel and rollback so you can run SQL queries on a point-in-time snapshot of your data, or rollback data to a previously known good version. Now, you can stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose. https://lnkd.in/g8g3FUah #Data #StreamingData #DataStreaming #Firehose #S3 #AWS #ApacheIceberg #Iceberg

50