A comprehensive guide on constructing LLMs using Databricks. Guide showcases the incorporation of LLM-specific tools and resources within the Databricks framework. It covers aspects such as Vector Indexes, open-source LLM models, and hosted endpoints. Find the guide here: https://lnkd.in/ehtTQhhc Seattle Data Guy
Malay B.’s Post
More Relevant Posts
-
🔥 Databricks Feature Serving is now GA! 🔥 Databricks Feature Serving helps ML developers make pre-computed features that are stored in Delta Tables, accessible with milliseconds of latency using Databricks Online Table (currently in Public Preview) 📡 to build real-time Al applications. 🏎️💨 All the deets on the link below ! 👇 https://lnkd.in/erRuZgXG
Announcing the General Availability of Databricks Feature Serving
databricks.com
-
Well, this is interesting - in the bucket of “didn’t expect to see that!” Regardless, now let’s just hope that Databricks doesn’t decide to kill off Iceberg. The open table formats, along with the advancements in analytics engines (e.g., DuckDB, StarRocks) that work on them, have been invigorating in bringing greater power, flexibility, and capability in the datalake/lakehouse engineering space. #iceberg #databricks #deltalake Link: https://lnkd.in/g48r2gdp
Databricks buys Tabular to win the Iceberg war – Blocks and Files
https://blocksandfiles.com
-
At Snowflake, we’ve been long-time advocates and supporters of Apache Iceberg, believing that customers should have choice and flexibility to use the same, single copy of data across different engines. As organizations try to bridge data systems and make the most of the data tools available to them, interoperability is more than a convenience - it’s a P0 requirement. That’s why it’s no surprise that the Ninth Annual BigDATAwire Readers’ and Editors’ Choice Awards recognized Apache Iceberg as the obvious winner in not one, but two categories: ‘Readers’ Choice: Top 3 Big Data and the AI Open Source Projects to Watch’ and ‘‘Readers’ Choice: Top Data and AI Achievement.’ In addition, Apache Polaris (Incubating) was also listed as a top open source project to watch. Iceberg’s model of transparent management and diversity of contributors helps customers regain control over their data assets. It’s an approach the industry should follow throughout the data stack, ensuring a more open, interoperable, and customer-centric data ecosystem. Snowflake increasingly advocates (and IMO leads) in this space, and it’s been exciting to see other vendors across the landscape start to do the same. #BigDATAwireRCA24 https://lnkd.in/gSmk7xdn
2024 Readers' & Editors' Choice Awards - BigDATAwire
bigdatawire.com
-
Wow, I wish we had DataBricks when we were building WTW's portfolio risk models! Actually, founded in 2013 the platform did not exist at the time and we had to do with the tools available to us back then. Having said that, the broad steps involved in the development and deployment of Machine Learning have not changed all that much. Just the computing power and tools now available are a massive improvement, and of course the terminology used to describe the process has changed.
Peter Keutgens' Statement of Accomplishment | DataCamp
datacamp.com
-
Sometimes in #DataEngineering it feels like there are as many tools and technologies as there is #Data in the world. This can really give you a headache, especually when you're new to the field. Thanks to Seattle Data Guy for giving a good at start point to navigate the tool jungle. I personally like to put concepts more in the foreground than actual technologies. That's why I recommend the technology-agnostic "Fundamentals of Data Engineering: Plan and Build Robust Data Systems" https://amzn.eu/d/7DGH3mP by Joe Reis 🤓 and Matthew Housley
It feels like the list of data technologies that data engineers need to know continues to expand...not contract. There are way too many tools and solutions you can pick from. But really, many of them aren't that different from each other in terms of the problems they are trying to solve. Here are 7 great videos and articles you can go through to get up to speed quickly on what is going on in the data tech world. 1. Snowflake vs Databricks - And the Battle For Iceberg https://lnkd.in/gq-KCFsK 2. Common Pitfalls in Deploying Airflow for Data Teams https://lnkd.in/gGepn-K5 3. Memory Efficient Data Streaming To Parquet Files https://lnkd.in/gpVwKhEK 4. Why is Polars All The Rage? by Daniel Beach https://lnkd.in/gUVATCj5 5. Apache Iceberg - What Is It by Julien Hurault https://lnkd.in/gfntMYZX 6. The Data Engineer’s Guide to CDC for Analytics, Ops, and AI Pipelines https://lnkd.in/esbcDi_q 7. Data Modeling Where Theory Meets Reality https://lnkd.in/g_8FCpuA What articles will you be reading?
-
🚀 Solving complex data engineering challenges! In my latest blog, I share how I optimized parallel processing for complex data workloads using Microsoft Fabric, achieving scalability and efficiency. Check it out for insights and strategies #MicrosoftFabric #DataEngineering #ParallelProcessing #Spark #FabricCapacity #Devoteam
Optimising Parallel Data Processing in Microsoft Fabric: A Case Study
https://www.devoteam.com
-
I've been deeply involved with Databricks and Azure services the past couple years, and would like to share that AI-driven tools are transforming data analysis in litigation and regulatory action. Using Databricks on Azure, AlixPartners Data Analytics team transformed a days-long process into an overnight task, dramatically improving efficiency and decision-making speed. Read the full article to learn about how our team leverages these cutting-edge tools to provide a competitive edge in handling complex datasets: https://lnkd.in/deCjWkDx
A real-world game changer: Getting to meaningful results overnight
alixpartners.com
-
Tech companies can't exfiltrate their data fast enough to services like Snowflake and Databricks to get all those good ML and AI driven BI insights. And then the hacks happen (AT&T) Time to start using services that work with your data on-premises. The DBSnapper architecture is designed for exactly this purpose. #hack #dataprivacy #platformengineering #database
-
-
Databricks Delta Lake platform continues to lead with game changing innovation. Liquid Clustering is a game changer for Data Professionals looking to improve write and read performance while maximizing cost savings. #databricks #deltalake #liquidclustering
Liquid Clustering in Delta Lake is one of the fastest adopted features at Databricks, thousands of customers have adopted it and it was just announced GA today! No more low level tuning of Partitions and ZOrdering. This improves writes by around 7x and reads up to 12x! Check it out: https://lnkd.in/gaTdrtiY
Announcing General Availability of Liquid Clustering
databricks.com
-
Reposting this game changer feature: Liquid Clustering in Delta Lake is now GA. Massive Improvements 7x for Writes and 12x for Reads.
Liquid Clustering in Delta Lake is one of the fastest adopted features at Databricks, thousands of customers have adopted it and it was just announced GA today! No more low level tuning of Partitions and ZOrdering. This improves writes by around 7x and reads up to 12x! Check it out: https://lnkd.in/gaTdrtiY
Announcing General Availability of Liquid Clustering
databricks.com