Chelsea Ip

Follow along John Chun Hong Leung's new tutorial to learn how you can ingest, extract, and visualize time-series data by using PySpark on Databricks (implementation code included).

24

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 Comments

Predictive AI workflow with Kumo + Databricks Kumo operates directly on raw Databricks tables, utilizing Unity Catalog for data governance and security, generates predictions at scale, and writes them back to your #Databricks warehouse. The data warehouse native deployment for Databricks (in private preview) ensures that no data is stored on disk in a Kumo owned environment and any data leaving Databricks is transformed, encoded, and deleted after use. Reliably and securely create predictive AI to make predictions without data ever being stored outside your Databricks warehouse. Learn more and request a demo here: 🔗 https://lnkd.in/dYREGfvE. #PredictiveAI #MachineLearning #DataScience #GNN #LLM #Databricks #AIInnovation #BusinessIntelligence

13

Amazon continues to push the boundaries of large language models and conversational AI. At this year's ACL conference, Amazon researchers presented over 30 papers exploring innovative techniques to enhance language models - from improving model evaluation and robustness, to leveraging knowledge graphs for better reasoning, to advancing applications like code synthesis and multimodal understanding. This breadth of cutting-edge work showcases Amazon's deep commitment to advancing the state-of-the-art in natural language processing and building the conversational AI systems of the future. The company's world-class AI research continues to have a major impact on both the academic and industry landscapes. Looking to join a team at the forefront of the conversational AI revolution? Check out the latest openings on the Amazon Science careers page. #AmazonScience #ConversationalAI #NaturalLanguageProcessing #MachineLearning #ACL2024

3

🚀 Calling all data engineers! If you’re looking to get the most out of your Databricks Jobs clusters, check out these data-driven tips for optimizing compute clusters at scale. 💡Understand the tradeoffs of Photon and autoscaling 💸See how serverless compute compares to classic clusters 💪Learn how to save countless hours optimizing clusters More in this post: https://hubs.ly/Q02KQ1t50

8

We in the Silicon Valley have taken a position that the only thing we should be doing is injecting AI in to everything we do. And the big 5 Indian outsourcing companies are also drilling the same in to the brains of everyone in the organization. AI or DIE Greg Head does his homework and brings us to the reality. We have seen this in the past. Rewriting everything in Java or committing to J2EE architecture will not revolutionize business. Moving all data to Hadoop won't guarantee better business intelligence. Moving everything to Cloud will not cut IT costs by 95%. Moving data warehouse to AWS RedShift is not a sliced bread kind of invention. What are the questions people are asking in the Silicon Valley? Can we use LLM to retire our complex ERP systems? (Oracle, SAP, MFG-PRO, Infor, Sage etc) Can we use LLM to retire our expensive and clunky HRMS ?(Oracle, Workday, SAP, ADP etc) Can we use LLM to retire CRM ?(SFDC, Oracle, SAP etc). Can we use LLM to rewrite our homegrown applications for high throughput? Why not? LLM advocates say if you give your requirements to (ERP, HRMS, CRM, Custom applications) LLM, you can have your own software that will work for your requirements and it will cost nothing. And no expensive SaaS fees, No multi-million upgrades and no money for expensive consultants. If this becomes true, it will be the end of multi trillion $ software industries. Dream on LLM fans. We are not there at. Similar promises were made by Indian offshore outsourcing companies. That they will take over the IT and run it from India for 1/5th the cost.

26

6 Comments

If you're in the Seattle area don't miss this workshop tonight(6/12) with Niels Bantilan! Build a Custom Chatbot with LLMs + RAG. Most modern chatbots today use Large Language Models(LLMs), Retrieval Augmented Generation(RAG), and scalable workflows to create a unique conversational experience! This workshop will equip you with the skills to effectively build your own chatbot and scalable AI workflows for LLMs using and union.ai The modern MLOps tooling will also provide a reliable framework for your machine learning operations by streamlining processes, increasing efficiency, and adding reproducibility to your AI applications. https://lnkd.in/g3WeQjZt #workshop #llms #rag

3

Learn about DSPy, a recent innovation in #LLMops led by researchers from Databricks co-founder Matei Zaharia's research lab, in this blog post co-authored by JetBlue AI/ML engineer Dan Pechi.

64

2 Comments

📈 Retrieval Augmented Generation (RAG) is increasingly becoming recognized as a critical technology for enterprises exploring advanced Generative AI solutions. Madison Lee and I have written an article that provides an overview of various RAG implementation strategies. Whether your organization is seeking - - No-code solutions for streamlined and accessible implementation, - Low-code options offering a blend of flexibility and ease of use, or - Full-code approaches for ultimate control and customization, Our guide demonstrates how IBM watsonx.ai can support all of these methods. Check out our article to discover the most suitable implementation method for your enterprise. Link: https://lnkd.in/g4Af3JEQ

22

1 Comment

"The new Columnar capabilities enable rapid real-time intelligence for data analysis and quick insights, along with further opportunities for traditional ML and AI" ⚡ Check out this write up on Capella columnar from private preview user Will Thayer, MBA Technologist 📝 https://bit.ly/3JGo8UR

24

1 Comment

Llama3 on Databricks. Databricks is changing the game with making open models easily usable.

1

🚀 Automate Your CI/CD Pipeline with Snowflake and GitHub Actions 🚀 Our Senior Data Engineer, Jaime Salas, shares an insightful guide on integrating GitHub Actions with Snowflake to streamline your CI/CD pipeline. Learn how to: 🔹 Set up key pair authentication 🔹 Create and manage Snowflake users and schemas 🔹 Automate tests and merges using GitHub Actions Enhance your development workflow with these powerful automation techniques! https://hubs.ly/Q02BWXC10 👉 Read the full article #DataEngineering #Snowflake #GitHubActions #CICD #Automation

15

How did Dremio take on self-serve analytics with a "lakehouse" 🤔 Hear from Dremio Founder Tomer Shiran on how they made data more accessible through an open source data lakehouse approach using Apache Arrow and Apache Iceberg 💪 And built a $2B company while doing it 😲 Full episode 👉 https://lnkd.in/geQrmn2R

6

Dremio founder, Tomer Shiran, sat down with the Open Source Startup Podcast to talk about how we're making data more accessible through an open source data lakehouse approach using Apache Iceberg. Listen to it here!

7

📊 The Power of High-Quality, Diverse, Real-World Data Fine-Tuning LLMs 🌍 When it comes to training Large Language Models (LLMs), the mantra isn’t just “more data”—it’s better data. While having more data can be beneficial, it’s the quality, diversity, and real-world relevance of the data that truly drive impactful results. Here’s why: 1️⃣ Higher Quality: clean, well-curated data improves model accuracy and reduces noise, making the model more reliable and stable. Poor-quality data often leads to inconsistent responses and increased hallucinations. Unbiased labeling consistency is key. 2️⃣ Diversity: it ensures the model can generalize across different contexts and user scenarios. This makes the model more robust, safe and inclusive, which is especially important in NLP for avoiding bias and ensuring representation. 3️⃣ Real-World Datasets: training on real datasets helps models perform better in real-world applications, as it learns from actual patterns and usage scenarios, making the model more practical and effective. For LLMs, there’s a general playbook to prepare data effectively: (1) Collect Instruction-Response Pairs: gather examples of how tasks should be answered, covering various domains and complexity levels. (2) Concatenate Pairs: organize the data into structured sequences, allowing the model to learn from end-to-end interactions. (3) Tokenize: process the data into tokens, using padding and/ or truncation as needed to maintain uniform length. I particularly love 🤗 transformers library's AutoTokenizer to identify which tokenizer to use on every model. (4) Split into Train/Dev/Test Sets: partition your data into training, validation, and testing sets to ensure proper model evaluation and tuning. Ultimately, data quality prevails over quantity, specially as we live in the LLMs era when models have been pre-trained with almost all the public data available in the 🌎 . A well-prepared dataset tailored to your application will make all the difference in the performance and reliability of your model. #AI #MachineLearning #DataQuality #LLM #NLP #ModelTraining #DataPreparation

5

Amazon continues to push the boundaries of large language models and conversational AI. At this year's ACL conference, Amazon researchers presented over 30 papers exploring innovative techniques to enhance language models - from improving model evaluation and robustness, to leveraging knowledge graphs for better reasoning, to advancing applications like code synthesis and multimodal understanding. This breadth of cutting-edge work showcases Amazon's deep commitment to advancing the state-of-the-art in natural language processing and building the conversational AI systems of the future. The company's world-class AI research continues to have a major impact on both the academic and industry landscapes. Looking to join a team at the forefront of the conversational AI revolution? Check out the latest openings on the Amazon Science careers page. #AmazonScience #ConversationalAI #NaturalLanguageProcessing #MachineLearning #ACL2024

1

Recommender systems have become an integral part of modern digital experiences for personalized content. But many companies don’t have the resources to build large-scale systems. Learn how to train and deploy recommendation models at scale on Databricks in this new series. Part 1 focuses on distributed data loading and training. Read here: https://dbricks.co/3XsZaOX Asfandyar Qureshi Karan Jariwala Lu Wang Saaketh Narayan Ning Wang

274

5 Comments

Crafting effective prompts is an art itself, but getting it right can mean the difference between a great output and total garbage! Today, we’ll explore the 5 prompt engineering techniques that every AI engineer should know to get the most out of LLMs. #llms #llmops #chatgpt4 #rag #gemini #ai https://lnkd.in/d3an85KE

3

🌟 Join Me at the Databricks + AI Summit! 🌟 I'm thrilled to host a community talk at the Databricks + AI Summit in SF, June 10-13, 2024! 🚀 𝐃𝐨𝐧'𝐭 𝐦𝐢𝐬𝐬 𝐦𝐲 𝐬𝐞𝐬𝐬𝐢𝐨𝐧 𝐨𝐧 "𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐁𝐞𝐬𝐭 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬: 𝐋𝐞𝐭'𝐬 𝐄𝐱𝐩𝐥𝐨𝐫𝐞 𝐄𝐯𝐨𝐥𝐯𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬 & 𝐄𝐓𝐋 𝐃𝐞𝐬𝐢𝐠𝐧, at the Braindate lounge. This is a fantastic opportunity to dive deep into the latest trends and strategies in data engineering - from how to build high performing teams, to how to reduce technical debt with modern design trends.   📲 DM me or comment below for more info or to book a meeting. Looking forward to seeing you there!   #DatabricksSummit #Smoothstack #AI #DataEngineering #Braindate #ETLDesign

27