Nicolas Iderhoff

What Karl says. There's going to be two amazing talks about Hopsworks based #ai and #ml systems: * Build a personalized Bitcoin (BTC) virtual assistant in Python with Hopsworks and LLM function calling by Javier de la Rúa Martínez (https://lnkd.in/eQfWpFRU ) * Build TikTok's Personalized Real-Time Recommendation System in Python with Hopsworks by Jim Dowling (https://lnkd.in/eXEHePjG) If you are in the #berlin area next week - go and take a look!

7

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Kommentare

Tomorrow is the day, the Databricks Roundtable is taking place in Cologne.🚀 Join us for an exclusive hands-on session where Philipp Schwartenbeck will present a multi-agent system and Felix Xaver Krisch will demonstrate the complete lifecycle of a data product directly in Databricks! In the live demo, we will take you through the entire journey of a data project - starting with data exploration, followed by incremental data processing, workflow scheduling and scaling. We'll also show you how to implement CI/CD pipelines and monitoring tools to ensure seamless project delivery in Databricks. In the second live demo, we'll dive into the world of LLM-based multi-agent systems. See how autonomous collaboration between agents can solve complex tasks in real time, helping organisations improve the efficiency of their data processing and decision making! Find all the event details and register now... there are only 3 places left❗ 🔗https://lnkd.in/g_EyDuHp Looking forward to seeing you tomorrow! #databricks #dataengineering #AI #multiagent #LLM #cloudmigration #datagovernance #realtimedata #dataplatform #changemanagement

6

❄️ 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐒𝐧𝐨𝐰𝐩𝐚𝐫𝐤'𝐬 𝐂𝐚𝐩𝐚𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬 𝐟𝐨𝐫 𝐏𝐲𝐭𝐡𝐨𝐧 𝐃𝐞𝐯𝐬 𝐨𝐧 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 🐍 Ever wondered how to leverage the scalable compute power of Snowflake’s virtual warehouses to empower your data applications using your favourite programming language? In this blog post, we will take a deeper look into the Snowpark API and explore how you can efficiently build scalable #Python-based data applications including third-party and custom dependencies on the Snowflake Data Cloud. 🔍 𝐖𝐡𝐚𝐭 𝐘𝐨𝐮'𝐥𝐥 𝐋𝐞𝐚𝐫𝐧 · Introduction to Snowpark: Discover how Snowpark simplifies querying and processing large datasets in your favourite programming languages like Python. · Setting Up: Step-by-step guide on how to create a Snowpark session and connect to Snowflake. · Advanced Features: Dive into User-Defined Functions, utilizing third-party packages, and integrating custom libraries. · Practical Examples: Real code snippets on setting up your environment, creating and using UDFs, and handling dependencies. 💡 Explore how to enhance your data pipelines and make the most out of Snowflake's computing power using Python: https://lnkd.in/e7_isjD3 📚 This is blog post #8 in our inovex GmbH Snowflake series. Check out our previous articles here: https://lnkd.in/ew9pPBDk #DataEngineering #Snowflake #Snowpark #Python #BigData #CloudComputing

21

📷 New on YouTube: In his 2023 #SAGconf session, Matthias Niehoff delves into the latest developments in modern data architectures, including Data Mesh, Data Lakehouse, and the integration of analytics with operational systems, offering insights into their practical implementation. 👀 Watch now: https://lnkd.in/eHs3EkZa #SoftwareArchitecture #SoftwareArchitectureGathering #SoftwareArchitectureConference #DataArchitecture #DataMesh #DataLakehouse #ELTPattern #CloudDataWarehouse #ModernDataStack #DataStrategy #SoftwareEngineering

8

Dask DataFrame is fast now - Florian Jetter (Coiled) @ PyData Südwest Live stream of the PyData Südwest meetup https://lnkd.in/eU9G8rvu 20:15 Talk / Q&A 20:45 Talk Lightning Talks Ask questions via Slido: https://bit.ly/SW2405QA 📺 Dask DataFrame is fast now - Florian Jetter (Coiled) Dask is a library for distributed computing with Python that integrates tightly with pandas. Historically, Dask was the easiest choice to use (it's just pandas) but struggled to achieve robust performance (there were many ways to accidentally perform poorly). The re-implementation of the DataFrame API addresses all of the pain points that users ran into. We will look into how Dask is a lot faster now, how it performs on benchmarks that is struggled with in the past and how it compares to other tools like Spark, DuckDB and Polars. ⚡️ Lightning Talks

28

2 Kommentare

“After one year of reading and working daily with LLMs, here are the 𝘁𝗼𝗽 𝟭𝟮 𝗯𝗹𝗼𝗴𝘀 on 𝗟𝗟𝗠𝘀 & 𝗥𝗔𝗚 I 𝗿𝗲𝗮𝗱 𝘄𝗲𝗲𝗸𝗹𝘆 and 𝗿𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱 ↓”

8

1 Kommentar

Want to get started with AI vectors in the 23ai Oracle database ? Have a look at the post we wrote, my colleague Ulrike Schwinn and I ! In this blog post we cover a few key tasks using SQL, PL/SQL, and ORDS: - Loading an ONNX embedding model into the database - Generating vector embeddings using the loaded vector embedding model - Storing vector embeddings in a table with a data type VECTOR column - Using similarity search in SQL and ORDS Feel free to give it a try with the 23ai Free version and have fun! #oracledatabase #23ai #23aifree #oraclecloudinfrastracture #vector #similaritysearch #AI #ORDS #SQL #tutorial #embeddings Blog post: https://lnkd.in/eZBDRcfu

27

👇🏻 Check out how Erste Group is leveraging synthetic data to democratize #Data & #AI for everyone using MOSTLY AI integrated with the Databricks #DataIntelligence Platform 👇🏻

11

❄️ 𝐇𝐚𝐫𝐧𝐞𝐬𝐬𝐢𝐧𝐠 𝐭𝐡𝐞 𝐏𝐨𝐰𝐞𝐫 𝐨𝐟 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐍𝐚𝐭𝐢𝐯𝐞 𝐀𝐩𝐩𝐬: 𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚 𝐋𝐮𝐧𝐜𝐡 𝐏𝐥𝐚𝐧𝐧𝐞𝐫 𝐀𝐩𝐩 The Snowflake Native App Framework is revolutionizing the way developers package data and logic into shareable applications. In our latest blog post by Hiroshi Hamano, Jannik Bach, we dive deep into a practical example: a lunch planner app. 🍽️ 🔍 What we'll cover: · 𝐀𝐩𝐩 𝐒𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 & 𝐓𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥𝐬: How the app utilizes Snowflake's external functions to offer a seamless planning experience. · 𝐏𝐮𝐛𝐥𝐢𝐬𝐡𝐢𝐧𝐠 & 𝐒𝐡𝐚𝐫𝐢𝐧𝐠: Strategies for deploying through the Snowflake Marketplace. · 𝐀𝐜𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐃𝐚𝐭𝐚: How data flows from provider to consumer seamlessly within Snowflake. 💡 Learn how these features can streamline operations and offer a robust solution for your data needs, giving practical insights into using the framework effectively. Check out our blog post and start enhancing your development strategies with Snowflake: https://lnkd.in/eByY9jdk This is the latest insight in our inovex GmbH blog series on the #Snowflake Data Cloud. Explore previous posts here: https://lnkd.in/ew9pPBDk

23

Retrieval Augmented Generation (RAG) is a straightforward concept that any company can implement with little effort. You can think of RAG as an NN-based document summarization, like merging the results of a simple elastic search/apache solr query with a text summary. The original concept involved retraining the Llm to have a similar latent space, however in applied concepts the retraining has been dropped and replaced with a vector database and prompt engineering that can be paired with any Llm that exists. Based on Meta's LLAMA3, LlamaIndex, Qdrant, Ollama, Angular, and FastAPI, I built the RAG demo about funding lines for universities of applied sciences in hesse. The demo is available at (not well optimized for mobile use): https://lnkd.in/eBYQwKh4 This demo is a weekend project when the little ones were asleep, so it took me less than 3 days. Imagine your company creating a bot like this with more manpower and effort. Best of all, it's completely free and costs you nothing. Your data stays in your company without any security or GDPR violations. P.S.: On my own behalf: I also work as a speaker and consultant on AI topics. Through my research in context-aware fault diagnosis in industrial production lines, I know the state of the art in AI and have implemented various AI use cases myself using different neural networks under industry constraints. If you need some advice, just get in touch with me. I'm looking forward to talking with you about self-built solutions without buying expensive cloud services.

11

2 Kommentare

In the #StateofDataAI, Databricks analyzed the usage of Meta Llama and Mistral, the two biggest players of open source LLMs. Key findings:  - 77% choose models with 13B parameters or fewer - 76% of companies using LLMs choose open source, often alongside proprietary models. Find out how this translates to company priorities in this new report.

4

Join Databricks on November 7th for Data + AI World Tour Munich! Discover how leading companies like Fraport AG, Deichmann SE, Bayer AG, and CARIAD SE (A Volkswagen Group Company) are taking control of their data and building custom AI on the Databricks Data Intelligence Platform. Register now!

5

In the #StateofDataAI, Databricks analyzed the usage of Meta Llama and Mistral, the two biggest players of open source LLMs. Key findings:  - 77% choose models with 13B parameters or fewer - 76% of companies using LLMs choose open source, often alongside proprietary models. Find out how this translates to company priorities in this new report.

3

𝗖𝗵𝗮𝘁𝗚𝗣𝗧 𝗦𝗲𝗮𝗿𝗰𝗵: 𝗔𝗿𝗲 𝘆𝗼𝘂 𝘀𝘁𝗶𝗹𝗹 𝘀𝗲𝗮𝗿𝗰𝗵𝗶𝗻𝗴 𝘁𝗵𝗲 𝘄𝗲𝗯 𝗹𝗶𝗸𝗲 𝗶𝘁’𝘀 1999? ChatGPT Search from OpenAI has officially launched. What is ChatGPT Search? ChatGPT Search is your 𝗻𝗲𝘄 𝗴𝗼-𝘁𝗼 𝘁𝗼𝗼𝗹 𝗳𝗼𝗿 𝗴𝗲𝘁𝘁𝗶𝗻𝗴 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 by simply asking questions in everyday language. Say goodbye to the old ways of searching! 𝗞𝗲𝘆 𝗦𝘁𝗿𝗲𝗻𝗴𝘁𝗵𝘀 𝗼𝗳 𝗖𝗵𝗮𝘁𝗚𝗣𝗧 𝗦𝗲𝗮𝗿𝗰𝗵: 𝗨𝘀𝗲𝗿-𝗙𝗿𝗶𝗲𝗻𝗱𝗹𝘆 Ask questions like you would to a friend. 𝗙𝗮𝘀𝘁 Get reliable information quickly. 𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 Find what you need right when you need it. 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝗳𝘂𝗹 Gain deeper understanding through conversation. While ChatGPT Search offers great potential, it would benefit from an improved user interface and additional features compared to alternatives like Perplexity. So, which do you think is better: 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆 𝗼𝗿 𝗖𝗵𝗮𝘁𝗚𝗣𝗧 𝗦𝗲𝗮𝗿𝗰𝗵? Share your thoughts in the comments! 💬👇 ------- I'm Kamin C. 👋 Click my name + follow + 🔔 Like it? Hit it on. ♻️ Have any favorite AI resources? Drop a comment. 💬👇 Video credit: OpenAI Background footage: Kamin Chung

20

25 Kommentare

Join Databricks on November 7th for Data + AI World Tour Munich! Discover how leading companies like Fraport AG, Deichmann SE, Bayer AG, and CARIAD SE (A Volkswagen Group Company) are taking control of their data and building custom AI on the Databricks Data Intelligence Platform. Register now!

17

Participating in "The future of AI is open" Devpost hackathon was a lot of fun for the inovex GmbH team. 💫 For this hackathon, we asked ourselves what AI is good for if not to help humans learn new cool things? For instance about Snowflake! We all know that the Snowflake data cloud ecosystem is fascinating but also huge, it provides a lot of functionality. So wouldn't it be cool if we had an AI instructor to teach us about Snowflake, in short, a Snow Instructor? This Snow instructor could read all the documents from the official Snowflake documentation and generate quiz questions so that we can test our knowledge, just like in the show ”Who wants to be a millionaire?”. And that is exactly what we built! Check out our Snow Instructor ⛷️ under https://lnkd.in/eXGs2s4P and let us know what you think! Developing and deploying a full-fledged AI-based application with Streamlit and Snowflake really is a bliss and the new Snowflake Arctic LLM as well as the Cortex API in general, are easy to use and fun. We can't wait for the next AI on Snowflake project to start ❄️🏎 #snowflake #hackathon #python #snowpark

30

1 Kommentar

🚀 Last week, our team at msg attended the Databricks Data + AI World Tour in Munich. The event was a fantastic opportunity to dive into Databricks latest innovations and to strengthen our collaborations. We were particularly impressed by three recent features: 🔹 Publish to Microsoft Power BI Service: This feature enables users to publish datasets directly from Databricks to Power BI, streamlining the process of creating interactive reports and dashboards. 🔹 Lakehouse Federation: Lakehouse Federation allows querying across multiple data sources without data movement. By leveraging Unity Catalog, it provides a unified metadata layer, simplifying data access and governance across diverse systems. 🔹 Lakeflow Connect: LakeFlow Connect offers built-in connectors for ingesting data from enterprise applications and databases into Databricks. It also added support for SQL Server, Workday and Salesforce (public preview). Thank you to our partner Databricks and all the other amazing partners we connected with during this event! Special thanks to Heiko Thiele and Sebastian Grünwald for the great discussions and insights. #DatabricksWorldTour #DataInnovation #PowerBI #LakehouseFederation #LakeFlowConnect #msg #Databricks #DataAnalytics

33

If you want to learn about building agentic RAG systems this weekend, then we recommend looking into this comprehensive blog/tutorial series by Prince Krampah - starting with basic routing and function calling to multi-step reasoning over complex documents 🧠📑

1