Axel Schwanke

𝗗𝗮𝘁𝗮 + 𝗔𝗜 𝗪𝗼𝗿𝗹𝗱 𝗧𝗼𝘂𝗿 𝟮𝟬𝟮𝟰 𝗠𝘂𝗻𝗶𝗰𝗵 - 𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀, 𝗮 𝗳𝗮𝗻𝘁𝗮𝘀𝘁𝗶𝗰 𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻, 𝗶𝗻𝘀𝗽𝗶𝗿𝗶𝗻𝗴 𝗽𝗲𝗼𝗽𝗹𝗲, 𝗮𝗻𝗱 𝗮𝗻 𝗶𝗻𝗳𝗲𝗰𝘁𝗶𝗼𝘂𝘀 𝘀𝗽𝗶𝗿𝗶𝘁 … This is how our colleagues Frank Liebrand and Sebastian Moritz summed up yesterday’s event by Databricks. The message was clear: Databricks Data Intelligence Platform is the backbone of innovative AI use cases across various industries, enabling companies to seamlessly integrate and process even petabytes of data. It was inspiring to hear firsthand success stories from leading companies like 𝗖𝗮𝗿𝗶𝗮𝗱, 𝗙𝗿𝗮𝗽𝗼𝗿𝘁, 𝗔𝗹𝗹𝗶𝗮𝗻𝘇 𝗮𝗻𝗱 𝗕𝗮𝘆𝗲𝗿 𝗮𝗻𝗱 𝗺𝘂𝗰𝗵 𝗺𝗼𝗿𝗲. Each showcased their unique journey in harnessing Databricks to drive their data and AI initiatives. AI/BI was the big topic of the day and Databricks itself presented their journey on how they use AI/BI to gain faster and deeper insights from their own data. Beyond the informative presentations, the event provided a valuable opportunity to exchange experiences with different companies and discuss current challenges. As a partner of Databricks, we are inspired by the platform's versatility and its impact on empowering a wide range of customers in their data transformation journey. A big thank you to the Databricks team especially Felix Völker & Inga Roos for hosting such an engaging and insightful event. We’re already looking forward to next year!   𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗹𝗲𝘃𝗲𝗿𝗮𝗴𝗶𝗻𝗴 𝗮 𝗰𝗹𝗼𝘂𝗱 𝗱𝗮𝘁𝗮 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗮𝗻𝗱 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗻𝗴 𝗶𝘁 𝗶𝗻𝘁𝗼 𝘆𝗼𝘂𝗿 𝗲𝘅𝗶𝘀𝘁𝗶𝗻𝗴 𝗦𝗔𝗣 𝘀𝘁𝗮𝗰𝗸, 𝗳𝗲𝗲𝗹 𝗳𝗿𝗲𝗲 𝘁𝗼 𝗿𝗲𝗮𝗰𝗵 𝗼𝘂𝘁 𝘁𝗼 𝘂𝘀!   #dataplatform #AI #databricks #bixConsulting

40

1 Kommentar

Top 6️⃣ Database Models: Which One Have You Used? 🤔 Explore the unique features of these database models and discover which one suits your needs best! ⤵️ 🔹 Flat Model A simple, spreadsheet-like structure with data in a single table. Easy to implement but struggles with complex relationships. 🔹 Hierarchical Model Organises data in a tree-like structure with parent-child relationships. Great for clear hierarchical data, but not ideal for many-to-many relationships. 🔹 Relational Model The go-to for traditional databases, using tables with rows and columns. Known for its flexibility, SQL support, and efficient handling of complex queries and many-to-many relationships. 🔹 Star Schema Perfect for data warehousing and OLAP applications. Central fact table with surrounding dimension tables, optimised for fast, simple query performance. 🔹 Snowflake Model A more normalized version of the star schema. Reduces redundancy and improves data integrity, but can lead to more complex queries. 🔹 Network Model Uses a graph structure to represent complex relationships with multiple parents and children. Efficiently handles many-to-many relationships. Source: ByteByteGo 💬 Over to you: Which database model have you used or found most effective? Share your experience below! ⤵️ #DatabaseModels #DataManagement #TechTalk #DataScience #DatabaseDesign #SQL

16

1 Kommentar

What do #digitaltwins in data ecosystems have in common with landline telephones? ☎️ Our latest blog post https://lmy.de/iQtQy examines this seemingly random connection – and reveals 3️⃣ prerequisites that are the basis for #datadriven success. Want to call someone you don’t know in real life or rather unearth the treasures of your data? #wearecatenax #digitaltwin #cofinityx #boschsemanticstack

32

1 Kommentar

Andreas Scherbaum: PostgreSQL Development Conference 2024 - Review 🦸‍♂️[PostgreSQL Geeks Unite]🦸‍♀️ 🚀 Press pause on your index optimization and gather 'round! Breaking news from the PostgreSQL universe: PGCon is now PGConf.dev. Yes, you read that right. And no, you won't be catching that plane to Ottawa; get your passports ready for Vancouver, BC! 🎤 This is not a drill, my SQL wizards! New city, new vibe, new team, who absolutely nailed it this year. Imagine all your favorite extensions and modules compiled into one seamless conference experience. Shoutout to Andreas Scherbaum for the fantastic review. You had me at "PGConf.dev." Clever name, guys. Really rolls off the tongue, doesn't it? Whether you're replicating the good times or just trying to avoid deadlocks in your schedule, PGConf.dev is clearly the place to be. Pro tip: Book your flights, brush up on your CTEs, and consider this an official hint to present next year (yes, you, with the million row query and the trick up your sleeve). Let's keep those transactions committed and our spirits high. Until next time, keep your queries optimized and your connections pooled. #PostgreSQL #PGConfDev #DatabasesAreFun #TravelForTech 🔍👀 🧠 Credit: Andreas Scherbaum You can find the full article here: https://postgr.es/p/6x6

1

It’s not easy for #organizations to become data-driven, despite the aspirations many have to get there. Even organizations with #data #analytics teams often struggle to make effective use of their data. According to a recent Gartner survey, fewer than half of data and analytics leaders say their teams are effective in providing #value to their organizations. German computer scientist Michael Berthold became aware of the issue while a professor at University of Konstanz, where he consulted with a number of businesses trying to adopt data analytics practices. These businesses frequently expressed a desire for a platform that could help process and analyze their data, and so he and several colleagues set out to build one. “The initial goal was to create a modular, highly scalable and open data processing platform that allowed for the easy #integration of different data loading, processing, transformation, analysis and visual exploration modules, without focus on any particular application area,” Berthold said. “The software was designed to be professional-grade and also serve as an integration platform for various other data analysis projects.” The resulting #open #source platform, called KNIME, eventually morphed into a VC-backed startup of the same name, and the aforementioned colleagues (Bernd Wiswedel and Thomas Gabriel) joined Berthold on the founding team. Today, KNIME has 400 customers paying for the fully managed version of its platform, including Audi, AMD, Lilly, Novartis, Bayer, Sanofi, Genentech, the FDA, P&G and Mercedes-Benz. Annual recurring revenue has been increasing 30-40% per year since KNIME’s founding in 2008, according to Berthold, and now sits at about €30 million ($~32.35 million). #startups #dataanalysis #dataprocessing #techdevelopers #innovation #dataworkflows #opensourceanalytics #creativegeeks https://lnkd.in/g3zUvc96

2

Accelerate your decision-making without being an #ML expert! Join our lab and master #SnowflakeCortex ML-based functions 📈 Discover how simple #SQL commands can power forecasts and spot outliers with precision, all thanks to the magic of machine learning under the hood. In this session, our expert instructors will guide you through: - Building models that predict and forecast key metrics 📊 - Identifying anomalies in your data 🔎 - Evaluating your model's accuracy 🎯 - Automating model retraining and integrating email reports 🔄

86

1 Kommentar

#TBT to Marvik's presentation at the #AISummit by Databricks. "Generative AI at Scale Using GAN and Stable Diffusion" #GenAI is under the spotlight with its diverse applications, but deploying these models at scale presents many considerations. This presentation will explore various architectures and discuss optimization hacks for the sophisticated data pipelines essential to generative AI. https://lnkd.in/dC6AGNgx

28

Great blog on Spark AQE!

12

🙋 Going to PyConDE & PyData Berlin this year? Don't miss this session by Marie-Kristin Wirsching, data scientist at inovex GmbH, on how to build engaging data stories with Streamlit and Snowflake! Explore a proof-of-concept, tracing the conception of a data story to the implementation of a Streamlit app in Snowflake, by using open source datasets from Deutsche Bahn. 📊 👉 https://buff.ly/3vM50kX #Python #AI #DataScience

24

Generative AI models "tend to fail in the real world" when dealing with "messy" data and ambiguous human language. Databricks argues that compound #AI is the solution to the shortcomings of #GenerativeAI. How has Databricks improved on #LLMs? Find out:

1

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Kommentare

How to Train Your Own Large Language Model (LLM) by Databricks Check out the 26:50 mark for insights on feeding your own data into a model, including various methods and approaches. https://lnkd.in/g6h3zEgE #Technology #Data #Dataprocessing #artificalintelligence #AI #finance #manufacturing #realestate #healthcare #Future #Futurism

2

Have you ever struggled with really big tables - billions of billions of records? Have you wondered what partition keys to use and felt the pain of having to rewrite your table if your partitioning strategy needed changing? Liquid clustering eliminates the need for this. 🚀 Liquid clustering is a game-changing feature revolutionizing data management in Databricks. It eliminates the constraints of traditional partitioning on Delta, offering unparalleled adaptability for your data strategy. 👉 Simplifies data layout decisions 👉 Allows flexible redefinition of clustering keys without data rewrites 👉 Enables data layout evolution alongside changing analytic needs It's really worth using! Check it out and don't miss out on optimizing your data flows by up to 12x. Learn more about how Liquid Clustering can transform your data management: #DataManagement #Analytics #BigData #LiquidClustering #Databricks

3

In data partitioning, there are no universal rules. Tailor your approach based on the specific characteristics of your data and the problem at hand for optimal results. Check out Federico Rucci's latest article on creating appropriate test sets. #MachineLearning #DataScience

16

Do you know how to ensure #data_integrity in your #database? Join us on July 10 to explore integrity features of relational databases with Federico Razzoli (Founder, Vettabase Ltd) and Tianzhou Chen (CEO, Bytebase) on a #webinar: https://lnkd.in/eWrCgnF6 #rdbms #dbms #data

8