Joe Khoury

Real-Time #Analytics News from the Past Week: 1. Starburst announced a range of new capabilities for their Trino-based open hybrid lakehouse platform, Galaxy, including: -The general availability of fully managed streaming ingestion from #ApacheKafka to Apache Iceberg tables. -The public preview of fully managed ingestion from files landing in Amazon Web Services (AWS) S3 to Iceberg tables. -Multiple enhancements to performance and price-performance of their #lakehouse platform. 2. Cohesity introduced a patent-pending visual data exploration capability to Cohesity Gaia, its #AI-powered search assistant launched earlier this year. By providing customers with a visual categorization of the themes across documents and files within a #data set, the visual data explorer brings new context to the data and suggests queries that help users gain insights faster. 3. IBM announced the release of its most advanced family of AI models to date, Granite 3.0. IBM’s third-generation Granite flagship language models can outperform or match similarly sized models from leading model providers on many academic and industry benchmarks. 4. Confluent announced the launch of the Confluent for Startups AI Accelerator Program. The 10-week virtual program seeks to collaborate with 10 to 15 early-stage AI startups that are building real-time applications utilizing the power of #DataStreaming. 5. Semantic Web Company (SWC) and Ontotext announced that the two companies have merged to become the Graph AI provider, Graphwise. Read the rest on RTInsights ➡️ https://lnkd.in/eQKsrPCZ

12

🚀 Big News! 🚀 📚 I'm excited to announce that my latest book, "Kafka Troubleshooting in Production," is now available! 📚 If you're managing Kafka clusters, this book is your essential guide to achieving and maintaining Kafka stability. What You’ll Learn Inside: 🔍 Monitor & Resolve Issues: Proven methods to keep your Kafka environment running smoothly. 💡 Cost-Effective Provisioning: Efficient ways to provision Kafka clusters without overspending. 🔧 Root Cause Analysis: Techniques to identify and fix Kafka problems at their source. 🚀 Data Loss Prevention: Strategies to minimize and avoid data loss and delays. ⚠️ Forestall Failures: Proactively address and prevent common failure points. 📋 Troubleshooting Checklists: Step-by-step checklists to troubleshoot issues effectively. Perfect For: KafkaOps, SREs, and DevOps practitioners will find practical techniques and recipe-style checklists to detect, troubleshoot, and prevent real-world production issues. With insights from years of experience diagnosing and resolving complex production problems, this book will ensure your Kafka clusters run smoothly. 🙏 Special Thanks: A big thank you to Or Arnon for the detailed review of the cloud-related chapters, and to Uri Bariach for the valuable feedback on the on-prem chapters. 📘 Get Your Copy Today: • Amazon: https://a.co/d/ebEN1Ov • Springer Nature: https://lnkd.in/dVA3bYSA  • Barnes & Noble: https://lnkd.in/dFZvEqET • WHsmith: https://lnkd.in/dQjwbJ3v  Don't let Kafka be a mystery. Transform the way you manage Kafka in production by grabbing your copy today! #Kafka

58

8 Comments

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

Llama3 on Databricks. Databricks is changing the game with making open models easily usable.

1

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 Comments

Super cool new features for our SQL Server Capture connector! 💪 Lower impact on the source DB We’ve restructured the connector to only query tables with new changes which results in fewer overall queries. ⏩ Lower latency Lower latency in extracting data when many tables are being captured 🧹 Automatic CDC table clean-up As soon as we’ve read data, we have the option to delete it from your source, resulting in lower disk usage. 👨‍💼 Automatic CDC instance management We can automatically create new change tables and remove old ones as your schema changes so you no longer have to do anything in that case. Check it out: https://lnkd.in/dyqzBPia

12

1 Comment

I love being able to use #dbt to automate away tedious tasks! I'm sure I'm not the only one that dreads typing out the number for each column there is before and after aggregated columns. The built in dbt macro group_by works great in most cases. I added a few lines to help out in other cases when the aggregate columns aren't at the end. I wrote a short blog post about it: https://lnkd.in/gMXnmjKJ

5

Have you heard about the latest from Databricks and the Delta Lake community? They've introduced the Open Variant data type, a game-changer for Apache Spark and Delta Lake. This innovative feature significantly boosts schema flexibility and evolution within data lakes, tackling the challenges of managing diverse and evolving data structures head-on. Key Points: 1. Schema Evolution Made Easier: Open Variant allows for storing semi-structured and nested data flexibly within Delta Lake tables. This capability simplifies schema evolution by accommodating new data fields without requiring immediate schema updates across the entire dataset. 2. Optimized Data Management: With Open Variant, data engineers and analysts can seamlessly integrate new data formats and structures into existing Delta Lake tables. This flexibility supports agile data pipelines and analytics workflows, reducing complexity and improving productivity. 3. Compatibility and Interoperability: The Open Variant data type is designed to work seamlessly with existing Delta Lake features and Apache Spark APIs. It ensures compatibility with various data formats, making it easier to ingest and process diverse data sources. 4. Enhanced Data Lake Capabilities: By enabling efficient handling of complex, nested, and evolving data schemas, Open Variant enhances the capabilities of Delta Lake as a robust foundation for modern data lake architectures. This introduction marks a significant step forward in empowering organizations to manage and derive insights from increasingly diverse and complex data sets with greater ease and efficiency. Read more about the Open Variant data type and its impact on data lake management in the Databricks blog post. https://lnkd.in/dT6CxTxR #DeltaLake #ApacheSpark #DataManagement #DataEngineering #DataAnalytics #SchemaEvolution #BigData #DataLake

4

1 Comment

Are you looking to work with Delta Lake but want to avoid the cost and complexity of Databricks? The Rust-based "delta-rs" crate provides a compelling alternative, allowing you to create and append to Delta Lake tables without the need for a Spark cluster. Some key benefits of the delta-rs approach: **Cost Savings**: Run Delta Lake operations in cheaper, more scalable environments like AWS Lambda. **Operational Simplicity**: Abstracts away the complexity of Spark cluster management. -**Flexibility**: Use delta-rs in diverse contexts, like Kafka message ingestion. Learn more about how you can implement Delta Lake without Databricks in our latest technical blog post: https://lnkd.in/gncb_Dkj #DataEngineering #DataLakes #DeltaLake #Rust

8

Are you dealing with complex array transformations in your data pipelines & analytics? Simplify your workload with Snowflake's powerful new higher-order function: REDUCE. With REDUCE you can:  -> Simplify complex array operations -> Write clean, functional easily readable code -> Boost array processing performance In short, working with array data in Snowflake just became much more simple & efficient! Check out the latest blog by Yogitha Chilukuri & Wenqi Wei to learn more: https://lnkd.in/gtQfU54m  (Links to docs for this and other higher order functions in the comments)

133

1 Comment

What are collations, and why should you care (especially if you are migrating a data warehouse workload to the cloud)? In a #database, a collation is a set of rules that determine how data is sorted and compared, usually applied to how text data is sorted in different languages for the purpose of indexing text data and for making comparisons between text values. Different languages have different character sets and ordering. When you are migrating a traditional #datawarehouse workload to the #datacloud, robust collation support plays a pivotal role in ensuring your data behaves predictably, especially when dealing with case sensitivity. Some examples of where collations come into play: > Data Consistency: Collations define how the database sorts and compares strings. Choosing the wrong one could lead to unexpected results, impacting everything from reports to application functionality. > Case Insensitivity: Are you dealing with names, emails, or any data where case differences shouldn’t matter? Collations like _ci (case insensitive) ensure that ‘John’ and ‘john’ are treated as the same, simplifying queries and ensuring data integrity. > Performance: Optimal collation selection can significantly boost query performance by reducing unnecessary comparisons and sorting operations. > Global Reach: In a connected world, collations also affect how your data interacts across different languages and locales. Choosing the right collation ensures your data remains consistent and understandable regardless of where it’s accessed. At Snowflake, we provide a broad range of collations tailored to your specific needs, including the recently launched upper, lower, and locale-ci. Curious to learn more? Check Qinyi Ding's insightful deep dive into choosing the right case-insensitive collation in Snowflake: https://lnkd.in/dz9UaHXx

50

108 Recipies Snowflake Build: Just moments ago (Nov 12th, 12:30 PM CST), SF presented the Snowflake Dynamic Iceberg table using Polaris External CatalogSync." We have 20+ recipes in this e-book. Below is how to build summary tables as new data streaming into the base table. Snowflake

6

New feature release - Job metrics timeline view! Ever wonder why your Databricks job performance changes over time? You may ask yourself questions such as: “Why is my runtime increasing since last week?” “Is the cost of this job also increasing?” “Is the input data size changing?” “Is my job in danger of crashing?” Understanding why your job performance is changing is critical to both optimize costs and prevent crashes. In this post we show a new visualization timeline features to help engineers find root causes quickly. Try it out today with Gradient! #dataengineering https://lnkd.in/ebPjtU7y

7

5 Comments

It's not every day that I get to talk about something that I think will change the future of analytics. Today is one of those days. Introducing: Snowflake Native Amplitude One of the top requests we get from data leaders is to be able to leverage all the investment they've made in their Data Cloud with Snowflake by having Amplitude run directly on top of the data. Today we are able to deliver on that promise. Snowflake Native Amplitude is the same Amplitude Analytics our users know and love, completely replatformed on Snowflake. No ETL, no multiple copies of the data, no issues with data drift or mutability. Amplitude querying directly on top of Snowflake via SQL and returning the results in our point and click dashboards. Curtis Liu and our engineering team spent the last 3 years creating a version of Amplitude Analytics that works with SQL natively on top of Data Warehouses. This required a massive engineering effort. We rebuilt all of our chart querying capabilities, from segmentation, to funnels, to retention, to journeys, to event properties, to every other chart type on top of the data warehouse. We needed to ensure essential features like user timelines, complex segments, microscope, and cohorts worked effectively in the same way as they did on our existing custom Nova database. We then spent a huge amount of time optimizing the hell out of the performance. (Data Warehouses are SLOW if you don't tune them correctly, particularly on critical reports like funnels and user journeys.) As a result, Amplitude is now the first fully replatformed enterprise app on Snowflake. It just works. Sign up for free and start using it right away! We also believe in this so much that we are going big right out of the gate with the launch. Anyone can use full Amplitude analytics, up to 5 seats, completely free of charge. Data leaders have been begging me for this functionality from Amplitude for years. We are excited to finally deliver it today. Read more (and signup!) on Snowflake Native Amplitude here: https://lnkd.in/gbvPDy3P

795

82 Comments

If you're new to Databricks or are looking to further develop your skillset, the Databricks academy is a great place to start. Here you can learn about Databricks sales and technical training to get enabled on the Data Intelligence Platform. Before you can help your customers unlock the potential of their data, you need to master the Databricks Lakehouse Platform. The Databricks Academy is your go-to place for training and certification for all roles in your organisation. If you would like to discuss the resources available and where to begin, feel free to reach out and I can point you in the right direction. #learning #certified #databricks

23

Same Chicken (Iceberg), different Curries (Engines) Cloudera: Read Snowflake Data from Polaris External Sync Catalog I am able to "read" and integrate Snowflake data through Polaris with other data in Cloudera Spark. Cloudera Snowflake

17

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 Comments

If you ingest data from Kafka/Kinesis into Databricks, you should watch Charles Tan and Shawn Gordon's short vide on how to shift left your data pipelines to reduce cost and latency! #ApacheFlink #Kafka #StreamProcessing #RealTimeData https://lnkd.in/ggMsi9KU

14

1 Comment