Joon Kim

⚖️ Tech Comparison: Redshift vs Druid https://lnkd.in/e3ZCD3xb Like {hashtag|\#|Snowflake}, {hashtag|\#|Redshift} was built for traditional BI and infrequent reports. Not having multi-level indexing and time-based partitioning means inefficient compute with more data processed than necessary. When the analytics use case requires high performance on large data sets or concurrency requirements and streaming data, then Apache Druid is ideal.

10

Want to simplify your telemetry data? In #o11ycast ep. 74, Liudmila Molkova explains how semantic conventions make it possible. Listen now: https://hubs.ly/Q02Txlxw0

1

City Storage Systems (CloudKitchens) has transformed its machine learning infrastructure since adopting Daft. Their data processing and ETL tasks, previously managed on separate Spark clusters, now leverage Daft’s intuitive DataFrame interface — surpassing PySpark’s capabilities 💪 Tune in to their session at #RaySummit by Anyscale on Tues Oct 1 to discover how the shift to a consolidated Ray and Daft environment supports their end-to-end ML pipelines, from data engineering to model development. #Daft #Ray #training #ai #ml #dataprocessing #etl #PySpark #pipeline

16

🚀 Automate Your CI/CD Pipeline with Snowflake and GitHub Actions 🚀 Our Senior Data Engineer, Jaime Salas, shares an insightful guide on integrating GitHub Actions with Snowflake to streamline your CI/CD pipeline. Learn how to: 🔹 Set up key pair authentication 🔹 Create and manage Snowflake users and schemas 🔹 Automate tests and merges using GitHub Actions Enhance your development workflow with these powerful automation techniques! https://hubs.ly/Q02BWXC10 👉 Read the full article #DataEngineering #Snowflake #GitHubActions #CICD #Automation

15

Attention Snowflake users ! Wouldn't it be great if every time anything anomalous happened in your infrastructure it got flagged? Well in Orchestra that's pretty easy to set-up. Check it out below! https://lnkd.in/eP8bj6MU #dataengineering #anomalydetection #snowflake

7

Unlock the Power of Data Lake Table Formats with Alex Merced, Developer Advocate at Dremio! 📊 Join Alex Merced as he delves into "The Who, What, and Why of Data Lake Table Formats." 📅 Save the Date: https://lnkd.in/eFZ6m5HV 🔍 Key Learnings: - Explore how data lake table formats handle large datasets. - Learn how these formats optimize query performance on extensive data. - Understand the importance of data durability and recoverability. - Discover how these formats ensure data is auditable and compliant. Don't miss out on this opportunity to level up your data analytics game! 📈 #DataAnalytics #DataLake #Dremio #TechTalk

4

Snowflake Launches Arctic, An Open ‘Mixture-Of-Experts’ LLM To Take On DBRX, Llama 3 Snowflake announced the launch of Arctic, a large language model (LLM) optimized for complex enterprise workloads such as SQL generation, code generation and instruction following. Touted as the “most open enterprise-grade LLM” in the market, Arctic taps a unique mixture of expert (MoE) architecture to top benchmarks for enterprise tasks while being efficient at the same time. It also delivers competitive performance across standard benchmarks, nearly matching open models from Databricks, Meta and Mistral at tasks revolving around world knowledge, common sense, reasoning and math capabilities. https://lnkd.in/ghAZ5jNn

1

Back in March, #Databricks launched #DBRX—a super-advanced state-of-the-art open language model that's truly next-level. This new model marks a major step forward for #OpenSource #LLMs. What’s really cool is that DBRX outperforms nearly all established open source models and even rivals #ClosedSource giants like #GPT4, #Gemini, and #Mistral across various benchmarks. Want to know what makes #DatabricksDBRX so awesome? We've got you covered! Check out the full article below to learn everything you need to know about it. https://lnkd.in/g3w845G7

2

Discover how Wavicle helped Pilot Company transition to a unified data framework with Amazon S3, Databricks, and Snowflake, cutting latency and reducing costs for faster, cost-effective analytics and better decision-making. https://hubs.la/Q02KNPYD0 #cloudmigration

15

Say goodbye to vendor lock-in! Dremio now supports the Apache Iceberg REST Catalog Specification, championing an open, vendor-agnostic ecosystem. 🌐🔓 Our latest update empowers customers with unmatched flexibility and control over their data, driving enterprise interoperability at scale. 🔗 Dive into the details here: https://lnkd.in/gmZKkSTa #ApacheIceberg #OpenSource

101

1 Comment

🎶To the left, to the left… Shift your data processing and governance to the left. Learn how to eliminate data inconsistency, reduce processing duplication and costs, prevent downstream data quality issues, & maximize ROI of your data warehouse and lakes. → https://cnfl.io/3RHCc4x

175

Session recording 📹: ZHI LIU Product Manager at RisingWave discussed real-time data enrichment and analytics using RisingWave and Apache Doris: "Streaming process and OLAP are not like a one-to-replace-the-other relationship. It's more like: Each of the processing paradigm tries to solve a different use case. So when people first come up with stream processing, they ask the question like: In real world, data is generated continuously, so why don't we process data continuously so that we can acquire insights continuously? The idea is that you are facing an infinite stream of data where you have a fixed query, so whenever there are new data coming to the system, the query reacts to the data." "The idea of OLAP is more like: You have a finite set of data, and business analysts will try to explore what's contained in that data set, so they will come up with different queries. Based on the insights, the modify the queries and try to get more different insights." https://lnkd.in/ghRPu4-X With this, we’ve wrapped up all sessions from the Apache Doris Meetup @Singapore! If you’re still eager to connect with the Apache Doris community, don’t miss the upcoming online webinar. Colin Wang will provide an in-depth introduction to 🌟Real-Time Analytics with Apache Doris: Everything You Need to Know 🌟. Stay tuned and join us! https://lnkd.in/g4jeZvHt

8

In the last few months, we've added a lot of features to make it easier to manage dbt projects with Kestra. Sync your dbt Labs project from #Git, scale runs dynamically with task runners, and edit dbt code directly in Kestra’s UI. Whether you’re managing hundreds of models or handling intensive data transformations, Kestra’s task runners adjust resources on-demand, so you can build, deploy, and manage dbt workflows. Plus, our new Namespace File editor makes it easy to share and version code across environments. Check out the full post to see what Kestra can do for your dbt workflows 👉 https://lnkd.in/eEnEBa8A

24