Giannis Polyzos

The DataFusion SIGMOD paper is officially out. Congrats to Andrew Lamb and the other authors! The paper is great, accessible read covering the architecture of a modern, state-of-the-art query engine. Also exciting to see Arroyo (YC W23) get a mention! The coolest thing about the paper (and about DataFusion itself) is that it demonstrates how building the engine in an open, extensible way doesn't preclude great performance. Systems built on top of DF (like Arroyo) can easily add new functions, operators, tree-rewrites, optimizations, data types, and more while using the the same tools and infrastructure that the built-in functions/operators/etc. are built with. This is incredibly powerful, and has already led to a blossoming of powerful data systems including Comet (a Spark accelerator), LanceDB (a vector DB and query engine), and ParadeDB (which embeds an OLAP DB in postgres). Read the paper here: https://lnkd.in/dpknuPy4

33

2 σχόλια

We are back with another blog post! This new one explores whether to index a field in Apache Solr as a string or integer for optimal filter query performance. Read: https://lnkd.in/dYXznYJb

5

We're wrapping up September with style! 🧣 📐 chalk.functions​ now offers a ​cosine­_similarity​ function, which is handy for comparing vector embeddings. (example linked in the changelog). 📈 Offline query runs in the dashboard now show metadata and CPU/memory utilization metrics. 📥 We have a new offline query configuration option for recomputing features ​only​ when they are not already available in the offline store. This option is useful for workloads with computationally expensive features that cannot easily be recomputed. Please reach out if you’d like to try this feature. Comment to the full changelog in comments ⬇️

7

1 σχόλιο

This session by Jan Lukavský will introduce a platform created to bridge the existing gaps in data management while removing some of the complexities in existing Big Data ecosystem. The model is consistently applied across three abstract types of storages - streaming (e.g. Apache Kafka, Google Cloud PubSub), batch (e.g. Hadoop HDFS, S3, Google Cloud Storage) and random-access (e.g. Apache Cassandra, Apache HBase, Google Cloud BigTable). The platform also provides sources and sinks for data processing engines like Apache Beam or Apache Flink, so that sophisticated data transformations can be easily integrated into the platform as well. Wanna hear more about it? Join us at: https://lnkd.in/eJRNCHMz  #bigdata #accesspatterns #datastorage #datastreaming The Apache Software Foundation #apachekafka #hadoop #apachecassandra #apachebeam #apacheflink

1

It always puzzled my why Automatic DDL replication, while Standard in MySQL is so hard in PostgreSQL. Good to see some progress being made https://lnkd.in/ddi83nkN #postgres #mysql

25

9 σχόλια

In our latest blog, our CTO Gang Tao demonstrates how to use a random stream and just one line of #SQL to perform a #MonteCarlo simulation. By generating a large number of random samples, the Monto Carlo method helps estimate complex values - in this example, the value of pi. Curious how? Read on to learn how to calculate pi with Timeplus 🥧 #randomstream #montecarlo #pi https://lnkd.in/gP_NtwaP

4

49% of companies are stuck on outdated PostgreSQL versions - but here's what nobody's talking about: The most successful migrations we analyzed weren't about technical expertise. They were about timing and strategy. Our latest deep dive reveals the real patterns behind successful PostgreSQL upgrades in 2024, including: - Why "zero downtime" is often the wrong goal - The hidden costs nobody calculates - A framework that actually works https://lnkd.in/gzwJq5-z #PostgreSQL #Engineering #TechLeadership #Databases

5

🚀 New Feature: 𝗧𝗘𝗠𝗣 𝗧𝗔𝗕𝗟𝗘𝗦 in #Databend! 🔥 Create temporary tables that are automatically dropped at the end of the session. No cleanup needed, just seamless, session-specific data handling! Learn more in our docs 👉 https://lnkd.in/gyRHu6h7

1

Interesting blog post from Sijie Guo about CAP theorem for streaming data, and the impact of separating storage nodes from broker/serving nodes in reducing cross-zone traffic.

5

Just out! Chapter 2.3, Domain modeling, explores how you can defend against poorly thought out, poorly understood architectures. In this chapter of the playbook, we walk through applying domain modeling to our model use case. https://bit.ly/3z8Sc9y #modeling #DDD

1

With a simpler implementation than the outgoing code handling these tasks, tasks stuck in queued will no longer slip through the cracks and stay stuck. https://lnkd.in/d_cXFWGx

1

Bloom indexes in #PostgreSQL efficiently test for the presence of a set of values, especially useful for multi-column queries where traditional indexes may be less efficient. They are ideal for checking combinations across columns without needing explicit indexes on each. #PostgreSQLPerformance @MinervaDB #dba https://zurl.co/OKDl

5

Apache Airflow 2.10 is here! The latest release brings enhanced stability, improved performance, and new features that make workflow orchestration even more efficient. Highlights include Data-aware scheduling, which allows for more intelligent task management, and enhanced task logging, ensuring better traceability and debugging. Upgrade now to leverage these improvements and boost your data pipeline workflows! Follow link for more detailed information about it. https://lnkd.in/dU8EKNzR Youtube Link: https://lnkd.in/ds4x8Aea #ApacheAirflow #DataEngineering #WorkflowAutomation #TechUpdate

2

Apache Cassandra 5.0 is now General Availability (GA)! 🎉 This major release significantly improves performance, usability, and capabilities for the world's most powerful distributed database. Key features include: - Storage Attached Indexes (SAI) for improved query flexibility - Trie Memtables and SSTables for enhanced efficiency - JDK 17 support for better performance - Unified Compaction Strategy for improved node density - Vector Search capabilities for AI applications - Dynamic Data Masking for improved security 🔗 and more at: https://lnkd.in/e3k35MTK ⚠ As a reminder, the Cassandra 5.0 launch marks the end of the 3.x series. Users are encouraged to plan their upgrade strategy soon. 🔗 Blog: https://lnkd.in/eJT-VsuF ⬇ Download: https://lnkd.in/dU__DRmQ 🤝 Learn more about Cassandra 5.0 at CommunityOverCode in Denver, October 7-10, 2024. https://lnkd.in/gpY6fUkN

143

1 σχόλιο

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 σχόλια

Tomorrow at Confluent #Current24: Marc Farnsworth leads the Building Real-time Analytics Apps with Kafka and #Druid session. - Ingest streaming data natively at scale - Enhance and align data from multiple topics - Automatically detect data types and ingest into scalable tables - Execute SQL queries that combine real-time and historical data - Create interactive dashboards to visualize event data - Embed these dashboards into web applications https://bit.ly/4gvOkAH

5

Apache Kafka topics are immutable, so you can’t edit or delete their data. But there are a few things you can do to fix bad data in event streams. By Adam Bellemare, thanks to Confluent | #Data #DataStreaming

6