Tim Ellis

👋 🛠️ Say goodbye to complex troubleshooting. With Synthia, the AI assistant built into Sisense's Compose SDK, developers get real-time guidance—no technical jargon needed. Whether you’re a seasoned coder or new to the game, Synthia makes navigating Compose SDK seamless, solving challenges and optimizing workflows faster. Start your 30-day free trial 👉 https://bit.ly/4dGzK7j #AI #SaaS #DataScience

30

Your Datalake - in 2024 don't just consider the job done with Iceberg-ing and Object Storage-ing . Take a look at Apache Doris. We are coming full circle where 'integrated solutions' such as Clickhouse and Apache Doris are capable of doing the heavy lifting. They even include features such as federated querying. Apache Doris is my current favourite. This article by Dien, Tran Thanh gives you a great inside look into a migration from BigQuery to Apache Doris. https://lnkd.in/eGYbvCKx

12

The DataFusion SIGMOD paper is officially out. Congrats to Andrew Lamb and the other authors! The paper is great, accessible read covering the architecture of a modern, state-of-the-art query engine. Also exciting to see Arroyo (YC W23) get a mention! The coolest thing about the paper (and about DataFusion itself) is that it demonstrates how building the engine in an open, extensible way doesn't preclude great performance. Systems built on top of DF (like Arroyo) can easily add new functions, operators, tree-rewrites, optimizations, data types, and more while using the the same tools and infrastructure that the built-in functions/operators/etc. are built with. This is incredibly powerful, and has already led to a blossoming of powerful data systems including Comet (a Spark accelerator), LanceDB (a vector DB and query engine), and ParadeDB (which embeds an OLAP DB in postgres). Read the paper here: https://lnkd.in/dpknuPy4

33

2 Comments

Confluent's Tableflow is getting closer and closer to GA. Materializing Kafka topics as Iceberg tables, but with a great deal of thought around schema conversion, schema evolution and such. It's amazing how much work goes into making a product robust to schema changes. After the initial announcement I wrote two articles explaining Tableflow and its role in the future of bringing operational and analytics planes together. * Tableflow: the stream/table, Kafka/Iceberg duality: https://lnkd.in/gZbQ2cZ8 * Hybrid Transactional/Analytical Storage: https://lnkd.in/dTg6DKVy

102

4 Comments

I still remember that in 2017, when I was at the Reltio conference, it felt like the entire internet had ceased. It turned out to be an AWS-S3 outage, and that was when we all realized S3 is the backbone of the internet. Even today, S3 continues to hold the highest market share(31%), followed by Azure (25%) and GCP (11%). (Posted news in comment for those who forgot) As always 🎯 Mark Freeman II and Chad Sanderson have done a fantastic job explaining why DQ is so difficult for raw storage like S3. Excerpt from blog : "The strength of S3 is also its Achilles' heel: you can store virtually anything in it. Structured data, unstructured data, incomplete data, image data, audio data, Avro, Parquet, JSON—you name it, and you can likely store it in S3. This flexibility is incredibly powerful for data workflows involving machine learning or downstream workflows requiring diverse use cases. However, it also creates a long tail of potential data quality problems, with silent edge cases often causing the most significant issues." @Telmai, we have always believed that data quality monitoring should start at the data sources(shift-left), including landing zones (bronze) or data lakes. Hence, we natively support and automate DQ and monitoring of data lakes and all the raw formats. When you design your data reliability strategy, you should treat this as table stakes. I highly recommend this article and 🎯 Mark Freeman II's post on data quality and contracts to get such valuable insights. https://lnkd.in/gd27Zbik #dataquality #datalake #lakehousearchitecture #dataobservabily #dataobservability

48

5 Comments

The last two weeks were a BIG deal in the data world. For all my non-data-nerd friends, let me try to simplify why my industry is in an uproar over the events of the past few weeks and how it relates in a roundabout way to the exuberance over AI. To break it down very simply, a “database” is made up of two distinct parts: (1) Compute (aka Query Engine or Execution Engine) (2) Storage (aka Storage Engine) Historically, database companies have ALWAYS sold database products with both parts combined. All of the famous databases that you have ever heard of have both Compute and Storage together (Oracle, Teradata, IBM DB2, Snowflake, etc.). While that was the status quo for over 40 years, all of that has been changing recently with the embrace of an open source table format called Iceberg for the Storage layer. This means that users of Iceberg can store their data for FREE in a format that any other compute engine can query/analyze. For several years, this was a niche architecture reserved for only the most advanced tech companies (Netflix, Airbnb, LinkedIn, Apple, etc.) who used Iceberg because their data was so vast and so valuable that they were hyper sensitive to vendor lock-in. These tech companies used Iceberg in combination with an open source compute engine called Trino (full disclosure: Starburst is the Trino company) to form a complete analytical data warehouse (or Icehouse as it is sometimes called). However, in just the past few weeks, the two largest players in the data space (Databricks and Snowflake) have BOTH embraced Iceberg enabling their own customers to store data in this open format as well. I cannot overstate how HUGE this is for customers because with every major database vendor embracing Iceberg, this ushers in a new era of competition at the compute layer. Customers are no longer locked-in with their platform vendor and can now shop around for the best price/performance and functionality in the engine that they choose. This is now more important than ever as heightened interest in generative AI demands access to more and higher quality data, especially when implementing RAG. It’s all too common to see data warehousing costs skyrocket as data teams start wrangling hundreds of terabytes of data using bundled “all-in-one” platforms. When compute engines compete, you win. My colleagues Tom N. and Evan Smith a great post about this below: Anjali Norwood Kevin Liu Carlos Benavides Manish Malhotra

368

25 Comments

Given my experience with LangChain, building PDF document chunks and generating embeddings through the Palantir Build Pipeline is straightforward. Before you learn this tool, learn the following: Chunks, Embeddings, VectorDB, Similarity Search, Ranking, Prompt, etc in LangChain. Of course, Pyhton is your AK47 Palantir Technologies

10

🚀 New blog on Postgres write-ahead logs (WAL) We dive deep into the mechanics of Postgres logical replication and the role of WAL. We discuss how WAL works, why you want to use them, and their various components and settings. A few key items we discuss: 👉 How to use WALs to ensure your data remains consistent and protected, even during unexpected failures.  👉 The WALs role in logical replication with change data capture and important configurations to consider.  👉 Explore various WAL settings such as wal_buffers, checkpoint_flush_after, max_replication_slots, and max_slot_wal_keep_size to fine-tune database operations. Losing data is a nightmare for any data engineer. Whether you're a seasoned data engineer or just starting out, this blog is packed with essential insights to help you optimize Postgres for better performance and reliability. Check out the full post below. #postgres #writeaheadlog #dataengineering #datareplication

31

3 Comments

Blogged: Interoperability between lakehouse table formats with Apache XTable (Incubating). Scenario: Dremio -> Apache Hudi ❌ Dremio -> Apache Iceberg ✅ There are 2 analytics teams in an organization. Team A uses Hudi to manage some of their most critical, low-latency data pipelines. Team B focuses on ad hoc analysis, BI, and reporting, using Dremio's compute engine with Iceberg. ❌ Challenge: Team B wants to do analytics using data from the 2 different datasets within the organization. To do so, Team B has to use the dataset generated by Team A (stored as a Hudi table) and combine it with their dataset (the Iceberg table) ✅ Solution: Interoperability between the formats with Apache XTable. Detailed reading link in comments. #dataengineering #softwareengineering

105

4 Comments

How does an organization benefit from Strategic Data Modeling? 💥 Consensus and visibility of the broader business model 💥More productive conversations with business teams 💥Better quality of requirements 💥Higher signal, lower noise in technical conversations 💥Cross-platform, cross-domain, and widely understood conventions 💥Big-picture visual overview of the business and its database footprint 💥Preliminary designs become implementation blueprints 💥Accelerating onboarding of new team members 💥Making data more accessible and unlocking self-service within organizations 💥Keeping the database landscape manageable at scale 💥Getting a handle on complex data pipelines Check out Serge Gershkovich's book, 𝐷𝑎𝑡𝑎 𝑀𝑜𝑑𝑒𝑙𝑖𝑛𝑔 𝑤𝑖𝑡ℎ 𝑆𝑛𝑜𝑤𝑓𝑙𝑎𝑘𝑒, to learn more! #DataModeling #SqlDBM #Cloud #Snowflake https://lnkd.in/gsQsJ_yb

14

1 Comment

I'm very happy to share a technical article I co-created with Deeksha Gunde, that walks through enabling a federated Airflow environment using AWS SNS/SQS. https://lnkd.in/e4RwPirp #aws #dataengineering #analytics #bigdata

33

If you ingest data from Kafka/Kinesis into Databricks, you should watch Charles Tan and Shawn Gordon's short vide on how to shift left your data pipelines to reduce cost and latency! #ApacheFlink #Kafka #StreamProcessing #RealTimeData https://lnkd.in/ggMsi9KU

14

1 Comment

My TL;DR of database choices in 2024: - Pick Sqlite iff you don’t need the db to be accessed over network - Pick Postgres otherwise - Never pick NoSQL (and I did my PhD on NoSQL) Shout out to this awesome article I read this morning for more depth: https://lnkd.in/dwriene8 Happy Sunday! (Ignore this post and don't think about databases today 😂)

902

114 Comments

I presented about Apache XTable (Incubating) & how it brings interoperability to #lakehouse table formats at Dremio's Subsurface. Check out the recording of the talk in comments! #dataengineering #softwareengineering

48

3 Comments

Handling missing data points is not as easy as it sounds for #streamprocessing "Fancy" mechanisms like "watermark" won't show you new result if the count is 0 for a tumble/hop window. Check the nice video from Confluent. Dan mentioned a solution to add special events in the data stream(say Kafka), for 0USD transactions. Well, it works but will impact your COUNT/SUM/AVG, unless you add many filtering. I would suggest use DataGen or RandomStream in Timeplus as a heartbeat stream, UNION 2 streams together, run TUMBLE window with GROUP BY window and type, then you will get count=0 and fire alerts. SQL like this: WITH cte AS (SELECT 'real' as type FROM stream1 UNION SELECT 'datagen' as type FROM datagen) SELECT window_start, type, count(1) as cnt FROM tumble(cte,1m) GROUP BY window_start, type HAVING cnt=0 Leave comments below if you have any thoughts, or join timeplus.com/slack for more discussions.

8

Had a wonderful time talking with Kris Jenkins about Postgres internals and our work ParadeDB! Tune in below to learn more about Postgres, pgrx, custom index and table access methods, and more:)

20

1 Comment

We have often seen many Customers asking one of the very interesting question. 🔥 We have deleted a large amount of data from some of the tables, but we still do not see the space reclaimed to disk. Why ? 🔥 🎇 Some of the database engines may also see performance degradation (no improvement) eventually, as the live tuples/records are scattered across more number of data blocks than actually required. This may be because the data that got deleted was scattered across a lot of data blocks but not really stored sequentially within a few data blocks. Queries on such tables may involve scanning more data blocks for a very few records. We call this as Table fragmentation. 🐘 This concern may not be limited to #postgres, the same is also applicable for #oracle or #sqlserver or any other database engine. 👨‍💼 👩‍💼 Database and Application Architects generally implement techniques like Partitioning and Sharding wherever applicable so that the challenges of data deletion/archival are completely avoided with a simple partition drop instead of a time and resource intensive "delete" statement. We will extend a little more on database scalability sometime later. For now, let us see what can we do to reclaim this space, if you are using PostgreSQL. 💰 Considering the resources and expertise available for #Oracle and #SQLServer like database engines, users might have already been aware of solutions to rebuild tables online in Oracle or SQL Server. 💸 Do note that such options are available only with Enterprise versions for most of the commercial database engines. For users migrating from #oracle to #PostgreSQL or newly deploying #postgres as the database engine, you could use one of the popular OpenSource extension called : pg_repack, that can safely rebuild tables online and reclaim the space back to disk. Manisankar K from HexaCluster has published this article on pg_repack and also compared with solutions available for the Oracle Enterprise versions. If you have a similar challenge where you could not avoid a massive delete but still want to rebuild the table online to reclaim space, read this article to safely implement pg_repack for PostgreSQL. Contact HexaCluster for expert assistance for your PostgreSQL databases and for database migrations to PostgreSQL. #postgres #postgresql #oracle #sqlserver #hexacluster #databasemigration #databasemaintenance

9

"From PostgreSQL to Lakehouse using Hudi Streamer, Debezium & Kafka" Soumil S. joined us last week for the Episode 3 of Lakehouse Chronicles! He presented an amazing demo to tackle a real-world problem (CDC) of bringing data from operational sources to a lakehouse using Hudi Streamer. Here is a Github repo & Youtube recording 👇 🌟 Github: https://lnkd.in/dRarunaT 🌟 Youtube: https://lnkd.in/duhCKMed #dataengineering #softwareengineering

32

2 Comments