Kendrick Kester

Part 2 of working with Google Cloud. How to load data from GCS into Google BigQuery. This is a short read but hope you enjoy. If there is any interest on me doing an equivalent article of these for AWS, let me know. #dataengineering

19

6 Comments

AWS just released a technical guidance about how to run Apache Iceberg workloads and I have enjoyed to read it, if you run ( or thinking about it ) Iceberg workloads on AWS, this guidance is IMHO a must read. The section that I liked the most was about Reading Best Practices, because write operation most of the time is the not so difficult part of the process, the hard part begin when you need to read a LOT OF DATA, and this section provided to me a lot of insights. [1] [1] - Read Best Practices: https://lnkd.in/e3gCytpb #AWS #ApacheIceberg #DataArchitecture #DataEngineering #DataPlatform

23

1 Comment

I wrote a thing.... It's about How to Build Data Visualization Dashboards with Apache Superset and Pinot Check it out. #apachepinot #apachesuperset https://lnkd.in/e46uRmuQ

5

Y'all know I'm a huge fan of Y42. The value they add on top of the analytics engineering process is just insane. Check out the new data diff feature in beta, available to all customers now. Watch until the end because you'll see they baked in data diff as part of the pull request process. 🤯 If you are looking to up your analytics engineering game, contact me and I can show you the ins and outs of this incredible platform!

8

#aws S3: The Bedrock of a Data Engineering Workflow As a data engineer, having a reliable and scalable storage solution is crucial. That's where Amazon S3 comes in. Its ability to handle massive datasets and its seamless integration with other AWS services make it an essential part of the workflow. Here are some of the ways I leverage S3 for data engineering tasks: Lakehouse Architecture: S3 serves as the foundation of my data lake, storing raw and processed data in a cost-effective and accessible way. ETL/ELT Pipelines: I use S3 as a staging ground for data during ETL/ELT processes, facilitating efficient data movement between different tools and services. Data Sharing: S3 makes it easy to securely share data with colleagues and external partners. Disaster Recovery: S3's versioning capabilities allow me to recover data in case of accidental deletion or corruption. What are your favorite ways to use S3 for data engineering? Share your thoughts in the comments! #ProTip: Utilize S3 Lifecycle Management to automate data lifecycle transitions between storage classes for optimal cost optimization. #aws #dataengineering 𝐇𝐨𝐩𝐞 𝐭𝐡𝐢𝐬 𝐡𝐞𝐥𝐩𝐬 - 𝐈𝐟 𝐢𝐭 𝐝𝐨𝐞𝐬, save & 𝐫𝐞𝐩𝐨𝐬𝐭 ♻️ 𝐭𝐡𝐢𝐬 𝐭𝐨 𝐡𝐞𝐥𝐩 𝐨𝐭𝐡𝐞𝐫𝐬. ↳ Hit the ‘save’ button! Follow me, hit the 🔔 ↳ We will learn something together. 🔴 Subscribe for AWS Data Engineering content to groups with 800+ data enthusiasts! ↳ Check my comment below! P.S: ✅Version 2 of 𝐑𝐞𝐚𝐥-world 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 𝐛𝐨𝐨𝐭𝐜𝐚𝐦𝐩 on AWS Data engineering starting ON 20TH APRIL! ✅𝐈𝐟 𝐲𝐨𝐮 𝐚𝐫𝐞 𝐚𝐜𝐭𝐢𝐨𝐧-𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝, fill the form below to be get invited for the webinar ; make sure you join the Whatsapp groups.

126

2 Comments

🎯When newly joined resources mess up the pipeline: 😀Junior: "I updated the ETL pipeline code. It runs twice as fast now!" 😀You: "Yeah, because it skips half the data. Congrats, you've created a Partial Extract process." In starting it was me lol😀😀 Follow Harikesh Tyagi🎯 for more content. #DataEngineering #BigData #ETL #DataPipelines #DataOps #CloudComputing #DataWarehouse #ApacheSpark #SQL #DataScience #MachineLearning #DataAnalytics #DataTransformation #DatabaseManagement #TechHumor

42

43 Comments

AWS's announcement of S3 Tables and S3 Metadata isn't just another feature release - it's a strategic move in the data lakehouse space. Here's why this matters for CTOs & technology leaders: AWS's decision to build on Apache Iceberg is particularly interesting. Rather than pushing their own proprietary format or backing Apache Hudi (where they're the biggest contributor), they've aligned with what's becoming the de facto industry open table spec. ICEBERG ! This is a smart play that acknowledges a reality in modern data architecture: organizations want standards-based, interoperable solutions that don't lead to vendor lock-in and more of copy data syndrome. The performance claims are compelling - 3x faster queries and 10x more transactions per second. But what's more interesting is the automated maintenance features. The potential for 30-60% cost savings through automatic tuning isn't just about economics; it's about engineering efficiency. Your teams spend less time managing infrastructure and more time delivering value. Yes, you'll pay 15% more for storage and new fees for monitoring ($0.025/1000 objects) and compaction ($0.05/GiB). But let's be honest - what's more expensive: slightly higher storage costs or engineering time spent managing data optimization? For most organizations, the automated management will likely pay for itself in reduced engineering overhead. The hidden gem of S3 Metadata feature might seem less exciting, but it's solving a critical problem. Many organizations have built homegrown solutions to track what's in their data lakes. Having this capability built into S3 could eliminate a whole category of custom tooling that engineering teams have had to maintain. What Engineering Leaders Should Consider?? Adoption Strategy: If you're already using Iceberg, the transition path looks straightforward. If you're heavily invested in other table formats, you'll need to weigh the benefits against migration costs. Team Skills: While this simplifies many operations, it's still Apache Iceberg under the hood. Your team will need to understand Iceberg concepts to fully leverage these features. Cost Models: The pricing model rewards efficient data organization. It's worth reviewing your current data patterns to estimate the real cost impact. The integration with IAM for table-level RBAC, automated maintenance, and metadata management suggests a future where data lakes can be both powerful and manageable. For technical leaders, the key question isn't whether to use these features, but how to integrate them into your data strategy. The potential for reduced operational overhead and improved data governance makes this a compelling option, especially for organizations already heavily invested in AWS. One thing's certain: Iceberg won the open table format war, but the data lakehouse wars just got a lot more interesting. *Views are mine. #datastrategy #dataarchitecture #dataengineering #lakehouse #modernization #ai

126

11 Comments

Introducing Databricks LakeFlow, a new unified, intelligent solution for data engineering with everything you need to build and operate production data pipelines. LakeFlow simplifies data engineering—from data ingestion to transformation and orchestration—so data teams can scale to the ever-growing demand for reliable data and AI.

8

Data engineering is solutions engineering, much like software engineering. Story 1/3 to collaborate this. Re-Architecting as primary + Cost Efficiency as a huge bonus. I was handed an airflow job that ran once a day. What it did was move data (a few GBs) from AWS S3 to Redshift. It finishes with creating some SQL views that analytics directly worked with. These views will be update periodically given new data requests from analysts. The airflow server was hosted on an EC2 machine. It was all good until the until the workers started dying intermittently. It got to an extent where I would have to manually restart them daily just to get them to run. I did everything in the books that I could to keep them up to no avail. Assume I didn't have enough Airflow knowledge to fix it. That's ok. There was no other option for me here than to change up the whole implementation. No, not a fresh setup but totally axing airflow. So I set forth. What I did was to refractor the entire DAGs in such a way to fit into AWS Lambda, then snapped it to a cron via an EventBridge events rule. To compensate for Airflow's nice interface for visibility, I added to the service, a full after run report that details how many tables and views were successfully moved/recreated or not and how long they took. This is dispatched to my email per run. This way, I replaced a full EC2 machine with a single AWS Lambda, was able to go to sleep and not wonder if the pipeline completed. A few things made me able to achieve this: 1. I have a software engineering background. 2. I did not actually need permission from anyone to implement this. As a matter of fact, I kept this running for up to 2 months before I informed the Solutions Architect that the data team had no more use of the EC2 instance. Next: 2/3 Replacing a Kafka instance on Confluence with a combination of AWS services. - Conscious cost move. #AskPyFesh

21

4 Comments

I recently deployed a partial Data ML stack in the Hetzner cloud using bare-metal Kubernetes. Currently, 24 developers and ML engineers from the Basel infrastructure team are using it. To ensure we stay within a strict budget of $200, with a maximum threshold of $500, I implemented several optimizations: - For each JupyterHub user, I allocated 1 CPU and 2 GB of memory. To optimize resource usage, I disabled the terminal and provided pre-installed Jupyter templates. These templates are heavily optimized for specific jobs and do not support all pip libraries, keeping the image size small and controlling compute resources. - I'm running Apache Superset on just 2 CPUs and 4 GB of RAM. Surprisingly, it's performing well, although traffic isn't heavy. I configured middleware to reject new logins if CPU or memory usage reaches critical levels. - Auto-scaling is enabled with extremely fast node boot-up times. The cheapest nodes proved less useful, so I switched to medium-sized nodes. We average 10 nodes, with peaks up to 25 nodes. - I used Hetzner volumes for persistence and leveraged LocalStack for AWS services. This setup works amazingly well and is extremely fast. Each user has their own 5 GB of storage space. - DuckDB has a dedicated 100 GB space for testing, costing around $1.50. It's configured to use an entire node with affinity settings, allowing us to monitor traffic and CPU spikes. I may consider using a dedicated node for DuckDB in the future. - Streaming data is now reduced to 1 MB/sec, and data is retained for two days before being dropped. This helps control egress costs, and we appear to be within budget.

60

4 Comments

I went back to day 1 on Zach Morris Wilson's bootcamp to dive more into Run Length Encoding. The way he pointed it out made me realize that most data engineer's probably don't think about it. I ended up finding a separate supplementary youtube video where he went more in-depth using Trino and Apache Iceberg! Choosing the right PARTITION columns for a dataset made sense to me, basically it is the same process as picking a primary key. What is the main way the data is accessed? Choosing the right SORT columns was new to me though and it is awesome that he demonstrated how sorting directly impacts run length encoding compression and how much space can be saved, but also how less bytes are accessed through processing from downstream pipelines. Not everyone is thinking about it, but given my disposition on efficiency, I am now! Join the free youtube bootcamp here: https://lnkd.in/ggwSmMek Video Link: https://lnkd.in/gsFSJ2hs #datamodeling #dataengineering 

6

+1 on SQLMesh for NOT making me write a ton of yaml and jinja. SQLMesh keeps them minimal. Model audits and metadata go right into the model files, which makes a lot of sense! #dataengineering #analyticsengineeeing #sqlmesh

50

5 Comments

Ever wished for a faster way to build advanced SQL resources? Check out what I am using - https://lnkd.in/d77waC-H That was me until I started using You.com. I recently needed to create a table of advanced SQL formulas to help senior engineers work more efficiently—normally, a time-consuming task. But with You.com's AI Agents, it was a breeze! The results were spot-on, giving me exactly the formulas and concise descriptions engineers need to tackle complex queries in less time. This tool has transformed how I create valuable resources, making the process faster and more tailored than ever before. For any engineer or data pro looking to maximize productivity, I can’t recommend You.com enough. It’s like having a co-pilot in your workflow. #you #ravenaondata

49

7 Comments

If you are at Amazon Web Services (AWS) (AWS) #reinvent and haven't stopped by the dbt Labs booth [booth 1795... you should. If you’re serious about making the most of your AWS investment for data analytics, here’s why dbt should be a standard piece of your data stack: ✨ Analysts Need Freedom to Answer Business Questions Fast: - Why is it so hard? Business logic is buried in BI tools, slowing analysts down with manual processes. - How dbt Helps: dbt’s Visual Editor empowers analysts to build transformations easily, removing bottlenecks for faster, more agile insights. ✨ Scaling Data Transformations Across AWS (and Beyond): - Why is it so hard? Each platform—whether Redshift, Athena, Snowflake, or Databricks—has unique requirements. Managing transformations across them leads to fragmented workflows and wasted time. - How dbt Helps: dbt centralizes your transformation logic, allowing teams to work consistently across all platforms. This means less complexity, faster insights, and a stack that scales with your business. ✨ Teams Struggle with Consistency Across Data Tools: - Why is it so hard? Metrics often differ across tools, leading to delays and a lack of trust in data. - How dbt Helps: dbt’s Semantic Layer ensures consistent metrics everywhere, enabling seamless collaboration and aligned insights. 📍 Stop by booth 1795 and see how #dbt can transform your #AWS #analytics stack!

47

1 Comment

When managing Snowflake at scale, it can be difficult to balance cost and performance. Capital One Slingshot makes it easier, building on tools we created internally to help our own teams streamline provisioning, optimize workloads and reduce query inefficiencies. See how Slingshot can help you maximize your #Snowflake investment. https://bit.ly/3AK2gGZ

1

S3 Table Iceberg: The credential vending issue still exists. remove_orphan_files failed-- I need to give AWS creds. I had this issue in Snowflake too. All other maintenance procedures are working--Expirte, rewrite, etc. In snowflake, spark.sql(f""" CALL s3tablesbucket.system.remove_orphan_files(   table => 's3tablesbucket.prod.example_table' ) """).show(1000,1000) Py4JJavaError: An error occurred while calling o36.sql. : java.io.UncheckedIOException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3" How to fix this issue: In Spark Session, we need to provide AWS creds. --conf spark.hadoop.fs.s3a.access.key=<> \ --conf spark.hadoop.fs.s3a.secret.key=<>\ --conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem Roy Hasson Amazon Web Services (AWS)

18

6 Comments