David G. Simmons 🇺🇦’s Post

AWS just launched S3 Tables, bringing native Iceberg support to S3. After spending some time digging into the details, here's what stands out: The integration is impressively deep - this isn't just another service layered on top of S3. AWS has added a new type of bucket (table buckets) with built-in table management and maintenance. The pricing looks reasonable too - storage is just 15% more than standard S3, and for typical analytics workloads, the monitoring costs are minimal. But what really gets me excited is how this lowers the barrier for building tools that write Iceberg tables. Until now, writing Iceberg tables well (with proper compaction) has been limited to big players like Databricks and Snowflake. Maintenance complexity was a blocker. S3 Tables could change that. The most interesting long-term impact might be on Iceberg catalogs. Instead of running separate catalog services (and dealing with all that operational overhead), we might be moving toward a world where object stores just handle this natively. The authentication story becomes much cleaner too - same AWS IAM policies that grant access to data can handle catalog operations. I suspect we'll see other cloud providers follow AWS's lead here. The fact that they called it "S3 Tables" rather than "S3 Iceberg" suggests they're thinking about supporting other table formats too. If you're building data systems on AWS or considering Iceberg, this is worth a serious look. I write one post for software engineers everyday. Follow Pratik Daga so that you don't miss it.

Using an event-driven architecture and serverless components in your AWS apps can help with decoupling, demand scaling and more. Using AWS Step Functions to fan out work to other services works well in many cases. AWS Step Functions is a workflow management service that offers many types of control flow choices. You can run steps in a sequence, in parallel, or other apporaches and require user input when needed. The steps in the workflow can call many other AWS services or external APIs to do their work and can take advantage of using services like the Elastic Container Service (ECS) and Fargate compute or other services like AWS Batch which takes care of scheduling and running batch jobs for you in ECS or EKS. The example below from Daniel Alejandro Figueroa Arias shows how they changed their architecture to use the Eventbridge Scheduler, Step Functions, and AWS Batch to process their ETL jobs and reduce coupling while improving reliability. https://lnkd.in/eQQWW8Bf

Using an event-driven architecture and serverless components in your AWS apps can help with decoupling, demand scaling and more. Using AWS Step Functions to fan out work to other services works well in many cases. AWS Step Functions is a workflow management service that offers many types of control flow choices. You can run steps in a sequence, in parallel, or other apporaches and require user input when needed. The steps in the workflow can call many other AWS services or external APIs to do their work and can take advantage of using services like the Elastic Container Service (ECS) and Fargate compute or other services like AWS Batch which takes care of scheduling and running batch jobs for you in ECS or EKS. The example below from Daniel Alejandro Figueroa Arias shows how they changed their architecture to use the Eventbridge Scheduler, Step Functions, and AWS Batch to process their ETL jobs and reduce coupling while improving reliability. https://lnkd.in/eBuX9rPD

This is most awaited features, will help with small file problem in big data

Started exploring more about serverless architecture using AWS for one our project. Serverless architecture is actually good and cost effective for small scale startups and projects , and for individual developers who are building their own projects and want to solely focus on application logic without worry much about the server management. Few important aws services that mostly required for us to build a basic serverless architecture from aws are:- ✅𝐀𝐏𝐈 𝐆𝐚𝐭𝐞𝐰𝐚𝐲 for creating routes , deploying and managing APIs for our application. ✅𝐋𝐚𝐦𝐛𝐝𝐚 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐬 - Service to allow our backend code to run virtually without managing servers. ✅𝐈𝐀𝐌(𝐈𝐝𝐞𝐧𝐭𝐢𝐭𝐲 𝐚𝐧𝐝 𝐀𝐜𝐜𝐞𝐬𝐬 𝐌𝐚𝐧𝐚𝐠𝐦𝐞𝐧𝐭) - To create user groups and roles for securing and controlling our AWS resources. ✅𝐂𝐥𝐨𝐮𝐝𝐖𝐚𝐭𝐜𝐡 - For collecting and tracking metrics ,logs and events from AWS resources. ✅𝐃𝐲𝐚𝐧𝐨𝐃𝐁 - Fully managed NoSQL db that provides simple and fast data models and scalability. #serverless #awslambda

🔄 Automatic Offset Commit in AWS Lambda for Kafka Did you know? When using AWS Lambda with Kafka, offset management becomes a breeze! Key points: - Lambda automatically commits the offset when it finishes processing with a status code 200 - No need for manual offset management in your code - Ensures exactly-once processing semantics 💡 Pro Tip: Don't forget to set up a Dead-Letter Queue (DLQ) for messages that can't be processed. This helps manage errors and ensures no data loss. Learn more about Lambda and Kafka integration: GitHub: https://lnkd.in/dwaSJ_cK Substack: https://lnkd.in/dPxFa5FN #AWSLambda #ApacheKafka #EventProcessing #Serverless #DataEngineering

Discover the simplicity of Managed Apache Flink on AWS! Our latest blog provides an easy guide with examples to implement real-time data processing. Elevate your skills today! #ApacheFlink #AWS #DataProcessing #TechTutorial

AWS Lambda logging is crucial for monitoring and troubleshooting serverless applications, but it can be challenging to set up and manage effectively. The key takeaway is that by implementing the strategies and tools discussed, developers can gain valuable insights, troubleshoot issues efficiently, and build more reliable serverless applications. * Understand AWS Lambda log types and access them in CloudWatch * Enable structured JSON logging for easier analysis * Optimize log levels and retention for cost management * Enhance logging with third-party frameworks using Lambda Layers * Simplify log analysis and monitoring with Better Stack #AWSLambda #ServerlessComputing #LoggingBestPractices #CloudwatchLogs #BetterStack #Monitoring #Troubleshooting #ServerlessDevelopment #CloudComputing #Observability https://lnkd.in/e7Q8rVCh

Here's a quick tip on how to make your #aws glue jobs give a little more detail in #cloudwatch logs

Learn how AWS #Mainframe #Midrange #Moderization new Application Features in Blu Age can help customers better understand the codebase and plan their modernization journey. https://lnkd.in/gJJsZhBy