Optimizing AWS Step Functions: A Case Study on Reducing Latency and Cost Preview: In the world of software engineering, scalability and efficiency are paramount. One of the challenges I faced in my recent project was managing a rapidly growing user base and the associated AWS Step Functions events. This blog post details my journey in optimizing our system, reducing latency, and cutting costs. I'll also explain how I utilized AWS Database Migration Service (DMS) and Kinesis to ensure real-time event processing. Key Highlights: Reduced event count by 3000% Significant cost savings Implementation of a scalable, loosely coupled system To read more about the detailed implementation and lessons learned, read the full post on Medium.
Dishank Oza’s Post
More Relevant Posts
-
🎉 Synthesis and Scoped Logging in the AWS CDK! A new video! 🎉 If you're interested in learning how CDK synthesis works, check out this video. I use scoped logging to demonstrate how Synthesis generates CloudFormation templates including: - Aspects! - CfnElement cloudformation generation! - CfnResource property processing! - Token resolution! - Post-resolve tokens! - Integrating Scoped Logging with AWS Lamba! - How logging objects are stored and shared in the construct tree! - Log levels! https://lnkd.in/gG5T6GjD
Synthesis and Scoped Logging in the AWS CDK
https://www.youtube.com/
To view or add a comment, sign in
-
Patrick Ames from AWS has more experience than just about anyone with managing thousands of Ray clusters and Spark clusters and processing exabytes of data. There are tremendous operational and performance challenges. - managing thousands of clusters - scaling to many petabytes or exabytes - latency and cost The original blog post is here: https://lnkd.in/g-pJhFei https://lnkd.in/gQsCrXPH
Is Apache Spark Too Costly? An AWS Engineer Tells His Story
https://www.youtube.com/
To view or add a comment, sign in
-
LogProcessing with AWS StateMachine https://lnkd.in/dp26myKv This application helps in resuming AWS StepFunction stuck because of some issues i.e., downstream application not available or the process has some issue which needs to be fixed to resume the Step Function or if some process needs human approval. The video explains in detail.
LogProcessing with AWS StateMachine
https://www.youtube.com/
To view or add a comment, sign in
-
Last week I ran a hands-on workshop looking at the AWS Fault Injection Service, which is a chaos engineering tool that allows you to setup and run experiments to test how resilient your infrastructure in an AWS account (or multiple accounts) is. The recording is available at https://buff.ly/3CihgMT and the code we used in the workshop is at https://buff.ly/4hFIX2r #AWSCommunity #DevopsPlayground #ChaosEngineering
To view or add a comment, sign in
-
We've had buckets of announcements this week, and I'm sure lots of you have eleven 9's of excitement about this one! 🪣✨ AWS has expanded S3 bucket scaling in a big way - here's what you need to know: S3 Bucket Scaling: What's New • Default quota now 10,000 buckets per account • Request increase up to 1 million buckets • Applied to all AWS Regions • No action needed for default increase Practical Benefits for Developers • Create buckets for individual datasets • Implement granular security policies • Enable default encryption per bucket • Use S3 Replication more easily Optimize Your S3 Architecture • Separate concerns with dedicated buckets • Improve data isolation and management • Remove barriers to scaling • Enhance overall storage architecture Cost and Scaling Considerations • First 2,000 buckets at no cost • Small monthly fee for buckets beyond 2,000 • Use Service Quotas for limit increases • New default applies to all AWS accounts Read all about it: https://lnkd.in/e2BPCnpy Could this S3 update solve any challenges in your current projects? Tell me all about it in the comments below 👇 👩🏻💻 Follow me (Brooke Jamieson) for the latest AWS + AI updates. 📍 save + share! #AWS #AmazonS3 #CloudStorage #CloudComputing
To view or add a comment, sign in
-
🚀 Excited to share that I recently completed the Amazon Web Services (AWS) APAC Solutions Architecture program on Forage! 🚀 In this simulation, I had the opportunity to design and implement a scalable hosting architecture tailored to a client experiencing rapid growth and slow response times. Leveraging a range of AWS services, I crafted a solution that balanced performance, scalability, and cost-efficiency. Key components of the architecture included: 🔹 Amazon Elastic Beanstalk: Simplified the deployment and scaling of applications by automating infrastructure tasks such as provisioning, load balancing, and health monitoring. 🔹 Amazon RDS: Enabled seamless setup, operation, and scaling of relational databases, with built-in features for automated backups and software patching. 🔹 Amazon S3: Provided durable, secure, and highly scalable object storage for static content, making it ideal for content distribution and data archiving. 🔹 Amazon Route 53 & Elastic Load Balancing: Enhanced application availability and fault tolerance by distributing incoming traffic across multiple instances, while Route 53 managed DNS routing efficiently. 🔹 AWS CodePipeline: Streamlined the CI/CD process by automating code build, test, and deployment, enabling fast and reliable updates. Throughout the simulation, I ensured the architecture was clearly communicated to the client, detailing both the technical functionality and cost structure in simple terms. Check out the full simulation experience here: https://lnkd.in/d7NmHzRd #AWS #SolutionsArchitecture #CloudComputing #Forage #ScalableSolutions #Innovation
To view or add a comment, sign in
-
Exciting S3 announcements to scale your storage architecture.
Senior Developer Advocate @ AWS, ex AWS ML Hero- Industry Mentor & Speaker across Data, Design and Strategy
We've had buckets of announcements this week, and I'm sure lots of you have eleven 9's of excitement about this one! 🪣✨ AWS has expanded S3 bucket scaling in a big way - here's what you need to know: S3 Bucket Scaling: What's New • Default quota now 10,000 buckets per account • Request increase up to 1 million buckets • Applied to all AWS Regions • No action needed for default increase Practical Benefits for Developers • Create buckets for individual datasets • Implement granular security policies • Enable default encryption per bucket • Use S3 Replication more easily Optimize Your S3 Architecture • Separate concerns with dedicated buckets • Improve data isolation and management • Remove barriers to scaling • Enhance overall storage architecture Cost and Scaling Considerations • First 2,000 buckets at no cost • Small monthly fee for buckets beyond 2,000 • Use Service Quotas for limit increases • New default applies to all AWS accounts Read all about it: https://lnkd.in/e2BPCnpy Could this S3 update solve any challenges in your current projects? Tell me all about it in the comments below 👇 👩🏻💻 Follow me (Brooke Jamieson) for the latest AWS + AI updates. 📍 save + share! #AWS #AmazonS3 #CloudStorage #CloudComputing
To view or add a comment, sign in
-
A good 101 course on AWS WAR basics and benefits within organisations.
To view or add a comment, sign in
-
🚀 Building Event-Driven Architectures with AWS 🌐 Excited to share a recent project showcasing a serverless event-driven architecture using AWS! Here's how it works: ✅ Amazon S3 acts as the event source, triggering actions whenever events (like file uploads) occur. ✅ An AWS Lambda Function processes these events with custom logic, enabling seamless automation and data handling. ✅ IAM ensures secure and fine-grained access between services, keeping the workflow secure. ✅ Amazon CloudWatch Logs provides monitoring and debugging capabilities, ensuring the system runs smoothly. This architecture is scalable, cost-efficient, and highly responsive, perfect for real-time workflows like file processing, data analytics, and automated notifications. A special thanks to my mentor, QASIM HASSAN, for his invaluable guidance and support throughout this project! 🙏 📈 Ready to scale your operations with serverless technology? Let’s connect and discuss how! 🌟 here is my youtube video in which I completely explain what we do in this project: https://lnkd.in/d-pNPtFD #AWS #Serverless #CloudComputing #EventDrivenArchitecture #Automation
To view or add a comment, sign in