Michael Antonio’s Post

Last week I ran a hands-on workshop looking at the AWS Fault Injection Service, which is a chaos engineering tool that allows you to setup and run experiments to test how resilient your infrastructure in an AWS account (or multiple accounts) is. The recording is available at https://buff.ly/3CihgMT and the code we used in the workshop is at https://buff.ly/4hFIX2r #AWSCommunity #DevopsPlayground #ChaosEngineering

LogProcessing with AWS StateMachine https://lnkd.in/dp26myKv This application helps in resuming AWS StepFunction stuck because of some issues i.e., downstream application not available or the process has some issue which needs to be fixed to resume the Step Function or if some process needs human approval. The video explains in detail.

Optimizing AWS Step Functions: A Case Study on Reducing Latency and Cost Preview: In the world of software engineering, scalability and efficiency are paramount. One of the challenges I faced in my recent project was managing a rapidly growing user base and the associated AWS Step Functions events. This blog post details my journey in optimizing our system, reducing latency, and cutting costs. I'll also explain how I utilized AWS Database Migration Service (DMS) and Kinesis to ensure real-time event processing. Key Highlights: Reduced event count by 3000% Significant cost savings Implementation of a scalable, loosely coupled system To read more about the detailed implementation and lessons learned, read the full post on Medium.

Annnnd AWS Step Functions released workflow variables for storing state level variables. Learn more: https://lnkd.in/gaSQnP6n

What a great tool Datadog has released for detection engineering - it just works! Try it out: $ export AWS_REGION=us-east-1 $ grimoire shell -c "aws s3 ls" -o /tmp/logs When you see "Found event", press Ctrl+C. $ cat /tmp/logs Profit! #detectionengineering #Grimoire

Patrick Ames from AWS has more experience than just about anyone with managing thousands of Ray clusters and Spark clusters and processing exabytes of data. There are tremendous operational and performance challenges. - managing thousands of clusters - scaling to many petabytes or exabytes - latency and cost The original blog post is here: https://lnkd.in/g-pJhFei https://lnkd.in/gQsCrXPH

Another great Meetup for the books 🙌 A big thank you to 👨💻 Eugene Tolbakov for the great presentation, to Duco for hosting us, and to Axiom for sponsoring! You can find the recording from our last event here: https://lnkd.in/eCmVU674 In this talk, Eugene discusses the process undertaken by the Observability team at Chase UK to manage AWS OpenSearch clusters effectively. Utilizing Infrastructure as Code(Terraform), they have streamlined cluster management for efficiency and ease. He elaborates on their approach for defining index templates and patterns, configuring roles, and leveraging ingestion pipelines to streamline cluster management. Eugene also outlines the enhancements they've implemented to ensure a stable platform and enhance the overall observability experience and share key insights and learnings from their journey toward operational excellence with AWS OpenSearch management. #observability #devops #sre #aws #opensearch

I recently assisted a colleague of mine with migrating an existing process to AWS, which involved understanding of the application’s system design. Also helped in optimising a long running process just by a choice of using Segmented Tree with node having 2 values with- (1st referencing to the calculated interval and 2nd for a reference to original index) instead of an alpha-numeric ID mapped as a dictionary, reducing the execution time for ~190K records from O(n^2) to O(nlogn) (including the AWS Glue instance invocation time). I believe the choice to right data structure is crucial for improving execution time in such scenarios.

AWS has announced two new features for Step Functions that aim to improve productivity and developer efficiency: Variables: Allows developers to assign and reference data across different states, streamlining state payload management. JSONata data transformations: Enables advanced data manipulation and transformation, including date/time formatting and mathematical operations. These additions to AWS Step Functions are designed to: 1/ Simplify the development of distributed serverless applications 2/ Enhance payload management 3/ Reduce the need for custom code. Here are the links to learn more: https://lnkd.in/gtx4aH6m https://lnkd.in/ganzc3YD #aws #serverless #stepfunctions

We are also glad to have worked with the Datadog team to ensure that their instrumentation fully support the new JSONata payloads for standard tracing. Also, with JSONata, you can now get deeper trace context when using AWS Step Functions. Here is the blog post from Datadog to learn more: https://lnkd.in/gxz2_fWd