Jack Liu’s Post

Combining fewer components to build your application can save you time but may not always be the best approach. API Gateway on AWS has a number of direct integrations with other services but you may need to write mapping templates using Velocity Template Language (VTL). You can combine most AWS components using Lambda functions and write the code needed to transform data if needed. Some components allow you to more directly interact without having a Lambda function. The example below from Khai Bui shows how you can directly integrate API Gateway with DynamoDB but you have build mapping templates to make it work. You are using fewer services but you have to determine if having those VTL templates is worth it for the overall solution. https://lnkd.in/e-5ZMH9P

Many AWS services support direct integrations with other ones. In the cases where this isn’t supported people usually put an AWS Lambda function in between. When API Gateway is used and needing to interact with other services directly you can use Mapping Templates which are blocks of Velocity Template Language (VTL) code. Many people try to avoid VTL but if the logic isn’t too complicated it is worth trying out in many cases. This article from Arpad Toth shows how you can interact directly with DynamoDB from API Gateway without having a Lambda function in the middle and improve performance. https://lnkd.in/ezhNw4Gi

How to build an end-to-end content repository using AWS services with a simple and dynamic access control based logic over unstructured data. https://lnkd.in/gh-JBNuk

How to Manage Large Data Writes in DynamoDB #aws #dynamodb #awsdynamodb #serverless

Excited to see published this two-parts blog post I co-wrote with my colleague Stefano Sandona'! We describe in detail how to implement AWS Lake Formation third party services and custom applications integration. In part 1, we lay out the theoretical background and create a sample analytical engine with AWS #lambda. In part 2, we create and deploy a web application as the front-end for the engine, we use AWS #amplify with a GraphQL API implemented with AWS #appsync. Full code is provided, so you can try and test the solution in your accounts. AWS Lake Formation third-party services integration is perfect for customers and partners alike that want to leverage AWS data platforms' governance and security layers. part 1: https://lnkd.in/dWURvipZ part 2: https://lnkd.in/dAxfaNhu Let us know what you think :)

🔥 Logical operators and conditional expressions for DAG scheduling 🔥 Combination of Dataset and time-based schedules 🔥 REST API endpoints to programmatically create Dataset events 🔥 Custom names for Dynamic Task Mapping and much more as #Amazon #MWAA announces support for Apache Airflow 2.9.2 environments. What makes this release more special for the MWAA team is that they're also open-sourcing MWAA's Airflow image. You can use it locally if you want to run a MWAA-like environment for testing, experimentation, and development purposes. Give it a try here: https://lnkd.in/eKh39EzT  Read the launch blog post Parnab Basak and I wrote: https://lnkd.in/ecPKhkkD Kudos to the team!

𝗠𝗮𝗽𝗽𝗶𝗻𝗴 𝗬𝗼𝘂𝗿 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 𝗙𝗲𝗲𝗹𝘀 𝗟𝗶𝗸𝗲 𝗮𝗻 𝗜𝗻𝘀𝘂𝗿𝗺𝗼𝘂𝗻𝘁𝗮𝗯𝗹𝗲 𝗧𝗮𝘀𝗸? If you've ever tried to map your network, you know how daunting it can be. Here’s what the process typically looks like: 1. 𝗦𝗰𝗼𝗽𝗶𝗻𝗴 𝘁𝗵𝗲 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁: Are we mapping just a single Kubernetes cluster, or multiple? Do we need to include Kubernetes-to-AWS calls? 2. 𝗜𝗱𝗲𝗻𝘁𝗶𝗳𝘆𝗶𝗻𝗴 𝗿𝗼𝗹𝗲𝘀 𝗮𝗻𝗱 𝗽𝗲𝗿𝗺𝗶𝘀𝘀𝗶𝗼𝗻𝘀: Should we map just the principals and services they use, or do we need a detailed, per-resource mapping? Did we even configure principals for our workloads? (I remember seeing that one prod cluster where all services share the same principal...) 3. 𝗔𝗹𝗶𝗴𝗻𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: Do these principals match the architecture map we built last year? (Because those names like “checkout-service-prod-us-east-1” sure look distinct, but when did we deploy the checkout service on us-east?) 4. 𝗖𝗼𝗻𝘀𝘂𝗹𝘁𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀: Should we ask the developers to help with that? Wait, that could take ages. 5. 𝗥𝗲𝘃𝗶𝗲𝘄𝗶𝗻𝗴 𝘁𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁: After two months of back-and-forth, we finally have a network map that’s already outdated as soon as it’s ready. 𝗘𝗻𝘁𝗲𝗿 𝗢𝘁𝘁𝗲𝗿𝗶𝘇𝗲 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 𝗠𝗮𝗽𝗽𝗲𝗿. Otterize takes the complexity out of network mapping with zero configuration. Here’s how it helps: • 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗰 𝗠𝗮𝗽𝗽𝗶𝗻𝗴: Otterize automatically maps your network traffic into human-readable graphs and data. • 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗶𝗮𝘁𝗶𝗻𝗴 𝗪𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀: Otterize distinguishes between workloads sharing the same principals (e.g., two pods using the same AWS IAM Role). • 𝗙𝘂𝗹𝗹-𝗦𝗰𝗼𝗽𝗲 𝗠𝗮𝗽𝗽𝗶𝗻𝗴: Otterize maps calls to the major clouds (AWS/Azure/GCP), databases (MySQL, PostgreSQL), message brokers (Kafka), and more. Stop struggling with outdated, incomplete network maps. Let Otterize do the heavy lifting, so you can focus on what really matters.

🐾Week 66: DynamoDB: how to choose right capacity🐾 🤓 While working with clients, I noticed that many of them tend to use DynamoDB provisioned capacity. Their motivation is to pay less, because they afraid of spikes and uncontrolled scaling events. It’s sounds fair as provisioned capacity sets a cap on how much capacity units (CU) you will use, and therefore it sets a cap on your bill. But the reality is that for most cases you will spend much more with provisioned mode. Let’s dive deeper into DynamoDB capacity modes and pricing. Full post: https://lnkd.in/dJKG_PCr __________ 💬 Anything you would like to add or ask? Let me know in the comments 🔔 If you like this post, you can follow me on Substack for more: https://lnkd.in/dMcCFRep #aws #awscommunity #data #machinelearning #mlops #database #dynamodb

Mixing things up in this post, I talk about a fun debugging problem me and the team worked on where we had inconsistent failures when inserting records into DynamoDB. https://lnkd.in/e-rgf-U9 How would you have troubleshot the issue? Would you have done anything differently? #aws #debugging# #readthedocs

Optimizing Search Queries with Shards in AWS Open-Search Service Imagine you have an index containing 100,000 documents, with an average document size of 1KB and your goal is to achieve search latency 30 ms or less. To accomplish this, you can leverage the power of shards in Open-Search. By distributing the data across multiple shards, you can parallelize search queries and reduce latency. Here's how you can optimize your shard config: 1. Determine the number of primary shards, ex. split your index into 5 primary shards. Each shard will hold approximately 20,000 documents, ensuring balanced data distribution. 2. Set config of replication for high availability. Set up a replication factor of 1 - replicas improve both fault tolerance and search availability under load.. 3. Analyze search performance. With 5 primary shards and 1 replica shard, Open-Search can process search queries in parallel across all shards. If each shard responds in 6 ms, and you have five shards, the query time should remain close to 6 ms because the query runs in parallel across the shards. 4. Monitor and optimize - As the data grows, keep an eye on search performance and shard utilization. By adjusting primary shards or increasing replica count you can scale for higher query loads and availability. So, next time you're working with AWS Open-Search Service, consider how you can harness the potential of shards to optimize your search queries and deliver lightning-fast results! Keep Learning, Keep Solving :) #AWSOpenSearchService #Shards #SearchPerformance