Anderson Chaves’ Post

How does Dask compare to Spark? We’ve re-run the TPC-H benchmarks and noticed some interesting results. Dask is often faster than Spark, both locally at 10 GB scale and on a 10 TB dataset on the cloud. Dask more reliably completed the queries, especially at large scale. PySpark failed or ran into an hour-long timeout for several queries that Dask successfully completed. We’re not Spark experts though, and would welcome critique here. We talk through these results, and other ways in which Spark and Dask differ in more detail in our blog post: https://lnkd.in/gTXynzrT

🚀 Excited to share my latest Medium post on "Overcoming AWS Step Function Payload Limitations"! Discover how we tackled large-scale data processing with AWS Step Functions and Amazon S3, ensuring efficiency and scalability in the cloud. 🌐 👉 Read the full story here: https://lnkd.in/eSC2d7pc ✅ Find out how to streamline your workflows and embrace the challenges of cloud computing. If you enjoy the read, please clap, comment, and share! #AWS #CloudComputing #DataEngineering #TechInnovation

Lightning-fast performance meets stellar recall rates with the power of the cloud. 🏃☁️⚡️ https://go.aws/4bIEpEf Amazon MemoryDB, an in-memory database with native vector database capabilities, delivers the fastest vector search performance at the highest recall rates among popular vector databases on AWS. In the race for speed, relevance, and performance, #AWS lets you have it all! #AWSSummit

What if I told you that increasing memory allocation in AWS Lambda could actually lower your costs? 👇 It sounds counterintuitive, but in some scenarios, it's the key to supercharging your performance while cutting down expenses. In my latest blog post, I reveal a real-world example where doubling the memory slashed processing time by over 50% and reduced overall costs by 17%. This isn't just theory — it's a strategy you can start using today. 👉 Curious to learn how? (Link in the first comment) 👈 #AWS #cloud #finops

Infrastructure as code in simple? Yes, that works! 💡 What sounds like an oxymoron has become a reality since the latest changes to AWS' IaC Generator. Read our blog post, written by our Strategic Digital Architect AWS Kevin Feuer, to find out how you can easily implement #IaC on #AWS, even if your workloads are already in the cloud. Time for a change of perspective. 🔄 Read our blog post here: https://bit.ly/49OjW0g

Amazon Web Services (AWS) Graviton4 new x8g instance. A very interesting blog article. "Here’s a long-term snapshot from one of the clusters that was used to support development of Graviton4 in late 2022 and early 2023. As you can see this cluster runs at massive scale, with peaks as high as 5x normal usage" EDA in the Cloud. Graviton on Graviton. Awesome. https://lnkd.in/eRNtZzu3

As I was alluding to yesterday, here's how brain-dead easy it is to have #duckdb query #gcs...literally less than 10 lines of code. I'll have a substack article coming out soon that shows a good bit more on how to read/process/write back to GCS using your local compute. I think this is a good way to handle small-to-medium datasets and save on your cloud compute bills 😁 #dataengineering

Did you know DynamoDB had a self destruct feature? TTL lets you do that. You choose when you want the item to be deleted in the time (UNIX epoch) specified along with the item in the specified TTL attribute. Wanna know more? Article link in the comments. #dynamodb #aws #cloud

New Vecima White Paper -- Through a collaborative effort between Vecima and AWS, the Entra vCMTS has been successfully trialed in the Cloud, offering operators the flexibility to choose from multiple deployment models that suit their specific needs. Read Douglas Johnson's white paper to learn more. https://lnkd.in/e8GM4Twi