Building the future of AI infra at NVIDIA. Apache Software Foundation Member; Former Head of Open Source (Ray) + Head of Field Engineering @ Anyscale.
I'm really excited to share this new blog on Amazon's Exabyte-Scale migration from #Spark to #Ray! 🐘 1.5EiB of data processed in a quarter 🚀 82% better cost efficiency 💸 $120M saving per year OK you might ask: isn't #BigData processing Apache Spark's bread and butter? 🤔 Well, this is a story about software abstractions.. Like Patrick Ames wrote in the blog, "... (Amazon engineers) had limited options to resolve performance issues due to Apache Spark successfully (and unfortunately in this case) abstracting away most of the low-level data processing details". So, the takeaway: if you need more flexibility in data processing (e.g. need GPU, or #unstructured data like video), let's talk! https://lnkd.in/gesSyxHF to get started https://lnkd.in/gfj2Xp_2
Akshay Verma
4mo
Really helpful!
Malathi Sankar
4mo
Fascinating read. Thanks for sharing!
Pritam Pan
4mo
This is another excellent use case for Ray beyond machine learning. Fascinating read!
Hire FAANG talent on Discord 🕹️ | Trusted by top VC backed startups | Send me a DM for access 👋
4mohttps://discord.gg/learnmutiny