Zhaofeng Li

🎉 Big congratulations to our incredible team (Haoyu Chen, Nick Greenquist,Yuxing Wang, Sridevi Maharaj, Ruiyang Wu and many more) for delivering several quarters-long impactful features on the Play Store! 🚀 We’re thrilled to have launched key updates like Search Home, Sports Moment, and WearOS recommendations, all designed to enhance personalization and help billions of users worldwide search and discover content they love. 🌍✨ Check out the Play Store Q3 updates: https://lnkd.in/dn5wx8Mi #GooglePlay #Personalization #Recommendation

73

1 Comment

Understanding Develocity Build Data with Honeycomb https://lnkd.in/gmGSrkVB

4

🚀 Exciting News Db2 Users! 🚀 ⌚ Amazon Web Services (AWS) has introduced on-demand Db2 licenses through AWS Marketplace! 🕺 This innovative solution enables Db2 users to procure database licenses for RDS Db2 workloads with ease. #AWS #AmazonRDS #DB2 #DatabaseManagement #Innovation #CloudComputing #TechNews #AWSUpdates

9

Recently, Amazon Q Developer announced expanded support for account resource awareness with Amazon Q in the AWS Management Console and the AWS Mobile Application. Now, from the Amazon Q tab in the AWS Console Mobile App, customers can ask Q questions like “list my running EC2 instances in us-east-1”, “list my S3 buckets”, and “what related resources does my ec2 instance <id> have?” and Amazon Q will respond with their specific resource details on their mobile devices from anywhere. #aws, #awscloud

10

SWE job growth declined for the first time since 2000 The SWE market has had tremendous growth. Though now is the first time in two decades we’re seeing shrinkage in the number of job openings. Gergely Orosz wrote a wonderful, in-depth article on the state of the SWE job market. Learn more in the latest issue: https://buff.ly/4f5ldDe If you’re new to my content, every week I share a spreadsheet of recently posted, hand-picked software engineering roles to make your job search a little easier. To get the weekly jobs via email, subscribe here: https://buff.ly/4hkLOh0 Photo: Live Data Technologies

7

1 Comment

I'm thrilled to see Amazon Q generally available. Amazon Q has revolutionized my work with Amazon Connect (contact center technology), empowering me to effortlessly build a sophisticated agent assistant and seamlessly integrate it with the Bedrock database, enabling powerful Retreive-Augment-Generate (RAG) Generative AI use cases. This powerful tool has not only simplified complex integrations but also enhanced efficiency. Additionally, I use Amazon Q as a personal assistant during my whiteboarding sessions, providing instant answers and insights about AWS Cloud services. The ease of use and robust capabilities of Amazon Q exemplify the pinnacle of modern technology, driving innovation and optimizing customer experiences. #TechInnovation #AWS

20

Come join us tomorrow in SF to learn more about the latest updates to run AI workloads on Google Cloud!

11

1 Comment

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

Waymo is probably both more expensive and longer waits on average than Uber or Lyft today, but they are close enough, and rapidly getting better. These markets will flip to fully autonomous when the tech is ready.

1

Meta's taking AI hardware to the next level! 🛠️ With our commitment to open and adaptable AI infrastructure, we're working to make cutting-edge technology accessible to all. Check out how these innovations are paving the way for an inclusive AI future. #AIForAll #MetaEngineering #TechForGood

1

Two facts: First: Jack and the Storefront team handle some of the most interesting problems at Recharge if you're into caching, synchronizing, and lots of data puzzles to solve. Second: I work with the team as Principal so we'd get to have some fun while we do the first part.

17

Interesting article from Bloomberg about life at Google's X in a time of layoffs and cost reduction at https://lnkd.in/gmw3GG7C. X is a component of Alphabet (Google's parent company) that incubates new ideas. They have been responsible for some great successes - getting Waymo and a lot of Google's AI work started - but they've also had projects that didn't survive - like Google's attempt to provide Internet access via weather balloons. When you work at a company that has hundreds of billions of dollars in revenues, it's hard to get a lot of interest in an entirely new business idea that brings in hundreds of millions. If you have a project that improves ads revenue by that amount, or decreases capital expenditures, you can do it, but if you want to, say, build wearable tech, it becomes hard to attract the attention of marketing or product. So, new ideas often die on the vine. For the most part, this is good and proper. Companies may have the financial wherewithal to branch out, but more often than not, they consider it a distraction. And they are probably right - it's hard to argue that Meta was right to divert such a huge chunk of their resources to chase after the metaverse. They've (rightfully) pulled back, and are more focused on their core business again. On the flip side, if you are in the tech sector, and you don't innovate, you are likely to get left behind at some point. So, you need to make and manage these investments somehow. When I joined Google, the CEO had a 70-20-10 rule - 70% of effort went to core businesses, 20% to broader efforts to improve Google, and 10% to stuff that's completely blue-sky. This is probably not completely right, but it is also probably not completely wrong. X is the division of Alphabet that is *specifically* designed to bring blue-sky products to life. Unsurprisingly, they've had to pivot and rethink their strategy many, many times. The article interviews a number of people who have worked at X over the years; it's well worth a perusal, for folks who are interested in the story.

18

2 Comments

Recently, I came across an article about Amazon Prime shifting from microservices to a monolith. While it amazed me at first glance, it has also reportedly saved 90% in architecture costs. This struck a chord with me because it echoes the challenges I’ve faced in my own journey with microservices. ✅ Advantages I’ve seen with microservices: - Scalability - Flexibility - Fault isolation ❌ Challenges I’ve faced: - Increased complexity - Operational overhead - Data consistency issues There were moments when I questioned the path I was taking, following microservice architecture because it was widely regarded as a "best practice". And yes, statistics backs up this statement. This Amazon Prime example reminds me of an important lesson. There is no one-size-fits-all solution. Architectural choices should always align with the problem at hand, not just because they are old or considered the best. If you want to know more about what happened at Amazon Prime, go ahead and check out this article : https://lnkd.in/gtPgZf4J

12

1 Comment

Managing and delivering a robust CI/CD pipeline at scale is a complex and evolving challenge. The term “CI/CD” holds varied meanings across different work cultures, reflecting diverse priorities and practices. As the technology landscape continues to grow, CI/CD has become just one component of the broader and increasingly significant developer enablement ecosystem. Read more at https://lnkd.in/gPwGFrTk

12

Coffee & Systems (Study Group - Bengaluru) Week - 2 (Streaming Processing - Part 2) September 21, 2024 4:00 PM → 5:30 PM - Starbucks RMZ Ecoworld --- NOTE - Registration now requires approval as event space is limited. I will try to bulk approve at the end of the day, prioritising folks who have attended the previous session/who I feel can contribute to the discussions(mostly based on YEO, Job Profile) --- Week 2 is more of a continuation of the first week and we dive further into stream processing. For this, we will mostly focus on Streaming 102. The article is quite dense and forms the base of thinking about correctness, latency and the tradeoffs in streaming systems. We will also briefly see how Apache Flink implements these concepts, tying theory with practice. - Streaming 102: The world beyond batch - https://lnkd.in/g8mxR_tj - Timely Stream Processing in Apache Flink - https://lnkd.in/gik3qA2h - (Optional) Stateful Stream Processing in Apache Flink - https://lnkd.in/gPzjzYPy Going through the materials presented in the previous week (especially Streaming 101) is thoroughly recommended before starting on the articles listed above

61

7 Comments

🏋️‍♂️ DoorDash's Microservice Challenges 🏋️‍♂️ In early 2020, DoorDash faced rapid user growth, expanding both in terms of customers and operations. Their existing Python monolith backend struggled to keep pace with this scale. To handle the increasing complexity and demand, DoorDash transitioned to microservices. However, with this transition came a series of unexpected failures that are common in distributed systems. 🚨 Failure Types Encountered 1️⃣ Cascading Failure: DoorDash faced cascading failures where upstream services were impacted by the downstream ones. The issue occurred when downstream services had different latency expectations, leading to upstream bottlenecks. 2️⃣ Retry Storm: Transient failures triggered a retry storm where multiple clients, encountering a failed service, retried simultaneously, overwhelming the system and creating a thundering herd scenario. 3️⃣ Death Spiral: When one instance behind a load balancer failed, latency in bringing new nodes online caused remaining nodes to become overwhelmed. As nodes continued to drop, this resulted in a death spiral where the load crippled the entire system. 4️⃣ Metastable Failures: Once DoorDash’s system entered a failure state, even after conditions normalized (such as traffic surges decreasing), the system remained in an unhealthy state and required manual intervention to recover. ⚙️ Countermeasures Implemented To address these issues, DoorDash employed the following strategies: ✅ Load Shedding: Prioritized higher-value requests when resources were under heavy load. ✅ Circuit Breakers: At the consumer service level, decisions were made to limit requests based on local or global resource load, letting only critical requests pass through. ✅ Auto Scaling: Used predictive auto-scaling to horizontally adjust resources according to expected traffic patterns, allowing for more efficient handling of traffic surges. ⚠️ Shortcomings of Local Countermeasures ➜ While effective in some cases, the local countermeasures had limitations: They only responded to the health of local nodes, but many outages impacted the entire system globally. ➜ Local metrics (e.g., memory, CPU usage) didn't account for the latency caused by dependencies on downstream services. ➜ Overall, these remediations proved insufficient for broader system issues. 🌍 Enter Aperture: Globalized Control for Reliability ➜ To move beyond these local fixes, DoorDash adopted Aperture, an open-source reliability management system. Aperture introduces a global reliability control layer, moving beyond local heuristics and improving system stability by considering the entire architecture. With global visibility, Aperture can prevent cascading failures, retry storms, and other systemic issues across DoorDash’s microservices. For more interesting engineering stories, follow me, Abhilash G Raja #Microservices #Cloud #DistributedSystems

12

1 Comment