Ryan Peterman

Did you know there are no "roles" (SDE, SDET, TPM, …) when you get to L8 IC (Senior Principal) at Amazon? It’s one of Amazon’s peculiarities. [Side note: in general, Amazon SDEs tend to be "jack of all trades" where we pragmatically do whatever is necessary to make our organization successful at that given moment (within reason). In fact, I believe being well-rounded and able and willing to wear a number of hats is one of the things that makes me valuable to an employer. We don’t over-index on "that’s not part of my role doc." Other companies where I worked (eg Google) have very clear swimming lanes and people stick to them pretty religiously, eg. "oh no I don’t talk to customers, I’m an SDE and talking to customers is for TPMs." Whereas at Amazon I never once had a manager tell me, "what you did (although impactful, important and necessary) was not really your job." ] My little side note about this critical piece of amazon culture explains why as you get to L8 the concept of a "role" no longer really makes sense. The boundaries between roles get fuzzier and fuzzier as you go from Senior Engineer (L6) to Principal Engineer (L7), and they become unimportant when you get to L8. We’re all just “Senior Principals” - whether you came via the SDE path, the TPM path, or in my snowflake case, the SDET path. In fact I think most L8s are snowflakes in that their path to L8 was unique and the value they bring to the company is also unique. In phonetool (our amazon directory) you can have whatever unofficial title you want (also within reason...), so I chose to keep the "SDET" title to reflect my background and passion for testing, engineering and operational excellence, and developer tools. I also hope that by keeping the title, others in my (old?) role can see that SDETs at Amazon do not have a "ceiling" in their career. And lastly, it's a fun conversational piece ("you're an L8 SDET???") (The image here is "Peccy" - a beloved little icon that has represented Amazon's Peculiar Ways for years) [Edit: Strictly speaking there are a few L8 IC roles that do have a different job code, like PMT or Scientist - but I think we generally view all L8 ICs in the same lens]

929

41 Comments

Amazon-to-Google-to-Amazon #a2g2a thought of the day: the way Amazon thinks about its Principal Engineers is unique - and I love it. “I always thought of you (and my other Principal Engineers) as Amazon employees that were on loan to me.” Yesterday I was catching up with my good friend and ex-manager Dan Sommerfield. Dan promoted me to Principal Engineer back in 2014 and was my manager for a number of years. As we were laughing about some of our shared past, Dan said that and it stuck with me. A few weeks ago I posted in my a2g2a series about Swimming Lanes. I’ve also posted in the past about the Amazon Principal Engineering community (see links to both of those in my comments). Today’s reflections combine those. Amazon Principal Engineers tend to have a healthy disregard for organizational and role boundaries. And Amazon’s DNA gives them the freedom to do so. At these big faang companies, for better or for worse, most orgs tend to optimize for themselves. It makes sense in terms of business needs: they need to move fast and expand their business. But this is often how we end up with duplicative work siloed usually at Director level. I’ve seen, time and time again: Principal Engineers look across organizations and connect dots that others don’t. The Amazon Principal Engineering community is a giant organic mesh. It worked incredibly well as a big mesh when I became a PE in 2014 and we were about 250 PEs in the entire company, but I’m continuously amazed that it still works in 2024 at today’s scale. This is a testament to the culture that some of our founding figures set decades ago- and our Distinguished Engineers continue to carry the torch today. It’s not that this doesn’t happen in other companies; it did happen at Google and Microsoft too. But it’s so intrinsically embedded in the DNA of an Amazon Principal Engineer that it’s a lot more natural and expected here than anywhere else where I’ve worked or considered working.

592

24 Comments

Next week, I will be the Principals of Amazon (PoA) talk presenter. It would be fun to have folks join (and ask good questions!) if you are an Amazonian.   Talk Abstract: Code reviews can be a painful ordeal, leaving both reviewers and authors feeling frustrated. But it doesn’t have to be that way! In this practical talk, we’ll explore techniques to transform code reviews into an enjoyable learning experience. You’ll learn how to give and receive feedback constructively, focusing on the code rather than personal attacks. You’ll gain strategies for structuring feedback, addressing comments, and handling disagreements without devolving into office brawls. Whether you’re a junior or mid-career developer, this talk will equip you with the skills to make code reviews a collaborative, educational, and (gasp!) enjoyable process. Join us to get a stress-free ‘ship it!’ on your next CR.   Here is a sneak peek at what we will cover. You might notice a few Principal Tenets in here. I use these daily in my work, in addition to the more famous Amazon Leadership Principles. The Principal Tenets are a framework for thinking about my work as a Principal Engineer and always raising my game. You can learn more about them here: https://lnkd.in/esPSssWv   Code Reviews that Don’t Suck   Learn, Educate, and Advocate You are NOT your Code Start with a Self-Review Consider Team Standards Check the Obvious Things Make it Atomic / One Change per Commit Respect what Came Before Make it Readable Provide Clear Context Go Talk to a Person, in Person Work Hard; Have Fun; Make History   How about giving Code Review feedback that doesn’t suck? Lead with Empathy Don’t be too Harsh But Be Clear Use Radical Candor Give a Full Review Be Prompt… or Delegate Act as a Co-Owner Coaching a New Engineer Hold High and Consistent Standards Use Comment Conventions Include Praise

300

9 Comments

What's the worst quality in an engineering manager? In my opinion, it's not wanting to be a manager. Too often, technical people end up in management roles because it's the "next logical step" in their careers. While some of these new managers thrive, many of them wish they were solving technical problems instead of people ones. Staying in this role is a disservice to yourself and your team. There are many ways to advance your technical career without being an engineering manager. For an interesting story about how an EM at Amazon "stepped back" into building, check out the latest blog post from DevPath: https://educat.tv/3VlczIZ

16

Am I Doing “No Real Work”? Are You? Alright, so just to start: this isn’t like, defending Google specifically. The author especially targets Google, but condemns technology companies overall. Are there people who “aren’t contributing” or not doing so at the same level as peers? Yes, that’s absolutely true. Maybe it’s situational and can change, maybe it’s terminal and they aren’t a good fit. That is not really what the article is about, though or not exclusively. My main takeaway was that we are working on things that don’t make money. We build things that never launch. We research things that aren’t profitable. The author seems to have run a successful, technical business. Maybe it had laser focus on one product, and what to build was well understood. This isn’t always true. Technology is a new application of scientific understanding. We don’t always have the scientific understanding we need to build a particular technology, and they require spending on research. Sometimes a promising research project can bear no fruit. Key learnings can come from this, unrelated projects can be improved, research techniques can be made more robust. Investing in research is not “wasting money” that could be “returned to stakeholders”. If human hours spent must equate directly to revenue, don’t invest in companies that invest in research. Invest in businesses with billable hours like law, or physical product production where hours on the line yield salable products. I think it’s really unhelpful to conflate “wasting money” on “unproductive workers” and a company spending money on R&D, which is inherently risky. Google Cloud was not well-understood, or a guaranteed success. It yields billions of dollars now. That was one bet. You can debate how many bets should be made, and at what scale, but we are at a stage where, to move the needle at a company this size, the bets have to be very big, and made fairly frequently. Standing still will not mean we keep making the same amount of money so we can “return value to investors”. I do think in the “white collar”, big tech environment, there is a lot of “silent value”. Maybe that does end up yielding nothing for shareholders. Training and guiding more junior engineers so they can contribute to more complex projects has a very ambiguous value. They can leave. They can never really uplevel skills. They may be tasked with working on something with “no shareholder value”. The cost of not doing it is skill and brain drain, and an inability to continue innovating. If innovation stops, the world moves on around you and shareholder value drops to zero. Mostly I think while I don’t mean specifically to defend Google (because most big tech make these kind of investments), I may have taken this personally. I am contributing to shareholder value? When I turn off this 15 year old system, does the stock price go up? No, but it frees us up for faster innovation. Is it worth what I get paid? That’s a very different question.

69

15 Comments

Distinguished Engineers (E9) at Meta make $4m/year. That's 16x the Bay Area median ($255k) Let's break it down [output = input x leverage] We all have 24 hours in a day, so there is a cap on [input] But hey, let's say they manage to have 2x-3x more focused time, so their [input] is 2x-3x There is still 5.3x-8x to go And that's [leverage] Here is the [leverage] they have: 1️⃣ Writing Every piece of text they write is clear to everyone, and they write a lot, for a lot of people 2️⃣ They create something from nothing And that something has clear value to the business 3️⃣ They influence other Engineers around them They are often described as having a "Midas Touch" in terms of their ability to uplift and develop other Engineers around them The best indicator is that the Engineers in their teams get promoted faster 4️⃣ They are well known and have an impact beyond Meta, promoting the brand 5️⃣ They have industry-leading experience in some areas There are rumoured to be <50 E9s at Meta

13

2 Comments

🚀 **System Design Spotlight: The Power of Event-Driven Architecture** 🚀 As a senior software engineer with the privilege of having worked at Google and Microsoft, and having taught thousands of engineers at Scaler, I’m constantly fascinated by the innovations in system design that drive our industry forward. Today, I want to share insights on one of my favourite topics: **Event-Driven Architecture (EDA)**. ### Why Event-Driven Architecture? In today’s fast-paced tech world, the ability to process and react to events in real-time is crucial. EDA decouples event production from event consumption, leading to highly scalable and maintainable systems. ### Key Benefits: 1. **Scalability**: Components scale independently, essential in microservices architectures. 2. **Resilience**: Asynchronous event communication enhances system resilience. If one component fails, others continue operating. 3. **Real-Time Processing**: Ideal for applications requiring real-time data processing, such as financial systems, IoT applications, and social media feeds. 4. **Flexibility and Extensibility**: New features or services can be added without disrupting existing ones. ### Practical Example: Consider a modern e-commerce platform. When a user places an order, several events are triggered: - **Order Placed**: Inventory updates. - **Payment Processed**: Accounting records the transaction. - **Order Shipped**: Customer notification. Each event is processed by different microservices asynchronously, ensuring a seamless user experience. ### Tools of the Trade: - **Apache Kafka**: Handles high throughput with distributed streaming. - **AWS Lambda**: Executes code in response to events in a serverless environment. - **RabbitMQ**: A robust messaging broker for reliable message delivery. ### Conclusion: Event-Driven Architecture is a game-changer for building scalable, resilient, and flexible systems. It’s a powerful paradigm that aligns perfectly with modern software development demands. Whether working on microservices, IoT, or real-time data processing, EDA offers a robust solution for complex workflows. If you’re passionate about system design and want to dive deeper into EDA or other advanced topics, let's connect! I love sharing knowledge and learning from this incredible community. 🌟 #SystemDesign #SoftwareEngineering #EventDrivenArchitecture #Microservices #TechInnovation #Scalability

99

2 Comments

🔍 Demystifying Distributed Systems: Raft's Secret Sauce for Reliability As an engineering manager, I've seen firsthand how critical consistency is in distributed systems. Today, let's unpack the Raft consensus algorithm's approach to leader election and consistency. Trust me, it's simpler (and cooler) than you might think! 🎭 The Players: Leaders and Followers Imagine a high-stakes game of "follow the leader" where servers are the players. In Raft: One server is the leader The rest are followers But what happens when the leader stumbles? 🏁 Leader Election: A Race Against Time The leader sends regular heartbeats to all followers If followers don't hear a heartbeat, they call an election Followers nominate themselves and request votes The first to get a majority vote wins and becomes the new leader But here's the kicker: Not just any follower can become leader. They need to prove they're up-to-date enough to lead. It's like ensuring the new team lead knows about all the recent project decisions! 🔒 Consistency: Keeping Everyone on the Same Page Once we have a leader, how do we ensure all servers agree on the data? All changes go through the leader The leader replicates changes to followers Once a majority confirm, the change is committed The leader then tells everyone it's official Think of it as a team decision: once most agree, it's locked in, and everyone gets updated. 💡 Why This Matters: In the world of microservices and distributed databases, Raft's approach ensures: There's always a clear decision-maker Data remains consistent even if some servers fail The system can recover quickly from failures Real-world impact? Imagine an e-commerce platform that never loses your cart items, or a banking system that always knows your correct balance, even during server updates! 💾 Real-World Database Examples Different distributed databases handle consistency in various ways: 1. etcd & Consul: Use Raft for strong consistency. They're like a team that always makes decisions together. 2. Apache Cassandra: Uses a different approach called "eventual consistency." It's more like team members working independently and syncing up later. Pro: Super fast writes and high availability Con: Might temporarily have inconsistent data across nodes 3. MongoDB: Offers a spectrum of consistency options: In replica sets, it uses a consensus algorithm similar to Raft Also provides eventual consistency options for faster performance 4. CockroachDB: Uses Raft for strong consistency while scaling horizontally. 🤔 Question for you: Have you encountered consistency issues in your distributed systems? How did you tackle them? #DistributedSystems #Engineering #TechLeadership #Raft 🔗 Part 2: https://lnkd.in/dRhU3B63 Join the Journey with Vikas K. 🔥 Follow for a World of Insightful Content!

9

Meta uses monorepo to manage its code. I was skeptical about this approach when I joined the company a few years ago. How could it work with such a massive codebase? But I was wrong. I found working in Meta's monorepo to be a pleasure.  If you want to learn what working in Meta's monorepo is like (and why we are not using git), check out my latest post. https://lnkd.in/guaDRris #softwareengineering #productivity

4

"Blazing Fast: Uber Serves 40 Million Requests per Second, Slashing CPU Cores by 95% from 60K to Just 3K, with Integrated Cache" 🚀 Don't Just Band-Aid Your Performance: Cache Like a Pro 🛠️ Cache is every developer's first line of defense against sluggish apps. But simply throwing a cache at a problem is like slapping a bandage on a gushing wound. It might offer temporary relief, but it won't fix the underlying issue. Uber’s epic struggle: While trying to make their apps faster and able to handle more users while stuck with old-fashioned storage. When there's suddenly a massive increase in people wanting to read data, it pushes them right to the edge of disaster. ❌❌ A BIG TECH HURDLE❌❌ Solution: Docstore comes to the rescue. Beyond the Quick Fix: 1.DocStore Query patterns - It turned out that more than 50% of the queries coming to Docstore are ReadRows requests. Build what’s required and what makes sense. 2.Bypassing Cache - Bypassing the cache in Uber's microservices ensures strong consistency for critical flows like retrieving items in an eater's cart, while leveraging cache for less critical tasks such as fetching a restaurant's menu. 3.Brilliant Cache Invalidation Strategy using CDC and Flux - Cache Invalidation is one of the critical problems in Software Development. This strategy ensures cache consistency within seconds, preventing the risk of uncommitted transactions affecting cache integrity. 4. Special Redis Codec for Row Reads: Unique row values in MySQL are cached in Redis using a special codec to ensure compatibility. This facilitates sharing cache resources across databases while maintaining data isolation. 5. Cache Warming and Cross-Region Replication: Instead of using inbuilt Redis replication, Leveraging the Query Engine for Cross-Region Replication ensures consistency. 6. Sharding Differently: Sharding Redis clusters differently from the database scheme optimizes caching for diverse data access patterns, enhancing overall system performance and scalability while resolving potential inefficiencies in cache management and data retrieval. 7. Adaptive Timeouts: Optimizing Redis Performance with Dynamic Timeouts. This prevents issues like premature failures or excessive latencies. Percentile latency ranges: P75 latency decreases by 75%, and P99.9 latency drops by over 67%, all while curbing latency spikes. ⏱️ As of the current date, CacheFront supports over 40M requests per second across all production Docstore instances, with continuous expansion. 📈 Blog Link : https://lnkd.in/ghvAY3_Z This is undoubtedly one of the finest reads I've encountered on distributed cache systems. I highly recommend giving it a read. 📚 Big Thanks to Preetham Reddy Narayanareddy Afshin Salek Piyush Patel Zurab Kutsia Eli (Oleg) Pozniansky #Systemdesign #distributedcache #cache #p99latency #systemdesign #design

21

1 Comment

Falling back in with a side-quest Colab about LLM evals (high level) - its not just about prompting. Including my absolute favorite: Cosine Similarity. I love this algorithm (or is it technically just a function? ratio? idk/care). This colab is a bridge to some concepts I'm hoping to get to soon: like how to automate quality evals, and auto-reject outputs that do not closely follow the prompt instructions. These are things that really should be handled by code when creating LLM-backed services for production purposes. Again, this is using the smallest Llama 3.2 1B model, and there's no training step this time, so you can run it on just a CPU runtime without much fuss. LMK what you find (anyone who runs this should get different results, and those results are probably always going to be interesting)

17

3 Comments

Great compilation of resources! These are definitely some of the best materials out there for prepping for tech interviews. I've personally found the "System Design Interview" books to be incredibly helpful for grasping complex design principles. Also, "Neetcode" has a great collection of problem solutions that are super handy for quick review sessions. One resource I'd add to the list is "The Pragmatic Programmer" book. It's not directly interview-focused, but it provides invaluable insights into software development that can help deepen your understanding and improve your problem-solving skills. For behavioral interviews, practicing the STAR method with real-life examples from your experience can be a game-changer. It's all about storytelling and demonstrating your impact. Curious to hear what others think about these resources. What has been the most effective material for your prep? #TechInterviews #InterviewPrep #SoftwareEngineering

2

🎉 New YouTube video just dropped! Come hang out with me as I walk through how to answer the common system design interview question: Design LeetCode! This question can also be referred to online as Online Judge, Online Coding Competition, or Live Leaderboard. In the video, I break down what are bad, good, and great answers to many of the problems that come up when designing a system like this. Enjoy! https://lnkd.in/gGMMb54F

69

3 Comments

Calling all IT leaders, developers, and practitioners! Ready to explore the future of data in the age of GenAI? Join Google Cloud for Database Day: Sunnyvale on August 27th! **🔥 Here's what you can expect: ** Two tracks packed with insights: one for IT Leaders & Executives, and another for Developers & Database Practitioners. Expert-led sessions on topics like building GenAI apps with Google Cloud databases, maximizing PostgreSQL ROI, and mastering Cloud SQL. Hands-on labs and demos to level up your skills and connect with peers. Don't miss this chance to build your GenAI future. Register now - spots are filling up fast! #GoogleCloud #DatabaseDays #GenAI #DataInnovation

1

Hello, colleagues, this Yahoo article is interesting and relatable for professionals. Here is a teaser: "Amazon has an incredibly granular list of criteria that need to be met before you can be considered for promotion, so I worked with my manager to identify the gaps in my performance, ... and I created a project for each gap that, if completed, could serve as evidence of my ability. I worked backward to create a strict timeline to finish my projects ... that helped me get systematic about my approach moving forward." https://lnkd.in/dsfvPeAk Take a lesson from this person. I seems Amazon gets very analytical when it comes to promotions. And in a company that size, it is impossible to not promote without rigid check-off type policies or whatever XYZ job promotion software. But, hey, there is a lot I will do for a $500,000 salary. Please take note of this last sentence, if anyone works for Amazon and wants a Quality Engineering Manager/Director anywhere in the world.

1

1 Comment

Recently I got to chat about some of the most important things to know about developer productivity with the folks at Cortex on their podcast, “Prefeontal.” Among other things, we talked about: - Why does quality matter even more for internal tools than external products? - Why should we measure team productivity over individual productivity? - How do you present data about developer productivity to different audiences? - Is there a danger of measuring PRs Per Developer? Is there ever a good time to do it? - Is there a conflict between developer experience and developer productivity? - How do you communicate about software development and its challenges with non-technical leaders? And way more. You can find it on their website: https://lnkd.in/gcRYaTkM Or on Spotify: https://lnkd.in/giFSNhTH Or Apple Podcasts: https://lnkd.in/gBNHWW8h

38

1 Comment

sharpening writing skill is one of killer skill!

2

I spent last week exploring the internals of DragonflyDB - a drop-in replacement of Redis, and it does a staggering 6.43 million ops/second on a single machine 🤯 DragonflyDB is a multi-threaded implementation of Redis with a bunch of insane optimizations. Some of the architectural design decisions and internals of data structure implementation just blew my mind. To understand it better, I went through their blogs, docs, and even some fragments of source code. I compiled all my learnings in a series of videos and trust me you will absolutely love going through it. the videos will change how you look at data structures :) subscribe to my channel, and the video drops every Friday evening. youtube.com/c/ArpitBhayani ps: Until then watch other System Design and Database Internals videos on my channel; they are all no-fluff. #AsliEngineering #DatabaseInternals #SystemDesign

1,040

24 Comments