Ishan Khanna ⚡️

Observation: 9/10 SWE's quit at “reading the docs”. Cut through the BS with this easy trick: Take action and build. Writing code is 100x easier than the docs make it sound. “Reading the docs” gave me crazy imposter syndrome: -Git -WPF -Kafka -OAuth -Angular -GraphQL -RabbitMQ -Kubernetes In ALL cases, I conquered my fears by building. Except for WPF. I gave up on that. And Kafka, that's for another day. Forcing yourself to build/code/deploy melts analysis paralysis. Then you'll say "oh that was pretty easy". Trust me, we aren't building rockets. Practice AND Study build true mastery.

22

1 Comment

People tend to believe that there is more of a trade-off between Quality and Speed than there really is. Most of the time, if you’re doing a good job of refactoring and writing tests as you go, the final product doesn’t take any longer than it would have to do it “the hacky way,” if by “the final product” we mean “a system that actually works and does what its users need.” True, once in a while there are huge problems that you can’t tackle right now. For example, maybe the right thing to do is to redesign your application with a whole new data store, but that could take years. You can’t do it right now. That said, _most_ of the problems that users experience with software don’t come from these decisions to set aside these huge projects for later. Most issues come from a series of small decisions that grew over time into a complexity that nobody understood or managed correctly. Often they come from a developer just _feeling_ rushed, like they can’t refactor this right now, they can’t test this right now, etc., when really they do have the time to do it. Let me give a specific example. I once caused a fairly significant production outage at YouTube. I was mid refactoring a very complex piece of code, and I had not written a test at the highest level to make sure all the pieces still worked well together. Had I written the test, the issue absolutely would have been caught. So egg on my face, right? Well, there are two lessons to take away from this, actually: 1. The shallowest lesson here is that one should have tests that cover what one is refactoring. 2. A deeper lesson is this: the function I was refactoring had grown incrementally into a hairball that was basically impossible to understand. Every person who had worked on it (many people) had chosen _not_ to fix its structure over a period of many years. They also had never written a test for its behavior—you couldn’t even look at a test to understand what it was _supposed_ to do. The function was basically Pandora’s Box; some day somebody was going to touch it and cause something to break, and the person who did that happened to be me, because I wrote about 100 changes to it (incremental refactorings) and one of them happened to have a bug. So should I have written a test? Maybe. Honestly, until I went through the process of refactoring it, it was really hard to tell what it was _supposed_ to do in all sorts of various edge cases that it covered. Am I personally responsible for that outage? Of course! But am I the _only_ person who contributed to that outage occurring? No. Every time somebody says, “well, I’ll refactor that later,” and doesn’t, “we really ought to have monitoring eventually,” “I should write some tests some day,” THAT is where most problems start. And the wild thing is—almost always, those problems didn’t have to start! If everybody had maintained the system well as they went, it’s unlikely there would have been any impact to the speed of delivery.

153

24 Comments

The reality of system design no one will tell you is: Companies and interviewers exist on a spectrum somewhere between opinionated and dogmatic: about tech choices, products, and you. Telling a system design interviewer from Amazon that you want to use SQL, is akin to telling a hiring manager at Apple you prefer Android. Who knew? It's obvious once it's spelled out, but only because we sometimes assume these big brand-name tech companies (along with their interviewers and interview processes) are faultless. This was an eye-opening clip, Victor Moreno and Kevin Landucci! But I like the full interview even more (link in the description) 🛎 Subscribe to ApplyPass for more content like this

22

2 Comments

Fascinating summary of why Figma decided to move from ECS to EKS (#Kubernetes) here are the four key points that stand out to me: 1. Hiring ECS experts is very hard! I believe this. 2. Using open source softwares, we find that most things nowadays are available as a Helm chart distro, while very few OSS projects support ECS officially. (I understand this viscerally, Helm was essential to my Kubernetes onboarding journey and still is.) 3. Autoscaling - the ECS platform supposedly does support it, but they were not using that feature at all. Why? (Might have to read the source document to understand) this turned out to be very costly, anyway. 4. Logging through CloudWatch, Datadog, and Snowflake also turns out to be very costly. Dynamic scaling is a well traveled road in the Cloud Native community, with projects like Karpenter, KEDA, the built-in solutions like Horizontal Pod Autoscaler... I have never used any of this in anger, but I have studied it and shared it for the KubeCon audience. Does this resonate with anyone else? I too would like transparency into the version of software running and to be able to easily access that information across my clusters. This is a valuable property of GitOps solutions like #Flux when implemented well, but the implementation of a multi-cluster and/or multi-tenant use case is still a challenging one, requiring more than a bit of study to master. Does anyone in shouting distance have good or bad experiences to share with that in mind, on either platform or others? (I'm interested to hear from you!)

9

𝗥𝗟𝗙𝗢#𝟯𝟲: 𝗛𝗮𝗰𝗸 𝘁𝗵𝗲 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝗲𝘀𝗶𝗴𝗻 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 I never likes system design interview questions, at least in the form of books and articles that supposedly train you to pass the system design interviews. They start with an ambitious and vague question, like “design Instagram”, and the answer is usually formulaic: whats the average size of this, what's the expected number of request to access that, and a few calculations that result in sentences like “the database size should be 30 Terabytes”… sentences written by the utterly deranged. In practice, you (almost) never start with 1 billion users, so you will (almost) never need a 30 TB database to begin with. No, you start with a simple app that stores JPEGs for a few 100s of people, and then start optimizing for 1000s and hundreds of thousands, before thinking about millions and billions. The “Hack the System Design Interview” books seem to me like they teach starters to start designing microservices before even considering simpler alternatives. But hey, people love them. I find it more productive and effective to learn about system design by building systems and studying how others built similar systems, rather than to read those stupid “Hack the System Design Interview” books and articles. https://lnkd.in/dHFsBzin #raft #paxos #distributed #systemdesign #cloud #aws Subscribe to the newsletter: https://rlfo.substack.com/

15

I've led 100+ System Design Interviews in my career. Here's how I'd crack the SDI at a place like Microsoft if I had to do it again today: 👇 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗮 𝗯𝗼𝗶𝗹𝗲𝗿𝗽𝗹𝗮𝘁𝗲 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝗲𝘀𝗶𝗴𝗻 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸: The RESHADED framework is a handy way to quickly break down the key elements of any design problem. 𝗗𝗼𝗻’𝘁 𝘂𝗻𝗱𝗲𝗿𝗲𝘀𝘁𝗶𝗺𝗮𝘁𝗲 𝘀𝗼𝗳𝘁 𝘀𝗸𝗶𝗹𝗹𝘀: Define the scope of the problem, don't be afraid to ask your interviewer plenty of questions… and then be prepared to communicate and rationalize the tradeoffs you make. 𝗧𝗮𝗸𝗲 𝗮𝗱𝘃𝗮𝗻𝘁𝗮𝗴𝗲 𝗼𝗳 𝗯𝗮𝘁𝘁𝗹𝗲-𝘁𝗲𝘀𝘁𝗲𝗱 𝗶𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗽𝗿𝗲𝗽 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: I recently wrote a completely free course where I captured all my best insights from interviewing candidates at Meta and Microsoft. You can check it out here: https://educat.tv/4cydH2b Good luck — and happy learning! #systemdesign #softwareengineer #interviewprep

30

1 Comment

The easiest way to escape danger in the deep dive of a system design round is to use understatement. The deep dive is usually the toughest part of system design interviews partly because you have to defend your tech choices. The simplest justification can be the most effective. The one that seems unstable at first, because it's so simple, is, in reality, solid enough to get you out of any jam. You can say "I'm choosing NoSQL because that's what I have the most experience in." 🤯 As a justification. In a system design round. That is a revolutionary idea for any software engineer who assumes that justifications have to be fancy, impressive, or backed up by technical reasoning. Nice job on the video Victor Moreno and Kevin Landucci 🎈 Subscribe to ApplyPass to learn more tips ✍️

20

1 Comment

Interesting conversation for those just beginning their careers, in engineering or beyond. Thanks for sharing Andy Greenwell!

3

1 Comment

My career as a software engineer boils down to two things: - Getting systems to talk to each other. - Getting people to talk to each other. Not necessarily in that order.

19

1 Comment

Why BigTech Engineers are paid a lot 💰 while it could suck to work there (sometimes)? I used to work at FAANG and whenever I designed a technical solution involving complex distributed systems problems, it was roasted by Principal Engineers in the review. 🥺 It’s not uncommon at a few Big Tech companies. ❌ Folks look forward to a 5-10-year vision, while the feature needs to be shipped “quickly” as per PMs’ timelines. ❌ This loop of "improve the doc", misaligned expectations, and pushback from sister teams leads to churn and process-driven burnout. This adds up to over-skepticism and slow down the company. ✅ But there is a strong reason why they are forced to follow such a suit. They operate at a huge scale. 🤞A small hiccup can cause a loss of billions of dollars and of customer trust built over years. And that is the reason, Engineers are paid top-notch at Big Tech. 👉 If you want to know if BigTech, Mid Tier or Startup is best for you, read my newsletter article I wrote few months ago. Link is in the comments.

17

4 Comments

Given this job market, companies are hiring more senior roles and less junior ones. This is going to be disastrous in ~5 years when the normal bench of junior engineers who could have leveled up to senior will be in short supply. There will be a real gap in our bench that used to be built out by both big tech and startups.

40

10 Comments

https://lnkd.in/gqUkq3BM is an excellent overview of major topics in LLMs. If you are getting started in this space, this video which is also generated by another LLM 😂 , is a great place to begin your journey in this exciting area of AI. It covers topics like pre-training, evolution of GPTs and other popular LLMs, fine tuning (SFT, PEFT like LORA, RLHF), inference optimizations (quantization, speculative decoding, distillation, flash attention), output sampling techniques, multimodal capabilities and finally some real world applications like math, coding, medicine etc. Loved their analogies even for core complex concepts like flash attention - "it's like finetuning the engine of a car, to make it more fuel efficient, without changing how the car drives"

24

I've been saying this for years: Developer cycles cost more than processor cycles! The average HTTP response time is about 0.5 secs. Database queries often run into seconds. Moving your stack to a 'fast' language so you can shave a few hundred millisecs off your server processing will cost you hugely and it will have a negligible effect on your customers. Most web-apps (CRMs, CMSs, marketplaces, etc) are IO-intensive, not CPU-intensive. The bottlenecks are on the network, datastores and external API interactions, *not* on the processors. And to solve these kind of problems your best bet is horizontal scaling and parallelism, not fast languages. #SoftwareEngineering #TechLead #SoftwareArchitecture

4

I woke up to a call at 3AM. “PagerDuty Alert: error anomaly detected.” It’s happening. Every engineer’s worst fear when they join the on-call rotation. My heart rate teleports to 120. I stumble toward the monitor. No errors. 7 minutes pass. Still no errors. The heck is going on? I rub my eyes. One alert says the house is in flames. All the others say “get back in bed, dingus”. 4 more minutes. Should I escalate to my manager? What if I’m wrong? It’s 9:07 AM. My better wits have returned with sleep. I check DataDog again. “You’ve gotta be kidding me.” I was looking at the wrong tab. The errors were real. I just got lucky they went away on their own. Let this be a PSA to on-call engineers: This is why runbooks exist — steps to follow in emergency situations. You probably won’t need them in broad daylight. But when the glow of your monitor is the only thing holding your eyes open, You’re gonna wish you had a checklist to keep you sane.

364

52 Comments

Raphael Neves - I totally relate to this! In a previous project, we resisted addressing tech debt for as long as we could, thinking it wasn’t urgent and we could keep pushing features. But after a while, it felt like we were trying to drive with square wheels—progress slowed down drastically. We reached a point where the system became so clunky and error-prone that even small updates were taking forever, and the number of bugs was growing rapidly. Eventually, we had no choice but to tackle the tech debt head-on. It was only after we cleaned up some of the critical issues that development started to feel smooth again. Looking back, it would have saved us a ton of time and headaches if we had incorporated some of those strategies you mentioned earlier in the process. Prioritising it early makes all the difference. #TechDebt #CodeQuality #SoftwareDevelopment #Agile #TechLeadership #DevOps #CleanCode #Productivity #Refactoring #TechStrategy #SoftwareEngineering #DeveloperLife #Teamwork #AgileDevelopment #CodeMaintenance

10

1 Comment

Take this course! I cannot recommend this enough! Gibson Biddle's DHM model has dramatically improved my executive communication and cemented a number of poorly organized thoughts and observations of my own. He just says it much more clearly, much more concretely, and much more usefully than the mishmash in my head :) (And I'll take this moment to plug my own lightning course on using community to land your next job, happening tomorrow, in the comments.)

16

5 Comments

Almost everything I read on LLMs is either 1) vendor-driven that doesn't help me build a mental model for tradeoffs, or 2) tool documentation that assumes I know exactly what I want to do, just not how. This is an excellent post smack in the middle. https://applied-llms.org/

77

3 Comments

Non-software engineers can actually be like “some” software engineers. This is something I’ve already thought during this journey of technological change, backed by direct experiences helping companies solve real problems with software. Software is a solution to a problem. Thus, we usually need to understand the problem before starting to think of a possible solution. Traditionally, different people were dedicated to those different domains (understanding the problem and designing the solution). But things get a little blended nowadays. In some situations, non-technical members are in a better position to design (and implement) a solution. They just need to be a little technically-minded. The most effective solutions might actually come from non-technical individuals, as they possess domain expertise, which gives them a clear understanding of both the problem and their needs. With the help of modern tools, they are now able to create potential solutions themselves. The article is very interesting also because it basically demystifies the current adage about “we’ll all be replaced by AI agents” with a much more interesting take: “people can become more effective at solving problems” https://lnkd.in/de3b7vM5

7

1 Comment

Some things to keep in mind while designing systems in the real world: 1. Keep simplicity at the core of your design. There are cases where you may have to trade off simplicity for performance, availability, security, etc., but strive to keep it as simple as possible. 2. Make calculated assumptions and validate them with data. More importantly, document these assumptions. 3. Keep it an iterative process. Don’t try to design for future potential use cases. Keep it extensible, but avoid unnecessary components that solve 1% of your cases while creating 10 new problems and risks. 4. Understand the success metrics for your system. Make it highly observable. Implement as much telemetry as you can afford to make your life easier. 5. Understand all the failure modes of your system. Handle them as gracefully as possible and document them. Be a bit pessimistic—whatever can fail will fail, and your software may crash. Try to avoid this as much as possible. 6. Avoid using technology just because it is cool and fun. Opt for “boring” tech that works and has been battle-tested. Boring is simple and well-supported. To understand how real world systems work in depth take a look at our discussions at The GeekNarrator podcast. I am sure you will find value.

20

4 Comments