Anand Pawar

The payment gateway Stripe migrated 1.5 PB of data into new shards. They used the following algorithm. 1. Mark data to be migrated 2. Bulk export data into new shards 3. Simultaneously, copy live data into new shards (async events) 4. Check data for correctness 5. Make clients point to new shards 6. Remove migrated data Article link: https://lnkd.in/dGt_DGiD Migrating data from one place to another is a common use case (most cloud providers offer solutions around this). I've created a set of videos on this topic. 1. Database replication: https://lnkd.in/deahS5Qi 2. Write ahead logs and change data capture: https://lnkd.in/d9c_fmhg 3. Write amplification and split-brain: https://lnkd.in/d2W5qaB6 Have a great day! #SystemDesign #DatabaseMigration

293

11 Comments

Anything That Can Go Wrong, Will Go Wrong ~Murphy’s law Mentioned case sounds like an interesting failure Big rewrites are incredibly risky projects If they are critical they can be deadly I seen a few over the last 13 years Let’s unpack this T̶w̶e̶e̶t̶ X-Post → Company decides to rewrite system → Projects gets delayed → Blocked revenue → Bankruptcy In thread it seems the business jumped on SAFe train Someone probably read blog article about it Jumped straight to the implementation Other things happened as well 👇 🧠 Missing knowledge I assume the old system was at least 10 years old With job rotation there was no one knowing it New discoveries cause endless delays 🏢 Government ways of working Public sector is driven by fix term & budget contracts Those contracts are judged by cost The “law” forces SAFe/waterfall ⚠️ Poor risk assessment from both sides The company should know what to do if project fails The vendor should have safety plans in place Both parties faced the Murphy’s Law 💡 What can you learn from it? → Document early, document often → Prepare for not optimal ways of working → Always have plan B, if critical then C, D and E as well → Verify causality in people’s content 🙃 SAFe wasn’t the reason for the bankruptcy Someone getting hyped about golden bullet was -- Shout out to Mihai-Valentin Curelea For posting about it yesterday Check him out!

19

32 Comments

T-Shaped Developer aged like milk Market moved on and code isn’t enough You must stand out In the pile of 100 applications Or in an interview that can be the last You can’t change it You must adapt Code and communication are the baseline You can see the hype on LinkedIn Everybody gets it What else? → Domain → Business → Leadership → Psychology → Client relations → Marketing (CV) → Sales (interviews) → Product development Pick what you’re the best at and use it Who you are - gives you advantage Change from T to You You’re a gamer → work in gaming You like money → work in finance You like cars → go automotive Your passion will sell your skills Nobody can compete with you being you (and you’ll be happier doing what you love)

12

7 Comments

Ownership - Founding Fullstack Engineer 💰 Up to £140k base 📈 Generous stock options (5 vest 1 cliff) ⚙️ Tech Stack: Python & TypeScript 📍London Onsite (relocation offered) Having raised $3m in seed last year and already boasting an enviable client list this is a rare chance to join at the ground floor of a company that’s scaling rapidly towards a $1bn valuation. About the Role: 📍This isn’t your typical 9-to-5; it’s an onsite position where you’ll collaborate closely with the founders every day, helping to shape the company’s future. I’m looking for someone with a true owner’s mentality—someone who understands the high-energy demands of startup life and is passionate about driving impact everyday. In return you’ll be given ownership of tech direction and potentially life changing stock options from day one. Interested? Drop me a line in confidence. #Startup #London #Python #Typescript #FoundingEngineer #Ai

16

1 Comment

One simple way to retain software engineers. An end of year review. Goal setting for 2025. Proper 1-2-1 time. Where possible, ideally face-to-face. Sounds obvious. But many engineers I speak say it doesn’t happen. People want to feel valued. When was the last time you had a 1-2-1? #retention #endofyearreview #2025goals

8

What do API consumers really need without feeling overwhelmed: API contracts Documentation: - Getting started guides - Tutorials - Workflows Sandboxes, Interactive playgrounds, Notebooks SDK Sample Code Change log Versioning strategy Deprecation/Sunsetting strategy What else makes an API experience delightful? Is more the merrier? Is there a notion of what is minimally expected from an API producer?

50

14 Comments

My blog on building a high performing Software Engineering team. I have written this blog from my experiences over the past 12 years working with and building high performing Product Engineering teams. I hope you find this an interesting read. Please do share your experiences, comments about the blog, my writing style. Always eager to learn and improve. #Engineering #HighPerformance #Devops #ProductEngineering

28

𝗖𝗹𝗼𝘂𝗱 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗼 𝗮𝗱𝗱 $𝟭𝟴𝟭𝗯𝗻 𝘁𝗼 𝗨𝗔𝗘'𝘀 𝗲𝗰𝗼𝗻𝗼𝗺𝘆 𝗯𝘆 𝟮𝟬𝟯𝟯, 𝘀𝗮𝘆𝘀 𝗔𝗪𝗦 Fantastic to hear that the continued growth of Cloud adoption in the Middle East is set to add $181bn to UAE's economy. This will allow the UAE to accelerate it's digital transformation and generate many new opportunites within this very exciting sector. For businesses, moving to a cloud system hosted by companies such as Oracle, AWS or SAP – is more economical than creating their own infrastructure of servers, hardware and security networks, industry experts said. It also brings down the overall cost of ownership. #CloudAdoption #DigitalTransformation #AspireTechnology https://lnkd.in/ddYiAKrc

10

Leadership in Platform Engineering is not for the faint-hearted🤯😅 If you are a Platform Engineer considering moving into a leadership role, here is some key advice you should take on board📢 I spoke to a Platform Engineering leader last week who has led multiple teams of up to 30 people whilst still being hands-on themselves. One big thing changed for them when going from a self sufficient engineer to a team leader🤔💭 ☁️Your responsibility now covers the WHOLE PROJECT ☁️Your objective is to achieve the BUSINESS GOALS ☁️You are not just responsible for your OWN DELIVERABLES ☁️Your COMMUNICATION SKILLS have to be excellent (even more than your technical ones) ☁️You have to be willing to PUT THE BUSINESS OBJECTIVES FIRST #platform #devops #tethr #leadership #engineering

7

📢 - 𝐅𝐢𝐫𝐬𝐭 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞𝐬 𝐨𝐟 𝐒𝐭𝐚𝐫𝐭𝐮𝐩 𝐇𝐢𝐫𝐢𝐧𝐠 - 📢 On Engineers' Day I would urge all industry leaders, business and product alike - You will rarely get a clean slate to build your product / stack on. Do not make the mistake of fixing or building for short team (1-2 years) and set your hiring strategy to save money - The money you save now will come back and cost you astronomically more - not to mention time and loss of momentum. Be hands on, build for the longer term and champion the processes. #engineers #startup #firstprinciples

16

MONEY SPENT ON PRODUCT TALENT = GREATER LEVELS OF OUTCOMES? I missed the ProductTank where David Cox spoke but have read his blog post on Mind The Product website on the same subject Organisations starving on a diet of low-hanging fruit (link to post in comments) It's really interesting and the crux of the article focuses on the idea of marginal gains and the concept doesn't really apply in Product Because people can waste too much time on marginal improvements in areas that don't have much of an impact The piece discusses the marginal gains culture at British Cycling that led to improves Medal hauls at major championships That yes, it was part of the story, but the biggest factor was increase financial investment Which ties into a question I posed a Product leader last week........ Regardless of the Product itself, the most successful Product teams: are they successful because they are paying the best, so attract the best leaders and the best talent? My reasoning was that the best leaders and the best talent ask the difficult questions, know how to say 'no' and recognise that commercial outcomes are fundamentally linked to every task that you are committing to sprint-to-sprint. As such, that Product talent develop better track records and command better money. It's a lot like Dave's British British Cycling analogy or a Football Team: those that spend the most on talent, achieve the better outcomes. Except Manchester United of course 😂 But thinking locally to the better paying firms that invest not only in good wage levels to attract & retain, but also in their wider culture. They think about how to refine their hiring processes to weed out the mediocre talent that is invariably attracted by the money, rather the broader opportunity. Those businesses tend to perform really well, even if they weren't the first Product of their type, to go to market Cheaping out on talent rarely ends in outstanding outcomes!

9

2 Comments

I disliked JIRA when I was an engineer Updating tickets and writing them was a nightmare There are 3 reasons why I changed my mind 1. Saving time for the business The main reason why JIRA has a bad rep is because of the time it takes to update tickets. It's what engineers see. As a manager, I learned that reporting on what's being done is essential and impossible if we use simple tools. 2. Everybody knows JIRA Most engineers work with it, and it takes no time to find yourself around. Atlassian did a fantastic job on UX. It's one less onboarding topic and an easy win for me. 3. Async work I need to know how the work is going and report it to the rest of the business. What happened, when did it happen, and if any problems need my attention. I could organise a meeting, but having a JIRA board to see everything at a glance is better. Paraphrasing Churchill: JIRA is the worst project management tool except for all others that have been tried What tool are you using?

24

16 Comments

There’s something strange happening lately. There are arguably more software engineers than ever. But there are a huge number of unfilled roles. Thousands of software engineers are searching for work, but companies claim to be struggling to fill the roles that they need to fill. So what’s the issue? Some significant skills gaps have grown roots. Many people know how to code, but there are few software engineers. What matters most is the right algorithm, the systems we build, and the business problems we’re looking to solve, not what programming language or cloud service we use. Many new developers entering the field these days are trained primarily as web developers, but the most in-demand skills aren’t related to web development. They relate to AI & ML, DevOps, cyber security, data analysis, and cloud architectures. So, ok, what’s the solution then, Andrew? Hiring for a strong foundation of ‘hard skills’ and investing in your people. It is, specifically, getting them specialized training when it’s necessary. With that strong foundation built, it’s much more manageable for your best people to learn specific technologies and do so consistently. Businesses are looking for real software engineers, not just developers. What do you think about the relationship between the skills you hire for and those you’re willing to train? Tell me about it in the comments below.  

20

18 Comments

In case you missed it while prepping for Flag Day 🇺🇸 Happy State of Engineering Management Report Day happened this week! The annual thought leadership piece from Jellyfish has dropped. Hear from Engineering leaders, managers and frontline ICs .... we cover a range of topics that matter to great R&D leaders. Key Points: 1️⃣ Engineers are now strategic partners! 🙌 2️⃣ Burnout continues to be a challenge 😱 3️⃣ The talent shortage is spiking back up. 4️⃣ Many teams have embraced #AI 🧠 5️⃣ Others have legit concerns about AI ⁉ If #developerproductivity #genAI #engineeringexcellence are important to you .... have a read! (link in the Comments)

2

1 Comment

Has anybody used or created a framework for assessing observability maturity across teams? I found a post from Honeycomb that was posted recently, but it’s quite leighweight and a work in progress. I’m interested in understanding how to quantify how far along on their observability journey teams are. The output would help identify those that need greater levels of support etc. Let me know 🙏🏼 #observability #monitoring #softwareengineering

1

Are you sure you want to hire a CTO?    I often get Founders/CEOs approach me to help with hiring a CTO but when we get into the finer details they are describing the responsibilities of vastly different roles.    Here are the four most common roles advertised incorrectly as CTO.   Technical Architect:    The company are looking for someone to design and guide the development team during the implementation stage. This role might be more hands-on in designing and implementation rather than the broader business and leadership responsibilities associated with a CTO.   Head of Engineering:    Instead of a CTO, a company might need a Head of Engineering who can lead and manage the software development team, drive technical excellence, and ensure timely delivery of projects. While there may be some overlap with the CTO role, the primary focus here is on the engineering aspect rather than the broader strategic and business concerns.   Product Manager with Technical Background:    The company is looking for someone to bridge the gap between technical teams and product development whilst being involved on the ground daily. In this case, they might advertise for a CTO but a Product Manager with a strong technical background who can guide the development process and prioritise features based on technical feasibility and market demand may be more suited.   Chief Innovation Officer (CIO):    Instead of a CTO, a company might be seeking someone to spearhead innovation initiatives, drive R&D efforts, and explore emerging technologies. This role might involve less day-to-day involvement in technical decisions and more focus on long-term strategic planning and innovation.   So before hiring a CTO, consider whether what you’re looking for actually fits into a different role.   This will eliminate any confusion.   #CTO #Recruitment 

42

14 Comments

How to make a disproportionate impact as a software engineer? If you are a Software Engineer, remember the fact that you are a creator. So often the mirage of solving 1000 leet code problems, optimising code for linear time complexity, getting memory utilisation to bare minimum, getting highly opinionated on the right framework, obsessing over best practices, makes us forget that we are building software that solves problems for human beings. Start thinking about Software Engineering as a creative process just like a creator. Think about creating something that makes an impact. This thought process alone will give any Software Engineer a huge edge over the crowd. There is always a huge demand for Engineers who can put themselves in the shoes of the users of the product that they built, over an Engineer who cares more about runtime complexity. Get creative, create stuff that people love. #softwareengineering #product #creator

38

3 Comments

Tech Industry Shake-Up: Gergely Orosz on Job Market, AI Revolution & New Realities Over the past 18 months, the tech industry has undergone a seismic shift. Here’s a breakdown of the key changes and how they impact software engineers and engineering managers: 📉 **Job Market**: The hiring landscape is tougher than ever due to fluctuating interest rates, changes in VC funding, and fewer IPOs. Finding the right job and closing deals have become more challenging. 🤖 **AI Revolution**: While AI is revolutionizing efficiency, it lacks a broad, initially free distribution channel like the smartphone boom had. This shift is more about enhancing existing workflows rather than introducing entirely new ones. 🔄 **New Norms**: Software engineers face slower career growth, a focus on simpler, well-established technologies, and added responsibilities. There's a shift away from trendy innovations back to foundational technologies like monoliths. 🔄 **History Repeating**: Drawing parallels to the Dotcom Bust of 2001, unexpected changes are the new norm. Prioritizing career security and adaptability is crucial in these uncertain times. 📚 **Advice & Insights**: The article offers practical advice on securing careers and understanding the tech landscape's ongoing transformation. Stay updated, stay adaptable, and navigate these challenges with confidence! #TechIndustry #JobMarket #AIRevolution #SoftwareEngineering #CareerAdvice #VCFunding #Startups #TechTrends

1

Top tip for "No Code" integration solutions: Only use them if it can be done internally (i.e. someone in the business is has the technical ability to build and manage them) and the monthly platform bill is < £500. If you are paying a third party to build no code integrations - you may as well get them custom built so that you avoid the high ongoing fees and scalability issues that come with a no code platform. As the business scales you'll reach a point where it is more cost effective to do this anyway.

4