Adron Hall

Anthropic released another paper on their ongoing work towards interpretability of large foundational models like their Claude 3 family, built on previous work by many. Their work continues to be enlightening and inspiring. The findings in this work are generally similar to previous findings on smaller models but scaled up to Claude 3 Sonnet: 💡 they find large numbers of "features" corresponding to combinations of neuron activations and these features map to concepts that we recognize/understand: for example, features for transit infrastructure, particular people, concepts (like bias and "objects lacking sentience"). This might sound familiar from classical machine learning, but the interesting thing is that there is no single thing/neuron encoding each of these concepts, it's a particular combination of neuron activations that embeds these concepts in "superposition" in the network. 💡they observe a sort of "distance" of these feature activations that arrange themselves "near" semantically similar features. When they use their sparse autoencoder approach to learn more and more features they observe that the level of abstraction in the detected features becomes more granular, in ways that are interpretable and meaningful. 💡when they artificially fire these activation patterns during inference, it affects the output in ways that relate to the "feature". For example, asking Claude "what's the most interesting science" normally it responds Physics, but if you clamp the feature activations for the feature "Brain Science" to 10x their max values then Claude responds: Neuroscience. Ways to interpret _and_ influence the model. Great readable work [1] along with an interactive explorer of some of the "features" themselves [2] 🎂 There's an easter egg too: if you scroll in the feature explorer tool and try and select one of the "Randomly selected features from 1M" you get a message "if you want to see the rest of the features, we're hiring!" with a link to their recruiting page :) [1] https://lnkd.in/gZbparwp [2] https://lnkd.in/gC9eus53

19

This is a great video about how our personal ideology impacts the way we process data and information. For me, it shows the importance of fighting against populism and "framing" in the media and teaching our children in critical thinking. https://lnkd.in/eXQdB4fR

2

A few months ago, Jean Ducrot, CPWA recommended that I read the book, "What Every Engineer Should Know About Digital Accessibility" by Sarah Horton and David Sloan. #Accessibility is something that I've always had a lot of insecurities about; and I think so much of that stems from the notion that as an engineer I wanted to have the _engineering tools_ to make any kind of design accessible. What I love about this book is how much they drive home the reality that accessibility isn't _just_ an engineering problem—it's a team problem. And, the requirements of accessibility must be baked into the entire product development life-cycle, from ideation and design through to implementation and testing. Read more: https://bennadel.com/4738 This book is not about tricks for technical implementation. And, it's certainly not going to show you the "one weird trick" that makes products accessible. If anything, the book is a bit tedious. But, the value of the tedium is that it underscores the deep thinking that people have done on this complex and nuanced topic. If there is "one weird trick" to accessibility, it's thinking holistically about the customer experience; and keeping things simple; and seeing complexity as a creative constraint, not a technical constraint. I still have _so much to learn_ about accessibility. But, I do feel like this book has really shifted my perspective.

32

5 Comments

"In solving the specific problem of their function, they had lost sight of the holistic forest through the trees. I've observed companies where the product or design org and the engineering org have their own core model. The models use completely different languages; they don't interact with each other, and no one speaks the same language. They do this because there's a lot of pressure on each org to produce something, and collaborating cross-functionally is difficult. Meanwhile, the business is searching for one simple model to rule them all, which doesn't really exist. As a result, without a viable alternative, the company uses metrics like revenue and NPS (a travesty of statistics and research)." https://lnkd.in/gbk48eS7

35

3 Comments

In part 3 of the series "Building Services at Airbnb" lets see how resilience was built into the standard service platform. While the new SOA increased ownership and development speed, it added complexity in coordinating a distributed system. Resiliency of independent services needed better coordination, as simple measures like request timeouts and retries were found insufficient. New measures were 🔹 Async Server Request Processing : The earlier Dropwizard based service framework used synchronous request processing this means threads waited idly on I/O intensive requests. Leveraging the async response feature in the latest Dropwizard framework increased throughput. As a result fewer I/O threads could handle more concurrent requests hence better absorb spiky traffic, it also prevented starvation of threads by I/O intensive requests. 🔹 Request Queuing : Typical request workloads for Airbnb were I/O intensive and not compute intensive, this meant the chances of server running out of memory than CPU were higher. Therefore request queuing proved an effective technique to allow services absorb bursty request load and prevent services from failing due to resource exhaustion. More specifically, they implemented a controlled delay (CoDel) queue with adaptive LIFO (last-in first-out). 🔹 Load Shedding : The production incident postmortems cite aggressive client retries as the most common cause of cascading failures because the retry storm does not give an overloaded service any room to recover. So a smarter resilience feature is for the client to know when to stop sending requests altogether. Hence techniques such as fast failing with Service Backpressure error response when server side request queue is overloaded which prevents the retry storm and gives room for recovery. Another technique was API request deadline propagation. 🔹 Client Quota-based Rate Limit : A service owner can define client quotas. The quota-based rate limiting works well with service back pressure in production. When a caller misbehaves, rate limiting protects the service from unexpected load from abusive clients, but prevents service back pressure errors for other clients. 🔹 Dependency Isolation and Graceful Degradation : For an unresponsive downstream dependency which happens to be a soft dependency the behavior was to serve partial responses. 🔹 Outlier Server Host Detection : Server host failures are a frequent threat to uptime. Client-side smart load balancing tracks success rate and response latency of each host, and avoids routing requests to outlier hosts with low success rate and elevated latency. They added Envoy to their service discovery stack, which supports outlier detection. https://lnkd.in/gZrH6Bap For more interesting engineering stories, follow me Abhilash G Raja #Microservices #Monolith #SoftwareArchitecture #Engineering #Airbnb

3

Scaling a team is often necessary as demand grows, but it’s essential to ensure it’s done effectively. Before adding headcount, consider if scaling is the best approach—sometimes addressing process inefficiencies or architectural issues can yield better results. If scaling is the answer, keep these factors in mind: * Define clear goals for each team. * Ensure accountability and support. * Minimize dependencies and avoid inefficiencies. * Scaling impacts people, process, and technology. Successful team growth requires careful planning to maintain efficiency and productivity. https://lnkd.in/eTkqykmG

15

4 Comments

Data platform folks need to get hip to open-telemetry and observability. Lineage is useful, and interesting, but when data teams really learn to use software-grade observability capabilities, they make a leap forward in system reliability and data trust. Look at your existing tooling and start experimenting with how you can switch to using an OTEL-based stack. It’s a lot of what you’ve been wanting. #meatbasedengineer

31

3 Comments

I'm fascinated at how much I don't know. Let me tell you a story about how #TIL my design approach preferences and the way I've learned to work are related to a book called Cheaper by the Dozen, which was later made into movies. We (me and everyone I work with Kaizen I/O) use a human-centric perspective in our design discussions. If you're in the office, you'll hear questions like: - Who can cancel a policy? - Are they the authority for cancelling or is that someone else? - What team is that person on? - And then what do they do next? - Do they need to tell anyone that it's been cancelled? - and so on. With our expertise in architecture and technology, this line of exploration enables us to create software imbued with empathy for humans ;) (this statement is still evolving, so bear (bare?) with me). Anyways, I've been thinking about how to succinctly articulate this approach, which led me to read more about a discipline called Human Factors Engineering. That's a whole other rabbit hole, but let's stay focused on why I'm writing this post. Lillian Moller Gilbreth, an engineer, psychologist and mother of twelve who invented the Pedal Bin, was an early pioneer in applying psychology to time-and-motion studies. "A genius in the art of living". She's noted as pioneering ergonomics on the Engineering Psychology wikipedia article. Her two children wrote Cheaper by the Dozen which tells the story of their family life and describes how time-and-motion studies were applied to the organization and daily actives of their large family. BAM! There is so much to learn. https://lnkd.in/gxgRxGWu

7

Consolidated my Notion homepage on a custom domain. I'm surprised it's not yet a native feature. I've realised that while pointing my personal website (https://angeloreale.com), I had lost most of my few years old SEO authority. So then, ... we just build it again. :) Recommending this Notion wrapper, and here is my referral link (I get some bucks if you sign-up using it): https://lnkd.in/dNKCuppi

Emily burning down the myth that the bottleneck for making digital products is at the keyboard, and AI (maybe we need a Mavis Beaconbot? 🤔) will fix that. “The efficiency improvements were ~5%; not nothing, but also nothing I couldn’t gain your company simply by deleting 2 meetings per week off everyone’s calendar” Understand how value flows before blindly automating your work #leanthinking

16

🌟 New Article Alert! 🌟 I've just published an article on Medium: Demystifying Clean Architecture in Go: Separating Fact from Fiction. 🎉 https://lnkd.in/e6pnqEPz In this piece, I dive into the fascinating (and often misunderstood) world of Clean Architecture, exploring how its principles can be adapted to the simplicity and power of Go. From flat project structures to more elaborate designs, I've included practical examples to illustrate how we can balance clarity, scalability, and testability. Whether you're a seasoned Go developer or just curious about software architecture, I hope you'll find this article insightful and relatable. 💬 I'd love to hear your thoughts, experiences, or even challenges you've faced while applying Clean Architecture principles in your own projects. Let's keep the conversation going! #golang #go #cleancode #cleanarchitecture #softwarearchitecture Check it out and let me know what you think! 👇

6

"We thought we'd solved scheduling in Outlook when we invented 'free/busy' time... Fast forward two decades and now we have systems where one side structures a calendar, and the other side just picks an open time off a website. It’s not unlike what we did with free/busy. I don’t know about you, but I’m 100% on meetings being rescheduled when I pick the time off someone else’s calendar without their input." Automation is a gigantic PITA. If you think it’s not, you’re wrong. Automation is never straightforward. It's always handling exceptions—and almost everything is an exception. It's about abstracting way complexity - and every abstraction leaks. This is a good article. Give it a read: https://lnkd.in/gaRKwR6Z.

2

🏎️💨 Making of a Ridiculously Fast™ API client in R. Why my package is **~17x faster** than the community alternatives that call the **exact same API endpoints**. 👇🏽**design choices matter** ✨ https://lnkd.in/ggi_2XyQ

129

2 Comments

Technology is often accelerated by pressing need, which is currently most apparent in how we’re tackling the #climatecrisis. The damage we’re inflicting on our planet could hardly be more severe. Already, we’ve passed the 1.5 degrees threshold, and it’s looking as though temperatures might reach up to 4 degrees by the end of the century. So, while sending parasols into space to block the sun’s rays might seem like something out of science fiction, it’s ideas like these that could be our last hope. #sustainability #ESG

1

The Aha! Moment 🎉 🎊 As a software engineer, we often find ourselves deep in the trenches, wrestling with complex concepts that just don't seem to click. It's frustrating, exhausting, and sometimes, you wonder if you'll ever get it. I've been struggling to understand some very difficult concepts in Python for a long time. It felt like hitting a wall over and over. But I persisted, kept pushing through the frustration, and today, something amazing happened. Suddenly, all the pieces started to fall into place. What once felt like chaos now makes perfect sense. It's like a fog lifting, revealing how everything is connected. This is why we keep going. These moments aren’t just about grasping a single concept; they’re about seeing the bigger picture, recognizing patterns, and realizing how it all fits together. It’s a breakthrough that changes how you approach problems. So, if you're in the thick of it right now, feeling stuck, just remember: the aha moment is out there waiting for you. Keep pushing, keep experimenting, and don’t be afraid to make mistakes. They often lead to breakthroughs. What's your most memorable aha moment in coding? Share your story—I’d love to hear how you broke through the tough spots! #TechCommunity #CodeStory #SoftwareEngineering

3

1 Comment

Devs, want one tip for improving your design communication? Add diagrams! I just finished reading a 12-page document describing a distributed-database crash-recovery protocol and there wasn't a single picture! It may all be clear in your head, but describing complex relationships in prose isn't going to get the same level of understanding as if you draw out the relationships as well, especially if there are temporal components. When I was leading EBS, one of my colleagues suggested a rule: that every change proposal had to have at least one diagram or it wouldn't be considered. Now occasionally this meant that a very simple proposal would have a silly diagram just to meet the retirement. However, more often what the proposer thought was a simple explanation wasn't so simple to the readers and the diagram really helped!

97

5 Comments