Matthew Weier O'Phinney

If you're the kind of person who's excited about all the recent S3 feature announcements (bucket policy unlocks via temporary root sessions, up to millions of buckets per account, pagination and region filtering for ListBuckets, conditional writes, ListCallerAccessGrants API) and you're attending re:Invent, then you should come to our storage track session, STG306 Advanced security patterns on Amazon S3. Learn how to use all the exciting new stuff securely in the most interactive format available at re:Invent!

94

2 Comments

This is a bittersweet post (more on that later) and long. When I joined The ASSISTments Foundation over 4 years ago, our platform was a hodgepodge of code and frameworks accumulated over decades. Ruby on Rails, GWT, JSP, Java, an API with no standard, and a database and data structures that had long since lost their ability to scale with the kinds of things we needed it to support. All running on a few physical servers at WPI. There was SO much work to do if we were ever going to have a foundation on which to truly innovate. Over the last few years, we’ve moved our entire infrastructure to the cloud, rebuilt the teacher experience and student experience in Vue.js from the ground up, and launched a brand new product for school admins and instructional leaders. But the last step was the biggest: we had to refactor all five of our products to use our new RESTful API (SDK 3.0), migrate terabytes of content to our new database structures and build an entirely new content management and authoring system to support everything. Plus, we wanted to include some of the first new features not possible on the old system: new item types and an alpha version of teacher authoring for our longtime ASSISTments 1.0 users. On Tuesday, after well over a year of the most complex refactoring you can imagine, the team delivered flawlessly. Even though this is not the Oscars (and I certainly am not “the winner” of this), I’m now going to go over my time limit and praise the amazing team responsible for this: First, Director of Engineering Ryan Emberling and Senior Product Manager Cindy Starks organized and managed the entire project through the many, many curve balls we were thrown and pivots we had to make, building the management systems on the fly as we figured out this herculean task, with help from Michael Weingarth owner of our district admin product. Lead Designer Kyle Scheuing, working with our Director of Content and now Product Owner Dawn Peterson designed an intuitive but amazingly powerful authoring environment that is legitimately worthy of a design award if such things were given for authoring environments. Our engineering leads, Christopher Donnelly, Joan Wong, and David Magid—along with Ryan—self-organized in true agile fashion to lead the rest of the engineering team through countless hours of analysis, architecture decisions, code review, and God-knows-what-else. And finally, our amazing engineering team: Danita Forbes-Sample, Manuel Gonsalves, Rownok Nowrose, Brian Rojas, Jaime Torres, Jose Velazquez did months and months of work rewriting vast swaths of our codebase. I am so proud of this team, and honored to be able to work with them. Their achievement has set the stage for innovation at ASSISTments for years to come. The bittersweet part of this is that I will not be with them for that journey. I will miss this team incredibly, and I hope our paths will cross again.

44

8 Comments

I've been an engineer on call in both sprawling enterprise IT settings as well as narrower settings such as running Apache Pulsar and BookKeeper at scale. What I'm reading about wide events, canonical log lines and what Charity Majors is calling Observability 2.0 has me saying YES! But... Yes, wide events and canonical log lines are obviously a great user experience for the on-call engineer under pressure. I worked at Splunk for a while, I've seen what a powerful slice and dice machine over structured logs can do. But it is expensive! What I haven't seen yet is a good discussion of infrastructure costs. How this can be done without exploding costs? Have we designed observability data systems capable of storing huge amounts of structured wide events, with low latency queries and reasonable costs? I'd love to see a paper or a blog post or something that explores wide events from an infrastructure cost perspective, discussing the design space, factors such as scale, cardinality, query patterns, and all the associated trade-offs.

49

Super interesting (and technical) breakdown on how the folks at ClickHouse are transforming the simple, flexible JSON format into something that can be leveraged at hyperscale. An excellent example of how sometimes scale can make you do crazy things.

1

Ever have a problem migrating your production sized statistics to your development systems? Why would you want to? How would you do it? Come listen to Todd Mollenhauer discuss this lively topic! Register today! https://lnkd.in/gD8gTW5p

19

Meta needed a survey to tell them what every team should already know… Delays are problematic. Phase gates are problematic. Large queues are problematic. Large batch sizes are problematic. This *should* be 101 stuff in our industry, but it isn’t. Most teams do pull requests out of blind obedience to “best practices” without seeing them for what they are. The way to reduce cycle time isn’t to nag people to review pull requests, it’s to eliminate the pull requests. And no, you can’t do that alone, you need practices in place that ensure you review code in real time and invest in people so they have a better sense of the target intrinsically. It’s not a quick fix. Meta sought a quick fix and decided to effectively tell people stuck in traffic to drive faster. That’s not how context switching works. It’s not how queuing theory works. It gets worse though. Meta did this and now this article was written lauding it and plenty of people in the comments wishing that they too could introduce auto-nag tooling as a quick fix for their cycle time problems. There is no instant pudding.

7

Having a ton of fun writing an ORM for V2 of Case Kit. Typescript has such an incredible type system. I have autocompletion for column names, the ability to include relations, relations of relations, etc etc etc... all returning properly typed data (no classes in sight). It's been really satisfying seeing it coming together, and it's been great getting my hands dirty with some (relatively) complex generics. And to head off the obvious question… why aren’t I just using Prisma? Well there’s a couple of things I’m doing differently that I think are marginally better: there’s no code generation involved here, it’s just types (inspired by Drizzle); there’s no Rust binary you need to deploy, and it hews much closer to Postgres behaviour and syntax (the aim here is not to abstract away which database you’re using). But the real blocker to me using Prisma was that its extensions don’t support modifying queries of nested relations - a hard requirement for me due to the way I have multitenancy and soft deletes set up. It’s not shown in the video but the Case Kit ORM lets you write simple middleware that will be applied to every level of a nested query. Perfect for setting timestamps or adding clauses on tenant id or things like that.

21

"No More Blue Fridays" and eBPF, it's just a one-minute reading: https://lnkd.in/gAAmNU4D if you haven't seen it. That's precisely what I speculatively said to my colleagues, without even reading anything about it, answering their question about how that could happen: "BlueFriday was most likely triggered by the ring0 kernel driver accessing the invalid memory. It's hard to say without knowing details, but Ring0 could be the only way on Windows to access a particular functionality, like Network Stack." Undoubtedly, eBPF is a great tech, but is it a silver bullet? Can it replace Ring0 or kernel drivers? I do not think so. It's excellent for its niche, but is it safe? No. Is it safer than kernel drivers? Yes, it is way safer (as well as any other indirection layer that provides an infiltrated interface).

3

This post from Dan Lorenc is spot on. Publishing code gives you no obligations, but participating in a community does. I'll expand the definition of "community" further, though, and say that if you have a public issue tracker associated with your code, that's implicitly an invitation to community with bring some obligations. I'll think about this some more and probably write a blog post once my thoughts have coalesced.

4

If my team had $1 for every time someone said that they were lucky to have a technical writer embedded on the team, we’d all be millionaires! 🚀 For many companies, technical writers are not embedded into engineering teams. They probably should be, here’s why and how you should do it: Read more: https://lnkd.in/eJmuP8XK #technicalwriting #techComm #techWriting #ContentStrategy #documentation #informationdevelopment #devrel #DeveloperRelations #DevOps

8

Interesting discussion with Dave Farley and Emily Bache. There are a lot of things that I agree with in it, however the one main sticking point that I have is that a true "engineer" in the more common incarnations, Mechanical, Civil, etc, require registration and licensing from a local state/district and/or governing body. I'd LOVE for programming to have such a system, but we currently don't. That's one reason why I settle with code craft and call myself a crafter. I agree it's not a perfect term, but to me its better than giving the false impression that there's a standard of skillset or understanding like other engineering disciplines. What do you think? https://lnkd.in/gXqmkden

15

31 Comments

Given my day job washing bottles at GitKraken, I often have conversations with other execs that gravitate to developer experience (DevEx). Is it possible to define & measure it? What does my company get out of it? Is DevEx just code for delivering gourmet lunches to my engineers' homes? Matt, how are you so consistently charming? Fine, the last one hasn't happened (yet, but I'm hopeful 🤞), but the others come up multiple times each month. Here’s the thing: DevEx isn’t just a concern for developers - or even just engineering leaders. It’s a critical factor that impacts everyone, from CEOs & CTOs to finance teams & investors. And most of all, it's critical to the customers who pay your bills. When we prioritize DevEx, we’re investing in the success of our teams and our products. We’re removing friction, adding features, enhancing collaboration, and driving the kind of innovation that creates real, sustainable success. In this era where software drives our daily lives, the developer experience shapes the success or failure of entire organizations. Because, at the end of the day, DevEx is about much more than just writing & shipping better code (or in-office massages for pets). It’s about building stronger companies, more connected employees, and happier customers - and that's what ultimately creates happy investors. So for the executives in my network, I'm going to set aside some time each week to share real-world insights on what DevEx truly means and why it matters - stats, trends, studies, as well as my own thoughts & examples. And for the devs in my network, get some popcorn and follow along as I convince your CEO and CFO that investments in improving the developer experience pay for themselves many times over. #DevEx #DeveloperExperience #DigitalExperience #CustomerExperience

52

5 Comments

Matt Watson, this is a really insightful post with a strong foundation in cutting edge use cases, which is exactly where businesses thrive in a fast-moving market. I agree that clients prioritize communication and ownership over code elegance. That said, there's a case to be made for the role of 'beautiful code'—not for the client, but for the engineers. As products mature and stabilize, maintaining clean, well-structured code can make it much easier for engineering teams to iterate, onboard new talent, and stay adaptable to evolving customer needs. Refactoring as part of the R&D cycle can keep costs manageable and ensure your team remains productive long-term. Would love to hear everyone's thoughts on how they have balanced this in practice! #CTO #Leadership #TechStrategy

2

Just launched a HUGE update to SciSummary. You can now ask questions about figures and tables in papers and get answers specifically from those tables/figures. It's able to interpret the data in the tables within the context of the paper itself, not as a standalone entity. You can even see here it's smart enough to extract data from the tables! Upload any PDF of a research paper and try it out for free today on SciSummary. https://scisummary.com #DataExtraction #Research #ArtificialIntelligence #Science

24

1 Comment

Donald Raab and I recently had a conversation about his upcoming book Eclipse Collections, Categorically, his 40 years of programming experience, and the sheer joy he still gets from writing code. Whether you’re looking to up your Java collections game, learn career or life lessons from an industry veteran, or just want to be entertained, tune in to my latest podcast episode (link to episode in the comments)

105

3 Comments

Best session about S3. Do not miss it!

4