Shaun O'Reilly

How are you deploying AI? Ensuring the effective deployment of Artificial Intelligence Systems in a secure manner requires a clear understanding of the complexity of the AI system and infrastructure used. Congratulations to the Australian Signals Directorate and the Australian Cyber Security Centre for collaborating with other global cyber authorities to provide clear and actionable guidelines for deploying and operating AI systems. ttps://https://lnkd.in/gXtJgg7Y

20

😲 One skill that separates levels is your ability to communicate complex requirements to different stakeholders. As a junior engineer, you’re still working on engineering skills. As a senior engineer, you’re working on implementing complex ideas, mentoring other junior engineers, and assisting in architecture. As a principal engineer, you’re now leading the architecture for some of the most complex needs inside the organization and leading teams to successful implementations. As a distinguished engineer, you’re now leading cross organizational architecture, working with competing priorities, pitching to vice presidents, and mentoring principal engineers. At each of those levels you’re communicating with different stakeholders. 💡 It’s your responsibility to communicate effectively and take feedback seriously so you can produce the best outcome. 🤔 Communicating with other engineers is drastically different than communicating with vice presidents. They want to know different information. The very best engineers can communicate to all of them. Your ability to write code doesn’t determine your vertical ceiling. Your ability to sell and communicate your contributions will.

11

DuckDB What the Duck! Matt Martin has been talking about DuckDB for a while now and has written some excellent posts on it. Last week, he mentioned it to me again, and I replied, "Is DuckDB even a real database?" That comment led me to spend a good portion of my weekend digging deeper into DuckDB. WOW. It is not what I expected. Yes, it is a database that runs in a single process, but when you consider how powerful a single process can be, things get really interesting. My old laptop, with 12 vCPUs and 64GB of RAM, can handle a significant amount of data. Now, think about what you can get in the cloud. AWS allows you to spin up a Fargate container with 16 vCPUs and 120GB of memory, and if you're willing to spend some money, you can get EC2 instances with over 192 CPUs and 1.5TB of memory. My favorite part of DuckDB is it can be used like any other library in Python, Java, JavaScript, Node.js, Go, Rust, etc. There's no server to set up, no connection to manage—it runs directly in the main process. There is also a CLI. The possibilities of what you can do with the CLI are endless. You can read in a CSV or JSON file with a single command and run full SQL commands against it or you can pipe data from the console. Imagine the next set of log files you need to query on the server and how easy it would be to find what you're looking for with a few simple SQL commands. With a simple command, you can load a directory of CSV, JSON, or Parquet files. By enabling a small extension, you can remotely read Parquet files from S3, hit an API endpoint, and more. The value as a transformation engine over Spark and Pandas for processing smaller (multi-GB) files is mind blowing. The full power of SQL embedded within your existing code. One of the Python features that sold me is the ability to create a Python function (full Python code) and execute it as an UDF within a SQL statement. Now that is powerful. One feature I can't wait to play with as a front-end guy is the WASM install. Yes, because DuckDB's zero dependencies you can run it in the browser. Checkout DuckDB's website for an example: https://shell.duckdb.org/. I truly see DuckDB becoming part of my go-to solutions for a number of tasks, big and small. My hat is off to the engineers of DuckDB. You solved a problem I didn't even know I had. Big shout out to Mark Needham, Michael Hunger, and Michael Simons for writing DuckDB In Action (https://a.co/d/4ZGcDoo). It has been a great read and opened my eyes to what is possible. #data #duckdb #dataingestion #sql

96

31 Comments

🧑‍💻 For the technical folk 🤓 I did some simple refactoring of some cloud functions yesterday, and the cost gains have been surprising. By simply removing unused packages, importing parts of libraries instead of the full library (think import 'lodash/get' instead of import 'lodash'), the functions have been running faster, with less $ costs. When you're building an MVP, you have to be messy for speed, but it's important to come back and clean that mess in a cost-conscious manner. This is especially true when working with Cloud Services. If you're running Firebase Cloud Functions - here's an article that helped me quite a bit: https://lnkd.in/dQMCm3Rq

117

13 Comments

A good use I've had lately with ChatGPT is when I'm debugging code and I've spent more than 20 minutes and can't crack it. 9 times out of 10, I have a missing parenthesis, a variable type issue, or a param passed incorrectly to a function. I will simply paste my code in ChatGPT and say "Fix this" and it usually pinpoints the problem on the first try. I don't have to tell GPT what coding language it is or even what the error message I'm getting is. It is a huge productivity boost. It has saved me a ton of headaches. ..the end. #dataengineering

72

30 Comments

Came across a new idea in https://lnkd.in/gZFBXWbJ ``` ... the later in development an issue is detected, the more likely it is that it's addressed tactically rather than structurally. Quality issues detected late are more likely to drive improvement in specific correctness (fix the test case) rather than in fundamental approach (redesign the architecture).``` In other words, the sooner we spot something, the more likely we are to fix it the right way, rather than apply some kind of band-aid. I'm usually advocating for moving feedback earlier in dev cycles for the sake of maximizing speed and minimizing bugs that make it out into the wild, but this is different. We've all experienced hassles late in some process and taken shortcuts to smooth things over. Fatigue and sunk-costs are pretty persuasive when you're close to the finish line. Yet another reason to make quality visible sooner!

3

I had recently published a few posts that showed how to do some DE tasks with C, comparing it to other programming languages. Subsequently, I've gotten several DM's from people that are confused if they should learn and use C for data engineering projects. To help bridge that gap, I created a write-up on a basic tutorial for C from a data engineer's lens. This should help ground you on what C should and should not be use for. Link below. Enjoy #dataengineering https://lnkd.in/e3NvkNAC

10

1 Comment

Parsing XML can be such a PITA. I know that there are many packages out there to work with it, but what happens when the XML documents have varying counts of attributes and sub elements throughout? I’m going to publish a new article soon with my twist on how to handle XML at scale when the documents are very complex and not consistent. Stay tuned. #dataengineering

33

20 Comments

𝗪𝗮𝗻𝘁 𝗮 𝗳𝗿𝗲𝗲 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝘁𝗼𝗼𝗹 𝗳𝗼𝗿 𝗚𝗼𝗼𝗴𝗹𝗲 𝘀𝗵𝗲𝗲𝘁𝘀 𝘁𝗼 𝗺𝗮𝗸𝗲 𝘁𝗿𝗮𝗰𝗸𝗶𝗻𝗴 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝗮 𝗯𝗿𝗲𝗲𝘇𝗲? This sheet automatically generates alerts on due and overdue steps so you know exactly which task and step needs attention! Here’s the kicker: You can jump directly to any alerted cell with a simple click! Plus, filter features display only the rows needing attention. Once you add the workflow steps as columns, just start filling each row with the tasks that will follow those steps. For each task row, its first cell will be automatically colored if any of its column cells has an alert background color. 🌟 Highlights: 1. Smart Navigation: Jump right to alerted cells using clever hyperlinks. 2. Visuals: Conditional formatting paints expired tasks red and upcoming deadlines yellow using just a single conditional formatting. 3. Use of an "ARRAYFORMULA" to set values of an entire column (see cell B2 "week") based on other columns. 4. Interactive Feedback: Get real-time updates with friendly "toast" messages. 5. Shows a dynamic installation menu only when needed, making it easy to copy the spreadsheet to create a new workflow. 6. Optimized code to maximize performance. 7. Flexible Design: Tweak columns as your workflow evolves; the script code adapts seamlessly. 8. Multiples users can fill the workflow at the same time. Go to Github to see the code or make a copy of the spreadsheet: https://lnkd.in/eFdy5G7E Enjoy! Made with ❤️ using #GoogleSheets and #GoogleAppsScript #WorkflowManagement #googleworkspace #ProductivityTools #OpenSource

12

GraphQL servers with nodejs - code first or schema first development? I’m torn between these choices, as I originally made an API that is generally great with hasura but I find I’m having to work around it when doing extra specific things like image and media storage. So that’s great for graphql instant generation, but not so sure about the code base so far. Other alternatives I’ve seen are supabase, postgraphile, but I’m mainly toying with a straightforward nestjs and prisma build - anyone here found their best route with this?? https://lnkd.in/gDnxRX8c

3

1 Comment

Testing often gets a bad rap among developers, seen as a time-consuming and complex layer added to the development process. Here’s why it's actually a cornerstone of successful projects: 🔍 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗦𝗹𝗼𝘄𝘀 𝗧𝗵𝗶𝗻𝗴𝘀 𝗗𝗼𝘄𝗻 It might seem that way, but integrating regular testing actually speeds up the development cycle by catching bugs early, preventing costly fixes later. 🧩 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗔𝗱𝗱𝘀 𝗨𝗻𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆 𝗖𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆 Testing clarifies the project requirements by forcing us to think through the use cases and potential failures, leading to a better understanding and a cleaner codebase. 💡 𝗪𝗵𝘆 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗶𝘀 𝗘𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹? 🚀 Ensures the final product works as expected, enhancing user satisfaction and trust. 🔧 Catching issues early means less time spent on revisions after the project’s release. 🔄 Promotes writing cleaner, more maintainable code, which makes future updates easier and more reliable. Testing isn't just about finding bugs; it's about ensuring a smooth, efficient, and cost effective development process. Embracing it can save much more than it costs. 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳𝘴, 𝘩𝘰𝘸 𝘥𝘰 𝘺𝘰𝘶 𝘪𝘯𝘵𝘦𝘨𝘳𝘢𝘵𝘦 𝘵𝘦𝘴𝘵𝘪𝘯𝘨 𝘪𝘯𝘵𝘰 𝘺𝘰𝘶𝘳 𝘸𝘰𝘳𝘬𝘧𝘭𝘰𝘸𝘴 𝘵𝘰 𝘮𝘢𝘹𝘪𝘮𝘪𝘻𝘦 𝘣𝘦𝘯𝘦𝘧𝘪𝘵𝘴?🧐 #development #testing #QA #softwareengineering

3

Functional builders in Java with Jilt #Java #Jilt

9

We need to talk more about the current inequality between Malicious Threat Actors and Australian SME Businesses. Because it is not even close to a fair fight at the moment. Malicious Threat Actors are investing and using as much Tech and even AI to assist their Cyber Attacks as they can. Australian SME Businesses are investing and using as little Cyber Security and AI as they can. It’s literally like sending the local U12’s side out to play the Australian Team ⚽️ #cybersecurity #cyberinsurance #infosure

5

Based on the post I made a little while back, I put this module together. It will let you create HTTP handlers that do server side events with NATS.io. https://lnkd.in/dJeVKM2H

22

I spent some time and completely rewrote the whole NATS micro framework for Piggybank. Now instead of headers, it's more of an idiomatic NATS approach using the verbs in the subject. I also updated the subjects used for database actions and secret actions. The binary now also includes a subcommand for database and secrets actions. I don't think anyone was actually using this so I figured completely changing things around wouldn't matter. I think I'm going to actually try to make this a real project. https://lnkd.in/eDuaWtrY

19

There has been a lot of hype around the Rust programming language. Most articles and blogs I read are heavy on promises, and a little light on testing. I’ve done a simple comparison between a lambda written in Python vs the same lambda written in Rust. Both use the official AWS SDKs. Boto3 is well established but the Rust SDK is new. The official release was about 8 months ago at reinvent. I’ve load tested both using K6. Why? Well, because I want to see for myself if I really do get the 8x improvement in performance and equivalent reduction in cost that people promote. What were the results? Come along and find out! https://lnkd.in/gfzVfDXY

8

10 Comments

⚠️ Attention Merchants ⚠️ Our partner, Australia Post, has scheduled maintenance that will result in rating, shipment, and tracking API outages for their services this weekend. The Shipping & Tracking APIs will not be available during the following times: 🔴 Friday, 2 August 11:00pm AEST - Saturday, 3 August 12:00am AEST 🔴 Sunday, 4 August 10:30pm AEST - Monday, 5 August 12:00am AEST Please note this maintenance impacts all Shipping & Tracking APIs, Parcel Send, and MyPost Business. Services may resume earlier than 12am. ShipperHQ Customers — remember to switch on a Backup Carrier to avoid rate disruptions. Check the comments for more information on preparing for carrier outages like this. #Ecommerce #Shipping #AustraliaPost

7

1 Comment

As developers, we're always seeking innovative solutions to streamline our workflows and enhance collaboration. Dev Tunnels offer a powerful way to securely expose local development environments, enabling seamless testing and debugging across teams. Check out this post to learn more about how Dev Tunnels can revolutionize your development process. #DevTunnels #Development #Collaboration #Innovation

4