Greg Whitescarver’s Post

Don't forget, AI is slow and expensive! I have to withhold the specifics, but we recently built an application that processes text documents of many different shapes and sizes. It identifies, normalizes, and stores a certain category of information so that it can be used in real-time applications. We started by writing an old fashioned application that could handle the most common shapes and sizes. It's extremely fast and it runs on very cheap infrastructure. There's basically zero incremental cost to process a thousand more documents. For the long tail of documents our old school application can't handle, we've engineered prompts for an LLM that allow it to identify and extract the same data reliably. Whereas our old school application processes a document in under 10ms, the LLM is likely to take half a second. The LLM's incremental cost per document can be a whole penny in some cases, which is a lot! And plus there are rate limits; we have to feed the LLM on a schedule to make sure we don't get shut down. Moral of the story: AI is an order of magnitude slower and more expensive than a regular-ass program. So if you CAN do something the old fashioned way, you SHOULD. And AI will help you write the code for that anyway ;-)