Scott Persinger’s Post

View profile for Scott Persinger, graphic

CEO at Supercog AI, prev at Tatari, Stripe, Heroku, founder at CloudConnect

One of the main key choice points when doing GenAI app development is choosing your foundation model. Should you go “closed-source” frontier model, like GPT-4 or Claude 3.5, or try to stick with an open source model like Llama3? Or does your use case fit a more purpose-trained model like Gorrilla or Stability? This mostly comes down to testing, and understanding the economics of your app (assuming it’s a commercial effort) to decide which model to build upon. But waiting behind this decision is another key choice - choosing what level of *abstraction* on which to build. The simple approach is to use the native API provided by the model itself. This can be a good choice, but strongly limits your ability to support other models in your app. Prompt instruction syntax, tool calling syntax, and JSON or markdown generation can vary pretty wildly across different models. So introducing some abstraction to represent the LLM offers the possibility of supporting or swapping different models depending on your needs, without requiring heavy surgery to your app. Unsurprisingly there are lots of options vying to abstract your LLM interface, including LangChain, LLamaIndex, vllm and many more. Testing and picking amongst these is a project in and of itself. Generally you evaluate each solution across a few different axes: Does it support commercial or open source models, or both?  Does it look to support “advanced” features of different models, or rather attempt a “lowest denominator” approach to preserve compatibility? What other features (agent behavior, RAG, indexing…) is it trying to offer? How much will you be bound to those implementations if you choose this abstraction? I haven’t seen it yet, but this would be a great opportunity for a “bake-off” type evaluation to produce a matrix to help you understand the different options and trade-offs. Maybe I’ll go ask ChatGPT to write it up… #ai #genai #llm

Scott Persinger

CEO at Supercog AI, prev at Tatari, Stripe, Heroku, founder at CloudConnect

5mo

Heh, ok GPT-4o did a passable job, with a very simple prompt: https://chatgpt.com/share/a8582be8-4ace-4d9f-b3ab-f4dbff8f7c09 Maybe you can come up with someone better?

Like
Reply

To view or add a comment, sign in

Explore topics