Graham Innocent

I ran across this post in the Changelog newsletter: https://lnkd.in/eCdjVekq - I like the name slop for unwanted AI-generated content. I love LLMs. They are part of my daily workflow, and I've been exploring them more seriously while on my sabbatical. I wouldn't misrepresent generated content without critical review as my own, but the web seems to be filled with more generated garbage (or 'slop')—content that looks fine at first but isn't accurate or helpful. LLMs (and other generative AI) should help us to make things of higher quality, not just churn out slop more quickly.

11

1 Comment

Meta's Llama 3 dropped just a few days ago, but already there are close to 1,000 variants publicly available on Hugging Face. Most people read base model evaluations and think they apply to all these deployments, but that ain't the case. Or at least the extent to which it's true is poorly understood. I want to see more eval work focus on the performance and risk shadow cast by base models on their downstream progeny. Basically, if base models aren't being used as-is, do their evals matter? Or are they poorly predictive of what will come of them post-fine-tuning? I don't have time/resources to work on this right now, but I'd love to see someone take this project on: across all variants posted on HF, how do evals on fine-tuned versions compare to evals done on the original base model? What kind of relationship is there between the two? Is the latter a constraint or directional anchor for the former? The best set-up here needs to find some way of ignoring some amount of noise (e.g., devs creating their own copy but not doing any fine tuning or doing some kind of terrible fine tuning) in order to focus on more serious attempts to fine tune for stronger performance on specific use cases, stronger safety and security, or something else. This question is increasingly important because fine-tuning is becoming easier and faster to do. That means it's more and more likely that developers will tune models to their own liking. Evals done on prior upstream versions are potentially meaningless... 'potentially' is the key word because we don't have a good understanding of the persistence of performance and risk, the elasticity of learning. By the way, if you're interested in working on hard questions around LLM evaluation, like these ones, we're hiring! Our Trust & Governance research team is looking for a Staff Applied Research Scientist. Come help us tackle these challenges! Link to the job posting in the comments. #artificialintelligence #trustworthyai #llm

51

9 Comments

🚀 💡Gemma 2, the next generation of open AI models, is now available to researchers and developers. It offers exceptional performance and efficiency with sizes of 9 billion and 27 billion parameters. Gemma 2 supports various AI tasks and integrates seamlessly with popular frameworks like TensorFlow, PyTorch, and Hugging Face. The models are optimized for various hardware, including NVIDIA GPUs and Google Cloud TPUs, ensuring versatile applications across different projects. This release is designed to drive innovation and accessibility in AI development, providing powerful tools for creating advanced applications. https://ow.ly/fGHp50Sx3UU #Gemma2 #GoogleCloudTPUs #MachineLearning #TensorFlow #PyTorch #HuggingFace

13

1 Comment

I see more and more chatter from people these days claiming that LLMs don’t actually work very well for programming. At the same time I noticed that an opensource project I have been using with customers had many open GitHub Issues going unresponded to even weeks after they were posted. With some time to kill during some long running deployments I picked a hand full and replicated each problem locally. With zero additional language, context or prompts the solution to each ticket could be found by just pasting the GitHub issue into regular chat gpt and trying the solution proposed. All I then did was paste each of those outputs as a comment and the author of each ticket popped up to say the issue was resolved. When people come and ask me about prompting I ask to watch them use an LLM first. After about two minutes I tell them what I have been doing on GitHub and then wait to see if they figure it out. The reason GitHub issues make good prompts is because when you sit down to write them you are asking a question hoping to get the attention of another human being to help you solve the problem you write the appeal and share the information in a particular way and that is one of the best ways to prompt.

4

Well said! As parents, we all want to ensure our children’s online safety, especially when it comes to popular games like Roblox. KidsNanny’s Screen Scanner feature adds an extra layer of protection by automatically detecting inappropriate content on your child's screen in real-time, whether it’s through conversations, in-game interactions, or suspicious offers. This ensures parents are immediately notified if any harmful behavior occurs, allowing them to take swift action. KidsNanny.ca - AI-powered Parental Control App

7

Spotify announced generative AI powered playlists in April in beta, but only recently seems to have started mass scale rollouts. I haven't had time to try them out till earlier this week. And I was able to prompt it with "create a playlist with soulful blues and jazz which match singers like Norah Jones and Melody Gardot with the depth of Adele's voice. I'd like a deep bass rhythm as well". And it produced excellent results. I *love* genAI usecases that reduce manual toil and this is one of them that hits the mark. Now obviously, as I listen to the music, the engineer in me leads me to always dig into the how - and while I wanted to deep dive on their GenAI approach, I landed on a more fundamental post on annotation scaling that I resonated with and is actually the underpinning to great outcomes. Better annotations, and being able to scale annotations beyond pure manual tagging leads to better fine tuning which in my experience is often a critical step over and above prompt engineering when you want to actually launch into production. https://lnkd.in/epDFCxCb #genai #music #experiencesthatreallyimprove

69

1 Comment

Hyperoptimisation from a product that works perfectly well seems to be incredibly value destructive for all concerned. Not sure why it happens so much in tech I learned today that something coming up imminently on our engineering road map is updating all our lambdas to be compatible with new AWS standards. Apparently this is something that will happen every 18 months or so forever and GCP is apparently even worse. If we don’t do this by the time AWS deprecates this functionality Spot Ship will stop working. However doing it will lead to zero improvement in functionality for us. AWS has literally spent money on engineering that has had no effect but to waste our engineers time and subsequently p*ss me off. They are not alone. There comes a time in a product life cycle where it feels engineering dollars are being spent for the sake of it. Why not either downsize workforce, or give the guys and girls a very long very well paid holiday (if trying to keep them out of competitor hands)??? While our product has got so much better over the last year (primarily by ticking off the most crucial of existing paying customer asks), I want to stay aware that at some point it will be essentially “done”. Any further “improvement” will just annoy clients. Engineers love to say SAAS is alive and never “done”…but I think this may be largely talking their book I’ve seen it a lot in consumer (e.g. having to update the Uber app when you just want to get a car back from the pub once every two weeks…despite them having added no noticeable new functionality for 10 years). This is the first time I’ve noticed it so clearly in B2B. #shipsandshipping #maritime #founder #vc

121

7 Comments

I’ve been thinking a lot about this. The post below highlights a troubling trend: SaaS companies explicitly opting out of serving the EU market because of regulatory challenges. And honestly, I can relate. As an EU citizen, it’s often easier to start something targeted at the US market and distribute it there than it is to operate within Europe. The US market is massive, vibrant, and—crucially—easier to navigate from a regulatory perspective. On the other hand, Europe, with its patchwork of regulations, feels like a minefield. GDPR, while well-intentioned, often becomes a barrier rather than a facilitator for innovation. Take the cookie banners, for example—those intrusive pop-ups we’re forced to navigate on nearly every site. They’ve become a meme at this point, something everyone ignores or blindly clicks through. What’s the point of regulations like this if they alienate users and stifle innovation while doing little to actually protect privacy? We’re seeing innovation bypass Europe. How did it come to this? Where is Europe heading if the cost of compliance outweighs the reward of serving the market? It’s sad to see, but it’s the reality we’re living in. Unless something changes, I fear this trend will only accelerate. Europe needs to rethink its approach if it wants to compete on the global innovation stage. Would love to hear your thoughts on this. Have you had similar experiences, or do you see a way forward?

4

1 Comment

I'm also taking the under. Generative AI is making rapid advances in how engineers write and deploy safe code. But it is not innovative, and barring massive advances in mathematics (i.e.: we're at least solving P=NP with this one) it never will be. As an industry, we in GenAI do not have succinct, independently verifiable definitions on common phrases like "AGI" or "AI Superintelligence." Even within technical circles we don't uniformly understand the sweeping differences between how machine learning powering modern LLMs work vs. human intelligence. The biggest risk IMO in AI isn't removing humans from the equation in creating technology: it's letting poor marketing set unreasonable expectations for our technology based solely on messaging meant to drive content/influencer engagement. I'm excited about where this next frontier in computing is taking us. I know it is going to (and already having) a massive impact on how humans interact with/manage/deploy automation in computing. But if we want to get there, we need to be honest with where we're going.

9

7 Comments

See from experimental results why open text formats work best for training LLMs and the problems AI has with generating complex, structured information. https://lnkd.in/etQS5JbE #LLM #Text #GenAI

23

1 Comment

At ScreensAI, we're focused on driving as much accuracy as we can with LLMs. But for contract review and playbook execution, it's often the case that vague instructions and subjectivity are as much to blame for undesirable results as the LLM's reasoning ability. Say you want to use an AI system to help determine if the following standard is met across thousands of contracts in your repository: 📜 The contract should limit the vendor's liability to 12 months’ fees. From our recently published Screens Accuracy Evaluation Report: "While any given individual contract reviewer may have a reasonable understanding of what they’d expect to pass or fail this standard, we see many edge cases in practice. If preferences for those edge cases are not clearly articulated, the LLM is left to assume and make a judgment call. Consider these scenarios: - What if there are major exceptions to this liability cap that weaken it so substantially that it might as well not exist? - What if this liability cap doesn’t apply to beta features? Or some specific set of features? What if the service is free? - What if the liability is capped at either 12 months’ fees or some fixed amount, whichever is greater? Or whichever is lesser? What if the fixed amount is very large? Or trivially small? - What if there is a secondary cap or a backup cap that applies under certain conditions at a much different amount than 12 months’ fees? - What if the cap is only for fees paid for the services that gave rise to the liability rather than fees paid in general under the agreement? In any of these cases, the LLM is left to assume. The leading LLMs are capable of reasoning through this type of nuance, but if they are told they must make a decision as to whether or not the standard passes, they might not make the same decision that you would if you don’t expose your preferences." 📊 If you want quantitative aggregate results to assess the risk of a portfolio of contracts, thousands of verbose "it depends" responses won't cut it. Contract professionals have strong opinions, institutional knowledge, and well-defined playbooks. ScreensAI helps them articulate this to the platform up front, systematically validating the types of LLM-powered results they'll get for the given set of expressions of their preferences. Then they can deploy this expression of preferences at scale in perpetuity. 🧙‍♂️ We think this is where a lot of the magic happens in LLM-powered contract review.

21

5 Comments

Smaller and more informed (#RAG/LORA) models are faster and more useful in business practice. You shouldn't treat #LLM's as a general purpose silver bullet or you will have never create a product of any value. Always pick a composite of the right model and data for the problem. This three point strategy (Model, #Data, Problem) will help you create better results for you and your consumers. Don't be afraid to chain results together to generate a larger encompassing solution.

3

Enjoyed reading Ernest Davis's critique today (shared by Gary Marcus) of the recent study by Porter and Machery on preferences by many people for AI-generated poetry over the work of famous poets - Davis - https://lnkd.in/eNTfD8fN Porter and Machery - https://lnkd.in/etqJhbs7 The image is my own contribution to the genre, which displays the same characteristics as the examples discussed by Davis: an arresting original on the left and a banal pastiche on the right. One of P and M's conclusions is that "the simplicity of AI-generated poems may be easier for non-experts to understand, leading them to prefer AI-generated poetry and misinterpret the complexity of human poems as incoherence generated by AI." That sounds a reasonable interpretation, though in some respects they go further than that, in ways that Davis criticises. The analogy with legal work is that, on something complex and subtle, such as a written argument in a dispute 1. there's a huge gulf between what looks plausible to the casual, non-expert reader and something produced with expert insight and judgement - in which less is very often more; 2. of course, there's a lot of mediocre hackery out there in law as in poetry; 3. but even mediocre lawyers can mostly engage somewhat with the facts of the matter, and the needs of their clients - they're not just playing word games, most of the time (yeah, I know, feel free to give your favourite counter-examples...); 4. and a really bad legal hack at least has some prospect, admittedly rather slight, of being held to account in some way for failure to engage with facts and law; 5. an open question is whether AI will start to supplant mediocre legal hackery with something worse or whether it will act as a stimulant to an increase in standards by human lawyers. A bit of both, perhaps.

11

8 Comments

🔍 Does Intelligence in AI Equate to Economic Value? I've been wrestling with this question for a while now. As I've worked with LLMs, it's become clear to me that intelligence doesn't necessarily drive economic value in AI—utility does. LLMs don’t need to “understand” to be valuable. They can iterate, co-create, and assist in ways that save time, streamline processes, and boost productivity. The real impact comes from how useful these tools are in everyday tasks, not from any notion of human-like intelligence. In my latest blog, I explore this concept further and discuss why the value of AI lies in its ability to provide practical, iterative solutions that enhance human capabilities. Whether AI gets smarter is secondary to the immense ROI it already offers today. Read the full blog here: https://lnkd.in/g2JjwGET If you've been pondering the relationship between AI, intelligence, and economic value, I’d love to hear your thoughts. Let’s discuss! #AI #ArtificialIntelligence #EconomicValue #GenerativeAI #LLMs #UtilityOverIntelligence #Iteration #8thLight

9

2 Comments

Sharing TechCrunch’s analysis of Altman’s latest post, calling his claims as just Hype or actually Believable. I want to believe, really :) Yet when someone from the US presumes to speak on behalf of the entire world (news flash: the US is not the world), the entire planet, improving the lives of billions of people, that’s when you lose me. Why? Just pause and see what’s going on today, right now. The world is kinda still mad. Wars. Poverty. Selfishness. Us versus them. Willful ignorance of blatant violation of human rights happening daily. If we were serious of using tech for good, we would consider pausing and fixing up the world. Today is Heritage day in South Africa. See what’s possible when diverse people put their history aside, acknowledge difficult truths, forgiveness and promoting Ubuntu, striving to be stronger together as one nation. Yes, we still have lots of issues to address. We’re aware of our current reality and can speak candidly about it. How many people can do this in the world today? South African stood bravely and had the courage to speak out, fight injustice and call humanity to wake up. Can AI accomplish the same? How can AI solve the world’s hardest problems? Maybe it will. But will its human subjects raise above themselves and do what’s best for ALL humankind? Time will tell.

10

1 Comment

Well, that's an interesting, but using more sophisticated ChatGPT o1 model for coding actually leads to exponentially more time spent for debugging and understanding the code. Here we're facing a problem with CoT approach - perhaps it really scores high on benchmarks, yet the problem with benchmarks is that they all assume just theoretical right/wrong answers within known knowledge, and all the tests and benchmarks assumes concise answer. Yet the software development is far above that as most of the modern software systems which are created for humans and humans only includes myriad aspects which should be tailored just right, so magic is happening when users are using the software. Although using o1 may benefit for solving concise tasks which can be formulated in the concise way it still far from even moving toward replacing e.g. software engineers and perhaps other jobs as well. It is yet again far from even be more useful than ChatGPT 4. So unfortunately we're facing again all this AI marketing hype. Yes, the new approach is interesting, but it suffers from diminishing usefulness. It generates more code, explanations to the code, but... this generated output is getting harder and harder to use right away with assumption that it will just work. AI can be integrated really into the processes as already a lot of ML is, yet we need to have humans capable of understanding what does it do to make sure that what it does, suggests, writes can be applied e.g. for solving a business task and it won't do more harm than good. OK strawberry you've passed PhD tests which I didn't even hear about. But now you remind me rather crazy professor which should be kept on guard so even before you can be integrated we need to hire two another guys into our company to keep you on check make sure your output is rather useful at all

1

So no doubt everyone has been talking about the latest llama3.1 large language model that has been released by Meta today. It's supposedly equal or better than ChatGPT 4o or Claude Sonnet 3.5. I've just taken it out for a spin on our GPUs and despite all the energy used to built this model it still can't do simple maths! What to know why? Then like and comment below and that'll keep me motivated to finish my next presentation on this topic!

13

3 Comments