Brian Genisio’s Post

We test more than 1,750,000 engineers every year as part of pre-hire assessments for some of the top technology and finance brands. We ran some of the most cutting-edge LLMs (including 🍓) through the same tests and found some mind-blowing insights. 1/ Our benchmarks show OpenAI’s 🍓 (o1-preview) dramatically outperforming all other AI models. However, the race between all other models is incredibly close with Claude Sonnet beating GPT4o in absolute score and GPT4o getting the upper hand in solve rate (i.e. getting to a fully working solution). 🏎️ 2/ While top AI models outperform most engineers in coding, the top 20% of engineering candidates can't be challenged even by the all-mighty o1 model! 💪 3/ AI performance improves drastically with multi-shot settings (especially for `o1-preview` and `o1-mini`), too many shots can lead to diminishing returns. The sweet spot seems to be 3-5 shots for models like 🍓 and GPT-4o. Also, some models can go off the rails when given too many shots. 💉 4/ The future of software engineering and technical assessments? Human-AI collaboration. 🤝 We’re building tools to ensure engineers are ready to co-pilot with AI, where the synergy between human intuition and AI strength will drive innovation. See comments for a link to the full research. #technology #innovation #future #artificialintelligence

Interesting finding, what is your take away from this? 1) AI cannot match humans 2) AI can augment average coders to the level of the Top 20 Perspective, a human trait, will make all the difference in how you see leveraging these AI tools.

The conversations about "AIs taking over the world" have recently felt exaggerated. While generative AI is undeniably a powerful tool and a valuable source of intelligence, it’s essential to remember that human creativity is something we have yet to fully understand and explore—each individual is unique, shaped by personal experiences, knowledge, and senses. A recent study by CodeSignal provides impressive insights into how candidates and AI perform on their evaluation platform. The results are quite impressive! "While AI models handle many coding tasks with impressive efficiency, human intuition, creativity, and adaptability offer a distinct advantage, especially when tackling complex or unpredictable challenges." https://lnkd.in/eYXApMpX

So happy to officially announce the results of the benchmark I helped develop! If you're at all interested in LLMs and how they're going to shape the future of engineering work, I hope you'll take a look. We discovered some really cool insights through this research and there's always more to do. LLMs are here to stay, and it's a question of how companies are going to leverage them to augment human capabilities to push the boundaries of accelerated growth in the months and years to come. Some key takeaways here are that LLMs are (in our opinion) not going to be replacing human engineers anytime soon. What IS going to happen, though, is that engineering work is going to rapidly become more and more complex as each new model is iteratively released and those capabilities expand. Without a solid plan of how you and your organization are going to embrace AI, you may find yourself falling behind the curve. We hope that this data and more insights to come as we continue to research the ways LLMs are shaping the future of our industry will be helpful for everyone, from AI enthusiasts to those who are just getting up to speed. Feel free to reach out to me if you have any questions about our research or how CodeSignal can help your organization find the right talent for your organization and to develop the skills required for success in this new AI era.

So interesting to see the differences in model performance on more than one dimension in a real human coding assessment from CodeSignal!

Ever wonder how human software engineer coding performance stacks up to the capabilities of LLMs? So did we. Check out what we found in our AI Benchmarking Report https://lnkd.in/gKErn4bZ

Sneak peek at my latest article at The Daily Singularity. Link to the full article in my page. -------------------------------------------------------------------------------------------------------------------- Factory AI has announced its latest product in direct competition to DevinAI with an AI that allows software engineers to be more autonomous. This will provide software engineers the ability to speed up their work, be more autonomous and get things done quicker by having an AI to do the monotous and repetitive tasks that are core to the industry. However, what is more interesting is what this means for other industries? #TheDailySingularity #tech #business #AI #news #FactoryAI #DevinAI

🚀 Exploring the Frontiers of Machine Learning Engineering 🚀 In today’s rapidly evolving tech landscape, Machine Learning Engineering stands at the crossroads of innovation and application. It’s not just about building models; it’s about transforming data into actionable insights that drive real-world impact. 🔍 Why is Machine Learning Engineering crucial? • It bridges the gap between data science and software engineering, ensuring that machine learning models are not only accurate but also scalable and robust. • ML engineers play a pivotal role in bringing AI from the lab to production, making it accessible and valuable across various industries. 💡 My Focus Areas: • Optimizing Models: Fine-tuning algorithms to balance performance with computational efficiency. • Deploying at Scale: Ensuring that ML solutions are reliable, reproducible, and adaptable in dynamic environments. • Real-world Applications: From enhancing e-commerce platforms to predictive analytics, the potential applications are vast and transformative. I’m passionate about leveraging these skills to solve complex problems and create innovative solutions that push the boundaries of what’s possible with AI. Whether it’s through personal projects, collaborations, or industry roles, I’m committed to making an impact with machine learning. Let’s connect and discuss how we can harness the power of ML to drive meaningful change! 💻🤖 #MachineLearning #AI #Engineering #Innovation #Tech

Generative AI is significantly influential in automating and refining business processes. The technology's ability to automate coding and streamline decision-making processes is particularly notable. "There is recognition in the industry that this will meaningfully improve the productivity of our engineers, data scientists, and software engineers," highlighted Shub Bhowmick, pointing out the productivity gains from automating routine tasks. This shift allows companies to allocate more resources to higher-level tasks, thereby enhancing innovation. Read more- https://lnkd.in/gTa3rhfy #generativeai #datascientist

At Intellectual Software, we're excited to be at the forefront of integrating AI and Machine Learning (ML) into our software development processes. AI and ML are not just buzzwords; they're transforming the way we build, test, and deploy software. By harnessing these technologies, we're able to create smarter, more efficient applications that can learn and adapt over time. From predictive analytics to automated code generation, AI and ML are enabling us to streamline development, reduce errors, and deliver superior products to our clients. Our team is leveraging cutting-edge tools like TensorFlow and PyTorch to develop models that enhance user experiences and drive business insights. We believe that the future of software development lies in intelligent systems that can evolve with user needs. Stay tuned for more updates as we continue to innovate and lead the charge in this exciting field! #AI #MachineLearning #SoftwareDevelopment #Innovation #TechLeadership