lakeFS

Exciting times at lakeFS: We just released what was probably the most requested feature since its inception: the lakeFS UI now supports Dark Mode 🌒✨ Upgrade and Enjoy!

🎓𝗗𝗲𝗳𝗶𝗻𝗶𝗻𝗴 𝗞𝗲𝘆 𝗣𝗹𝗮𝘆𝗲𝗿𝘀 𝗶𝗻 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄🎓 The world of data-driven projects involves three pivotal roles: #dataengineers, #MLengineers, and #datascientists. Each plays a unique part in building successful AI/ML models, and data version control is crucial for each. 🛠 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: The backbone of any #datapipeline. They handle the ingestion, transformation, and storage of data. With data version control, they ensure every dataset is consistent and reproducible across different environments, avoiding data drift and ensuring accurate outputs. 🧠 𝗠𝗟 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: They build and deploy #machinelearning models. Efficient version control for both models and datasets allows ML engineers to test and validate multiple iterations with confidence, speeding up model development and reducing costly errors. 📊 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝘁𝗶𝘀𝘁: Their focus is on extracting insights from #data. For data scientists, data version control means they can rely on consistent data, track every change, and maintain integrity across experiments, driving more accurate and reliable analyses. Data version control is the glue that keeps everything together across these roles. It’s not just about managing files—it’s about ensuring trust and traceability across your entire #datalifecycle.

Whether it’s a chatbot, product recommendation engine, or BI tool, LLMs have progressed from proof of concept to production. However, LLMs still pose several delivery challenges, especially around maintenance and upkeep. Implementing LLM observability will not only keep your service operational and healthy, but it will also help you develop and strengthen your LLM process. Let's dive into the advantages of LLM observability and the tools teams use to improve their LLM applications today. This article covers some necessary steps: 🔍 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘆𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹: Stay on top of what's happening inside your LLM, from data input to output. 📊 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗺𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴: Real-time insights to track performance and catch issues early. 🔄 𝗜𝘁𝗲𝗿𝗮𝘁𝗲 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗹𝘆: Use observability data to refine and improve your models faster. Learn more here 🔗 https://lnkd.in/dYmqYcgJ #ai #ml #llm #observability #datascience

Struggling with versioning ML models and datasets? This tutorial is here to help! 🔢 You'll learn how to 💡 Track and manage datasets seamlessly ⚙️ Ensure consistency across training and production 🚀 Optimize your machine learning workflows 🔍 Gain full control with reproducibility and transparency Check out the full article below 👇 https://lnkd.in/deuQb4HP #modelversioning #datamanagement #machinelearning #datascience

🗣📢 Can't miss #BigDataLDN track! While there have been many advancements with the rise of open table formats, there remains a gap between current data versioning practices and the requirement for tabular dataset versioning. In this talk, Tal Sofer of lakeFS will introduce the concept of a versioned catalog as a solution to these gaps. If you plan to attend the event, make sure to register and attend this track, 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗩𝗲𝗿𝘀𝗶𝗼𝗻𝗶𝗻𝗴 𝗶𝗻 𝘁𝗵𝗲 𝗔𝗴𝗲 𝗼𝗳 𝗢𝗽𝗲𝗻 𝗧𝗮𝗯𝗹𝗲 𝗙𝗼𝗿𝗺𝗮𝘁𝘀.   And of course, don't forget to swing by our booth, Stand 748 and connect with the lakeFS team! Check out the full session details below 👇 https://lnkd.in/dgbWqsrw

💡Feature spotlight!💡 Improve your deep learning workloads performance with lakeFS Mount. Here's how lakeFS Mount 𝗿𝗲𝗱𝘂𝗰𝗲𝘀 𝗼𝗯𝗷𝗲𝗰𝘁 𝘀𝘁𝗼𝗿𝗲 𝗿𝗼𝘂𝗻𝗱𝘁𝗿𝗶𝗽𝘀 𝗯𝘆 𝟵𝟬% by locally caching committed object metadata. Network I/O happens on demand, caching objects as they are read. We've seen typical AI/ML training jobs reducing 70% of time spent doing I/O 💥 to learn more, check out the link in comments

The lakehouse architecture has become the backbone of modern big data operations but it comes with some specific challenges. ✖ Ability to Write-Audit-Publish to test and verify changes before releases ✖ Rolling back changes to a consistent and well-known state ✖ Creating reproducible workloads that encapsulate multiple tables (and code!) ✖ Creating cost-effective, ad-hoc dev/test environments with no data copies Fortunately, there are ways to overcome these issues. In this article, we'll demonstrate how by implementing Git-like semantics, Delta Lake and lakeFS can work together to improve time travel for lakehouses. Read & learn here 👇 https://lnkd.in/dvEtUd66 #lakehouse #datalake #deltalake #lakefs #bigdata

Who's coming to Ai4? We are! ✋ Meet Jas Bajwa, Iddo Avneri and Oz Katz at Booth 2️⃣0️⃣3️⃣ and get to know the team behind lakeFS We'd love to talk about your ML and AI challenges and discuss how our data version control system can help you accelerate your development process. If you're attending, make sure to book some time to meet us at our booth (203): https://lnkd.in/d62eCayR #machinelearning #ai #artificialintelligence #dataversioncontrol Ai4 - Artificial Intelligence Conferences

📚Good read!📚 Eyal Trabelsi shares a comprehensive testing strategy for #machinelearning systems. As he states, testing ML is hard because it's "probabilistic by nature, and must account for diverse and dynamic real-world conditions." In this article, Eyal shares the foundation for the stages that will help you overcome these challenges : 1. 𝗖𝗜 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 🔄 : Automate the testing process to ensure consistency and efficiency. 2. 📝 𝗦𝘆𝗻𝘁𝗮𝘅 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: Validate the code to catch syntax errors early in the development cycle. 3. 𝗗𝗮𝘁𝗮 𝗖𝗿𝗲𝗮𝘁𝗶𝗼𝗻 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 👌: Verify the data generation process to maintain data quality and integrity. 4. 🦾 𝗠𝗼𝗱𝗲𝗹 𝗖𝗿𝗲𝗮𝘁𝗶𝗼𝗻 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: Evaluate model performance using various metrics to ensure accuracy and reliability. 5. 𝗘𝟮𝗘 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 ♾️: Conduct end-to-end tests to simulate real-world scenarios, ensuring the entire ML pipeline works seamlessly. 6. 📦 𝗔𝗿𝘁𝗶𝗳𝗮𝗰𝘁 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: Test the final artifacts to confirm their readiness for production deployment. Read the very detailed, in depth article here https://lnkd.in/dXMbArkt

Whether you're in Vegas for the shows, events or other [distractions], in this case we encourage you NOT to follow the adage of "𝑤ℎ𝑎𝑡 ℎ𝑎𝑝𝑝𝑒𝑛𝑠 𝑖𝑛 𝑉𝑒𝑔𝑎𝑠 𝑠𝑡𝑎𝑦𝑠 𝑖𝑛 𝑉𝑒𝑔𝑎𝑠." 🤫   The lakeFS team is heading to #Ai4 where we'll be joined by #machinelearning and AI experts from across all industries and companies who are ready to tackle new innovations at the forefront of #artificialintelligence.   On Tuesday, August 13, between 10:35 to 11:20 AM, Oz Katz (co-creator and CTO at lakeFS) will be joining a panel discussion: 𝗙𝗶𝗻𝗱𝗶𝗻𝗴 𝗩𝗮𝗹𝘂𝗲 𝗶𝗻 𝗬𝗼𝘂𝗿 𝗗𝗮𝘁𝗮 where Oz and fellow panelists will delve deep into strategies for making sense of your data and deriving value from this often untapped resource.   We hope to see you there so be sure to swing by our Booth#203 to say hi and pick up a sticker, a magnet, a plushy, and even some tips to help your #MLdevelopment scale up! https://ai4.io/