Megagon Labs’ Post

... Meeting ML practitioners where they live...

Tired of tedious #LLM annotation processes? Seeking to streamline your machine learning workflow? Try data annotation with MEGAnno, where human expertise meets #LLM capabilities for seamless, high-quality annotations! Many existing data annotation tools focus on the annotator, enabling them to annotate data and manage annotation activities. Instead, MEGAnno is an open-source data annotation tool that puts #ML practitioners first. It enables you to bootstrap annotation tasks and manage the continual evolution of annotations through the machine learning lifecycle. Key Features of MEGAnno: 🔧 Enhanced Data Scientist-Centered Annotation: a tailored experience for data scientists, facilitating direct annotation management within Jupyter notebooks. This feature enables the utilization of existing Python functions and our suite of built-in power tools, optimizing the annotation process. 🔍 Human Verification and Label Exploration: Seamlessly incorporate both human and LLM data labels with verification workflows and integration to popular LLMs. This enables LLM agents to label data first while humans focus on verifying a subset of potentially problematic LLM labels. ℹ Unified Data Management: MEGAnno integrates a robust back-end service acting as a single source of truth, managing the evolution of annotation information throughout the lifecycle. 🧐 Comprehensive Data Exploration and Selection: MEGAnno provides powerful tools to explore datasets and choose the most suitable data for labeling. Incorporating features like active learning, it optimizes the labeling process, ensuring efficient prioritization of tasks. 🔄 Informed Decision-Making Through Label Distribution Analysis: MEGAnno enables users to analyze label distribution and annotator behavior, facilitating informed decisions for subsequent labeling batches. To address the diverse challenges in LLM annotation workflows, MEGAnno prioritizes: ✅ Convenience: Automating annotation workflows, customizable model configurations, and robust error handling. 🔍 Selectivity: Empowering users to selectively verify annotation candidates based on confidence scores and metadata. 🔄 Reuse: Facilitating the storage and reuse of LLM agents, labels, and metadata for future projects. Try Out Our Demo or use MEGAnno locally! Visit https://megagon.ai/meganno for more info and documentation. #AI #MachineLearning #DataAnnotation #phd #datascience #MEGAnno #LLM #LLMs

MIT spinout DataCebo helps companies bolster their datasets by creating synthetic data that mimic the real thing. | Click below to read the full article on Sunalei

Great news! 🤩 Our #paper titled "DeepTSF: Codeless machine learning operations for time series forecasting" has been published in the #SoftwareX journal (Elsevier), in the context of I-NERGY Project. Along with Theodosios Pountridis, Giorgos Kormpakis, George Lampropoulos, Vagelis Karakolis, Spiros Mouzakitis, and Dimitris Askounis, we developed a comprehensive machine learning operations (MLOps) application for automated and codeless time series forecasting. DeepTSF is based on #u8darts and #MLflow and supports various machine learning and deep learning models for time series forecasting including lightgbm, random forest, arima, temporal convolutional network, temporal fusion transformer, lstm, nbeats, and nhits. Since DeepTSF is open-source, it can serve as a reference production tool for stakeholders such as data scientists, and domain experts that need to develop, optimize, evaluate, serve and monitor ML and DL models for time series forecasting. DeepTSF makes MLOps for time series forecasting easy for everyone by unifying the ML lifecycle development and monitoring. This is achieved through a user interface that guarantees a user-friendly and codeless experience which is its main contribution to the industrial time series forecasting domain. Additionally, CLI capabilities are also available for data experts and ML engineers that require custom model training procedures and flexibility, given a specific degree of coding expertise. To view the paper, follow the link:  👉https://lnkd.in/dxGAP9NR #codeless #app #forecasting #mlops #deeplearning #machinelearning #lightgbm #randomforest, #arima #temporalconvolutionalnetwork, #temporalfusiontransformer #lstm #nbeats #nhits

Great news! 🤩 The #paper titled "DeepTSF: Codeless machine learning operations for time series forecasting" has been published in the hashtag #SoftwareX journal (Elsevier), in the context of I-NERGY Project. To view the paper, follow the link: 👉https://lnkd.in/dxGAP9NR

Notes from experimenting with LangGraph: Structuring Agent Intelligence Lately, I've been diving into LangGraph to create agents that can solve problems in a structured manner, leveraging my own thinking encoded in graph form. I imagine a spectrum where on one end you have fully autonomous agents, and on the other, you have rigid, task-specific agents. LangGraph strikes a middle ground by structuring an agent’s thought process—enabling both flexibility and control. This combination allows for dynamic problem solving, yet keeps the process grounded in predefined paths. Their diagram is very true but you need to experience it to understand what it means. One takeaway from this experience? I’m convinced we’re about to witness a MapReduce resurgence, where LLMs get embedded in Directed Graphs (DG) workflows (note how I dropped the 'Acyclic' in DAG). This could be the next leap in handling distributed, structured AI tasks for Big Data Processing. One small hitch: LangGraph's abstractions make debugging without LangSmith quite a challenge (by design). You might find yourself needing the paid version sooner than anticipated, especially for larger agent deployments. While it’s great for LangGraph’s business model, I foresee that those managing high-volume agents will eventually develop their own frameworks. #AI #LangGraph #AgenticAI #DAGWorkflows #MapReduce #AIinBusiness #LLM

At the BioTechX conference, I had the opportunity to meet several fascinating professionals and companies in the product space. AI and GenAI were, of course, hot topics, but it was clear that many of us were grappling with similar challenges, often without realizing it. The most common concerns revolved around data privacy and balancing costs against deliverables. One particularly interesting discussion focused on clinical LLM models, built on platforms like Llama or from scratch using private company data. Some argued these models improved patient engagement, though there wasn’t solid data to back up those claims. On the engineering front, the familiar challenge arose: non-technical users facing too many options when interacting with software platforms. I stood by my belief that it’s best to use bare metal solutions without unnecessary layers—if it's a Jupyter notebook, keep it simple and don’t overcomplicate it. Regarding data, there was debate over whether 1TB qualifies as "big data." While modern databases can handle this with ease, it doesn’t mean languages like R and Python can process it just as smoothly. I threw out multiple options, and that's where many ran away 😂! Tomorrow, I’ll be presenting our tech stack and how we're building AI products with modern tools, all while keeping user needs front and center. It’s a tough process, but ultimately very rewarding! 🚀

RAG has come a long way since its introduction in 2020 by Meta at NeurIPS. Moving on from basic retrieval and using LLM just to answer questions on the retrieved texts, various techniques like Query Expansion techniques, Rerankers, Maximum Marginal Relevance(MMR) have helped in enhancing the utility of RAG as the primary text extraction design for enterprises. Building upon this , the latest buzz is around "Agentic RAG". Agentic RAG is a first step in autonomous text extractions flows, where LLM Agents decide on each step what should be the next step. Individual agents. capable of working with external sources (like searching web where needed and so on) enhance the output from a RAG system significantly improving the quality of the final output. From where I see, in a way, Agentic RAG is old wine in a new bottle. It gradually brings in the same concepts used in a traditional software (like graphs , decision points etc), but rather than coding in the decision logic, LLMs take the center space here. This provides an interesting twist to building data intensive softwares. We would see more and more of such traditional software architectures being disrupted by introducing LLMs at each decision points instead of hand crafted code. #llmagents #GenerativeAI #machinelearning #datascience

fast retrieval of recognized images is a base level technology which will affect several areas of science and business applications. https://lnkd.in/gS4v_y5j

A foundation model development cheatsheet, from foundation model developers. The folks over at EleutherAI, in collaboration with MIT, AI2, HuggingFace, Stanford, Princeton, and more, have released the foundation model development cheatsheet, a quick-start guide to familiarize new developers with useful tools and resources for developing new open models. You can use it to easily find, for example, pre-training data sources, tools and frameworks for data preparation, model training, model evaluation, and more. This is a great initiative aligned with EleutherAI's mission to lower barriers to entry of research and development of foundation models. Cheat sheet (interactive): https://fmcheatsheet.org/ Github repo: https://lnkd.in/dSDkbY-K Paper: https://lnkd.in/d5DRns4i Blog: https://lnkd.in/dg4drU_3 #genai #generativeai #research #openllms