Neal Oman

Quality data, tailored solutions. Cogito's enterprise grade data labeling solutions (EDLS) are domain-specific, compliant & backed by diverse annotation teams. We prioritize data authenticity & integrate with RLHF. Scale with confidence and unlock the true potential of your AI/ML models. #datalabeling #dataannotation #ai #ml #datasolutions #RLHF

9

JSON Handling and Type Management with Large Datasets in Go -- Ep. 4 is out now! In this episode, Miki Tebeka tackles the complexities of working with JSON data, especially when dealing with extensive datasets and optimizing type management. Here, Miki introduces key strategies for managing large JSON documents, avoiding unnecessary type definitions, and using Go’s encoding/json package to handle custom serialization needs effectively. Things you will learn in this video: ✅ Streamlining JSON Handling: How to avoid excessive type declarations by using anonymous structures and selective decoding for large JSON datasets. ✅Implementing Custom Serialization: Techniques for defining custom JSON serialization in Go with the Marshaler and Unmarshaler interfaces. ✅Managing Mixed-Type Arrays: Strategies to work effectively with JSON arrays containing various data types, ensuring smooth integration with Go’s type system. 📰Watch Now: https://lnkd.in/ghjz58pC Stay tuned for more episodes from Miki Tebeka in this mini series, highlighting Intro to #JSON for Engineers & Managers.🤓 #golang

23

💡 "Onehouse isn't about competition (between Apache Hudi, Iceberg, and Delta Lake) but rather about interoperability" 👐 Yes! We're building the most open data lakehouse, which also means we support all three formats ➡ it's all about user choice, freedom to choose the right format for the right task, and avoiding lock-in. Great article from VentureBeat highlighting our Series B and two latest product launches LakeView and Table Optimizer. #dataengineering #bigdata #data #datawarehouse #datalake #datalakehouse #nolockin

27

Snowflake’s Arctic-TILT model, powering our Document Al, beats GPT-4 with just 0.8B parameters, securing a top spot in the standard benchmark for document understanding DocQVA. This efficiency not only reduces operational costs but also enhances the scalability of solutions built on Arctic-TILT. This also means that it does not require large GPU instances or resources to leverage the Arctic-TILT model in Document AI, making it more accessible and adaptable to different customer sizes and needs.

452

7 Comments

City Storage Systems (CloudKitchens) has transformed its machine learning infrastructure since adopting Daft. Their data processing and ETL tasks, previously managed on separate Spark clusters, now leverage Daft’s intuitive DataFrame interface — surpassing PySpark’s capabilities 💪 Tune in to their session at #RaySummit by Anyscale on Tues Oct 1 to discover how the shift to a consolidated Ray and Daft environment supports their end-to-end ML pipelines, from data engineering to model development. #Daft #Ray #training #ai #ml #dataprocessing #etl #PySpark #pipeline

16

Just updated our doc to provide an overview of the Apache Doris capabilities in log storage and analysis. You will learn about: 💪 What makes the Apache Doris solution stand out 🛠 How to build a log analytic platform with Apache Doris https://lnkd.in/gr2nZ9-S #Filebeat #Logstash #fluentd #fluentbit #Kafka #Grafana #Superset #log #database

43

3 Comments

"Data Pipeline Observability is just a fancy alert system" or "Data Pipeline Observability is only for big pipelines"—statements like these are sure to make heads turn in disbelief. 😲 It’s time we bust these myths wide open. Mona Rakibe, CEO and Co-Founder of Telmai, takes these misconceptions head-on breaking down why data pipeline observability is far more than just an alert system. She highlights how data pipeline observability provides continuous oversight of your pipeline, constantly monitoring for potential disruptions and adapting workflows on the fly to maintain data quality and operational efficiency. Curious to learn more? The full article is just a click away—find the link in the comments to know more! 🧐 #DataPipelineObservability #DataPipelines #DataQuality #Datahumour

6

1 Comment

Deepa Ganiger's blog offers beneficial insights into implementing schema change detection and evolution in a streaming scenario with Snowflake. Gain a high-level understanding of the process and learn how to handle complex JSON data. phData is here to help! Reach out to our team of experts today. 😊 https://lnkd.in/g9KntHZV #DataScience #Snowflake #Kafka #DataProcesses #DigitalTransformation

11

This week on the blog, Ami Ikanovic discusses the "DIY is better" mentality for #datamasking and how it might seem like a quick win initially, but the long-term costs – time, complexity, and risk – are too high. https://lnkd.in/eUXyXKQf #datasecurity #datasecops #datagovernance #Snowflake

19

❇️ Curious to see Sonar AI Code Assurance and AI CodeFix in action? 🌐 Register for our Oct. 16 webinar: https://bit.ly/4ewtBv2 Manish Kapur, Sonar Senior Director, Product and Solutions, and Alexandre Rage, Sonar Product Manager will explore strategies for improving code quality and security throughout the SLDC by combining GenAI’s strengths with Sonar’s comprehensive code analysis. Whether you want to ensure secure, high-quality code with AI Code Assurance or quickly resolve issues with AI CodeFix, this session will provide all the insights you need. You will learn how Sonar can improve your coding process by minimizing risks associated with LLMs. #AI #GenAI #innovation #DevOps #programming #Coding #AICodeAssurance #AICodeFix #CleanCode #SonarQube #SonarCloud #SonarAISolutions

43

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

WorFBench: A Benchmark for Evaluating Complex Workflow Generation in Large Language Model Agents Large Language Models (LLMs) have shown remarkable potential in solving complex real-world problems, from function calls to embodied planning and code generation. A critical capability for LLM agents is decomposing complex problems into executable subtasks through workflows, which serve as intermediate states to improve debugging and interpretability. While workflows provide prior knowledge to prevent hallucinations, current evaluation benchmarks for workflow generation face significant challenges. Read the full article: https://lnkd.in/eqv7pQCt Paper: https://lnkd.in/eWDxh6Vz

5

Quick Video Tutorial: 🎥 Visualize your Great Expectations data quality tests in OpenMetadata. Centralize your data quality metadata alongside your data discovery, governance, lineage, insight and more: https://buff.ly/3zK6Hkv PS: Stay tuned for upcoming Anomaly Detection in OpenMetadata!

14

🏋️‍♀️ 2x performance improvement 💰 70% of the cost 🏆 RTA+ capabilities spanning ETL, OLAP workloads, ML, and geospatial analytics Don't just take our word for it -- download the Ocient vs. Druid Benchmark today ⤵️ https://hubs.la/Q02H7pgy0 #realtimeanalytics #rta #dataanalytics #roi #costsavings

22

Materialize’s Chief Scientist Frank McSherry discusses how to create live data by just using SQL in our latest blog post. In the post, Frank builds a recipe for a generic live data source using standard SQL primitives and some Materialize functionality. Then he adds in various additional flavors: distributions over keys, irregular validity, foreign key relationships. It’s all based off of Materialize’s own auction load generator, but it’s written entirely in SQL and can be customized as your needs evolve. By the end of the blog, you’ll discover that the gap between your idea for live data and making it happen is just typing some SQL. READ BLOG -> https://lnkd.in/eFJcB_un

10

🚨 New blog post alert: https://lnkd.in/g9h8Sdy8 Achieve superior data observability with DQLabs: semantic layer integration for precise anomaly detection, detailed data tracking, and improved metadata management. #DQLabs #moderndataquality #dataquality #dataobservability #datagovernance #Semantics #GenAI

18

🦙 Jamba-Instruct is now LIVE on LlamaIndex 🦙 Check out latest LlamaIndex blog examining why #RAG without a long context window isn't sufficient for enterprise use cases. The solution: Long Context + RAG. For an in-depth guide on how to leverage Jamba-Instruct's 256K context window on LlamaIndex, read here:👇 https://lnkd.in/dTAkqMix

53

1 Comment