Women Startup Stories’ Post

Dave Melillo takes on building a Data Platform in 2024 and Wannes Rosiers recapped it well! You can see how open source is all over data platforms. See what was shared about DataHub: "Similar to data orchestration, data observability has emerged as a necessity to capture and track all the metadata produced by different components of a data platform. This metadata is then utilized to manage, monitor, and foster the growth of the platform. Many organizations address data observability by constructing internal dashboards or relying on a single point of failure, such as the data orchestration pipeline, for observation. While this approach may suffice for basic monitoring, it falls short in solving more intricate logical observability challenges, like lineage tracking. Enter DataHub, a popular open-source project gaining significant traction. Its managed service counterpart, Acryl Data, has further amplified its impact. DataHub excels at consolidating metadata exhaust from various applications involved in data movement across an organization. It seamlessly ties this information together, allowing users to trace KPIs on a dashboard back to the originating data pipeline and every step in between." #dataplatform #datahub #dataengineering

"In 2021 I failed to emphasize the importance of data orchestration, it has emerged as a natural compliment to a modern data stack." Timing-wise I might not agree: orchestration has been crucial for a long time, but Dave Melillo does a great job describing the evolution of the modern (?) data stack in his article https://lnkd.in/eGmA4d8q . Don't forget to also read his article about building a data platform in 2021. Comparing data platforms over just a small period of time, Dave's summary is spot on: "What HASN'T changed?" The data world is still evolving rapidly. In his 2021 closing he states "Although it will happen, I am willing to bet that it will take several years before one vendor distills the entire data stack into one unified platform." He might be right: as a summary of Big Data LDN last year I wrote: consolidation of vendors won't be happening soon. But what to do in this fast moving world of data and which emerging strategies to follow? 🧩 Modularity is core. Or as the 2021 article summarizes: "Understand that flexibility and agnosticism are the main takeaways." New innovative approaches are still entering the market. To make sure you can tap in on this evolution, you need the flexibility of a modular approach to adopt new technologies. But... If users always need to learn new technologies, you have a repeated learning curve. That's where agnosticism taps in: make a sufficient amount of abstractions to have a unified and recognizable development experience. 🧩 part 2. Modularity is core. Dave describes dbt Labs dbt mesh as "This innovation enables the tethering and referencing of multiple projects, empowering organizations to modularize their data transformation pipelines, specifically meeting the challenges of data transformations at scale." Good to mention that he acknowledges the impact of dbt being so profound that it's given rise to the role of analytics engineer. By the way: modularizing data transformation pipelines brings you close to the notion of reusable data products. 🪄 Orchestration is magic. I could write a hole post on orchestration, but Dave summarizes it perfectly: "Without an orchestration engine, the ability to modularize your data platform and unlock its full potential is limited. Additionally, it serves as a prerequisite for initiating a data observability and governance strategy, playing a pivotal role in the success of the entire data platform." 🔍 Observability (next to orchestration) has been added to the article cover image. Observability in general should cover data, pipelines and your platform. Where platform monitoring sits separate, data (and pipeline) observability are core to data catalogs. To use Acryl Data as an example: Datahub excels at consolidating metadata. It seamlessly ties information together, allowing users to trace KPIs on a dashboard back to the originating data pipeline and every step in between. #dataplatform

Good read. Observability and Orchestration are a must. As is the ability to support different types of data flow (batch, stream, event).

It’s 2024, and you have a mountain of data to organize - and learn from. How do you do it? Two real-life examples: In 2013, UPS upgraded their data warehouse with petabytes of structured data. This powered a project to dynamically optimize delivery routes. They analysed large amounts of logistics data to make rapid real-time route adjustments, ultimately significantly cutting shipping miles and carbon emissions. In 2021, Coca-Cola Andina leveraged AWS to build a data lake, consolidating 95% of its disparate business data and integrating analytics, AI, and machine learning. Because data was all in one place, the analytics team spent less time talking to data owners to find what they needed - increasing productivity by 80%. This fostered a culture of data-driven decision-making across the organization, as well as increasing revenue. These show the two dominant data organization patterns: data warehouses and data lakes. Here are the main differences: Data types • Data warehouses are primarily for structured data. • Data lakes can store any data type: structured, semi-structured or unstructured. Flexibility • Data warehouses require setting up a data schema upfront. This streamlines querying, but limits the ability to pivot to new data or use cases that don't fit the original plan. • Data lakes let you ingest raw data from diverse sources without prior organization - no matter its type or structure - and decide how to use it later. This approach is highly flexible but can increase complexity. Scaling cost • Data warehouses are intended for smaller amounts of operational data, and tend to require upfront investment, especially on-prem. Because storage and compute are coupled, costs tend to increase with scale - and past a certain size, large datasets become prohibitively expensive. • Data lakes tend to be more cost-effective off the bat, especially with pay-as-you-go cloud offerings. As storage is inexpensive and decoupled from compute, they can leverage serverless elasticity to scale up and down automatically, and store operational and archive data in one place. Use cases • Data warehouses are great for high-speed querying and reporting on structured datasets - ideal for critical decision-making with tools like PowerBI and Tableau. Generally a smaller group of business professional users. • Data lakes are best for vast amounts of diverse raw data, from CSV files to multimedia. This breadth allows for exploratory analysis, predictive modeling, statistical analysis, and ML. Variety of users, from analysts to data scientists. The choice depends on what kind of problems you want to solve. If you're after fast analysis within a predefined structure, data warehouses could be your go-to. On the flipside, data lakes offer flexible insights across a broad range of data, more easily scalable to large datasets. For those who’ve faced this choice before - what made you pitch for one or the other?

🚀 Transform Your Data Strategy with Apptad Inc. + Databricks, a Gartner® 2024 Leader! 🚀 🎉 At Apptad Inc., we are incredibly proud to leverage Databricks' a Leader in the 2024 Gartner® Magic Quadrant™ for Data Science and ML to drive unparalleled success for our clients. 🔍 Why Databricks and Apptad Inc.? In today's fast-paced, data-driven world, the right technology partnership can be a game-changer. Here’s how we’re making waves: 🔹 Unified Analytics Powerhouse: No more silos! Our integrated lakehouse platform brings together data engineering, analytics, and ML, eliminating costly pipeline breaks and ensuring a seamless data flow. 🔹 Open-Source Flexibility: Built on Apache Spark, we avoid proprietary lock-ins, offering the flexibility to leverage open data formats, allowing you to innovate and scale effortlessly. 🔹 Cloud-Optimized Performance: Auto-scaling clusters and cloud-native architectures deliver unmatched cost-efficiency and performance at scale, maximizing your infrastructure on AWS, Azure, or GCP. 🔹 Streamlined Data Science: From exploration to production, our integrated notebooks and end-to-end ML capabilities accelerate your data science workflows, reducing time-to-insight and enabling rapid iteration. 🔹 Vendor-Agnostic Cloud Support: Our platform runs seamlessly on AWS, Azure, or GCP, giving you the freedom to choose the best cloud environment for your needs. 💼 Real-World Success Stories: 💡 CPG Industry: Our post-payment audit solution recovered ~$19M from LATAM by identifying and recovering excess payments. 💡 Healthcare: Our AI-powered insights in the Procure to Payment cycle increased cost and timeline compliance by up to 3X. 💡 Supply Chain: Our working capital analysis solution improved cash utilization efficiency by 36%. 💡 Retail: Effective deduction management processes potentially recovered 5-7% of revenue, boosting profitability. At Apptad Inc., leveraging Databricks' top-tier platform allows us to deliver exceptional results and drive innovation for our clients. We believe Databricks' recognition in the Gartner Magic Quadrant reflects our joint commitment to excellence and the transformative power of data. 🌟 Ready to Transform Your Data Strategy? Contact us today for a complimentary assessment and discover how our Databricks solutions can revolutionize your data strategy. 📧 [email protected] 🌐 What's your biggest data challenge right now? Let's discuss in the comments! Let's unlock the full potential of your data together! #DataScience #MachineLearning #BigData #CloudComputing #DataAnalytics #AI #DataStrategy #BusinessIntelligence #DigitalTransformation #DataEngineering #Innovation #TechNews #EnterpriseTech #GartnerMagicQuadrant

Building a Data Platform in 2024: How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: * What’s changed? * The Platform * Integration * Data Store * Transformation * Orchestration * Presentation * Transportation * Observability * Closing What’s changed? Since 2021, maybe a better question is what HASN’T changed? Stepping out of the shadow of COVID, our society has grappled with a myriad of challenges — political and social turbulence, fluctuating financial landscapes, the surge in AI advancements, and Taylor Swift emerging as the biggest star in the … *checks notes* … National Football League!?! Over the last three years, my life has changed as well. I’ve navigated the data challenges of various industries, lending my expertise through work and consultancy at both large corporations and nimble startups. Simultaneously, I’ve dedicated substantial effort to shaping my identity as a Data Educator, collaborating with some of the most renowned companies and prestigious universities globally. As a result, here’s a short list of what inspired me to write an amendment to my original 2021 article: * Scale Companies, big and small, are starting to reach levels of data scale previously reserved for Netflix, Uber, Spotify and other giants creating unique services with data. Simply cobbling together data pipelines and cron jobs across various applications no longer works, so there are new considerations when discussing data platforms at scale. * Streaming Although I briefly mentioned streaming in my 2021 article, you’ll see a renewed focus in the 2024 version. I’m a strong believer that data has to move at the speed of business, and the only way to truly accomplish this in modern times is through data streaming. * Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration. This time around, I have a whole section dedicated to orchestration and why it has emerged as a natural compliment to a modern data stack. The Platform To my surprise, there is still no single vendor solution that has domain over the entire data vista, although Snowflake has been trying their best through acquisition and development efforts (Snowpipe, Snowpark, Snowplow). Databricks has also made notable improvements to their platform, specifically in the ML/AI space. All of the components from the 2021 articles made the cut in 2024, but even the familiar entries look a little different 3 years later: * Source * Integration * Data Store * Transformation * Orchestration * Presentation * Transportation * Observability Integration The integration category gets the biggest upgrade in 2024, splitting into three logical subcategories: * Batch… #MachineLearning #ArtificialIntelligence #DataScience

Azure Series Part 13: Dive into Data Lakes and Addressing ACID Challenges 🚀 Welcome back to our Azure series! Today, we're focusing on Data Lakes in Azure, their inherent challenges, particularly around ACID guarantees, and how these challenges can be addressed. ➡️Understanding Data Lakes Data Lakes are vast pools of raw data stored in their native format until needed. They are designed to store, process, and secure large volumes of structured and unstructured data, offering massive scalability, flexibility in data types, and seamless integration with Azure's analytics services. 🔹 Massive Scalability : Designed to store petabytes of data, accommodating growing data needs.   🔹 Flexibility in Data Types : Handles structured, semi-structured, or unstructured data, supporting diverse analytical operations.   🔹 Integration with Azure Ecosystem : Offers comprehensive analytics solutions through integration with services like Azure HDInsight, Azure Databricks, and Azure Synapse Analytics. 🔍 Challenges of Data Lakes One of the primary challenges of Data Lakes is the lack of ACID (Atomicity, Consistency, Isolation, Durability) guarantees, crucial for maintaining data consistency during transactions. This absence can lead to several issues: 🚫 Data Reliability Issues : Without ACID guarantees, ensuring data reliability becomes challenging.   🚫 No Data Validation : There's a lack of mechanisms to validate data, leading to potential inaccuracies.   🚫 Data Corruption : In case of failures, data may become corrupted, affecting data integrity.   🚫 Difficulty in DML Operations : The absence of ACID guarantees complicates performing DML (Data Manipulation Language) operations.   🚫 Data Quality Issues : Ensuring high data quality becomes difficult without consistent and atomic transactions.   🚫 No Data Versioning : Without ACID properties, implementing data versioning for historical analysis is challenging. Scenarios Where ACID Guarantee is Not Achievable 🚨 🔸 Job Failing While Appending Data 💥: Affects data atomicity and consistency.   🔸 Job Failing While Overwriting Data 🌪️: Impacts data atomicity, consistency, and durability.   🔸 Simultaneous Reads & Writes 🔄: Affects data isolation, leading to potential inconsistencies.   🔸 Appending Data with a Different Schema 🧩 : Can lead to data integrity issues. 🚀 Addressing ACID Challenges with Delta Lake Delta Lake offers a robust solution to the ACID challenges faced by traditional Data Lakes. In conclusion, while Data Lakes offer a powerful tool for storing and analyzing vast amounts of data, they come with their set of challenges, particularly around ACID guarantees. That's it for today's part of the Azure series! Stay tuned for more exciting content. Don't forget to share and spread the knowledge! 🙌 #Azure #Databricks #DataEngineering #DataScience #BigData #CloudComputing #DataLake #BigData #Analytics #DeltaLake #ACID

Picking the right data technologies is crucial, especially when being in an early stage company. The data warehouse plays a particularly important role. It's typically the place where you store structured data for analytical use cases, the solution can also encompass the query engine. When picking a data warehouse you'd typically want minimal overhead while keeping the flexibility to facilitate many use cases. For example, BI, customer facing analytics, reporting, machine learning, etc. It's however far from arbitrary to pick one solution to rule them all. ❄️ Data warehouses such as Snowflake and BigQuery are great picks when query and streaming data latency are not top of mind. Both are managed services which means minimal infra maintenance. In addition both offer an extensive ecosystem, with many native integrations being available (e.g. Spark, Kafka). However, while both support real time use cases, it's often other technologies which have the upper hand when it comes to latency and throughput. Also cost might be a consideration as both mainly support a pay as you go model, meaning bigger data and more queries can cause significantly higher costs. ⚡There's an abundance of Open source alternatives, e.g. Druid, ClickHouse ,Trino, Pinot, DuckDB. Each of these options optimise for particular use cases. For example, in Druid you'd probably want to store pre-joined data and keep queries simple. When the characteristics of your data and queries are right, Druid can shine. Nevertheless, setting up and managing Druid can be a beast. In comparison, Trino is a more flexible query engine but relies on external data storage. This makes it a great pick if you have existing Hive/HDFS infra. DuckDB on the other hand shines if your data is relatively small and fits into memory. ✨ New managed and open source solutions are coming into the picture. For example Rockset is a relatively new player on the market but is setting a new standard for an out-of-the-box blazing fast data warehouse solution. Another example is StarRocks which offers a highly flexible and low latency solution with the ability to connect with various data storage solutions while also offering its own. On the other hand solutions which support vectorised data for AI are becoming increasingly popular. The data warehouse (and query engine) space is rich in its options and keeps on evolving which makes it far from trivial to pick a solution. There's not one solution to rule them all and many companies use multiple. How is your company tackling these problems? Which warehousing/query engine gets your preference and when? #dataengineering #datawarehouse #olap

🔍 Why Businesses are Moving Towards Data Lakehouses 🔍 Pros & Cons of Data Warehouses: ✅ Pros: Structured Data: Perfect for handling structured data (like transactions, customer data, etc.) with predefined schemas. Optimized for Queries: Enables fast and efficient SQL-based querying, ideal for analytics and reporting. Data Governance: Strong schema enforcement and ACID compliance ensures data quality and integrity. Data Integration: Excellent for integrating data from multiple structured sources. ❌ Cons: Limited Flexibility: Struggles to handle unstructured or semi-structured data like videos, images, or social media posts. High Cost: Expensive to maintain, with high storage and processing costs for growing datasets. Complexity: Managing large-scale data warehouses can be complex, especially for real-time analytics. Pros & Cons of Data Lakes: ✅ Pros: Supports Big Data: Designed for large volumes of raw data, including structured, semi-structured, and unstructured formats. Cost-Effective: Built on scalable cloud storage systems, data lakes offer a cheaper storage solution for massive data sets. Flexibility: Perfect for data scientists and machine learning applications, allowing teams to work with raw data in any format. ❌ Cons: Lack of Governance: Without schema enforcement, data quality can be inconsistent, leading to unreliable analytics. Slow Performance: Querying raw, unorganized data can lead to slow performance, especially for analytics-heavy use cases. Complex Processing: Requires extensive ETL (Extract, Transform, Load) work to prepare data for analytics. So, Why Do We Need a Data Lakehouse? 🏞️💻 Data Lakehouses offer a unified architecture that combines the best of both Data Lakes and Data Warehouses, overcoming the limitations of each: 🚀 Benefits of Data Lakehouse: Unified Data Platform: Handle both structured and unstructured data in a single platform. Cost Efficiency: Leverages the scalability of a data lake while offering the structured data capabilities of a data warehouse. ACID Transactions: Ensures data integrity and reliability, especially for analytics. Real-Time & Batch Data: Simplifies managing both real-time streaming data and historical batch data in a single system. Machine Learning & AI: Easier for data scientists and ML engineers to access high-quality data for modeling and analysis. The Databricks Lakehouse Platform is a powerful solution that aligns with this modern approach, providing organizations with a single source of truth, streamlined data management, and advanced analytics capabilities—all on one platform. 📊 In short, Data Lakehouses reduce complexity, save costs, and enable businesses to make faster, data-driven decisions. #Databricks #DataLakehouse #DataWarehouse #DataLakes #BigData #DataEngineer #AI

As Peter Parker was once told... 🕷 🕸 "With great AI comes great Data responsibility" Despite AI’s rapid growth, most enterprises struggle with foundational data readiness. Insight Partners recently revealed that a mere 11% of enterprises have scaled AI, with 60% still in the early stages of data modernization. And, as discovered by Vation Ventures, more than 70% of enterprises cite data quality & accuracy and security & privacy as their primary challenges in AI data preparedness 🐣 Which is why we can expect growing momentum behind Data Warehouse & Lakehouse modernization initiatives 👇 1️⃣ A majority of enterprises are prioritizing Data Warehouses & Lakehouses to facilitate AI-enabled data modernization More than 60% report undertaking Data Warehouse & Lakehouse modernization initiatives with... 👉 32% focused on data consolidation & migration 👉 30% focused on data transformation & operationalization What's more, just around 50% of all enterprises cite Data Warehouses and Lakehouses as their top data & AI priority 2️⃣ Data Warehouses are crucial for structured data enablement, allowing immediate ROI across key functional use cases Warehouses are optimized for structured data, essential for scenarios where real-time data integrity, speed, and reliability are paramount. Their structured nature makes them ideal for supporting BI & analytics functions that require fast, SQL-based queries on clean, well-organized datasets. And, importantly... 👉 BI analytics & visualization represent the second leading data & priority for a majority of enterprises with less than $5bn in revenue 🔎 As a result, Data Warehouses will be a key focus for a majority of enterprises for immediate & tangible data & AI ROI. ❗With Warehouses, enterprises can quickly leverage existing structured data to develop impactful functional use cases - such as anomaly detection for IT assets & functions - that generate quick ROI to prove business value & secure further executive support 3️⃣ Data Lakehouses offer a unified & flexible unstructured solution consolidating disparate data types for AI workloads By supporting the storage & processing of structured & unstructured data in a single platform, Lakehouses can streamline the data pipeline, reduce redundancy, and enhance the efficiency of AI-driven insights - an important consideration give that... 👉 48% of enterprises are reporting moderate fine-tuning of closed-source foundational models & customization of open-source models 🔎 Lakehouses will become an increasingly important solution facilitating more advanced & streamlined data & AI functionality, use case application, and operationalization ❗With Lakehouses, enterprises can develop an unstructured collection & enrichment data fabric with synthetic data generation to expedite the time-to-value for unstructured data Notable Innovators: ClickHouse, VAST Data, Dremio, Starburst, Incorta #Innovation #Data #Technology #AI