Malay B.

Best data news? It's right here 🎺 https://lnkd.in/ecYCuxYD Check it out: Medium 🧠 🧠 The BigQuery Qualify clause @matteoBennio 🧠 How Notion handles 200bn data entities Vu Trinh 🧠 A scalable ingestion framework in MS Fabric Piethein Strengholt 🧠 Migrating and testing out pyairbyte Felix Gutierrez 🧠 Snowflake vs Databricks; a useful comparison for 2024 Hugo Lu 🧠 People do not like this latest article “The rise of the notebook engineer” (sorry Daniel Beach!) **Editors Pick** 🧠 How Pinterest do Analytics Kapil Bajaj LinkedIn🕴 🕴 BigQuery release continuous Queries and real-time rELT (also see Richard Shenghua (升华) He's excellent post on this recently) 🕴 Difference between Junior and Senior engineers 🕴 Trio of Leaders leave Open AI #dataengineering #roundup #dataroundup #bigquery #streaming

14

3 commentaren

Data Vault modeling is often preferred for implementing the data medallion architecture: 1. Decoupling of Raw and Processed Data • Bronze Layer (Raw Data): The Data Vault’s raw vault captures source data with minimal transformation, just like the Bronze layer stores raw data from various systems. • Silver Layer (Standardized Data): The Data Vault business vault applies business rules and transforms the data into a standardized format, aligning with the Silver layer’s role of preparing cleaned and structured data. • Gold Layer (Aggregated Data): Data Vault outputs are then aggregated into a dimensional model or other formats, fitting the Gold layer’s requirement for analytics-ready datasets. 2. Scalability and Flexibility • Data Vault Strength: Designed for large-scale, enterprise-wide systems, Data Vault supports agile and scalable data pipelines. • Medallion Architecture: The architecture is meant to accommodate growing datasets and diverse data sources, which Data Vault handles well by adding new sources without redesigning the model. 3. Historical Data and Auditing • Data Vault Strength: Tracks data lineage and history using satellite tables without overwriting data, enabling easy auditing and debugging. • Medallion Architecture: This is critical in the Bronze and Silver layers, where data lineage and transformations need to be traceable. 4. Integration of Multiple Sources • Data Vault Strength: Uses hubs and links to integrate and track relationships between entities from multiple sources. • Medallion Architecture: The Bronze layer ingests raw data from diverse systems, and Data Vault provides a systematic way to integrate them for further processing. 5. Agile Development and Maintenance • Data Vault Strength: Agile, iterative development makes it easy to add new data sources and adjust models as business needs change. • Medallion Architecture: Built for adaptability, it benefits from Data Vault’s ability to evolve rapidly. 6. Alignment with Cloud Architectures • Data Vault Strength: Its design philosophy is well-suited for distributed computing and cloud-based environments like Databricks. • Medallion Architecture: Designed with modern cloud platforms in mind, it pairs seamlessly with Data Vault’s distributed and modular approach. Example Workflow 1. Bronze Layer (Raw Vault): Store raw data from various sources in a normalized Data Vault schema (hubs, links, satellites). 2. Silver Layer (Business Vault): Apply business rules and derive meaningful relationships while preserving lineage. 3. Gold Layer (Aggregates): Generate star/snowflake schemas or other formats for reporting and analytics. https://lnkd.in/eAAS7mdc

3

Looking for inspiration on how to drive an improved data-quality agenda in LeanIX? Don't miss out, and make sure to join this webinar on June 4th!

1

Mastering Architecture Diagrams and Technical Presentations: A #Data Architect’s Guide

1

Matillion Democratizes GenAI with No-Code Cortex Components on Snowflake AI Data Cloud https://lnkd.in/dvJ6mq5d #DataCloud #GenerativeAI #Matillion #NoCodeCortex #Snowflake

19

EK's new blog by Heather Hedden discusses the important role that taxonomy and information architecture play in building a semantic layer that is intuitive and comprehensive. #taxonomy #semanticlayer https://lnkd.in/ebtbB3xV

13

📚 𝐌𝐨𝐝𝐞𝐫𝐧 𝐃𝐚𝐭𝐚 𝐏𝐢𝐜𝐤𝐬: 𝐓𝐡𝐢𝐬 𝐰𝐞𝐞𝐤'𝐬 𝐌𝐮𝐬𝐭-𝐑𝐞𝐚𝐝𝐬 Dive into this week's top resources & be at the forefront of the data world! 🚀 𝐃𝐚𝐭𝐚 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 𝐓𝐡𝐞 𝐑𝐢𝐠𝐡𝐭 𝐖𝐚𝐲 Arup Nanda discusses various approaches to structuring data architecture teams based on evolving data landscapes. He contrasts traditional monolithic database designs with modern decoupled architectures like microservices and streaming systems. Read more: https://lnkd.in/dG2XJUMQ 🚀 𝐓𝐡𝐞 𝐑𝐢𝐬𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐃𝐚𝐭𝐚 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 Pedram Navid discusses the evolution of data roles, emphasizing the shift from data scientists to data engineers and now to data platform engineers. Initially, data scientists were key; then, data engineers emerged to handle data ingestion and transformation. Today, data platform engineers focus on creating frameworks allowing others to build custom pipelines. Read more: https://lnkd.in/dJAADeDM 🚀 𝐌𝐢𝐱𝐞𝐝 𝐌𝐨𝐝𝐞𝐥 𝐀𝐫𝐭𝐬 - 𝐓𝐡𝐞 𝐂𝐨𝐧𝐯𝐞𝐫𝐠𝐞𝐧𝐜𝐞 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 𝐃𝐢𝐬𝐜𝐢𝐩𝐥𝐢𝐧𝐞𝐬 Joe Reis 🤓 compares the evolution of mixed martial arts (MMA) to the field of data modelling. He argues that MMA fighters must be proficient in multiple fighting styles, and data professionals must master various data modelling techniques. Read more: https://lnkd.in/d9SAiGJy 🚀 𝐓𝐡𝐞 𝐈𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐜𝐞 𝐨𝐟 𝐚 𝐌𝐨𝐝𝐞𝐫𝐧 𝐃𝐚𝐭𝐚 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 𝐟𝐨𝐫 𝐒𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐀𝐈 𝐏𝐫𝐨𝐣𝐞𝐜𝐭𝐬 Daniel Lambert, M.Sc. discusses the critical importance of modern data architecture for successful AI projects. He highlights the exponential growth of data, necessitating agile and flexible architectures capable of handling vast amounts of information from diverse sources. Read more: https://lnkd.in/dBrvZ33m 🚀 𝐇𝐨𝐰 𝐝𝐚𝐭𝐚 𝐨𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐟𝐢𝐭𝐬 𝐢𝐧𝐭𝐨 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐬𝐭𝐚𝐠𝐞𝐬 𝐢𝐧 𝐭𝐡𝐞 𝐝𝐚𝐭𝐚 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 Mikkel Dengsøe discusses the critical role of data observability across all data pipeline stages. He emphasizes real-time issue detection in data ingestion, storage, transformation, and analysis to uphold data quality and reliability. Read more: https://lnkd.in/d9sNF9AF 🚀 𝐄𝐬𝐬𝐞𝐧𝐭𝐢𝐚𝐥 𝐓𝐢𝐩𝐬 𝐟𝐨𝐫 𝐁𝐞𝐭𝐭𝐞𝐫 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬 In the latest Women in Data Podcast episode, Karen Jean-Francois and Cecilia Oliveira delve into frequent challenges in data analytics and mitigation strategies. They discuss everything from comprehending requests and validating data accuracy to documenting workflows and delivering consistent updates in detail. Listen here: https://lnkd.in/d6c4-D3S Discover MD101's resource page offering the best data wealth, curated just for you. Get weekly updates to stay ahead of modern data trends: https://lnkd.in/dWRBjVCb 🔔 Follow 𝐌𝐨𝐝𝐞𝐫𝐧 𝐃𝐚𝐭𝐚 101 & stay updated with our weekly highlights from the data space.

14

2 commentaren

The modern data lake, sometimes referred to as a data lakehouse, is one-half data lake and one-half Open Table Format Specification (OTF)-based data warehouse. Both are built on modern object storage. https://lnkd.in/gD5PQ8mE #DevOps #GenAI #DataArchitecture by Keith Pijanowski thanks to MinIO

1

A comprehensive guide on constructing LLMs using Databricks. Guide showcases the incorporation of LLM-specific tools and resources within the Databricks framework. It covers aspects such as Vector Indexes, open-source LLM models, and hosted endpoints. Find the guide here: https://lnkd.in/ehtTQhhc Seattle Data Guy

12

Snowflake released the #SnowflakeArctic embed family of models and made them open source under an Apache 2.0 license. The Massive Text Embedding Benchmark (MTEB) Retrieval Leaderboard shows that the Arctic embed model with only 334 million parameters is the only one to perform better than the average retrieval performance of 55.9. The five models, ranging from x-small (xs) to large (l), can be used right away on Hugging Face. Companies use private datasets along with #LLMs as part of a Retrieval Augmented Generation (#RAG) or #semanticsearch service. The impressive embedding models put into practice the technical know-how, search capabilities, and research and development that Snowflake got from Neeva in May. Snowflake Copilot is integrated into Snowflake Cortex, their #AI service, and can be utilized for a variety of tasks such as question answering, and summarization. With these announcements, Snowflake is breaking ground on open LLMs and showing it is continuously innovating for their customers on the AI journey.  

8

Insightful article by Plain Concepts on the evolution of Data Architecture #data #architecture #evolution #future

11

#DataArchitecture: Top 15 Questions for the Leadership before opting for #DataMesh #DataProduct 1.    What are our main business drivers for adopting Data Mesh? 2.    Do we have the necessary cultural mindset to treat #dataAsAProduct? 3.    What are the metrics for success when delivering Data Mesh data products? 4.    How decentralized is our organization, and how mature are our domain teams? 5.    What is our strategy for building cross-functional, data-literate teams? 6.    How will we incentivize domains to take ownership and maintain high-quality data products? 7.    How are data and #analytics currently managed across domains? 8.    How will we handle the operational complexity of distributed data ownership? 9.    How can we standardize #APIs and interfaces for our data products? 10. Do we have clear #DataGovernance practices to ensure consistency and #compliance? 11. How do we manage #datasecurity and #dataprivacy in a distributed data mesh environment? 12. How will we handle cross-domain data dependencies and integration challenges? 13. What infrastructure investments are needed for self-serve #dataplatform? 14. Do we have the right tools for observability and monitoring of data products? 15. What are the risks of adopting Data Mesh, and how can we mitigate them? Details are available in our Premium Content. To subscribe to our Premium Content, please either DM on LinkedIn or email: [email protected] for subscription details. Transform Partner is your trusted navigator in the turbulent seas of data and digital transformation. In an era where businesses are inundated with information yet starved for #insights, we stand as a beacon of clarity and #innovation. Our consultancy brings together a team of seasoned experts with deep industry knowledge spanning government, banking and finance, healthcare, retail, and manufacturing sectors. When you partner with us, you're not just getting a roadmap; you're gaining a co-pilot for your digital journey. We provide hands-on support at every stage, from strategic planning and change management to technical implementation and post-launch optimization. Let's collaborate to turn your digital vision into reality. With Transform Partner, you're not just keeping pace with the digital revolution – you're leading it. Reach out today to schedule a personalized consultation and take the first step towards a transformative future. Image Source: Thoughtworks #TransformPartner – Your #DigitalTransformation Consultancy

313

14 commentaren

The Choice of Schema Impacts Data Storage and Insights Retrieval In data warehousing, schemas dictate how information is stored, accessed, and analysed—ultimately shaping how businesses make decisions. There are several types of schemas to consider: 1. Star Schema  ⭐ A simple design featuring a central fact table surrounded by dimension tables. Known for fast query performance, but can lead to data redundancy. 2. Snowflake Schema ❄️ A more normalised version of the star schema, reducing redundancy but increasing complexity. 3. Fact Constellation (Galaxy) Schema 🌌 A complex structure that supports multiple business processes with multiple fact tables, making it highly flexible but harder to maintain. 💡 Understanding these schemas can be the difference between slow, cluttered queries and lightning-fast analytics. #DataWarehousing #Analytics #DataStorage #BusinessIntelligence

1

Traditional RAG, or Baseline RAG, uses vector similarity to improve LLM outputs with real-world data, a revolutionary approach, while GraphRAG, is now setting new standards by utilizing knowledge graphs for superior performance, especially in complex scenarios. GraphRAG: https://lnkd.in/gcnDvrVk Addressing Business Challenges 🧩 ----------------------------------- Baseline RAG has been effective in enabling LLMs to handle private datasets such as proprietary research and business documents. However, it faces critical limitations: * Connecting the Dots: It struggles when answers require synthesizing scattered information. * Holistic Understanding: It performs poorly with large data collections or detailed documents. The Game-Changer: GraphRAG 🚀 ----------------------------------- Microsoft Research’s GraphRAG addresses these challenges by constructing a knowledge graph from your data. This approach enhances the AI's reasoning capabilities, leading to smarter and more accurate outcomes. Here's how GraphRAG can revolutionize your business: * Enhanced Data Insights: ** Comprehensive Analysis: GraphRAG provides a holistic understanding of large datasets, making it easier to derive meaningful insights. ** Efficient Querying: It enables more accurate answers by connecting related information across different parts of your data. * Improved Decision-Making: ** Informed Decisions: By leveraging community summaries and detailed entity relationships, businesses can make better-informed decisions based on comprehensive data analysis. ** Advanced Use Cases: From analyzing complex research reports to synthesizing vast business communications, GraphRAG excels where traditional methods fall short. GraphRAG is redefining AI capabilities, enabling businesses to tackle advanced use cases with unprecedented accuracy and depth. Embrace GraphRAG to drive smarter, data-driven decisions and stay ahead in the competitive landscape #GraphRAG, #RAG #AI #GenAI

13

In enterprise #AI, there are two main types of models: discriminative and generative. Discriminative models are used to classify or predict #data, while generative models are used to create new data. [' Even though #GenAI has dominated the news of late, organizations are still pursuing both types of AI. Organizations should not build an infrastructure dedicated to AI and AI only while leaving workloads like Business Intelligence, Data Analytics, and Data Science to fend for themselves. It is possible to build a complete data infrastructure that supports all the needs of the organization—Business Intelligence, Data Analytics, Data Science, discriminative AI and Generative AI. AI/ML SME Keith Pijanowski reviews the Modern Datalake Reference Architecture and highlights the capabilities it has for supporting AI/ML workloads. https://hubs.li/Q02r63pv0

7

How does a modern data architecture allow you to use a data catalog, data fabric and data observability to build resilient DataOps pipelines to create a data mesh of reusable data products published in a data marketplace? Join Mike Ferguson on 25-26 June in Amsterdam for a two-day course on modern data architectures. Topics covered: ✅ What is a data architecture and why do you need one? ✅ New data architecture and technology options ✅ How to assess your existing environment, define future requirements, and design a new modern data architecture ✅ And much more! https://lnkd.in/ekAasCVz #datafabric #datacatalog #datamesh #dataarchitecture #datastrategy

3