Paul Howarth

For large organisations using LLMs (AI) having access to all internal documents (such as Copilot 365 or Google workspaces + Gemini) there is a potentially major data governance/protection side effect. Currently organisations can refuse FOI or Subject Access Requests where the cost exceeds certain thresholds or is too difficult to answer. However, plugging LLMs into all of your internal data is going to mean that some of these requests which would have taken hours or days to comply with can be answered in seconds. Refusals to release information will become harder to justify.

16

5 Comments

𝐌𝐚𝐬𝐭𝐞𝐫𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞: 𝐓𝐡𝐞 𝐔𝐥𝐭𝐢𝐦𝐚𝐭𝐞 𝐆𝐮𝐢𝐝𝐞 An efficient data pipeline is critical to unlock insights and drive innovation. Let’s break down the essentials of building a rock-solid data pipeline: 🔘 𝐈𝐧𝐠𝐞𝐬𝐭: Gather data securely and efficiently. ↳ Authentication - Ensure data access is secure. ↳ Authorisation - Control permissions effectively. ↳ Scheduling - Automate data collection timing. ↳ Scaling - Handle increasing data volumes smoothly. 🔘 𝐂𝐥𝐞𝐚𝐧: Prepare data for analysis by removing inaccuracies. ↳ Deduplication - Eliminate redundant data. ↳ Missing Values - Address gaps in your dataset. ↳ Formatting - Standardise data appearance. ↳ Outlier Detection - Identify and handle anomalies. 🔘 𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝𝐢𝐬𝐞: Make data consistent and comparable. ↳ Standardise Taxonomy - Use uniform terminology. ↳ Unit Conversion - Harmonise measurement units. ↳ Transformation - Alter data to the desired format. ↳ Map Data Types - Ensure data types match their intended use. 🔘 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐞: Ensure data quality and integrity. ↳ Schema Validation - Verify data structure. ↳ Type Validation - Confirm data types are correct. ↳ Range Validation - Check that the data falls within the expected ranges. ↳ Business Rule Validation - Ensure data meets business criteria. 🔘 𝐂𝐮𝐫𝐚𝐭𝐞: Enhance and structure data for analysis. ↳ Data Modelling - Create structured data representations. ↳ Aggregation - Combine data for summary insights. ↳ Summarisation - Condense data for quick insights. ↳ Denormalisation - Simplify data for performance. ↳ Enrichment - Enhance data with additional information. From ingestion to curation, every step is crucial in transforming raw data into valuable insights. 🤔 What’s your biggest challenge in your data pipeline journey? Share your experience in the comments 👇 PS: This is not a comprehensive design but a great starting point for understanding the critical activities in data pipelines. ~~~~~~~~~~~~~~ ♻️ Repost if you found this post interesting and helpful!   💡 Follow me for more insights and tips on Data and AI. Cheers!   Deepak #DataPipeline #DataManagement #BigData #DataScience #AI #MachineLearning #DataCleaning #DataValidation #DataCuration #TechTrends #BusinessIntelligence

546

44 Comments

#DataArchitecture: List of key #Interoperability challenges stemming from #DataPlatform and #DataEcosystem: 1.    Data format incompatibility 2.    Inconsistent #metadata and schema definitions 3.    Varying #DataQuality standards 4.    Differing #DataSecurity and access control mechanisms 5.    Incompatible #APIs and integration protocols 6.    Semantic inconsistencies in data interpretation 7.    Scalability mismatches between systems 8.    Temporal inconsistencies in data updates and synchronization 9.    Regulatory and #compliance variations across platforms 10. Differing #DataGovernance models 11. Inconsistent #DataLineage and provenance tracking 12. Platform-specific #DataTransformation requirements 13. Varying levels of data granularity 14. Incompatible data storage and retrieval mechanisms 15. Differences in data versioning and historization approaches To receive more details and similar framework, best practices, tools and techniques in data and digital transformation, subscribe to our Premium Content. You can either DM on LinkedIn or email: [email protected] for subscription details. Transform Partner is your trusted navigator in the turbulent seas of data and digital transformation. In an era where businesses are inundated with information yet starved for #insights, we stand as a beacon of clarity and #innovation. Our consultancy brings together a team of seasoned experts with deep industry knowledge spanning government, banking and finance, healthcare, retail, and manufacturing sectors. When you partner with us, you're not just getting a roadmap; you're gaining a co-pilot for your digital journey. We provide hands-on support at every stage, from strategic planning and change management to technical implementation and post-launch optimization. Let's collaborate to turn your digital vision into reality. With Transform Partner, you're not just keeping pace with the digital revolution – you're leading it. Reach out today to schedule a personalized consultation and take the first step towards a transformative future. Image Source: Research Gate #TransformPartner – Your #DigitalTransformation Consultancy

108

8 Comments

In today's data-driven world, the key to unlocking your business's full potential lies in smarter data management. At DB Results, our Data Management Services are designed to empower organisations with solutions that maximise the value of their data. Our proven framework ensures that your data strategy aligns perfectly with your business objectives: Discovery: We start by assessing your current data performance and exposure, crafting a tailored plan for efficient data management. Planning: Our experts work with you to develop a comprehensive strategy, ensuring that every byte of your data is utilised to its fullest potential, while maintaining robust governance practices. Implementation: From technical capabilities to UX/UI design, we deliver end-to-end solutions tailored to your business needs. Support: Our commitment doesn't end with implementation. We provide ongoing support, operations, and enhancement services. Ready to harness the power of smarter data management? Learn more at https://lnkd.in/gPhhkrEJ #DataManagement #BusinessTransformation #DBResults #BCorp

6

Matillion Democratizes GenAI with No-Code Cortex Components on Snowflake AI Data Cloud https://lnkd.in/dvJ6mq5d #DataCloud #GenerativeAI #Matillion #NoCodeCortex #Snowflake

19

Review the white paper, “Will The Real In-memory Database Please Stand Up?” on the advantages of a real in-memory database (IMDS) for data management. https://bit.ly/true-IMDS #DBMS #IMDS #fasterdatamanagement #tinyfootprint

5

👏 Opendatasoft joins Databricks partner program 🤝 Opendatasoft today announced its partnership with Databricks, the data and AI company, becoming a Validated Databricks Technical Partner, enabling it to deliver innovative new ways to access data, and help organisations maximise value from their data programs. 🔄️ As a validated Databricks Technical Partner, Opendatasoft has developed an integration connector to enable its customers to seamlessly access and reuse data assets. ""We are focused on accelerating data use to increase productivity, enable better decision-making and reduce risk. This partnership offer users innovative new ways to access and harness information"" - Jean-Marc Lazard, CEO and co-founder, Opendatasoft. 🔎 Read the full press release here: https://lnkd.in/ey2nQP5F #data #dataportal #datademocratization

10

How to go from a data mess to a data mesh? 🔗 In the last decade, companies recognised that data was locked away in departmental silos and legacy systems. This drove many to centralise their data management. This in turn, brought data into monolithic, centrally governed structure, which led to bottlenecks, complexity and slow time-to-value for data. Essentially: out of the frying pan, into the fire 🔥 This brings us to a key question: how do you reconcile agility and central governance for data? If departments own their data and can manage it independently, it can lead to silos, risks and inefficiencies. On the other hand, a centralised model can easily become convoluted, difficult to manage and slow. The solution? Enter the data mesh. Data mesh is an approach of organising data within an organisation: – It is a distributed system, where departments/teams are responsible for their own data. – Data is considered a product, instead of a by-product. – Instead of a data storage (data lake/warehouse), data mesh is more interested in data activation/usage. – The role of central governance is not to direct but to enable; to serve as a centre of excellence, rather than a decision-making body. – Instead of focusing on optimising the data pipeline, the data mesh focuses on providing domain-specific value. – The emphasis is on communication and sharing of the data products, so that the risk of siloed data is mitigated. #DataMesh #DataManagement #Data

7

📊 Hear from one of our Open-Source Engineers on unlocking valuable insights and cheap metrics from datasets! 📊 Dive into the world of data efficiency with our latest article on observing Spark aggregates. Discover powerful techniques to optimise your data processes and drive smarter decision-making. https://lnkd.in/eJKbdgnS #OpenSource #DataScience #Spark #Analytics

17

I am delighted to have been a contributor to Gartner’s most recent #EnterpriseArchitecture research led by my colleagues Andrei Sachelarescu, Auria Asadsangabi & Lucas Kobat. Clients subscribing to our insights on how to define future-state enterprise architecture (EA), focused on how to design reference architectures and models can read, “3 Steps to Enable a Future-State Architecture for a Data-Driven Organization” (7 Oct 2024) Read it now: https://lnkd.in/e2r6KAHh I want to share some background on this. Organizations seek a balance between managing cost and performance while investing to innovate and grow the business. EA practices often struggle to align with rapid technological advancements like #AI and delivering reference architectures maximizing business value while minimizing costs. Without reference architectures, technologies’ full benefits go unused and unnecessary expenses accrue by repeatedly designing, building and managing solutions for individual problems. Reference architectures streamline tech management and problem solving with reusable designs for similar solutions. This reduces the total cost of ownership and addresses many problems with consistent requirements. Heads of EA can guide business and IT stakeholders to improve profitability through technology innovation investments that enhance enterprise productivity. So when designing a #DataDiven organization, it requires an enterprisewide view infused with data and analytics (D&A) leaders and their teams’ data and technology expertise. Heads of EA and D&A must collaborate to design a robust future-state architecture that makes the data-driven organization a reality. Gartner’s Key Findings: 🔵 Heads of EA cannot design a trusted future-state architecture alone for a data-driven organization, as this requires strong expertise from D&A teams as well. 🔵 Designing a data-driven organization requires a good understanding of both business outcomes and data and technology considerations. 🔵 Even the best future-state architecture for a data-driven organization will not deliver on its promises if data cannot support #SelfServiceAnalytics and if users of data-driven insights do not have enough comprehension on how to use those insights. Our Recommendations: 🔵 Foster enterprise trust in the design of a future-state architecture for the data-driven organization by forging a robust collaboration with D&A to ensure the data-driven vision is actionable and scalable. 🔵 Clarify what a data-driven future architecture looks like by supporting the D&A team in building an architecture that ensures data flows seamlessly across the organization, is managed for quality, and is easily and quickly accessible to everyone when required. 🔵 Enable trusted insights from the implementation of the future-state architecture by supporting the D&A team to maximize the consumability of insights to all stakeholders and address data usability gaps.

23

4 Comments

Don't miss out on this blog written by Allyse Fristo, M.Eng., it is a master class in FinOps. FinOps and responsible spend is an extremely important, and overlooked activity. Artificial Intelligence is extremely expensive and extremely disruptive. Reducing spend in your data warehouse allows you to conserve budget for your AI activities. Meaning, FinOps is even more important today than it has ever been.

11

1 Comment

⛔️ Don't use a "current state assessment" before your first large Data project -✅Instead aim for the high visibility, maybe simple but most likely effective use case (1 to 2) to set the ball rolling. Malcolm Hawker makes a very interesting point below which to many is still quite controversial. A number of views including LLM's will state the need to make the initial assessment a priority first to gain clarity/direction/vision which are all not wrong either. The swift dose that Malcolm delivers as someone who comes from an Advisory/Strategy level is interestingly reality led rather than the largely theoretical LLM version. I may not be an expert Data Strategist however the facts are what they are and successful Data outcomes are still way behind where they technically should be for a huge amount of organisations. Getting hearts and minds on board in the business first with tangible results, even if not game changing (whilst the wider assessments are also considered at a different pace) is a concept that applies to many businesses and not just Data. However the adoption of Data and further investment is critically based on results not concepts. Is Malcolm right? #usecases #dataprojectlaunch #datainitiative Simon Asplen-Taylor, Kate Sargent, Eddie Short, Nick Bouch, Eddie Short, Tony King, Jason Foster, David Wong

6

6 Comments

Really like the idea of an art driven infographic! Trust, Value, Innovation, Ethics... Certain trends jump out the longer you look at the image. I enjoy the multiple AI conversation streams but who are the ones stating with the data? #cloud #cloudmigration #cloudmodernisation #dataengineering

10

To utilise #agentforce from Salesforce you need to integrate your data to Snowflake Lumilinks Group Ltd can provide you specialist data engineering resource to enable your business to utilise #ai By being a specialist we can achieve this in an efficient time and cost, enabling you to see value from your data

17