Abhishek Joshi

An excellent article on the emerging building blocks for AI Agents - LAM(Large Action Model - function calling), Vision-enabled Language models, Model Orchestration, SLM (Small Language Models for niche tasks) with RAG.

8

https://lnkd.in/dyZcR9RP I really think it’s a great pattern for building ETLs based on your operational RDBMs sources (use database replica if you can, unless you can ingest messages from topics or event buses, then switch to events for sure, they are immutable and easy to ingest): - deploy AWS DMS service to track your databases changelog on table level and write the results to storage buckets (parquet files, date partitioning, KMS encryption) - this can be done with terraform and CICD deployed - attach S3 buckets with CDC files to your Unity Catalog as volumes - again terraform automation is a great way to go - use Spark Streaming-based AutoLoader to load data in parallel way to bronze delta tables in your UC (materialized views, actually, you won’t see a differences unless you start using single-user clusters ;) ) - this is even easier when you use Delta Live Tables (much less configuration needed for AutoLoader) - the greatest feature is the checkpoints usage - you don’t need to load the same files twice or run complex state checks - plus AutoLoader pattern can be used both for streaming and batch loads (you don’t need to run it continuously, it can run on batch schedule too) - use SCD (Slowly Changing Dimensions) features in Delta Live Tables to rebuild source tables in UC (a great feature of UC - you’ve got time travel for every table and lineage too, this is so handy in debugging) - then you can build all ETLs on UC delta tables without any live connection with RDBMs live connection - which should be avoided at all costs on production systems - all your ETLs will then run with full power of Spark / Photon compute engines and fast delta reads - all your PySpark code can be deployed with testing and linting using your favourite tool, e.g. GitHub Actions and deployed to the Lakehouse with Asset Bundles automation As a result, you avoid direct database integration on source database system, you can track data changes continuously, your ETLs are reproducible and idempotent, and can be securely monitored (with AWS CloudWatch + Alerts, Databricks Workflows and Data Alerts) and in case of reprocessing - you avoid major bottlenecks and can scale resources up, plus all your configurations are safely stored and versioned in code :) #Lakehouse #AWS #CloudETLs #Databricks #DeltaLiveTables #UnityCatalog #PySpark #CDC #DataEngineering

10

The rapid pace of development around AI in Software Engineering is fascinating and a little overwhelming! I’ll be kicking off my Master’s study in a few weeks, starting with a simple survey to capture professional Software Engineers’ perceptions and experiences of how AI coding assistants are impacting their work. It is clear that there are so many opportunities for us to leverage these tools to reduce toil and improve productivity, but how exactly they will change the SDLC, what we spend time on, and our ability to learn and retain coding skills, remains to be seen. I’m really looking forward to gathering these insights and understanding how we can better prepare ourselves for a future where AI tools are a standard part of our toolkit in Software Engineering.

20

7 Comments

Definitely need to avoid 💩 in, 💩 out situations! Otherwise, not only is there zero return on your shiny and exciting AI investment, but arguably you've just made things a lot worse. AI on poor quality data compounds the poor output, and moves your organisation backwards when it comes to your data quality and hygiene. #data #dataquality #claranet #ai #roi #datareadiness

9

I am delighted to have been a contributor to Gartner’s most recent #EnterpriseArchitecture research led by my colleagues Andrei Sachelarescu, Auria Asadsangabi & Lucas Kobat. Clients subscribing to our insights on how to define future-state enterprise architecture (EA), focused on how to design reference architectures and models can read, “3 Steps to Enable a Future-State Architecture for a Data-Driven Organization” (7 Oct 2024) Read it now: https://lnkd.in/e2r6KAHh I want to share some background on this. Organizations seek a balance between managing cost and performance while investing to innovate and grow the business. EA practices often struggle to align with rapid technological advancements like #AI and delivering reference architectures maximizing business value while minimizing costs. Without reference architectures, technologies’ full benefits go unused and unnecessary expenses accrue by repeatedly designing, building and managing solutions for individual problems. Reference architectures streamline tech management and problem solving with reusable designs for similar solutions. This reduces the total cost of ownership and addresses many problems with consistent requirements. Heads of EA can guide business and IT stakeholders to improve profitability through technology innovation investments that enhance enterprise productivity. So when designing a #DataDiven organization, it requires an enterprisewide view infused with data and analytics (D&A) leaders and their teams’ data and technology expertise. Heads of EA and D&A must collaborate to design a robust future-state architecture that makes the data-driven organization a reality. Gartner’s Key Findings: 🔵 Heads of EA cannot design a trusted future-state architecture alone for a data-driven organization, as this requires strong expertise from D&A teams as well. 🔵 Designing a data-driven organization requires a good understanding of both business outcomes and data and technology considerations. 🔵 Even the best future-state architecture for a data-driven organization will not deliver on its promises if data cannot support #SelfServiceAnalytics and if users of data-driven insights do not have enough comprehension on how to use those insights. Our Recommendations: 🔵 Foster enterprise trust in the design of a future-state architecture for the data-driven organization by forging a robust collaboration with D&A to ensure the data-driven vision is actionable and scalable. 🔵 Clarify what a data-driven future architecture looks like by supporting the D&A team in building an architecture that ensures data flows seamlessly across the organization, is managed for quality, and is easily and quickly accessible to everyone when required. 🔵 Enable trusted insights from the implementation of the future-state architecture by supporting the D&A team to maximize the consumability of insights to all stakeholders and address data usability gaps.

23

4 Comments

Shipping a consumer facing AI product requires a complementary set of skillsets than what is required for setting up an AI inference pipeline / EDA dashboard for internal business stakeholders. If the aim is to differentiate your business on the basis of AI product offerings AND intend to take data backed smart decisions to drive day to day business decisions at the same time, both set of skillsets are equally important. Think of the following roles: - Data Engineer (Data Platform) - Analytics Engineer (BI & Automations) - Data Scientist (Research, EDA, inference pipelines, model CI pipelines) - AI / ML Engineer (Consumer facing Data Products) - Allied Roles (DevSecOps, SRE, PM, QA, Design, UX, Frontend) (Did I miss any? Add in the comments) The lines between each of these adjacent roles are extremely blurred, sometimes invisible - owing to the aim of businesses to have Data Generalists who could do anything and everything related to data. However there's always a trade-off that most businesses don't consider while asking their junior data scientists to expose mind-boggling inference results they saw during an internal meeting on a Data Scientist's Notebook to end consumers. What do you think these businesses are missing to consider? #foodforthought #aiproducts #datascience #analytics #ai #ml #data

36

1 Comment

🚨 Coming from a public sector background and now supporting NHS teams, security is critical when it comes to safeguarding data. With rising cyber threats, your data needs the best protection out there, and that’s where the Databricks Data Intelligence Platform comes in. 💪🔐 It offers powerful, cutting-edge security tools to define, deploy, and monitor your workloads, ensuring your data stays safe every step of the way. Ready to strengthen your defences? Explore how to leverage top controls and new features to build a robust, defence-in-depth strategy for your data and AI assets. ⚙️🔒👇

2

Explore https://lnkd.in/gYku2amJ to analyze various open-source and proprietary LLM models. From my experience, LLAMA3 is sufficient for many use cases and is much more cost-effective.

22

2 Comments

We just acquired WarpStream at Confluent, and I couldn't be more excited! But why? There's a lot of reasons to love the Cloud, and I've seen huge successes within customers like Curve and Funding Circle UK who have jumped headfirst into Confluent Cloud, but sometimes there are challenges that end up slowing down full adoption of a fully managed service. Confluent Platform offers an alternative for hosting your own data streaming platform, but the overhead of managing this can be a burden for smaller teams. WarpStream's BYOC option will allow us to help those stuck in the middle. Whether its a sovereignty issue when deploying in a new territory, or edge deployments where an easier management saves hiring another engineer, the option to have control of your data plane while Confluent manage the system control plane will allow businesses to accelerate their adoption of a #DataStreamingPlatform across their business. There's a bright future ahead having Warpstream in the team, and we're looking forward to seeing what comes next!

15

1 Comment

I’ve registered for Government Data & AI Summit 2024, so if you share my interest in getting government data right, and safely leveraging AI, in order to drive better, fairer decision making across government - join me and register: https://lnkd.in/eD7aX6-k Why attend? This conference flips the script - as speakers sit alongside delegates in small group discussions, letting everyone have the conversations that matter most to them. It’s a great way to save time and money and achieve genuine cross-government collaboration!

29

2 Comments

🔥 A distributed system is not an elementary endeavor by any stretch of the imagination. Here are the 10 commandments to abide by: 🔄 Distributed Transactions 🔁 Retry and Idempotency 🚧 Handling Service Failures 📈 Schema Evolution 🔍 Observability and Distributed Tracing ⚖️ Eventual Consistency and CAP Theorem 🔎 Service Discovery and Load Balancing 🧩 Data Partitioning and Sharding 📡 Asynchronous Communication and Event-Driven Architecture 🔒 Security in Distributed Systems And they don't include SDLC matters. #DistributedSystems #SoftwareEngineering #TechChallenges #CloudComputing

Adrian Bridgwater's FOCUS (FinOps Open Cost & Usage Specification) caught my attention recently. The approach helps firms traverse the often muddy waters of cloud finance. FOCUS aims to create a unified language for cloud cost and consumption statistics. This would provide greater control over cloud spending, higher transparency, improved cooperation, and scalability. FOCUS does not necessitate any specific software implementation. It's more like a set of best practices for data organisation and categorisation. While FOCUS is not a program, it necessitates changing how enterprises interpret cloud cost data. Here are various approaches to implementing FOCUS: - Work with cloud providers: Encourage your cloud providers to use the FOCUS standards to maintain data consistency across platforms. - Standardize Internal Processes: Define internal tagging practices for cloud resources that align with FOCUS guidelines. - Invest in Cost Management Tools: Look for tools that can ingest and analyse cloud cost data based on FOCUS standards. What are your thoughts on FOCUS? #CloudFinOps #CostOptimization #CloudBilling

3

🚨 𝗙𝗶𝗻𝗢𝗽𝘀 & 𝗘𝗥𝗣: 𝗔𝗿𝗲 𝗬𝗼𝘂 𝗔𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗻𝗴 𝗖𝗼𝘀𝘁𝘀 𝗼𝗿 𝗕𝘂𝗿𝗻𝗶𝗻𝗴 𝗕𝘂𝗱𝗴𝗲𝘁? Cost allocation in the cloud is trickier than ever—are you getting it right or letting inefficiencies drain your resources? 🔍𝗧𝗛𝗘 𝗗𝗘𝗘𝗣 𝗗𝗜𝗩𝗘 ⭕ FinOps empowers real-time cost allocation, breaking down cloud expenses by department, project, or service, so every dollar spent is tracked and optimized. ⭕ AI-driven insights help finance and IT teams make faster, smarter decisions on cloud investments, eliminating the guesswork from cost allocation. ⭕ SaaS costs are often hidden, but FinOps helps reveal and manage these expenses, ensuring your ERP integrates with cloud strategies that make sense financially. 🤔 𝗪𝗛𝗬 𝗜𝗧 𝗠𝗔𝗧𝗧𝗘𝗥𝗦 Without clear cost allocation, cloud spend spirals out of control. FinOps helps cloud engineers and finance teams align, ensuring that resources are allocated where they drive the most value. 🚀 𝗬𝗢𝗨𝗥 𝗡𝗘𝗫𝗧 𝗦𝗧𝗘𝗣𝗦 Review your current cloud cost allocation strategy—are you optimizing every dollar? Read further with this link: https://vist.ly/3mizefy #finops #cloudcomputing #costallocation

3

2 Comments

Here's my latest article on Snowflake's rollback features. This awareness is key for designing Transaction Management and idempotency.

5

What’s the collective noun for a group data and enterprise architecture experts? 📣 Veran Performance’s Data and Enterprise Architecture Community! 📣 Emily Hassall-Clarke Jaff Jarvis Leanne Adams Chris Holttum Robert Dean Oliver Dykes Amber Burke Christopher Finnerty George Tracey Kavya Vaishnav Tom Leitch Harry Gale These fantastic folks play a central role in every project – on data management, reporting, insights, migrating data and service management. Find out how they could help you by dropping us a message in the chat or emailing [email protected] #datamigration #enterprisearchitecture #data #AI #servicemanagement

24

1 Comment

Exploring and running large language models (LLMs) locally has never been easier, thanks to tools like LM Studio, which provides valuable insights into model size, prompt tokens, RAM, CPU usage, etc => https://lmstudio.ai/ 👍 ---- #ai

10

1 Comment