Pedro Figueiredo

Good article on practical tips to improve the #RAG pipeline - across indexing, retrieval and generation. • Indexing - level up with simple, semantic or #LLM based chunking • Retrieval - implement hybrid search, combining vector and keyword searches • Query Rewriting - optimize queries for better retrieval • Metadata - boost retrieval accuracy with rich contextual data • Autocut - filter out irrelevant information from retrieved content • Reranking - prioritize the most relevant content, enhancing retrieval precision https://lnkd.in/eJMvu3XJ

7

In todays #XataLaunchWeek demo, Andrew shows us to how add a NOT NULL constraint to an existing column with a multi-version schema migration, accessing both the old and new schema simultaneously until the migration is complete ✨ Watch now: https://lnkd.in/ekcuabS2

3

In this article I would like provide some high-level guidance for domain (data) modelling. This article is for you if you are relatively new to domain or concept modelling. It uses Accord Project Concerto but the principles should apply to many modelling languages. #datamodels #modelling #concerto https://lnkd.in/dxgbXdkZ

7

4 Comments

Interesting thoughts from Junta N. about #AI use cases in #insurance.

19

With all the noise around data and data products, it can be tough to cut through and really get what matters. 😥 But I stumbled over this article a while ago, authored by Dylan Anderson on 𝑶𝒑𝒆𝒓𝒂𝒕𝒊𝒐𝒏𝒂𝒍𝒊𝒔𝒊𝒏𝒈 𝑫𝒂𝒕𝒂 𝑷𝒓𝒐𝒅𝒖𝒄𝒕 𝑫𝒆𝒍𝒊𝒗𝒆𝒓𝒚, and this picks some pretty complex topics and breaks them down in a way that just clicks. This article brings out the crux of this landscape, which is being 𝐩𝐫𝐨𝐝𝐮𝐜𝐭-𝐟𝐨𝐜𝐮𝐬𝐞𝐝, and how this journey began vs. what it has evolved into. What grabbed my attention is the discreet 𝐜𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐬𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐝𝐚𝐭𝐚 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐬! More importantly, Dylan puts across the operational model of data products in an extensive yet understandable manner and breaks the model into three phases: 𝐒𝐜𝐨𝐩𝐢𝐧𝐠, 𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠, 𝐚𝐧𝐝 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥𝐢𝐳𝐢𝐧𝐠. Not only do you get to read what these phases cater to, but you also get insights into the 𝐝𝐞𝐥𝐢𝐯𝐞𝐫𝐚𝐛𝐥𝐞𝐬 𝐨𝐟 𝐞𝐚𝐜𝐡 𝐨𝐟 𝐭𝐡𝐞 𝐩𝐡𝐚𝐬𝐞𝐬 that give great clarity or the practical, real-world impact of these phases. And the way it explains creating feedback loops between teams? Practical and totally worth thinking about. 😎 If you're into data & data products, this one’s worth a read. It’s packed with insights without the fluff. 🙌 Here's the link to this Medium article: https://lnkd.in/gtxzpbpV

66

2 Comments

https://lnkd.in/dqY75rSb - This is a really nice article but I would to add a few words of advice from my own experience: - Clear separation of your environments and avoiding monorepo terraform monolith are the best ways to go - I recommend isolating you Unity Catalog metastore, workspaces, networking and VPCs, extra cloud resources, and groups / service principals that going be reused across workspaces - all into separate terraform projects with isolated environments (state files) - you don’t want to ruin your infrastructure with a single bad deploy - Plus place all your IaC code into reusable modules to make things DRY and benefit from for-each loops on module level, remember to use module inputs/outputs for managing cross modules deployment dependencies. - Terraform docs pre-commit hook will help you document your code (but you need to type all your variables and provide human-friendly descriptions) - Also use terraform objects maps/lists for configs, add pre-commit hooks for code checks and formatting - Finally, CICD everything with GitHub Actions or your favorite deployment tool :) - Seriously, be prepared to spend hundreds of hours with Databricks documentation, which is great but could be also quite challenging sometimes 😅 #Databricks #IaC #LakehouseDeployment #CICD #Terraform #EngineeringInfrastructure

7

"Undisciplined use of Kafka leads to poor quality data." That’s 100% true - not just for Kafka but for any data movement/integration tool. Hear hear Andrew Jones Almost everywhere I go I see a mixture of home-grown CDC, SaaS tools like Fivetran, Stitch, QlikReplicate, sprinkled with cloud tools and OSS like Airbyte and Airflow or whatever to orchestrate. The problem is very rarely the tools used. The problems always stem from a mix-and-match approach. No real holistic design … as Andrew says it is a problem of discipline. I’d go as far as say beware of any tool that makes your job of moving data easy. There will always be a catch, probably technical debt to pay down as the ELT pipelines proliferate! Paraphrasing Tolstoy “all happy data pipelines are alike, all unhappy data pipelines are unhappy in their own way”

3

2 Comments

𝐓𝐢𝐞𝐫𝐞𝐝 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 𝐢𝐧 𝐊𝐚𝐟𝐤𝐚 Kafka’s architecture is built on the concept of distributed, partitioned, and replicated logs, designed for high-throughput, low-latency streaming. It’s great at handling concurrent producers and consumers, but with the growing demand for real-time data and PubSub systems, the tight coupling of storage and compute can lead to challenges around scalability and maintenance. Pinterest's implemented an alternate architecture in this regard. In a nutshell, they’ve decoupled storage and compute by leveraging cheaper remote storage solutions like AWS S3, a concept they call Tiered Storage. By offloading data storage to S3, Pinterest ensures Kafka brokers focus on what they do best—streaming data—without the overhead of managing and serving stored data to consumers. This shift allows them to maximize the compute power of brokers while making the system more scalable and cost-effective. Huge shoutout to the Pinterest engineering team for the deep dive on this—definitely a must-read for anyone building real-time streaming applications #kafka #streaming #realtimedata #engineering #aws #s3 #decoupled #architecture #cloudcomputing #scalability #data #dataengineering Confluent

5

As a follow on to the original AI Optimizer in Db2 blog post, authors Calisto Zuzarte & Vincent Corvinelli discuss the high-level features covered in the first release of the AI query optimizer. Read all about it: https://lnkd.in/etDrreeK

31

❄ Snowflake Vs Databricks 🏗 I've noticed a bit of a shift in the last 6 months or so. Databricks requirements seem to be (for us at least) more prominent than Snowflake. I had a really interesting chat with one of the founders of a leading Snowflake partner at Big Data LDN on where he thinks Snowflake is heading in comparison to Databricks. Is Databricks going to rule supreme? Will Snowflake be back? Is Fabric going to overtake them both? What are your thoughts? #snowflake #databricks #fabric

13

5 Comments

That's right, we have a super exclusive DBRX & LLMOps themed event with the London Databricks Meetup next week! And no, none of us can believe it's May next week either! Come along, grab a drink & a snack, learn from some terrifyingly smart folks! See you there? #dataengineering #london #datascience #llmops #genai #geospatial #datalakehouse #databricks

19

1 Comment

A super interesting article on using dynamic tables with Snowflake. https://lnkd.in/esp_4buQ #ETL #ELT #Snowflake #Data #Cloud #Orchestration

3

Snowflake announced its new feature, Cortex Analyst, last month. Here’s our Panda’s 🐼 take on it. In this blog Sumanth Donthula covers: the importance of self-service in analytics, why Cortex Analyst, and how Cortex Analyst outperforms other tools. Read the full blog here ⬇ https://lnkd.in/gSRtsVYK

7

The snowflake saga continues... Mandiant published a blog with their findings https://lnkd.in/eMRM3e4D. Bottom line Snowflake wasn't breached in the classic sense, but this is perhaps something more interesting than that. It would appear that Snowflake has an extortion app for it now. What seems to have happened is that somebody created some automation to make use of stolen Snowflake account credentials and went around a bought up a good number of Snowflake account credentials from initial access brokers, some of which were several years old at the time of being used in the attack campaign. I think this detail is important since it means those credentials were exposed for several years, but likely not actually used prior to this attack campaign probably because there was not previously an automated mechanism to use them. The attacker hit a large number (at least 165 so far) of Snowflake customer accounts in a short time, including apparently TicketMaster. Which certainly could give the impression of a direct Snowflake breach when viewed from the outside, but I think it just means there is an app for it now. Credential stealers like Redline, LumaStealer etc. are deployed by specialized groups that harvest credentials - literal usernames and passwords, but also post authentication tokens from as many victim's computers as they can. They then bundle these up and sell them to other specialized groups that focus on extortion. (This is where I must put in the obligatory mention that UpSight.ai is particularly good at stopping credential stealers!). What holds this process back is how fast an extortion group can make practical use of those credentials. Which is why I say this might be something more interesting, somebody invested some time to create Snowflake specific extortion tools giving those account credentials new value to the extortion actors, and the initial access brokers that sell them. Net what I suspect this means is that if you happen to lose control of your Snowflake credentials, they are very likely to be used in the future.

1

Databricks offers granular access control mechanisms that empower organizations to enforce data governance and privacy policies effectively. Through column-level and row-level security, companies can safeguard sensitive data while maintaining flexibility in data access. The use of dynamic views enables Access Control Lists (ACLs) to tailor visibility based on user roles and privileges. This ensures that authorized users have full access, while restricted users receive filtered or redacted data, aligned with predefined governance rules. In scenarios where personal identifiers—such as email addresses, first names, and last names—are present, restricting field-level access becomes crucial to comply with data privacy regulations like GDPR. With dynamic views, organizations can dynamically adjust access without duplicating data or creating multiple datasets, making compliance scalable and efficient. This approach not only simplifies compliance with regulations but also ensures seamless data governance across departments, providing the right people with the right level of access. By leveraging Databricks' capabilities, organizations can strengthen data security, reduce risks, and optimize data-sharing practices within an auditable framework. #DataGovernance #Databricks #DataSecurity #AccessControl #DynamicViews #PrivacyCompliance #GDPR #AI #CloudSecurity

2

Exponam.Connect now fully supports the latest Delta Sharing changes implemented with Deletion Vectors earlier this month. https://lnkd.in/eC5krg_Y Databricks Delta Lake #deltasharing #D2O #ExponamConnect

1

Having recently had a heightened interest in stream processing and streaming sql. I came across KSQL and how it can be used with Kafka can anyone shed any light as to why this isn’t more supported from Confluent. There seems very little thought into how it would be productionised and how version controlled sql could be deployed to the servers. I have started my own attempt at firstly just getting sql stored in a git repo deployed to a KSQL_DB server instance running on docker with my Kafka instance and then secondly I want to try building a DBT adapter for this as this would essentially solve the version control issues. KSQL seems a really powerful and easy to use solution, I know there are alternatives such as apache flunk but that seems like a lot more over head for those not wanting to use confluent cloud. #dataengineering #kafka #datastreaming #ksql #dbt

7

6 Comments

Azure Cosmos DB design patterns – Part 2: Data Binning This post explores the data binning technique, which aggregates data into specific time intervals or 'bins' for streamlined analysis. https://lnkd.in/ebnWi8WQ

1

Snowflake Iceberg Table "Auto Refresh": 1. Snowflake Iceberg table Integration with Polaris Internal Catalog Table --Working. 2. REST Integration Snowflake Iceberg table with Polaris Internal Catalog Table--Auto Refresh is Working.

65

2 Comments