Ben Halfpenny

Its a jungle out there! 🌿 Take a look at this manuscript mapping similarities/differences in #RWE recommendations across #regulatory and #HTA guidance documents/frameworks. Becaris Publishing Limited #fitforpurpose #RWD #evidence #realworlddata #dataquality #bigdata #regulatoryaffairs #marketaccess #patientjourney #patientoutcomes #clinicaltrials #realworldevidence #drugdevelopment #pharma #FDA #EMA #reimbursement #healthcare #heor #healtheconomics #medicalaffairs #ISPOR #access #ISPE #pharmacoepidemiology

47

𝐓𝐢𝐞𝐫𝐞𝐝 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 𝐢𝐧 𝐊𝐚𝐟𝐤𝐚 Kafka’s architecture is built on the concept of distributed, partitioned, and replicated logs, designed for high-throughput, low-latency streaming. It’s great at handling concurrent producers and consumers, but with the growing demand for real-time data and PubSub systems, the tight coupling of storage and compute can lead to challenges around scalability and maintenance. Pinterest's implemented an alternate architecture in this regard. In a nutshell, they’ve decoupled storage and compute by leveraging cheaper remote storage solutions like AWS S3, a concept they call Tiered Storage. By offloading data storage to S3, Pinterest ensures Kafka brokers focus on what they do best—streaming data—without the overhead of managing and serving stored data to consumers. This shift allows them to maximize the compute power of brokers while making the system more scalable and cost-effective. Huge shoutout to the Pinterest engineering team for the deep dive on this—definitely a must-read for anyone building real-time streaming applications #kafka #streaming #realtimedata #engineering #aws #s3 #decoupled #architecture #cloudcomputing #scalability #data #dataengineering Confluent

5

One platform like Databricks for the whole #MLOps stack might not be a bad idea. I would have advised against it just a couple of years ago, but much has changed. Nowadays, Databricks has all the key components needed for MLOps. We are facing a real epidemic when it comes to the amount of tools in #Data and #AI and resume-driven development. - MAD (Machine learning, AI, and Data) landscape has grown from 139 tools in 2012 to over 2000 tools in 2024. - An average company just had 8 SaaS applications in 2015. In 2022 it was 130. And it is just the beginning: - The MLOps market is expected to reach USD 32 billion by 2030 (from USD 1.6 billion in 2022). - The AI market is expected to grow from an estimated USD 184 billion in 2024 to USD 826 billion by 2030. I know, all these arguments like vendor lock-in, less control, and possibly higher costs. But would you rather have 100+ tools instead? And how many of them will survive? If I were betting on one platform for MLOps, it would be Databricks. Even though there is no perfect solution, they listen to customer feedback and improve fast.

689

58 Comments

https://lnkd.in/dqY75rSb - This is a really nice article but I would to add a few words of advice from my own experience: - Clear separation of your environments and avoiding monorepo terraform monolith are the best ways to go - I recommend isolating you Unity Catalog metastore, workspaces, networking and VPCs, extra cloud resources, and groups / service principals that going be reused across workspaces - all into separate terraform projects with isolated environments (state files) - you don’t want to ruin your infrastructure with a single bad deploy - Plus place all your IaC code into reusable modules to make things DRY and benefit from for-each loops on module level, remember to use module inputs/outputs for managing cross modules deployment dependencies. - Terraform docs pre-commit hook will help you document your code (but you need to type all your variables and provide human-friendly descriptions) - Also use terraform objects maps/lists for configs, add pre-commit hooks for code checks and formatting - Finally, CICD everything with GitHub Actions or your favorite deployment tool :) - Seriously, be prepared to spend hundreds of hours with Databricks documentation, which is great but could be also quite challenging sometimes 😅 #Databricks #IaC #LakehouseDeployment #CICD #Terraform #EngineeringInfrastructure

7

Great blog on Spark AQE!

12

Is this the first time snowflake have officially acknowledged the memory allocation to warehouses? Snowpark-optimised warehouses now let you specify the memory ratio compared to a regular warehouse (link in comments). TL;DR, an XS has 16GB of memory, a standard snowpark optimised XS has 256GB

36

5 Comments

Over the holiday weekend I came across an insightful article [https://lnkd.in/eeQ6pNAT] by Julian Flaks of EQengineered on the importance of providing due consideration to data quality as data-driven software is developed. It's great to see more thought leaders in our industry emphasizing this critical aspect of data management. In my recent article, I explored similar themes, drawing parallels between data and a well-maintained car, highlighting the necessity of regular maintenance for optimal performance. If you're interested in diving deeper into how proactive data care can drive efficiency and reliability, check out my piece here: https://lnkd.in/eE6Nr4Hy Kudos to Julian for contributing to this important conversation. Let's continue to prioritize data health and drive our industry forward! 🚀 #DataManagement #DataQuality #PreventativeMaintenance #DataCare #TechTrends

4

Could you please provide some key considerations and best practices to improve streaming jobs performance? This is a super simple question from a learner of my Apache Spark Performance Tuning course (Visit my website link at the top). I have created this course with more than 50+ performance tuning scenarios with TB data volume so my students can master the performance tuning skills. However, I still keep answering many questions from my students because they ask good and interesting questions. Here is my answer to the above question. Here is a list of some critical things for Streaming job optimization. 1. Use Rocks DB as your state store provider. 2. Configure streaming queries to execute in separate scheduler pools. 3. Apply watermarks to limit the state store size. 4. Drop duplicates within the watermark. 5. Set spark.sql.streaming.noDataMicroBatches.enabled configuration to false to avoid running empty microbatches without data. 6. Set shuffle partitions to 2 times the number of cores and enable AQE. 7. Chain multiple stateful operations for a better execution plan. 8. Limit the input rate with maxFilesPerTrigger, maxBytesPerTrigger or maxOffsetsPerTrigger for predictive batch size. 9. Do not use autoscaling for streaming jobs. 10. Prefer DLT for streaming jobs if you need autoscaling and use enhanced autoscaling for DLT. Looking forward to more suggestions in the comment. PS: Master the most critical skill of Spark Performance Tuning with my advanced course. Visit my website for more details.

72

4 Comments

Have you tried the ASOF join in SQL? Although not widely supported yet, both DuckDB and Snowflake have added support for ASOF joins to help join data across tables with misaligned timestamps The ASOF join keeps the *closest* records in your joined table relative to the records in your base table. This is super handy for things like: - Keeping the latest event per user relative to information on a user - Using the most recent exchange rates for transactions relative to the transaction date - Finding the latest stock balance at a depot relative to when a delivery is made ...and probably loads of other scenarios! Check out my latest YouTube video for a full explanation of how ASOF joins work, some of the syntax rules, and how you can emulate the ASOF join in databases that don't explicitly support it (yet): https://lnkd.in/e7pXfijE Of the databases that I use, the ASOF join has the following availability: - DuckDB: ✅ (>=0.6) - SQLite: ❌ - PostgreSQL: ❌ - SQL Server: ❌ - Snowflake: ✅ (February 28, 2024) Do you know of any other databases that support the ASOF join? Let us know in the comments! #sqlwithbill #dataengineering

79

19 Comments

You don't become a data engineer because you're Databricks (or similar) certified and code in Python. You become a data engineer because you can collect, process and make data accessible for organisations and help solve actual business problems.🔥🔥 Thoughts? #DataEngineering #DataRecruitment #Snowflake #Databricks #Bigquery #Redshift #Azure

127

14 Comments

📢Excited to share our latest blog ''An Introduction to migrating from Oracle to PostgreSQL using ora2pg'' In this blog, Lokesh Mandyam explores Ora2Pg tool for migration from Oracle to PostgreSQL. Here is the blog : https://lnkd.in/g2eSiFWH 🚀Aiming New Postgres Horizons🚀 Stay tuned!! For more info, reach out us at OpenSource DB or https://opensource-db.com/ #PostgreSQL #Oracle #ora2pg #migration #schema #docker #OpenSourceDB #osdb

17

6 Comments

Postgres Women India thanks Lokesh Mandyam for the interesting blog on "An Introduction to migrating oracle to PostgreSQL using ora2pg". This blog discovers how to migrate from Oracle 21c to PostgreSQL 16 efficiently using the powerful ora2pg tool. This comprehensive guide walks you through setting up your Oracle and PostgreSQL environments within Docker, configuring ora2pg for schema export, and successfully importing the schema into PostgreSQL. Key Highlights: Detailed steps for installing and configuring ora2pg in a Docker container. Schema export from Oracle and import into PostgreSQL. Essential tips for data migration, including why ETL tools like Pentaho can enhance the process. Ready to explore seamless database migration techniques? Dive in and learn how to leverage open-source tools to broaden your database horizons! https://lnkd.in/gniKP7aw #PostgreSQL #OracleMigration #DatabaseMigration #OpenSourceTools #Careers #WomenInTech

9

When working with an unfamiliar JSON string and needing to identify all keys, using a JSON_KEYS extractor can be extremely useful. #bigquery https://lnkd.in/eY3nKTE6

6

✍️ A couple of things I've been working on have been published recently. One is an article for Harvard Data Science Review that I wrote with Elena Simperl and Sir Nigel Shadbolt. It makes the case for drastically increasing transparency around the data used to train foundation AI models. It's admittedly a little out of date, as we submitted back in 2023. But there's still plenty of work to do to ensure we really know what this wave of AI has been trained on. There's more interesting articles in this special edition, 'Future Shock: Grappling with the Generative AI Revolution'. https://lnkd.in/eKapqiBX The other is on intellectual property, with Gefion Thuermer, Ben Snaith, Thomas Carey-Wilson and others at the Open Data Institute. It argues that a fair intellectual property regime is key to creating a data and AI ecosystem that works for everyone. It recommends that an incoming UK Government avoids reaching for a one-size-fits-all solution to the copyright conundrum, supports collective rights management and seeks international alignment. https://lnkd.in/eJBJE8p4 Please share and let me know if you'd like to talk.

52

https://lnkd.in/dA9mc9qs - a nice guide for implementing slowly changing dimensions for reliable warehousing on Databricks with Delta Tables

5

🚀 𝐔𝐧𝐯𝐞𝐢𝐥𝐢𝐧𝐠 𝐭𝐡𝐞 𝐌𝐚𝐠𝐢𝐜 𝐨𝐟 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐋𝐋𝐌𝐬) 🎯 🔗 𝐑𝐞𝐚𝐝 𝐭𝐡𝐞 𝐟𝐮𝐥𝐥 𝐬𝐭𝐨𝐫𝐲 𝐨𝐧 𝐌𝐞𝐝𝐢𝐮𝐦: From Generic to Genius: The Art of Fine-Tuning Large Language Models Have you ever wondered how Large Language Models (LLMs) like GPT or BERT transform from being generalists into specialists for tasks like medical diagnostics, sentiment analysis, or legal insights? 𝐈𝐧 𝐦𝐲 𝐥𝐚𝐭𝐞𝐬𝐭 𝐌𝐞𝐝𝐢𝐮𝐦 𝐚𝐫𝐭𝐢𝐜𝐥𝐞, I take you on a storytelling journey through the art of fine-tuning LLMs. Discover: ✅ 𝐖𝐡𝐚𝐭 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐢𝐬 𝐚𝐧𝐝 𝐰𝐡𝐲 𝐢𝐭’𝐬 𝐚 𝐠𝐚𝐦𝐞-𝐜𝐡𝐚𝐧𝐠𝐞𝐫. ✅ 𝐇𝐨𝐰 𝐭𝐨 𝐩𝐫𝐞𝐩𝐚𝐫𝐞 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬 𝐚𝐧𝐝 𝐢𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐞 𝐜𝐮𝐬𝐭𝐨𝐦 𝐜𝐥𝐚𝐬𝐬𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧 𝐡𝐞𝐚𝐝𝐬. ✅ 𝐒𝐭𝐞𝐩-𝐛𝐲-𝐬𝐭𝐞𝐩 𝐜𝐨𝐝𝐞 𝐬𝐧𝐢𝐩𝐩𝐞𝐭𝐬 𝐟𝐨𝐫 𝐥𝐨𝐚𝐝𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥𝐬, 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠, 𝐚𝐧𝐝 𝐭𝐞𝐬𝐭𝐢𝐧𝐠. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐭𝐢𝐩𝐬 𝐭𝐨 𝐭𝐮𝐫𝐧 𝐠𝐞𝐧𝐞𝐫𝐚𝐥-𝐩𝐮𝐫𝐩𝐨𝐬𝐞 𝐋𝐋𝐌𝐬 𝐢𝐧𝐭𝐨 𝐭𝐚𝐬𝐤-𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐠𝐞𝐧𝐢𝐮𝐬𝐞𝐬. ✨ Fine-tuning isn't just about adjusting models; it's about unlocking their full potential for real-world problem-solving. 👩‍💻 Whether you're a 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫, 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐫, 𝐨𝐫 𝐞𝐧𝐭𝐡𝐮𝐬𝐢𝐚𝐬𝐭, this guide simplifies the technical steps and presents them as an engaging story. 🔍 𝐓𝐚𝐤𝐞 𝐚 𝐥𝐨𝐨𝐤 𝐚𝐧𝐝 𝐬𝐡𝐚𝐫𝐞 𝐲𝐨𝐮𝐫 𝐭𝐡𝐨𝐮𝐠𝐡𝐭𝐬: 📖 𝐑𝐞𝐚𝐝 𝐭𝐡𝐞 𝐚𝐫𝐭𝐢𝐜𝐥𝐞: https://lnkd.in/g2wEW2gr 🔄 Repost to inspire others in your network. 💬 Let’s discuss your fine-tuning experiences or challenges in the comments! #LLM #FineTuning #GenerativeAI #MachineLearning #AIForGood #TheLLMInsider

29

The next part of my Causal AI series is up on Towards Data Science, this time exploring how Linear Programming can be used to optimise treatment strategies: https://lnkd.in/eb2ErFgi

46

https://lnkd.in/dcC-kGpu - Dimensional Modelling vs. One Big Table in Databricks Lakehouse - a very interesting article on data engineering and analytics architecture

27