Ian Hopkinson

🌟 Exciting blog post from NHS England on Reproducible Analytical Pipelines  🥁 Keep banging the drum Sam Hollings 📈 NHS England are sharing their journey on Reproducible Analytical Pipelines (RAP) and how it's enhancing the quality of their data processes. 🔍 What is RAP? RAP aims to automate statistical and analytical processes, making data analysis more efficient, transparent, and robust. This means less manual work and more time for data analysts to focus on what truly matters - delivering insights that improve healthcare services. 🌐 Why It Matters for the Culture of Data Services? Adopting RAP aligns the commitment to enhancing the quality and reliability of data services. It's not just about technology; it's about fostering a culture where continuous improvement and collaboration are at the forefront. 🛠️ Progress NHS England Data teams are integrating RAP across the organisation and It's still early days, but the impact is tangible and transferable. 📈 Looking Ahead They are committed to expanding the use of RAP, enhancing data capabilities, and continuing to share their progress. Hoping to hear more of this type of work. #NHS #DataScience #Innovation #RAP #HealthcareTechnology #Teamwork #Data #Health #Collaboration Link to the source in the comments. Image created by DALLE-3

17

1 Comment

On #DataScience, the #scientific method and #truth… ⚛ ☯ I was pondering these words today and wondering if Data Scientists actually utilise the scientific method in their work or use a different, less rigorous approach. If you are a Data Scientist, I'd be interested in your thoughts on the methods used... I then came across a conversation which contained these words: 👊 “For those trying to piece together the meaning of what we see taking place in public (claims about various big systems based on data)… the scientific toolkit is the toolkit to use as it produces a product that allows us to evolve in the direction of greater insight. The product is predictive power. If you have a cognitive model (not a computer model) of the universe that has greater predictive power than competing models, then it is closer to the truth. So what we really should be tracking is how well people making various claims are at spotting things that nobody believes, but which turn out to be true given time”. This is taken from a conversation between Brett Weinstein (Evolutionary Biologist) and Ivor Cummins (Chemical Engineer) at the Brownstone Institute. I then found Patrick Hall's Medium article where he feels so strongly about this, that he came up with a heuristic, “the Data Scientific Method” and he goes on to say: 🔴 “Data science has a science problem. A lot of data science seems much more aligned with cargo cult science, a phrase coined by Richard Feynman to describe an endeavor with the trappings and appearance of science, but at its core, is pseudoscience or snake oil. Cargo cult science feels too familiar to me. Sure, we use computers, look at numbers, write down equations, and call ourselves scientists, but an anti-pattern has become so common for me I’ve come up with a heuristic to describe it — the data scientific method.” He hits a little more harshly 👀 in this but I do wonder if this could be one of the route causes as to data science doesn't always get the credit it deserves... Data Scientists not following or knowing the scientific method. A short comment on this topic from a Data Scientist suggested that business pressure drives this (non scientific method) approach, and line management who do not have a STEM background are less willing to put value on the work. Very interested to hear thoughts on this by #Datascientists or those overseeing Data Science projects.... #datascience #science #predictiveanalytics

4

3 Comments

A lot of my recent chat about building parameterised plots came out of this project for Bigger Brighter Bolder. A few months ago, George Swift and Tracey Miller invited me to speak to their membership, which is mostly comprised of non data business owners (although there are a few exceptions*) about how data and dataviz could be used to communicate what a business does and what impact it has for its clients. In preparing for the talk, I collected some data about the BBB members, so that we could talk about something they all had some experience of, and demonstrate how the data collection process and the dataviz design helps tell the story. Different groups had different questions to answer, but the pattern was the same - a resounding increase in confidence across the relevant areas for their stage of business development. It was great being able to reflect back to BBB what impact they've had on so many of their members, and the look on Tracey's face when the tooltips first did their thing was priceless. BBB have been hugely helpful to me in figuring out how to transition from freelancer to business owner - if anyone is looking for tips on that, they are running their last workshop before the summer tomorrow (Tuesday 18th June) at 12.30 UK time. Here's the sign up link: https://lnkd.in/e4gMHemi And if this type of plot is something you think could help showcase what you do for your clients - you know where I am! --- * I'm one of them.

90

22 Comments

🤔 What is a perceptron, and how does it work? --- Get our FREE data science cheat sheets in high resolution by subscribing to our blog today! https://lnkd.in/ejCdvUUA #machinelearning

5

Containerizing R to use in Production with out refactoring to Python.

3

How often does an opportunity come to work on a greenfield platform like Pydantic Logfire? Build a database that will scale (if we get it right) to petabyte scale and change how develoeprs understand their applications.

115

3 Comments

Get ready to use your #data #analytics skills for good! DataKind UK are running their next weekend #DataDive weekend on 6-7 July, kindly hosted by Monzo Bank. So if you want to deploy your data superpowers to help fight poverty you can team up with representatives from Citizens Advice and Money A+E and have some real impact. But be quick the deadline to apply is Friday 17 May. #data4Good #datascience #geospatial

10

1 Comment

Matching free text to a statistical classification is a key part of the production process for many statistics in ONS. We have started to develop a flexible toolkit, 'ClassifAI', which we hope will make this process more efficient and accurate. It uses Retrieval Augmented Generation and can match a custom set of free text fields to a custom classification. Initial tests matching anonymised labour market survey data to the Standard Industrial Classification have been promising, showing evidence of handling trickier cases (e.g. spelling mistakes or jargon) better than existing automated approaches. Our aim is to deploy the tool as a scalable, secure API service, available for use in the organisation. But this will require further testing and refinement. Access the code for the proof of concept work below (N.B. this is experimental so should not be used in a production setting). It contains tutorials and instructions for how to build your own application to test it out. Feedback is welcome on all aspects of the code repository, and on any wider uses you can think of. More information in the blog below! All credit to: Iva Špakulová Jyldyz Djumalieva Philip Lee Mat Weldon Edward Jackson Sam Stock https://lnkd.in/eb474aYH

69

3 Comments

Finally got the BigEarthNet test set loaded into Postgres. Used Colab since nuking GDAL killed GeoPandas 🐼 https://lnkd.in/ebm9JVVc Once again, I don't think embedding distance has much to do with physical distance. While this is cool, let's see if we can throw novel data at Prithvi and get off of Postgres... Oh yes here is a cool article: https://lnkd.in/edMGKJs6 Bulk load with psycopg executemany() to avoid iterating over dataframes. Seemed fine without using copy.

1

I think there also has to be a reckoning with the dramatic naming in quantum tech; If your solution takes 25mins to run, all-in, then it's a 25min solution. Doesn't matter if you used qubits fashioned from the wing-clippings of Hermes... If your cloud queue is slow, then It's all slow. Industrial Quantum Advantage has a yearning for a decent definition.

14

2 Comments

Food for thought and a reminder that data science is not just for geo - here are three picks showing how data science is applied in extraterrestrial settings from @Towards Data Science : 𝐌𝐨𝐧𝐨𝐜𝐮𝐥𝐚𝐫 𝐃𝐞𝐩𝐭𝐡 𝐄𝐬𝐭𝐢𝐦𝐚𝐭𝐢𝐨𝐧 𝐭𝐨 𝐏𝐫𝐞𝐝𝐢𝐜𝐭 𝐒𝐮𝐫𝐟𝐚𝐜𝐞 𝐑𝐞𝐥𝐢𝐞𝐟𝐬 𝐨𝐟 𝐌𝐚𝐫𝐬 - https://lnkd.in/d8m5pNub 𝐋𝐮𝐧𝐚𝐫 𝐂𝐫𝐚𝐭𝐞𝐫 𝐃𝐞𝐭𝐞𝐜𝐭𝐢𝐨𝐧: 𝐂𝐨𝐦𝐩𝐮𝐭𝐞𝐫 𝐕𝐢𝐬𝐢𝐨𝐧 𝐢𝐧 𝐒𝐩𝐚𝐜𝐞 - https://lnkd.in/damEi-Xr 𝐒𝐢𝐦𝐮𝐥𝐚𝐭𝐢𝐧𝐠 𝐚 𝐫𝐞𝐚𝐥 𝐬𝐨𝐥𝐚𝐫 𝐬𝐲𝐬𝐭𝐞𝐦 𝐰𝐢𝐭𝐡 70 𝐥𝐢𝐧𝐞𝐬 𝐨𝐟 𝐏𝐲𝐭𝐡𝐨𝐧 𝐜𝐨𝐝𝐞 - https://lnkd.in/d8XCfpbg #datascience #geospatial #spatial #datavisualization #space

3

Day 240 of the #365GenAIChallenge! 📅 I'd like to introduce you all to... a Vector Library!📚 Vector libraries, like FAISS or HNSW, are lightweight tools for efficient approximate nearest neighbour (ANN) searches, ideal for prototyping or static datasets. Not to be confused with vector, that are scalable systems designed for production-grade workloads, offering features like real-time updates, hybrid searches, and multi-tenancy.

2

An interesting post on how Skyscanner uses the combination of Dr Jekyll and its in-house statistical tool called WISE to ship experiment variants to production. Instead of using a frequentist approach as is done in most AB experiments, this framework uses a Bayesian approach to quantify the uncertainty in the final estimates by reporting credible intervals instead of confidence intervals. Summary: - Example of an experiment to test whether a pop-up, mentioning that ‘Skyscanner never takes a cut’ increases the ‘redirector rate’ or not. - If sufficient data has not yet been collected, WISE’s results will say ‘keep testing’. - WISE is able to provide a recommendation on the variant to ship. - WISE uses the Bayesian approach. They model their belief of the underlying parameter values as probability distributions, which are updated as new data is observed. - For conversion metrics, it uses a beta-binomial model - For revenue per user metrics, it uses a hurdle gamma-exponential model - What's the experiment stopping criteria: Through Monte Carlo simulations of thousands of experiments, they have determined a set of bespoke target 90% HDI widths to serve as their stopping criterion, which ensured an appropriate balance between the duration of experiments and the correct decision rate. https://lnkd.in/dv7BEHXe #statistics #analytics #rstats #python #peopleanalytics #experimentation

6

1 Comment

A data engineeering course recommended by Andrew Ng. Seems useful for gaining better understanding of how to build robust data pipelines and systems. 3 months, 10 hours each week, self paced. Tempted to take the course, would encourage my team to go for it given we have significant data works (or recommend better alternatives!). Dr. Yap Ghim-Eng 葉錦榮博士 (PhD, PMP®, CSM®, CSPO®) has better recommendations? https://lnkd.in/g5xrn4em

53

5 Comments

I'm exploring how Splink can be used for address matching. Does anyone know of any labelled datasets of messy address data that could be used to measure performance? (i.e. labelled to indicate which addresses really match) FWIW there are some rough demos of matching some open address data (EPC to price paid data, and food hygiene score to price paid data) here: https://lnkd.in/e_wHRDcm But without labels, difficult to quantify how good it is

19

6 Comments

Location parameter - Clearly explained! #statistics #eda --- Get our FREE data science cheat sheets in high resolution by subscribing to our blog today! https://lnkd.in/eXHcfqUT

5

Evaluation is always interesting!!!, thanks Aishwarya for the below

3