The Data Collective

a little Delta Lives Tables speedtest #Databricks #DeltaLiveTables #IncrementalRefresh

Unlock the power of your data with Databricks! Whether you’re a new customer, or already onboard, we’ll help you harness the Data Intelligence Platform to transform your business. Ready to accelerate? Let’s go! #databricks #innovation #dataintelligence

István M. Best day so far, Patrick does his job really well. I'm sure it's tough talking for hours to a faceless audience with very little real-time feedback. Well done for keeping us engaged! Today was really about how easy it is to misunderstand or misrepresent results: • we, people, tend to anthropomorphise and use wishful mnemonics: however LLM models 𝗱𝗼 𝗻𝗼𝘁 "𝘭𝘦𝘢𝘳𝘯" or "𝘳𝘦𝘢𝘴𝘰𝘯" or "𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥" the same way as we mean we talking about fellow human beings • even if such models 𝘀𝗲𝗲𝗺 like doing these activities, something totally different is happening in the background so it can be a mistake to interpret results how we would from another human • the 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲 concept contains a certain circular reference for me because in reality there is only so many models available and it can easily happen that your model is using the same that you use to interpret definitions given in natural language (e.g. what "professional" means) and the same one could be used to judge it. My conclusion is: 𝗯𝗲 𝗰𝗮𝗿𝗲𝗳𝘂𝗹 what you promise to your clients and be prepared - if your creation is public facing - that there would be players who want to abuse it (𝘦𝘷𝘦𝘯 𝘪𝘧 𝘰𝘯𝘭𝘺 𝘧𝘰𝘳 𝘧𝘶𝘯). #Databricks #GenerativeAI #training

István M. I admit today (Day 2) was more abstract for me as I don't yet have the hands-on Python experience with ML models. The concepts were laid out nicely in a straightforward way and again the obvious lesson came up that can't be emphasised enough: 𝗱𝗮𝘁𝗮 fed to an LLM model is 𝗰𝗿𝘂𝗰𝗶𝗮𝗹 and can make or break a deliverable. Another important difference from a developer perspective (𝘢𝘱𝘢𝘳𝘵 𝘧𝘳𝘰𝘮 𝘱𝘶𝘳𝘦𝘭𝘺 𝘗𝘺𝘵𝘩𝘰𝘯 𝘤𝘰𝘥𝘦 𝘸𝘪𝘵𝘩 𝘚𝘘𝘓 𝘯𝘰𝘸𝘩𝘦𝘳𝘦 𝘯𝘦𝘢𝘳 𝘪𝘯 𝘴𝘪𝘨𝘩𝘵 ;)) is that the usual boundary line between end users and data engineers is blurrier as the job of an AI engineer is 𝘯𝘰𝘵 to 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥 the tool (not necessarily even conceptually) and to make sure what's returned is fully accurate, but we are talking only about 𝘱𝘳𝘰𝘣𝘢𝘣𝘪𝘭𝘪𝘵𝘪𝘦𝘴 coming out of 𝗯𝗹𝗮𝗰𝗸 𝗯𝗼𝘅𝗲𝘀. I can only suspect what's happening behind the scenes whereas inside a Delta Live Table I know what's going in and what I could predict even if it's automatically done. That is something that needs time getting used to. #Databricks #GenerativeAI #training

István M. After Day 1 of 'Generative AI Engineering with Databricks' I first of all would like to thank Patrick Do who is a good trainer and knows his stuff. As promised, my highlights: • AI / LLM models are 𝗻𝗼𝘁 𝗲𝘅𝗮𝗰𝘁. When you do a chat-based search on the internet for a good taco place - that's fine. However in corporate data analysis the users need to understand that generative AI produces results that are 𝘣𝘦𝘴𝘵 𝘨𝘶𝘦𝘴𝘴𝘦𝘴 based on data that is never 100% accurate. The job of the data engineer is to increase that 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 as much as reasonably possible. The job of the data consultant is to communicate this and make sure that clients (analysts, managers, C-level people) actually 𝗴𝗲𝘁 𝗶𝘁. • 𝗔𝗜/𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿 and 𝗱𝗮𝘁𝗮 𝘀𝗰𝗶𝗲𝗻𝘁𝗶𝘀𝘁 are 𝘯𝘰𝘵 𝘵𝘩𝘦 𝘴𝘢𝘮𝘦, they require different expertise. There is an overlap but as an engineer one does 𝗻𝗼𝘁 need to know how to create a good model but to understand how to evaluate and find the best possible one for the actual use-case. #Databricks #GenerativeAI #training

Run only CURRENT statement in Databricks SQL editor #Databricks #DatabricksSQL

Gerard and István will attend the | Generative AI Engineering with Databricks | course next week. We really look forward to it - despite time zone differences making it a very early experience from 6am in the morning for four days, but that is a small price for living in this faraway country ;o) We plan to share our experience by publishing “highlight of the day” posts, 2 posts a day for 4 days. We are really interested to find out how much we can apply straight away - or with little effort - in extending our existing approach. And as the course includes a chance to sit the exam we’ll do our best to have another badge next to our names :-) #Databricks #GenerativeAI #training