Jean-Christophe Bouramoué

✨ Why you should consider using transform on Spark Article by Ange KOUAME ➡ https://lnkd.in/eJvb8YxZ - - - - - - - - - - - - - - - - - - - - - - - - - - - 💡 Follow for more great content! ✅ ✍️ Want to contribute? Connect with us. 🔗 Let's create together! ✅ - - - - - - - - - - - - - - - - - - - - - - - - - - - #dataengineering  #bigdata #dataanalysis #apachespark

4

Generate Unit Test with Intellitest Visual Studio teams works on integrating Intellitest as built-in feature. The main goal is to generate test by right click on your class name or method. Then you can pick up the test you willing to keep. The downside is, that only available in Entreprise preview version and .Net 6 and above for now. If you want to give a try, you have to get the last preview version of Visual Studio and activate the feature: 𝐓𝐨𝐨𝐥𝐬 > 𝐎𝐩𝐭𝐢𝐨𝐧𝐬 > 𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭 > 𝐏𝐫𝐞𝐯𝐢𝐞𝐰 > 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐓𝐞𝐬𝐭 𝐬𝐮𝐩𝐩𝐨𝐫𝐭 𝐟𝐨𝐫 .𝐍𝐄𝐓 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 𝐚𝐧𝐝 .𝐍𝐄𝐓 𝐂𝐨𝐫𝐞 #visualstudio #softwareengineer #testing #intellitest #dotnet

233

18 commentaires

Here's a secret Snowflake tip 🤫: with the ability to host Notebooks on the new container runtime, we can now install arbitrary packages using !pip. 🔥 Yes, we can also install packages not available on the Anaconda Snowflake repo (with all the risks that come with it 😅). For instance, now you can install the Cinemagoer package and search the IMDb for all the movies containing the word "matrix" in the title 👇

89

New paper from myself and Anne Laure Thibaud (Thieullent) on the topic of how the rise of operational AI is going to fundamentally alter the way that organizations need to look at data. Key points: 1) GenAI (and Agentic) will enable digital employees at operational speeds 2) Digital Employees need data that reflects the operational reality 3) You need to establish a digital operating model 4) You'll need to plan for people managing AI as well as people 5) The Business needs to take control of its digital reality When we started writing this paper it felt like this world was still a while away, since then we've had an explosion in applications embedding GenAI and Agentic solutions. So now it more feels like this is how you need to plan for 2025 rather than a vision for the medium term. Myriam Chave, Mark Oost, Sergey P., Franck Greverie, Eric REICH, Alex Marandon https://lnkd.in/gFw2w9aJ

54

7 commentaires

A useful price comparison of LLMs.

6

Watch Luigi Nardi's talk from #PosetteConf 2024 🌟 “Autotuning PostgreSQL on Azure Flexible Server” and Explore the world of #PostgreSQL parameter tuning & see a proof-of-concept for autotuning Azure Database for PostgreSQL Flexible Server 🌐

4

Will MariaDB Introduction of Open-Source Vector Search bring it closer to Become Default MySQL Option ? https://lnkd.in/e3DeKXhv #mysql #mariadb #opensource

5

⭐️ New video release 📺: AsyncApp. My contribution to hype Pythons asyncio a bit more Watch Jens Nie as he simplifies building asyncio based applications with powerful yet easy-to-follow patterns, making asyncio a go-to solution for beginners and experts alike. #asyncio #raspberrypi 📺 Watch the video on YouTube: https://lnkd.in/ejFqM_R2 Jens Nie, a physicist turned developer at Rosenxt Group, presented at AsyncApp. He discussed the increasing use of asyncio in Python but noted it's not the default choice for all projects, especially by beginners. Jens shared his journey creating simple yet powerful asyncio-based applications using lightweight and attractive patterns. Asyncio addresses I/O performance problems by preventing code from blocking concurrent elements in an application. While traditional methods like threading and multiprocessing can be complex, asyncio offers a simpler abstraction layer. Jens emphasized that proper threading and multiprocessing require a deep understanding and are often topics for experts. To showcase asyncio's efficiency, Jens demonstrated building blocks for creating a Raspberry Pi monitoring app. This app gathers sensor readings, logs them, stores data, and tracks CPU and memory usage. It can be configured through various methods and follows the UNIX philosophy of focusing on specific tasks. With asyncio, Jens highlighted the ease of developing effective applications without intricate knowledge of underlying processes.

4

Your Contribution for tonight ✨: Airbyte ⬇️ Co-founded by Michel Tricot and Jean (John) Lafleur, Airbyte is a data integration platform for ETL / ELT data pipelines. You can integrate from APIs, databases, data warehouses, data lakes and data lakehouses. Airbyte is available self-hosted and cloud-hosted. 🧪 Here’s why you should contribute to this repo: 1️⃣ Their tech leverages modern practices to help solve data engineering challenges, affecting hundreds of companies and data professionals. If you are a Python or Java developer and want to further contribute to a powerhouse repo (with over 13,000 stars!), this is your opportunity. 🦾 2️⃣ The repo "really" welcomes new contributors. Many projects have a contributing.md file but rarely accept new contributors. Airbyte brought on their repo 12 new contributors just this last month! If you are looking for a welcoming repo, Airbyte may be what you have been looking for. 😄 3️⃣ Nearly ~45% of contributors submit a second PR. This means the repo has a supportive environment for building upon past contributions. If you enjoy working on this repo, chances are you can continue to work on it! ⚡️ If you are interested in contributing to building the only tool devs need to move data, check more of Airbyte's stats on Quira (https://lnkd.in/eM7etquW) or access the GitHub page directly here: https://lnkd.in/e_4Gczrk

13

Frankly I find ` OOPS generally excessively complicated . The simplest ` object is an " address space " an ordered list . I've commented to Alan Kay long ago , eg: https://lnkd.in/gTSi-fRy https://lnkd.in/gJiuCyFQ , that that what ` objects miss is the ` geometric attribute that APLs , including K , naturally retain . In CoSy.com , evolved from K as an open vocabulary in Forth , the 3 cell header for an ` object is simply `( Type Count refCount )` ( where ` refCount handles memory management , free on return to 0 ) . Upon that , any more complex Types can be constructed ` inheriting the entire general list manipulating vocabulary , again , evolved & greatly simplified from APL via K .

I'm really fond of the AlgorithmWatch newsletter. The latest issue is both fascinating and terrifying. I was hooked by the title : how to design antifascist #AI. Spoiler alert : it does not seem possible yet because of how AI is structured.. Resources regarding AI in France are terrifying from automated facial recognition in public transport to automatically detecting swimming pools (and how it can go terribly wrong). Not that other countries seem to fare better. We hear more and more about the environmental impacts of AI and there's a (relatively) new topic emerging here : keeping in mind that AI will never be totally accurate. This is a huge problem and could lead to considerable harm. A few years ago, there was some focus on how to balance inaccuracy with the cost of training a model (for example to reduce environmental impacts). But the possibility for errors is sometimes not acceptable. As illustrated with the example of swimming pools, relying on AI to make some process easier/faster led to a huge increase of work for some very human employees. Robo-debt is another concerning example. And there are many others. Should AI be both the poison and the antidote ? I don't think so. https://lnkd.in/e-cGQj-g Nicolas Kayser-Bril

5

Headed to SciPy? Join Guen Prawiroatmodjo , Elena Felder, Alex Monahan, Mehdi Ouazza, and Nicholas Ursa from the MotherDuck team to learn more about DuckDB's highly efficient, in-memory engine. Tuesday, 7/9: All the SQL a Pythonista needs to know: Introduction to SQL & DataFrames with DuckDB https://lnkd.in/eaT9eKK4 Friday, 7/12: How to Bootstrap a Data Warehouse with DuckDB

7

Remember back in 2012, when PostgreSQL introduced JSON support ? https://lnkd.in/e_Qb8rpc Now in PostgreSQL 17 we have JSON_TABLE and new JSON functions !

3

[French version in my previous post] My network, I've been a bit distant lately, as I've had a lot of things to think about, prepare ... I've spent 9 great years at onepoint. From a professional point of view, I was able to strengthen my expertise and my network around Red Hat products (mainly OpenShift, Ansible and JBoss), I was able to work on several large "Move To Cloud" projects, with Azure, AWS, Google, Google S3ns and OVH, I was able to carry out technical and organizational audits, do “fire brigade” missions with expertise, create offers and training courses, run training courses, do R&D, presentations, videos, organize a stand from A to Z, and even work on a mini comic strip and videos. More recently, generative AI has been at the center of my daily life. From a human point of view, it's been years of fabulous encounters, laughter and travel all over France and Europe, with colleagues, customers and sometimes even friends. I'd also like to mention the young (and not so young) Padawans I've been able to support in their development (they'll recognize themselves) and the experts I've stood up to (not to mention CTO Jamal Moqaddem and our endless discussions about who was right ... me, of course, even if he's convinced it was him). And someone who gave me my chance in 2015 and has stuck with me to this day and probably beyond (Erwan LE BRONEC: Thank you). If you're a deduction champion, you'll have understood that a page is turning. Friday will be my last day at onepoint. Thank you Onepoint for 9 years. What's next? Already, a 15-day vacation with my family, an air-conditioned mobile home, swimming pools and almost no cell phone reception... enough to get my strength back. In September, I'll be self-employed... Freelance, as they say. But personally, I don't like that word. I prefer to say that I'm finally creating my own business (admittedly, I'm on my own, but who knows what the future will bring). What's going to change in my daily life? Not much ... I'll be doing the same work, making my estimates, organizing my travel ... I've been doing all that already. Well, there'll be the invoicing side and all the administrative hassles, but for now, motivation is here. So it's with confidence and desire that I embark on this new episode in my life, and I'll try to share the major stages of my project with you. And to do things properly, I've updated my CV, which you'll find at: https://lnkd.in/eBbyJJ5r (I'll be taking care of my company's website as soon as I get back from vacation). PS: If you'd like to contact me about projects, please wait until August 12, as I won't be available before then.

25

6 commentaires

You need to expand your data expertise and/or the data footprint of your organization. You may have adopted some software tool or even developped internal software librairy. BUT Have you had a look at Yavantha ? It is so easy to try : https://lnkd.in/eza7S_zV And awesome! I tell you : every one around you will ask you how you bring so much values of your data! 🤩⏳🔧🛫🙃👍🙈

2

⭐️ New video release 📺: Unleashing Confidence in SQL Development through Unit Testing Watch how Tobias Lampert explores the power of #SQL unit testing in ensuring data quality and boosting confidence in SQL development, offering insights into a robust framework tailored for BigQuery. 📺 Watch the video on YouTube: https://lnkd.in/dJG4B4_M Tobias Lampert, an Analytics Engineer with vast experience in Data Science and Data Engineering, delivers a compelling talk on "Unleashing Confidence in SQL Development through Unit Testing." In this enlightening session, he sheds light on the significance of SQL unit testing in ensuring data quality within Data Warehouses. By showcasing the challenges faced by data teams in maintaining complex SQL models, Tobias demonstrates how unit testing can validate modeling logic swiftly, preventing modifications from disrupting existing behavior. The talk emphasizes the importance of Lotum's SQL unit testing framework, meticulously designed in Python using pytest and tailored for BigQuery. By processing millions of events from mobile games daily, this robust framework facilitates efficient testing, guaranteeing the accuracy of SQL logic. Attendees gain valuable insights into defining test cases with small sets of mock data, enabling the identification of potential code errors with ease. Tobias underlines the necessity of a well-designed unit testing framework in collaborative data model development, highlighting its role in early error detection and maintaining data quality. He explains how SQL unit testing is particularly valuable during the refactoring of existing data models and integration of new data sources, ensuring the preservation of logic, even in edge cases. The practical implementation of a SQL unit testing framework is elucidated, focusing on Lotum's pytest-based framework. By delving into the internal workings of the framework, Tobias illustrates how mock data is injected into SQL statements for testing, allowing for case-by-case verification without manual query code modifications.

3