Chaouki Trabelsi

𝐃𝐞𝐚𝐫 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 🔥 Here I come with another PySpark Transformation Challenge 📣 𝐈𝐧𝐟𝐨: You have a DataFrame with two columns: 𝐢𝐝 and 𝐩𝐥𝐚𝐧_𝐝𝐚𝐭𝐞. 𝐘𝐨𝐮𝐫 𝐭𝐚𝐬𝐤 𝐢𝐬 𝐭𝐨 𝐜𝐫𝐞𝐚𝐭𝐞 𝐭𝐡𝐫𝐞𝐞 𝐧𝐞𝐰 𝐜𝐨𝐥𝐮𝐦𝐧𝐬: 1. 𝐈𝐒_𝐋𝐀𝐒𝐓_𝐒𝐈𝐗_𝐌𝐎𝐍𝐓𝐇𝐒: For each id, check whether the date is within the last 6 months compared to the previous row for that id. The first row for each id should always be "YES", and subsequent rows should check if the previous date is less than 6 months before. If it is within 6 months, mark it as "YES", otherwise, mark it as "NO". 2. 𝐈𝐒_𝐂𝐔𝐑𝐑𝐄𝐍𝐓_𝐅𝐈𝐍𝐀𝐍𝐂𝐈𝐀𝐋_𝐘𝐄𝐀𝐑: For each id, check whether the date belongs to the current financial year (April 1, 2024 - March 31, 2025). 3. 𝐈𝐒_𝐂𝐔𝐑𝐑𝐄𝐍𝐓_𝐘𝐄𝐀𝐑: For each id, check whether the date belongs to the current calendar year (January 1, 2024 - December 31, 2024). 𝐂𝐫𝐞𝐚𝐭𝐞 𝐒𝐚𝐦𝐩𝐥𝐞 𝐃𝐚𝐭𝐚𝐟𝐫𝐚𝐦𝐞 (Input Data): 𝘥𝘢𝘵𝘢 = [   (1, "2024-08-15"),   (1, "2024-03-10"),   (1, "2023-05-05"),   (2, "2025-02-24"),   (2, "2024-07-15"),   (2, "2024-03-15") ] 𝘤𝘰𝘭𝘶𝘮𝘯𝘴 = ["𝘪𝘥", "𝘱𝘭𝘢𝘯_𝘥𝘢𝘵𝘦"] 𝘥𝘧 = 𝘴𝘱𝘢𝘳𝘬.𝘤𝘳𝘦𝘢𝘵𝘦𝘋𝘢𝘵𝘢𝘍𝘳𝘢𝘮𝘦(𝘥𝘢𝘵𝘢, 𝘤𝘰𝘭𝘶𝘮𝘯𝘴) 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐂𝐨𝐝𝐞? (Expected Data) Let me know the answer in the comments 😀 #pyspark #dataengineering

81

6 commentaires

These are very important points to understand how to work in data engineering while handling BIG data. Thanks Zach Wilson for summarizing it.

2

[Java] developers can use any deep learning model as long as it is ONNX-compatible, without the need to rewrite the entire backend in Python. This can save valuable time for many teams, especially if your product backend is in Java. 💡 I have implemented this example, a U2Net model trained in Python can be seamlessly used in Java via ONNXRuntime Java. 🚀 - First, a mask is generated using the U2Net ONNX model. 🎭 - Then, a composite image is calculated using the inputs (image, mask). 🖼️ - Finally, the final image is generated using inputs (image, background). 🌟 #JavaDevelopment #DeepLearning #ONNX #Integration #JavaBackend

9

5 commentaires

In this great post Thierno Diallo shows you how to implement Slim CI with dbt core and Snowpark, check this out! https://lnkd.in/d9Jv6Vve #dbt #dataengineer #dataengineering #analyticsengineer #analyticsengineering

65

3 commentaires

If you are curious  to understand the evolution of Diffusion models, here is a list of key research papers to get you started. 📌 Related Prior Works: ➕ GAN : https://lnkd.in/gpNdEghh ➕ VAE : https://lnkd.in/gy_R79S2 📌 Basic Diffusion Models: Generate images by denoising a true noise image (Gaussian) employing Forward and Reverse diffusion techniques using UNet. ➕ DDPM: https://lnkd.in/gmS5yU6M  ➕ UNet: https://lnkd.in/gTMFumaN ➕ DDPM-Improved: https://lnkd.in/gBAG8thf 📌 Latent Diffusion Models (Stable Diffusion): Use latent spaces, instead of directly working on pixel spaces, to enhance the efficiency ➕ Stable Diffusion paper: https://lnkd.in/gzhxNDEC 📌 Text Conditioning: Guide the generation of images using textual descriptions ➕ CLIP:https://lnkd.in/g_pk5wrW ➕ GLIDE: https://lnkd.in/gswrJwUJ ➕ Dall-E 2 : https://lnkd.in/gdgqm7wj

13

1 commentaire

New tool bar with nice order, ability to fold/unfold and what I like even more : possibility to search a tool by name. Nice update on Amphi last release. Good job Thibaut Gourdel and thanks for listening to users (especially when the user is myself). For everyone, if you don't know Amphi : it's a (young and WIP) visual python-based ETL. https://amphi.ai/

9

When working with an unfamiliar JSON string and needing to identify all keys, using a JSON_KEYS extractor can be extremely useful. #bigquery https://lnkd.in/eY3nKTE6

6

Excited to share a RAG LLM Quiz Generator I worked on for next week's Databricks' #DAIS2024 ! Discover how LangChain and a vector database can come together to create dynamic, context-aware quizzes for any topic you input, complete with incorrect distractor answers, explanations for all answers and links to sources for each. Because we're going to be showing this off at DAIS 2024 next week, this flavor of the quiz is based on... Databricks! Try The Quiz For Yourself: https://lnkd.in/e9WiuUdj Read our blog post fully breaking down the techniques and technologies & services used to implement this on Databricks : https://lnkd.in/efyPyzJp Or if that's not enough, take the notebook that powers this pipeline and try it for yourself in your Databrick's environment: https://lnkd.in/eS7Fnxex We'll be there all week at Booth E9, if you want to know more about how we built this on Databricks, some of the other projects were working on or are curious how Rearc may be able to support your Generative AI endeavors - stop by and say hello! #Rearc #DataAISummit #LLMWhisperer

28

11 commentaires

Netflix have mentioned about their internal workflow orchrestrator since 2022 “‘Maestro” And they just opensource it Last week! Some key highlight ✨: - It can scale horizontally and suitable for large scale data processing - Support various kind of run script (Docker images, notebooks, bash script, SQL, Python, and more) - Support acyclic, cyclic workflows and come with built-in complex run strategy/pattern - Workflow definition is JSON format (Unlike Airflow that is python base) - Support dynamically parameterized workflows with Simple Expression Language (SEL) For those who use cloud data platform like Databricks, Azure Data Factory and other mighy feel that their is nothings new. All of these features can be done easily (with some extra pay 💸) But Maestro is opensource like Apache Airflow. Repository : https://lnkd.in/g4dT4VN5 Blog from Netflix : https://lnkd.in/gkTPhFfz

14

𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝗡𝗲𝘄𝘀 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠𝘀) 𝗦𝗽𝗮𝗰𝗲! PyTorch has officially released TorchTune, a new tool designed to fine-tune Large Language Models (LLMs) with ease and efficiency. This groundbreaking tool promises to streamline the fine-tuning process, making it more accessible to researchers and practitioners alike. https://lnkd.in/eMf6dNZJ 𝗧𝗼𝗿𝗰𝗵𝘁𝘂𝗻𝗲 𝗢𝗳𝗳𝗲𝗿𝘀: ▪ 𝗣𝘆𝗧𝗼𝗿𝗰𝗵-𝗻𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀: Build on familiar ground with composable building blocks for popular LLMs. ️ ▪ 𝗠𝗲𝗺𝗼𝗿𝘆-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗿𝗲𝗰𝗶𝗽𝗲𝘀: Train on consumer-grade GPUs with 24GB VRAM – no beefy machines needed! ▪ 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗮𝗻𝗱 𝗲𝗮𝘀𝗲 𝗼𝗳 𝘂𝘀𝗲:  YAML configs and support for common dataset formats streamline your workflow. 𝗙𝗼𝗿 𝘁𝗵𝗼𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗱𝗶𝘃𝗶𝗻𝗴 𝗱𝗲𝗲𝗽𝗲𝗿, 𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Code: https://lnkd.in/gEgETe5t Blog: https://lnkd.in/gP3mctA6 Tutorials: https://lnkd.in/gvh2eEsU This is a fantastic opportunity to explore and innovate, I'd be very interested in hearing your thoughts. ▪ 𝗪𝗵𝗮𝘁’𝘀 𝘆𝗼𝘂𝗿 𝘁𝗮𝗸𝗲 𝗼𝗻 𝘁𝗵𝗶𝘀?  ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝗶𝗺𝗽𝗮𝗰𝘁𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗼𝗿 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵? ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝘄𝗶𝗹𝗹 𝗶𝗺𝗽𝗮𝗰𝘁 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗜 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁? #AI #PyTorch #TorchTune #LLMs #MachineLearning

24

1 commentaire

Cisco Kafka interview questions for Data Engineers 2024. ➤ How do you create a topic in Kafka using the Confluent CLI? ➤ Explain the role of the Schema Registry in Kafka. ➤ How do you register a new schema in the Schema Registry? ➤ What is the importance of key-value messages in Kafka? ➤ Describe a scenario where using a random key for messages is beneficial. ➤ Provide an example where using a constant key for messages is necessary. ➤ Write a simple Kafka producer code that sends JSON messages to a topic. ➤ How do you serialize a custom object before sending it to a Kafka topic? ➤ Describe how you can handle serialization errors in Kafka producers. ➤ Write a Kafka consumer code that reads messages from a topic and deserializes them from JSON. ➤ How do you handle deserialization errors in Kafka consumers? ➤ Explain the process of deserializing messages into custom objects. ➤ What is a consumer group in Kafka, and why is it important? ➤ Describe a scenario where multiple consumer groups are used for a single topic. ➤ How does Kafka ensure load balancing among consumers in a group? ➤ How do you send JSON data to a Kafka topic and ensure it is properly serialized? ➤ Describe the process of consuming JSON data from a Kafka topic and converting it to a usable format. ➤ Explain how you can work with CSV data in Kafka, including serialization and deserialization. ➤ Write a Kafka producer code snippet that sends CSV data to a topic. ➤ Write a Kafka consumer code snippet that reads and processes CSV data from a topic.

12

𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐧𝐟𝐨𝐫𝐜𝐞𝐦𝐞𝐧𝐭 & 𝐄𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐢𝐧 𝐃𝐞𝐥𝐭𝐚 𝐋𝐚𝐤𝐞. 👉 𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐧𝐟𝐨𝐫𝐜𝐞𝐦𝐞𝐧𝐭:  - Suitable for scenarios where the source data schema is static. - Ensures that the source data being written into the target table matches the target table definition, thus providing clean data ready for analytics. - Delta Lake uses schema validation on 𝒘𝒓𝒊𝒕𝒆. This means schema compatibility is checked during the insert operation. - If a row's schema doesn't match the target table schema, the data is rejected entirely, and an exception is raised. - The table schema is not changed unless it is intentionally modified. 👉 𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧: - Suitable for scenarios where the source data schema is dynamic and you intend to change the table's schema automatically. - Allows users to easily change a table's current schema to accommodate data that changes over time. - Activated by adding .option('mergeSchema', 'true'). - Any columns that are in the source DataFrame but not in the target table are automatically added at the end of the target table definition. 👨‍💻 Below references helps you dive deeper on these concepts.  - https://lnkd.in/gWKx2WVD - https://lnkd.in/gsH4U7_w #datalake #deltalake #schema #databricks #deltatables #dataengineering

7

❄️ I just finished reading the Snowflake paper and I found these decisions of the authors very interesting and effective, explaining its wide adoption: • 𝐒𝐞𝐩𝐚𝐫𝐚𝐭𝐞 𝐬𝐭𝐨𝐫𝐚𝐠𝐞 𝐟𝐫𝐨𝐦 𝐜𝐨𝐦𝐩𝐮𝐭𝐞: Makes it possible to scale and optimize each service independently, but still be tightly coupled. • 𝐃𝐢𝐝𝐧'𝐭 𝐫𝐞-𝐢𝐧𝐯𝐞𝐧𝐭 𝐭𝐡𝐞 𝐰𝐡𝐞𝐞𝐥: Chose the already state of the art technologies S3, EC2 and CloudHSM from AWS and used the saved time to improve other features. • 𝐒𝐢𝐦𝐩𝐥𝐢𝐜𝐢𝐭𝐲: Use only T-shirt sizes (X-Small to XX-Large) for selecting computing power no complex configs. • 𝐀𝐧𝐭𝐢𝐜𝐢𝐩𝐚𝐭𝐞 𝐮𝐬𝐞𝐫 𝐦𝐢𝐬𝐭𝐚𝐤𝐞𝐬: They can be rolled back using "Time Travel" feature. I highly recommend checking it out!

10

Balancing Executors, Cores, and Memory in Spark Jobs :   When running Spark jobs, configuring your executors can have a big impact on performance and resource efficiency. In a 10-node cluster (16 cores, 64GB RAM per node), assigning 1 executor per core gives you 160 tiny executors. Each has 1 core and 4GB of memory.   1. Tiny Executors (One per Core)⚙️ Configuration: Total Nodes:10 Cores per Node:16    Total Cores:10 nodes x 16 cores = 160 cores    Memory per Node:64GB    Total Executors: One executor per core = 160 executors    Memory per Executor: 64GB / 16 executors per node = 4GB per executor (assuming even distribution) Explanation: Each of the 160 tiny executors uses 1 core and 4GB of memory. 🔸 Pros:    🔄 Handles many small tasks quickly    🏃‍♂️ Fast task execution   🔸 Cons:     💾 Excessive memory replication     🚫 Not ideal for large-scale tasks   2. Fat Executors (One per Node) 🖥️ In this setup, we assign 1 executor per node, using all 16 cores and 64GB of memory. Configuration:    Total Nodes: 10    Cores per Node: 16    Total Executors: One executor per node = 10 executors    Cores per Executor: 16    Memory per Executor: 64GB   Explanation: Each of the 10 fat executors uses all 16 cores and 64GB of memory from a single node.     🔸 Pros:    ⚡ High parallelism     📈 Handles larger tasks   🔸 Cons:    🧠 Risk of memory overload (garbage collection issues)    🐢 Slower file access (HDFS)    💔 Inefficient resource use   3. Optimized Executors (Balanced Approach) ⚖️ The best way to balance performance is to assign 5 cores per executor and leave 1 core for system processes. In this 10-node cluster, you get 29 executors (each with 18GB of memory). HDFS Throughput : assign 3-5 CPU per executor(Optimized) Configuration:    Total Nodes: 10    Cores per Node: 16    Cores Used per Executor: 5    Total Executors:** (15 available cores per node) / 5 cores per executor = 3 executors per node    Total Executors: 10 nodes x 3 = 30 executors (subtracting 1 for system processes and application management)    Memory per Executor: 64GB / 3 = 21GB (accounting for 7% overhead, effective memory = 18GB)   Explanation: Each of the 29 optimized executors uses 5 cores and 18GB of memory.   🔸 Pros:    ⚖️ Balanced performance and resource usage    🚀 Efficient parallelism    🛠️ Avoids memory overload   🔸 Cons:    🔧 Requires careful tuning     I learned this concept in an easy way from Grow Data Skills Shashank Mishra 🇮🇳, and it’s been a game-changer! 🙌   #Spark #BigData #DataEngineering #ApacheSpark #DataOptimization #PerformanceTuning #Learning #GrowDataSkills #ShashankMishra

29

1 commentaire

🚀 Exciting News for all the AI geeks ! Microsoft has open-sourced GraphRAG, a cutting-edge tool for complex data discovery and retrieval-augmented generation (RAG)! 🌐✨ GraphRAG leverages large language models (LLMs) to create knowledge graphs, enabling more structured information retrieval and comprehensive response generation. This innovative approach significantly enhances question-answering over private or previously unseen datasets. 📊🔍 Check out more: https://lnkd.in/gedp6EXV #Microsoft #OpenSource #GraphRAG #DataScience #MachineLearning #AI #KnowledgeGraph #llm #ai #datascience #machinelearning #ml

24

I recently deployed a partial Data ML stack in the Hetzner cloud using bare-metal Kubernetes. Currently, 24 developers and ML engineers from the Basel infrastructure team are using it. To ensure we stay within a strict budget of $200, with a maximum threshold of $500, I implemented several optimizations: - For each JupyterHub user, I allocated 1 CPU and 2 GB of memory. To optimize resource usage, I disabled the terminal and provided pre-installed Jupyter templates. These templates are heavily optimized for specific jobs and do not support all pip libraries, keeping the image size small and controlling compute resources. - I'm running Apache Superset on just 2 CPUs and 4 GB of RAM. Surprisingly, it's performing well, although traffic isn't heavy. I configured middleware to reject new logins if CPU or memory usage reaches critical levels. - Auto-scaling is enabled with extremely fast node boot-up times. The cheapest nodes proved less useful, so I switched to medium-sized nodes. We average 10 nodes, with peaks up to 25 nodes. - I used Hetzner volumes for persistence and leveraged LocalStack for AWS services. This setup works amazingly well and is extremely fast. Each user has their own 5 GB of storage space. - DuckDB has a dedicated 100 GB space for testing, costing around $1.50. It's configured to use an entire node with affinity settings, allowing us to monitor traffic and CPU spikes. I may consider using a dedicated node for DuckDB in the future. - Streaming data is now reduced to 1 MB/sec, and data is retained for two days before being dropped. This helps control egress costs, and we appear to be within budget.

60

4 commentaires

Are you working on a #dbt project and wondering how to keep your database clean and organized? This article, which I had the pleasure of contributing to, might just be the help you need. It offers valuable tips and advice on how to keep your database in check and ensure its long-term stability. -> https://lnkd.in/ekS9Sa5p Thank you, Revolt BI, for the trust and opportunity to be part of this project. 🙏 #dbt #databases #datamanagement #cleandata #projectmanagement

4

I invite you to check out my latest article: 𝗦𝘁𝗮𝘁𝗲𝗹𝗲𝘀𝘀 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗘𝗧𝗟 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀: 𝗨𝘀𝗶𝗻𝗴 𝗠𝗲𝘀𝘀𝗮𝗴𝗲 𝗕𝗿𝗼𝗸𝗲𝗿𝘀 𝗳𝗼𝗿 𝗛𝗼𝗿𝗶𝘇𝗼𝗻𝘁𝗮𝗹 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 On my site: https://lnkd.in/ed96X57K On Medium: https://lnkd.in/e3EcRCNK The article demonstrates how to leverage message brokers to create stateless distributed ETL pipelines. It does this through a practical scenario: • Scenario Overview: Extracting several Salesforce tables and storing the data in Snowflake for easy access by BI tools and ML pipelines. The pipeline runs multiple times a day to keep up with frequent updates. • Pipeline Setup: The first part explains setting up three queues using message brokers—one for data extraction, one for uploading to S3, and one for merging into Snowflake. This is accomplished by treating the message broker as a job/task server. • Module Breakdown: The second part builds upon the first by detailing how each module operates independently, allowing for transformations, duplicate handling, and efficient inserts/updates. The scenario provides a clear example of how to build a robust, scalable ETL pipeline using message brokers without the need for complex clustering or dealing with data duplication issues. I am open to consulting engagements if you need help building this or any other data solution. Feel free to email me directly at [email protected] #dataengineering #snowflake #consulting #datascience #aws #kafka #rabbitmq #ETL #data

2