Casper Claesen

🚀 DE Interview 📢 Cloud Optimization ⌛ 1-Min-Read Optimizing Cloud Storage Using Spark With a Housekeeping Script 🧔🏽‍♂️Interviewer: Can you explain how you optimized cloud storage using Spark with a housekeeping script? 👨‍💻Candidate: 1️⃣ In one of our projects, we were facing challenges with rapidly increasing storage costs due to accumulating data in AWS S3. To address this, I implemented an optimization strategy using Spark along with a housekeeping script. 🧔🏽‍♂️Interviewer: That sounds interesting. Could you walk me through the steps you took? 👨‍💻Candidate: 1️⃣ First, I analyzed the storage patterns and identified that a significant portion of the data was either stale or less frequently accessed. 2️⃣ I then developed a Spark job to classify this data based on its usage and relevance, segregating it into active and inactive datasets. 3️⃣ The inactive data was compressed using efficient algorithms, and a housekeeping script was created to automate the compression process and manage the data lifecycle. 4️⃣ The script would regularly run to compress, archive, and move infrequently accessed data to a low-cost storage tier while deleting unnecessary files. Versioning in S3 was enabled to safeguard against accidental deletion. 🧔🏽‍♂️Interviewer: How did this impact your overall cloud storage management? 👨‍💻Candidate: 1️⃣ The implementation of the Spark-based solution and housekeeping script led to significant cost savings by reducing the storage footprint. 2️⃣ It ensured that only relevant data occupied primary storage layers, improving data retrieval speeds and system efficiency. 3️⃣ Additionally, the automated process minimized manual intervention, reducing errors and operational overhead. Summary: 1️⃣ Problem: Escalating cloud storage costs and inefficient data management. 2️⃣ Solution: Implemented Spark-based optimization and automated housekeeping script for data lifecycle management. 3️⃣ Impact: Reduced storage costs, enhanced data management efficiency, and streamlined cloud operations. #CloudOptimization #Spark #AWS #DataManagement #CostSavings #Automation #TechInnovation #CloudComputing #S3 #DataLifecycle

14

💥 [New Blog] "Quick Test : Fabric AI Skills As A Function" Some thoughts and experiments on how #AISkills can be used as a function in different applications. What if you could ask NL questions to generate data in #MicrosoftFabric? Or use #AISkills as an Excel macro or a function? I experimented with the idea. Read below and let me know your thoughts. https://lnkd.in/gymWtrcP

23

Hello, everyone! I have witnessed experienced Data Engineers in the community, in particular 👨🏻‍💻 Jaco van Gelder, complaining that it is pretty unfortunate that we rarely see unit tests in data pipelines -- and they are right! Although Data Engineering is not Software Engineering, best practices such as testing and documentation are applicable and should be the standard. So, if your data team is still in this phase where it knows tests are important but still don't know how to start implementing them, check this article where we use Dagster and dlt to create and manage pipelines in a way that makes unit tests easy to implement and integrate in the development workflow. https://lnkd.in/eW3Ri_4R #dataplatformengineering #dagster #dlt #testing

94

10 Comments

New post, and new website look all in one ! In this post I go over how I figured out how to use Python code in Databricks to generate a dynamic list of tables using Delta Live Tables! One notebook to ingest thousands of tables ? Sounds pretty cool to me. Check it out and let me know your thoughts ! https://lnkd.in/d-NnMZEk #databricks #dataengineering #data #automation #deltalivetables #dlt

15

1 Comment

REALLY digging the Data Transporter tool by Bram Colpaert. This thing is making moving data from 4 different #dataverse environments all the way up to production a sinch! The beauty of it is it keeps the same GUIDS across all environments with options to create/update/delete.

7

1 Comment

🚀 DE Interview 📢 Cloud Storage Optimization ⌛ 1-Min-Read Optimizing Cloud Storage Using Spark With Small Files Fix Script 👉🏽 🧔🏽‍♂️Interviewer: Can you explain how to optimize cloud storage using Spark with a small files fix script? 👨‍💻Candidate: 1️⃣ In cloud environments, handling a large number of small files can lead to inefficiencies, such as increased storage costs and degraded processing performance. This issue is common in big data processing pipelines where data is frequently ingested in small increments. 2️⃣ To address this, I developed a Spark script that consolidates these small files into larger, more manageable ones. The process starts by identifying the small files in the cloud storage, usually on platforms like AWS S3. 3️⃣ The script then reads these files and combines them using Spark's distributed processing capabilities. The merged files are written back to cloud storage, significantly reducing the number of small files. 4️⃣ This optimization not only cuts down storage costs but also improves the performance of downstream data processing jobs, as larger files are more efficient to read and write. 🧔🏽‍♂️Interviewer: Why is reducing the number of small files important? 👨‍💻Candidate: 1️⃣ Small files increase storage overhead because each file has metadata that consumes space. Over time, the accumulation of small files can lead to higher costs. 2️⃣ Additionally, when processing large datasets, small files can cause inefficiencies because distributed processing frameworks like Spark are optimized for larger file sizes. This can lead to slower job execution and higher resource consumption. 🧔🏽‍♂️Interviewer: How did you implement this in a cloud environment? 👨‍💻Candidate: 1️⃣ I implemented the solution in AWS S3, using Spark running on an EMR cluster. The script was automated to run periodically, ensuring that small files were regularly consolidated. 2️⃣ Versioning in S3 was enabled, so original files were kept intact, ensuring data safety while optimizing storage. 3️⃣ This approach led to a significant reduction in storage costs and improved the efficiency of our data pipelines. Summary: 1️⃣ Problem: Inefficiencies and high costs due to a large number of small files in cloud storage. 2️⃣ Solution: Developed a Spark script to consolidate small files, reducing their number and optimizing storage. 3️⃣ Impact: Lower storage costs and improved performance of data processing jobs. #DataEngineering #CloudOptimization #Spark #AWS #S3 #BigData #CostEfficiency #TechInnovation #CloudStorage #DataProcessing

20

2 Comments

I share many of Niels Cautaerts views on software versus data engineering. There are some big differences that many people (CTOs, etc), processes (scrum, agile, etc) and business strategists/planners mis-handle. Read the link in the comments and lmk your thoughts.

12

4 Comments

LLMs to extract insights from structured relational data such as SQL databases. #Azure #Microsoft #OpenAI #LLM #SQL

1

🚀 Boosting Efficiency in Azure Data Factory! 🚀 I came across an insightful article on how to parallelize copy activities in Azure Data Factory to improve data integration performance. It’s a game-changer for anyone working with large datasets and complex pipelines. If you’re looking to optimize your ADF workflows, this is a must-read! Parallelizing tasks can drastically reduce processing time, enabling faster data migrations and transformations. 🌐 Whether you’re migrating to the cloud or handling big data, mastering this technique will streamline your data processes. 🔗 Check it out here: https://lnkd.in/eXC5mWuX #AzureDataFactory #DataEngineering #DataIntegration #ETL #BigData #CloudComputing #DataPipeline #Azure #DataWarehouse #DataMigration #AzureSynapse #DataAutomation #DataOptimization #SQLServer #DataManagement #AzureDevOps #CloudArchitecture #DigitalTransformation #DataProcessing #BusinessIntelligence #TechInnovation #DataStrategy #ParallelProcessing #DataAnalytics #MicrosoftAzure #ETLDeveloper #CloudData

1

I really enjoyed creating this github repo on how to implement #AzureCosmosDB as VectorStore and create a RAG solution using #AzureOpenAI and #AzureCosmosDB as Vector Store. Believe it or not, you can build a prototype for building a chatbot on your data in couple of hours. Read it all and try it yourself, you'll be amazed, I promise! Watch out this space for more exciting things coming soon !! README document has step-by-step implemention details. https://lnkd.in/gawGnct7

24

no plans yet tomorrow? do you want to learn how to #automate your #database #deployment for your #SQLServer or #AzureSQL database(s) through #AzureDevOps? come and join me tomorrow #virtually! more info via the link in the post below

5

When we talk about #DataIntegration there are multiple patterns In this video, Jeroen van Disseldorp explains what patterns you can leverage. Watch the full webinar here: https://lnkd.in/eTDpRE7P #EventDrivenArchitecture #ApacheKafka #Axual

7

After some time creating a SQL server from the Azure Portal instead of using Infra as Code, I see that SQL servers are created out the box with Public accessible to disabled. This is a change from the many years of most services being created as public by default. This is probably as a result of our SFI, secure future initiative. I am involved with Databricks and Data Factory BICEP infra as code on the Azure Verified Modules. We create code that is WAF aligned, but its good to see that when you create something in the portal that its also secure by default. #azure #securefutureinititative #microsoftsqlserver

12

Interesting discussion on Reddit the other day: Is Kimball still relevant today, in the age of columnar cloud databases? In other words, is star schema still relevant now that cloud storage and processing is so cheap? Why not have everything in one big denormalized table? Of course star schema is still relevant! To directly quote a Reddit user: "Think about your data model as being a public interface. It’s important that your interface is clean and, most importantly, makes sense to the business users. You can have the best performance in the world but if the business looks at the model and it doesn’t feel natural for them to slice and dice and natural for how their domain is modeled in their head then you’re gonna be in trouble" I would even argue that modeling data is more important than ever from a business user perspective. Especially in large organizations where more and more data is being added to the dwh. You can't just throw it all on a single pile and completely disregard your business users. Because if you do, "you're gonna be in trouble". #dataengineering #datamodeling #starschema #kimball

14

🚀 Semantic Search for Dutch Documents using Elasticsearch and Local Embeddings 🇳🇱 In my latest project, I built an intelligent search engine for Dutch documents using Elasticsearch with KNN Search and embeddings generated locally using the embaas/sentence-transformers-e5-large-v2 model. This combination allows us to perform semantic search, retrieving documents based on meaning rather than just keywords. The key advantage? By generating embeddings locally using the sentence-transformers library, we have full control over the process, making it scalable and highly flexible for our use case. 🔗 Check out the full implementation in my repo: https://lnkd.in/e-skZKZ8 Key Takeaways: • Local embeddings for more control and flexibility. • Semantic Search to capture the true meaning of queries. • Scalable with Elasticsearch to handle large datasets. Would love your feedback! #vectorsearch #embeddings #elasticsearch #onPrem #Deloitte #AIReady #Data #Dutch

24

2 Comments

Large Language Models (LLMs) are popular in tech. In Belgium and the Netherlands, the podcast "Nerdland" is a favorite for tech and science fans. It covers topics like bioscience, space, robotics, and AI. With over 100 episodes, "Nerdland" is a goldmine of information. So, why not create a chatbot for "Nerdland" fans? This chatbot uses podcast content to engage and inform users. It allows the "Nerdland" community to interact with the content in new ways and makes the information accessible in many languages, thanks to LLMs' multi-language capabilities. This blog post explains the project's technical details, including the LLMs used, integration process, and deployment on Azure. https://lnkd.in/ddQyZ3KG #Azure #OpenAI

2

𝗦𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗲 𝗬𝗼𝘂𝗿 𝗗𝗮𝘁𝗮 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝘄𝗶𝘁𝗵 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗙𝗮𝗰𝘁𝗼𝗿𝘆 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀! In today's data-driven world, efficiency and flexibility are key when managing diverse data sources and transformation needs. Recently, I implemented a dynamic pipeline in Azure Data Factory (ADF) to automate data ingestion from GitHub Raw—and the results have been game-changing! 𝗪𝗵𝘆 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀? • A dynamic pipeline in ADF allows you to: • Process multiple datasets or files without creating separate pipelines. • Handle schema, file path, or format variations with reusable templates. • Reduce pipeline development and maintenance time. • Simplify orchestration with parameterized inputs. 𝗠𝘆 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲: 𝗚𝗶𝘁𝗛𝘂𝗯 𝗥𝗮𝘄 𝗗𝗮𝘁𝗮 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 𝗛𝗲𝗿𝗲'𝘀 𝗮 𝗵𝗶𝗴𝗵-𝗹𝗲𝘃𝗲𝗹 𝗼𝘃𝗲𝗿𝘃𝗶𝗲𝘄 𝗼𝗳 𝘁𝗵𝗲 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻: 1️⃣Source: Data hosted on GitHub Raw (CSV files). 2️⃣ Configuration: I created a JSON file containing metadata for all datasets (e.g., file URLs, file names, and schemas). This JSON file serves as a centralized source for the Lookup activity, enabling seamless iteration over datasets. 3️⃣ Pipeline Setup: Used the Lookup activity to read metadata from the JSON file. Leveraged ForEach activities to dynamically process each dataset based on the JSON configuration. 4️⃣ Transformations: Applied specific data transformations for each file using Mapping Data Flows. 5️⃣ Destination: Loaded the processed data into Azure Data Lake for further analytics. 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝗥𝗲𝗮𝗹𝗶𝘇𝗲𝗱 𝗖𝗼𝘀𝘁 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆: Reusable components minimized repetitive efforts. Scalability: Easily scaled to handle more datasets without additional pipelines. 𝗠𝗮𝗶𝗻𝘁𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Centralized logic (via the JSON file) made updates a breeze. 𝗙𝗹𝗲𝘅𝗶𝗯𝗶𝗹𝗶𝘁𝘆: Quick adjustments to accommodate changes in the source structure. The use of a JSON-based configuration file brought an extra layer of simplicity, allowing me to decouple pipeline logic from the dataset definitions. If you're working with diverse data sources or multiple files, dynamic pipelines can be a game-changer for your workflows. Whether you’re processing real-time logs, integrating third-party APIs, or syncing GitHub data like me, the flexibility ADF offers is invaluable. What’s your favorite use case for dynamic pipelines? Let’s connect and share ideas on unlocking the power of Azure Data Factory! #AzureDataFactory #DataEngineering #GitHub #DynamicPipelines #DataAutomation #JSON #CloudComputing #Azure

17

🚦 Meet the Control Flow Crew of Azure Data Factory: Keeping Your Pipelines in Line! Building data pipelines in Azure Data Factory? Let me introduce you to the Control Flow Activities—the quirky team of operators who keep our data moving (and make our lives a little bit easier): 🤔 If Condition: The Overthinker – This one’s always evaluating and re-evaluating. Like the friend who triple-checks Google Maps for the best route, it constantly asks, “Should I do this, or…not?” Great for when your data takes unexpected twists and turns! 🔁 For Each Loop: The Marathoner – This activity takes your data and runs laps around it until everything’s sorted! For Each is like that dedicated runner on repeat, making sure no data gets left behind. Just when you think it’s done, it’s off to process the next item. ⏰ Wait: The Procrastinator – “You know what, let’s just take a break.” The Wait activity is all about patience, adding some time before jumping into the next step. Perfect for when data needs a moment to catch up (or just nap) before continuing on its journey. 🕹️ Switch: The Decision-Maker – This one’s ready for anything! When the data arrives, it checks the values, makes a call, and sends it in the right direction. Switch has big “choose your own adventure” vibes and loves a good plot twist! 🤝 Execute Pipeline: The Networker – Calling in favors and connecting with other pipelines, it’s the social butterfly of Azure Data Factory. Need something done but don’t want to handle it yourself? Call Execute Pipeline—it’s got connections. 📜 Until Activity: The Persistent One – This activity doesn’t give up! It keeps looping and looping until a specific condition is met, kind of like that friend who won’t stop texting until you finally agree to their plans. The next time you’re orchestrating data in ADF, remember—you’ve got a team of personalities ready to handle whatever your data throws at them! 😄 So let the Control Flow Crew do the heavy lifting while you sit back and watch those pipelines flow! For more such content follow #Shilpa Das #AzureDataFactory #DataEngineering #ControlFlow #DataPipelines #Azure #BigData

10