Andrés Santos

As I near the end of my internship, I look back on a decision I made 6 months ago, where I had to choose between my top 2 preferred companies to do my internship. The first company was one I had always dreamed of working for, and I applied for a position in their data department. However, during the interview, despite my questions, they didn't clarify the job scope well. It turned out the role was mostly administrative—organizing meetings and taking minutes—though they assured me I would still be part of the data department. (To avoid any misinterpretation, my point is that the job role that this company offered me was out of my field of study) On the other hand, PayNet (Payments Network Malaysia) where I am currently interning, provided a different experience. The interview was focused and technical, with clear explanations about the tasks related to data analytics and engineering. They even mentioned specific projects I could work on. My message to recruiters is clear: clarity in explaining job roles, especially to interns, is crucial. Internships are about learning and gaining relevant experience in our fields of study. While my time at PayNet has been challenging, it has been incredibly rewarding to tackle tasks within my field. It's better to face challenges that align with our studies than to struggle with tasks unrelated to our academic and career goals. At least we will always have something to share in our weekly report 😄 P.S. Just a random picture of me working inside a phone booth because it's too cold outside in the office hahaha 😂

233

8 Comments

A new certification achieved: Build Neural Networks in Python from Scratch on https://lnkd.in/eJKs_ARb And a big shout out to this course by Loek van den Ouweland.  The course says 3 hours, but if you really want to understand all the examples, explanation and play around with all the Python code he shows in the course, it takes at least a couple of days to really understand what's going on.  But as a start in a journey into Machine Learning and AI, this is an must-have certificate.

5

I recently had the pleasure of presenting element61's 𝐝𝐚𝐭𝐚 𝐯𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡 together with my colleague Marie Vandecavey at both our Microsoft Analytics & AI Day and the Dataminds Saturday event. If you were not able to attend one of these sessions, here is a short recap! At element61, we believe data visualization is the key to unlocking actionable insights and driving data-driven decisions. Our framework goes beyond creating charts – it's about crafting a compelling data storytelling experience. Here's a glimpse into what sets us apart: - 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 𝐭𝐡𝐞 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Through stakeholder interviews and mockups, we explore end-user needs before starting development. - 𝐀𝐬𝐬𝐞𝐦𝐛𝐥𝐢𝐧𝐠 𝐚 𝐒𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐑𝐞𝐩𝐨𝐫𝐭: Leveraging our storytelling expertise and User Experience (UX) best-practices, we aim to create a BI tool that goes beyond standard reporting. - 𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐀𝐜𝐭𝐢𝐨𝐧𝐚𝐛𝐥𝐞 𝐏𝐚𝐠𝐞𝐬: Adopting a minimalistic approach, we craft pages that are focused on takeaways that solve business problems. - 𝐂𝐫𝐞𝐚𝐭𝐢𝐧𝐠 𝐌𝐞𝐚𝐧𝐢𝐧𝐠𝐟𝐮𝐥 𝐕𝐢𝐬𝐮𝐚𝐥𝐬: We focus on building visuals that are appropriate, simple and effective. 👉 𝑊𝑎𝑛𝑡 𝑡𝑜 𝑙𝑒𝑎𝑟𝑛 𝑚𝑜𝑟𝑒 about how element61 can elevate your data visualization strategy? Check out our detailed approach on our website: https://lnkd.in/ecJHvcyZ 👉 𝑊𝑎𝑛𝑡 𝑡𝑜 𝑔𝑒𝑡 𝑠𝑡𝑎𝑟𝑡𝑒𝑑 𝑎𝑙𝑟𝑒𝑎𝑑𝑦? Dive deeper with our cheat sheets! We've created 𝟒 𝐜𝐡𝐞𝐚𝐭 𝐬𝐡𝐞𝐞𝐭𝐬 packed with practical tips to guide you through each of our 4-step data visualization process. 𝐃𝐨𝐰𝐧𝐥𝐨𝐚𝐝 𝐭𝐡𝐞𝐦 𝐧𝐨𝐰 and start creating impactful visualizations today: https://lnkd.in/enurw2XH

47

Check out my new post on Gradient Boosting Machines (GBMs) on Medium! #machinelearning #ensemblelearning #gradientboosting #datavisualization #datascience #artificialintelligence

9

My colleague Jeroen Overschie made me finally understand how you can feed relevant documents into a large language model to enhance its context and provide better information and how important a good, relevant search engine still is for this method. In short: if your search engine and the results it turns up are better, the quality of answers from the LLM will be better.

9

A nice opportunity to glimpse the “behind-the-curtains” of language models. In this lecture, Yann Dubois provided a comprehensive overview of how models like ChatGPT are created, covering both pre-training (language modelling) and post-training. This lecture is especially valuable for understanding everything (beyond the language model itself) that makes up these complex systems: from data collection to algorithms and evaluation methods. #MachineLearning #AI #LLMs #StanfordCS229 #genAI

5

1 Comment

Join our workshop on all you need to know about Machine Learning with Databricks in one day! 📍 𝐋𝐨𝐜𝐚𝐭𝐢𝐨𝐧: Amsterdam NDSM 📅 𝐃𝐚𝐭𝐞: May 15th 🕑 𝐓𝐢𝐦𝐞: 10am-5pm 🎟 𝐓𝐢𝐜𝐤𝐞𝐭𝐬: https://lnkd.in/eYBvrXPT #MachineLearning #Databricks #RevoData #Workshop

16

How amazing a time it is to be able to call yourself things like "AWS GenAI Data Engineer", right? I, for one, definitely prefer it over those old boring titles. 😛 Anyway, I started my career as an AWS GenAI Data Engineer at Emumba recently and I must say that I've got to learn a lot! Before diving into that, let me tell you what my job requires from me: AWS. Emumba, being an AWS Partner, works with AWS services a lot and collaborates with the AWS representatives to provide GenAI solutions and services to other AWS partners. Being a part of the AWS vertical at Emumba, my job is to know all-things-data on AWS so I can help design data-intensive applications on AWS more effectively. GenAI. Since the rise of GenAI, Emumba has been at the forefront, dedicating countless resources to the research and development of generative AI applications. We don't just play with prompts. We automate business units and I play a part in that! (Trade secret: Agentic GenAI 👀). Data Engineering. GenAI is extremely powerful but where does that power originate from? Yup, you guessed it- data! Data has almost become a mafia in the modern world where everyone wants a piece of it (preferably a larger piece). And that's for good reason. Indeed, data is what drives modern-day applications, providing a top-notch personalized browsing experience. It's the same for GenAI. GenAI works on data. The higher the quality of data, the more useful GenAI will be for your organization. TLDR; My role is to gather data from an entire organization's data sources and perform some magic tricks to make that data useful for modern-day GenAI-powered applications. Of course, I'm a long way from being an expert at this but I've started my journey, and Emumba is definitely helping me accelerate that process. Within two months, I've gone through books, videos and projects and I still feel like I learn something new every day. Let this post officially mark the start of my career at Emumba and the completion of an important foundational step in my new career- The Data Foundation for Generative AI learning path on Amazon Web Services (AWS)'s skillbuilder platform!

72

8 Comments

Buongiorno, bonjour, guten morgen and good morning! ⏰🌐 We are putting together a team that speaks many languages for the upcoming #PHUSE EU Connect event in Strasbourg. Not just our beloved native idioms, but also the programming languages that make us more productive when analyzing data and developing models. We are preparing for a 90' hands-on workshop that focuses on the integration of APIs in a Validated Environment. After a brief introduction on the platform and on the technology, you will be guided on how to best use APIs in the language you prefer: SAS, Python or R. I can roll my Rs, but I actually can't code in R 😅 David Weik, this one is for you! See here for more info: https://lnkd.in/dY7ZgFtQ.

65

2 Comments

Excited to share my recent experience with the interview process for a Data Engineer position at Capgemini! It was a great opportunity to delve into SQL concepts. Here’s a list of SQL questions I was asked during the interview: 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐣𝐨𝐢𝐧𝐬? Joins are used to combine rows from two or more tables based on a related column between them. They allow us to enrich our data by connecting different datasets. 𝐄𝐱𝐩𝐥𝐚𝐢𝐧 𝐭𝐡𝐞 𝐰𝐨𝐫𝐤𝐢𝐧𝐠 𝐨𝐟 𝐋𝐄𝐅𝐓 𝐉𝐎𝐈𝐍. A LEFT JOIN returns all the records from the left table and the matched records from the right table. If there’s no match, it returns NULL for columns from the right table. This is super useful when you want to keep all records from one table while including related data from another. 𝐄𝐱𝐩𝐥𝐚𝐢𝐧 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐈𝐍𝐍𝐄𝐑 𝐉𝐎𝐈𝐍 𝐚𝐧𝐝 𝐅𝐔𝐋𝐋 𝐉𝐎𝐈𝐍. An INNER JOIN returns only the rows that have matching values in both tables, effectively filtering out any records without a match. In contrast, a FULL JOIN returns all records from both tables, with NULLs in place where there is no match. This is great for getting a complete picture, even if some data is missing. 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐰𝐢𝐧𝐝𝐨𝐰 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐬? Window functions allow you to perform calculations across a set of table rows that are related to the current row. They are incredibly useful for analytics without collapsing the result set. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐰𝐢𝐧𝐝𝐨𝐰 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐬 𝐚𝐧𝐝 𝐭𝐡𝐞 𝐆𝐑𝐎𝐔𝐏 𝐁𝐘 𝐜𝐥𝐚𝐮𝐬𝐞? While both can aggregate data, the key difference lies in their behavior. GROUP BY reduces the number of rows returned, while window functions maintain the original row count, allowing for detailed analysis alongside aggregated results. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐑𝐀𝐍𝐊 𝐚𝐧𝐝 𝐃𝐄𝐍𝐒𝐄_𝐑𝐀𝐍𝐊? Both functions assign ranks to rows within a partition. The key difference is that RANK skips ranks after ties, while DENSE_RANK does not. For example, if two rows are tied for rank 1, RANK will assign the next rank as 3, whereas DENSE_RANK will assign 2. 𝐖𝐫𝐢𝐭𝐞 𝐚 𝐪𝐮𝐞𝐫𝐲 𝐭𝐨 𝐫𝐚𝐧𝐤 𝐞𝐦𝐩𝐥𝐨𝐲𝐞𝐞𝐬 𝐛𝐲 𝐭𝐡𝐞𝐢𝐫 𝐬𝐚𝐥𝐚𝐫𝐲 𝐰𝐢𝐭𝐡𝐢𝐧 𝐞𝐚𝐜𝐡 𝐝𝐞𝐩𝐚𝐫𝐭𝐦𝐞𝐧𝐭. SELECT employee_id, department_id, salary, RANK() OVER (PARTITION BY department_id ORDER BY salary DESC) AS salary_rank FROM employees; Well, this was my experience! Hope this helps in your learning journey. There's still so much more for me to learn too. Do follow for more useful information! #dataEngineer #data #SQL #Learning #interview

300

4 Comments

In today’s fast-paced development environment, maintaining an efficient and error-free workflow is crucial. This highly informative article published on Tilburg Science Hub brilliantly lays out three Git branching strategies that are game-changers for any team, especially data engineers like us. 1. Trunk-Based Development (TBD) - In this approach, all developers work directly on the main branch, or the "trunk," ensuring that it always holds deployable code. This strategy encourages frequent commits and the use of feature toggles to handle incomplete features. While branches are minimized, short-lived feature branches are used for tasks that require more time, maintaining a focus on continuous integration and delivery. 2. Feature Branching - This strategy involves creating separate branches for each feature, which keeps the main branch clean and reduces the risk of conflicts. It's perfect for continuous integration and makes peer reviews straightforward. 3. Git Flow - A robust model that uses two main branches with infinite lifetime: `master` and `develop`. This method beautifully supports planned releases and accommodates hotfixes with ease. These strategies can significantly improve your project’s organization and help manage contributions more effectively. I highly recommend checking out this article to any data engineer looking to enhance their Git practices: https://lnkd.in/dCrGkM5m #DataEngineering #VersionControl #Git

8

🚀 Day 71/100 - Data Engineering Journey While delving into the realm of Apache Kafka, one of the foundational concepts we encounter right from the start is that of topics. Topics serve as the central organizing principle within Kafka, acting as the channels through which data streams flow. However, what truly makes topics powerful is their underlying partitioning mechanism, which dictates how data is distributed and processed within Kafka clusters. Kafka partitions topics for parallelized data processing, facilitating scalability and fault tolerance. Each partition maintains an ordered, immutable sequence of records, ensuring strict order based on offsets. This partitioning underpins Kafka's distributed architecture, enabling clusters to manage large data volumes and high-throughput streams. Key Partitioning Strategies: 1. Key-Based: Records are assigned to partitions based on predetermined key values, ensuring consistent routing to maintain sequential processing of related data. 2. Round-Robin: Records are evenly distributed across partitions in a cyclic manner, promoting uniform data distribution and load balancing. 3. Custom Partitioners: Developers can implement custom logic for partition assignment, offering flexibility to tailor strategies to specific use cases or requirements beyond standard key-based or round-robin methods. Optimizing Data Distribution 1. Scalability and Throughput: Effective partitioning in Kafka is vital for maximizing scalability and throughput by evenly distributing data and enabling parallel processing. 2. Fault Tolerance and Reliability: Kafka's fault tolerance relies on partition replication across brokers, ensuring resilience against failures and data loss for continuous availability. 3. Data Locality and Performance: Well-designed partitioning enhances data locality, minimizing network overhead and latency while optimizing performance by co-locating related data. Kafka's partitioning strategies are foundational to its distributed architecture, enabling efficient data distribution, fault tolerance, and performance optimization. Mastering these techniques unlocks Kafka's full potential for real-time data processing and analytics, driving innovation in the digital age. [ An engaging read: https://lnkd.in/gYDPACE5 ] #Kafka #TopicPartitioning #DataDistribution #Scalability #Day71 #100DaysOfDataEngineering

11

𝗦𝗹𝗼𝘄𝗹𝘆 𝗖𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝗗𝗶𝗺𝗲𝗻𝘀𝗶𝗼𝗻𝘀 (𝗦𝗖𝗗) This is a quite popular concept in the world of Data and Data Engineering. It's about changes to dimension tables in your data warehouse. For example, you might have a customer dimension table with a favourite food attribute. How do we handle the situation when a customers favourite food changes? I explain the 4 main strategies commonly used in the article bellow :)

10

2 Comments

When invited to the Super Data Science Podcast, Ritchie Vink, author of Polars, mentioned an upcoming Polars book set to launch in January 2025: "Python Polars: The Definitive Guide" by Jeroen Janssens and Thijs Nieuwdorp. The official Github repository already has a lot of code examples that worth checking out: https://lnkd.in/dJGbwQnM The Early Release version can also be read on O'Reilly Learning platform: https://lnkd.in/daTEf8Tv The book is offering guidance for you to:  - Process larger-than-memory datasets at record speed - Apply the eager, lazy, and streaming APIs of Polars and decide when to use which - Transition smoothly from Pandas or Spark to Polars - Integrate Polars into your existing codebase - Work with Arrow and Parquet to efficiently read and write data - Translate complex ETL tasks into efficient and elegant queries I started reading the Early Release version and I am enjoying it so far. I am looking forward to the final version of the book. #Python #DataFrameLibrary #Polars

40

2 Comments

I’ve completed the Understanding Cloud Computing Course on DataCamp! throughout the this course I have gained knowledge on: * Cloud service models (IaaS, PaaS, SaaS) * Core cloud platforms (AWS, Azure, Google Cloud) *AWS Professional Cloud Services( AWS simple storage Service( S3), AWS Elastic compute Cloud(EC2),AWS Relational Database(RDS), * AWS Professional Data Services( Redshift ,Kinesis,Sagemaker) * Azure cloud Services( Azure Blob Services, Azure Virtual Machine, Azure SQL Database) * Azure Data Services( Data Lake Storage, Stream Analytics, Machine Learning) Excited To continue my learning towards cloud concepts and real time hands-on Experience. #Cloud #Cloudcomputing #AWS #Azure #Googlecloud #CloudSolutions

13

1 Comment

🚀 Just Launched: My Interactive Euro 2024 Scoring Insights Dashboard! 🚀 I'm thrilled to share my latest project—a Power BI dashboard that dives into scoring patterns and expected goals (xG) for Euro 2024! ⚽ This dashboard provides powerful insights for fans, analysts, and anyone curious about the dynamics of goal-scoring in this tournament. Here's a quick look at what it includes: 🌍 Shot Map – A custom Python visual to show the location and outcome of each goal, helping you spot scoring trends across the tournament. 🎯 Slicers for Teams & Players – Easily filter data to analyze specific team and player performance. 📊 Key Metrics with DAX: ● Total Average xG – Snapshot of average expected goals. ● Average xG per Match – Dive into expected goals per game. ● Goals Scored – Total goals scored so far. ● Total Matches Played & Average Goals per Match – Overview of team performance across all matches. Creating this dashboard was an incredible opportunity to combine Python and DAX in Power BI, resulting in a dynamic, data-driven experience. Seeing the expected goals (xG) metrics in action really highlights how data can add a strategic layer to understanding player performance and game tactics. 👉 Check out the video below to see the dashboard in action! I’d love to hear your thoughts and any feedback from fellow data enthusiasts. Let’s dive into Euro 2024 analytics together! #DataAnalytics #PowerBI #DataScience #Euro2024 #SportsAnalytics #FootballAnalysis #ExpectedGoals #DAX #Python #FootballData #InteractiveDashboard #statsbomb #UEFA #FIFA #Understat

27

7 Comments

Over the past years I worked on so many things. ➡️ I have some experience in data analytics where I primarily used Power BI and SQL for generating insights. ➡️ Then I have some experience in data engineering with azure services as the primary tech stack for building and managing data infrastructure. ➡️ Then I also have some experience in backend development with flask as the primary skillsets. And the best part is that all these things are not something different but interrelated. Its great to see how the data is getting ingested and processed from multiple data sources. Then how it is getting used for generating insights by analysts. Then how the same data is supporting the data needs of multiple applications and websites. These types of exposure definitely give a holistic view as before getting the exposure of all these, I used to think that the primary use of data is just for analytics purposes only. #dataanalytics #dataengineering #data

176

12 Comments

Ai, Ai, Ai,... Exciting news! 🎉 I’m happy to announce that I’ll be presenting at the 𝐀𝐈 𝐈𝐦𝐩𝐚𝐜𝐭 𝐒𝐮𝐦𝐦𝐢𝐭 by NCOI Learning! We all know everyone wants AI, but very few are ready to tackle the dirty work behind it. 🧹 Let’s face it: AI doesn’t thrive on billions of scattered data points; it relies on 𝐜𝐥𝐞𝐚𝐧, 𝐜𝐞𝐧𝐭𝐫𝐚𝐥𝐢𝐳𝐞𝐝, 𝐚𝐧𝐝 𝐚𝐠𝐠𝐫𝐞𝐠𝐚𝐭𝐞𝐝 𝐝𝐚𝐭𝐚. Without proper preparation, consolidation, cleaning, and organization, you’re building your AI on quicksand. During my session, I’ll introduce our 𝐃𝐚𝐭𝐚 𝐌𝐚𝐭𝐮𝐫𝐢𝐭𝐲 𝐌𝐨𝐝𝐞𝐥, a framework to assess how close (or far) your organization is from truly leveraging AI for predictive and prescriptive insights. We'll also dive into the crucial role of 𝐝𝐚𝐭𝐚 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 in ensuring data is clean and structured for AI teams, as well as highlight strategies for improving data quality and ensuring reliable analyses. Looking forward to seeing you there! 👋 --> Follow my company page, if you haven't already: Data Simplified --> Find more about the event on: https://lnkd.in/eQwBwd-Y #AIImpactSummit2025 #DataSimplified #DataMaturity #AI #BusinessIntelligence #DataQuality #DataEngineering #AISummit

63

1 Comment