Alex Vesa’s Post

Programming differs from Software Engineering and especially Data Science, but the question is what connects them and what should you strive to be? Read my take on it: https://lnkd.in/dWdpNY9G

I've always been curious about the behind-the-scenes activities of network traffic on my machine, including protocols, IPs, and active ports. To satisfy this curiosity, I decided to build a simple data pipeline after spending a few months exploring data engineering field. 𝗧𝗵𝗶𝘀 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗶𝘀 𝗱𝗲𝘀𝗶𝗴𝗻𝗲𝗱 𝘁𝗼 𝗰𝗮𝗽𝘁𝘂𝗿𝗲, 𝘀𝘁𝗿𝗲𝗮𝗺, 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺, 𝘀𝘁𝗼𝗿𝗲, 𝗮𝗻𝗱 𝘃𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗲 𝗻𝗲𝘁𝘄𝗼𝗿𝗸 𝘁𝗿𝗮𝗳𝗳𝗶𝗰 𝗱𝗮𝘁𝗮. Using technologies like Python, Kafka, PostgreSQL, and Power BI, my goal is to create an intuitive solution for analyzing and monitoring network traffic on a host machine. Find more info here: https://lnkd.in/dQRCYWfJ Check out the project source code here: https://lnkd.in/d8WsQvvP

MLOps question: I'm a dbt noob. Is anyone out there using dbt for data science? Would you be willing to show me your workflow? I'm thinking through whether it's worth adding another tool to a DS' life, when today they're calling SQL queries in scripts on their own. Here's my soup of questions lol - would it help DS communicate which project uses which data? - would it help an entire team/org be able to track the lineage of queries/tables? - can it be used as a (poor man's) data catalog? - do you run it in a DAG tool? What does that look like for you? - do/should you pay for a vendor? Or is it fine to use with an existing DAG tool? - are you using it for "data tests" of any kind? - is it worth integrating with GreatExpectations? What would that workflow look like? - is there substantial benefit over calling all the sql queries yourself with a python client? - would you ever NOT want to use dbt?

A Real World ML Problem... 𝗧𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 🤔 An ML model can only generate business value once you plug it to a *live data sources*. There is no static CSV file, but constantly flowing data, that needs to be processed and fed into your ML model. And this is precisely what a *feature pipeline* does. 𝗧𝗵𝗲 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 🧪 Is a program that 1️⃣ 𝗳𝗲𝘁𝗰𝗵𝗲𝘀 𝗱𝗮𝘁𝗮 from a data warehouse, Kafka topic or websocket, among others. 2️⃣ 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝘀 this raw data into features 3️⃣ 𝘀𝗲𝗻𝗱𝘀 these feature data to storage (e.g. Feature Store) so the rest of the system can use it. Depending on the frequency at which the feature pipeline runs, we can distinguish between 2 types. - Batch feature pipeline - Streaming feature pipeline ➡️ 𝗕𝗮𝘁𝗰𝗵 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 🕒 A batch feature pipeline is a program, often written in Python or Spark that fetches data and generates features on a schedule, for example: - daily - hourly - every 10 minutes To implement a batch feature pipeline you need - 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴, for example, a GitHub VM or an AWS Lambda function. - 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻, to schedule and trigger the execution of the pipeline. Popular options are Apache Airflow and Prefect. ➡️ 𝗦𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 ⚡ A streaming pipeline is a program that is 𝗰𝗼𝗻𝘀𝘁𝗮𝗻𝘁𝗹𝘆 ingesting data (e.g. from → an external web socket, or → a message bus like Kafka processing it, and serving it downstream, either to → a message bus (e.g. Apache Kafka), or → a Feature Store. Stream-processing can be implemented either with → Apache Spark Streaming (JVM) → Apache Flink (JVM) → Bytewax (Python on top of Rust) → Pathway (Python on top of Rust) → Quix (pure Python)

This blog explores, examines, and addresses a number of challenges faced by data engineers in the industry, especially in the Gas & Energy sector: 👉🏼 How to avoid building bespoke solutions for industry-specific and myriad data sources for collecting data for ingestion and examination. 👉🏼 How to extend PySpark Data Source API for custom Data sources (native and non-native) so you can use your de facto and scalable Apache Spark™ data processing engine--and that in Python--as your data source and sink. 👉🏼 How to extend the new Python Data Source API for your data sources, sinks, batch, and streaming. 📓Have read! Check out the docs links below, have a go at it, and let us know. cc: 💁♀️ Allison Wang & 💁♂️ Craig L. 🔗 https://lnkd.in/gtX_pZRc 🔗 https://lnkd.in/ghjRk-Tn https://lnkd.in/g5CbWcyE

ML is powerless without big data ML Experiment 1: (Big data) I was building this project using Kafka to ingest data from multiple APIs, send real-time data to the Kafka cluster, create a streaming pipeline to process the data, and store it in a vectorDB. Three Takeaways: 1. Use Kafka for any real-time application Used Upstash Kafka cluster for handling streams of real-time events and messages and it easily maintains the data coming from different API threads. 2. Pydantic models for any data validation Not only for this project, But have seen Pydantic used in every open-source project to handle errors, exception handling, and making the system consistent. 3. Use bytewax for creating a streaming pipeline Recently, Came across with bytewax, a cool streaming pipeline framework in Python used for creating data transformation steps. Find the code Repo: https://buff.ly/3Rf7Czc

RDD (Resilient Distributed Dataset) Low-level API. Requires detailed coding with functions like map, flatMap, filter, reduceByKey. Not very developer-friendly and lacks optimizations. DataFrame Introduced in Spark 1.3. A higher-level API that simplifies coding. Challenges: Does not offer strong typing, so errors appear at runtime instead of compile-time. Limited flexibility (e.g., restricted use of anonymous/lambda functions). Converting DataFrames to RDDs is possible when flexibility or safety is needed. Conversion from DataFrame to RDD can be costly, and some optimizations are lost. Dataset Introduced in Spark 1.6. Provides compile-time safety and more flexibility with lower-level coding. Conversion between DataFrame and Dataset is efficient and maintains optimizations. Let me know if you need any specific details or further explanation on any of these points!

I'm developing a Synthetic Data Generation Library designed to support diverse data and ML use cases, including e-commerce, healthcare, patient engagement, lead generation, and customer feedback. During development, I focused on scaling and optimizing to minimize resource usage. Even with Kubernetes as a deployment endpoint, I aim to control resource allocation tightly by fixing resources and pod numbers. - For data generation, I'm using the Faker library in Python, which is fast, simple, and supported by a robust community. - For data writing, I initially relied on Pandas for simplicity but am transitioning to Polars due to its superior performance. - To handle streaming use cases, I’m adding support for DuckDB. - Parquet is the default data format, but I'm also supporting Iceberg. Currently, I use Spark on a single node for Iceberg writes and am experimenting with Getdaft to write Iceberg tables. - For concurrency, I tested various approaches, including Redis queues and Python threading, but ultimately chose Celery as the executor, which moderates load effectively. - I switched from Redis to Dragonfly for caching, and all API compatibility checks have been successful, though I’m still evaluating performance. To ensure efficiency, cached data is reused instead of being regenerated, and I’m incorporating randomization across request entities in the data generation process. I'm also using GenAI with strict Pydantic models to maintain smooth execution within the library.

Good takeaways from our session with the one and only David Jayatillake about Semantic Layers. Thanks Data Action Mentor for having us!

🚨 Working on a large codebase without any tests can be nerve-wracking. One wrong line of code or a seemingly harmless library update can bring down your entire production pipeline! Data pipelines often start simple, so engineers might skip tests initially. But as complexity increases, the lack of tests can severely slow down your feature delivery speed. It’s especially daunting to introduce testing in a large legacy codebase with little to no existing tests. And in long-running data pipelines, bad code can take hours to identify and fix, leading to frustrated stakeholders! 💡 Imagine confidently pushing changes to production without the fear of breaking your pipeline. Picture a world where you deliver features quickly, keep your stakeholders happy, and empower your team to move fast. By incorporating tests into your pipeline, you can dramatically reduce stress, avoid bugs slipping into production, and maintain a smooth, efficient workflow. ✅ Testing might not catch every possible issue, but it can prevent a significant number of production problems. In this post, we’ll explore the different types of tests and how to effectively test PySpark data pipelines with `pytest`. We’ll walk through creating unit, integration, and end-to-end tests for a simple data pipeline, covering key concepts like fixtures and mocking. By the end, you’ll know how to identify and test critical parts of your data pipeline, giving you the peace of mind to push changes with confidence. #dataengineering #testing #pyspark #pytest #unittesting #techtips #devops https://lnkd.in/gCgZCsi9