Piotr Czarnas

🔹Make the PySpark fly 🔹Magdalena Nowak Writing efficient Spark applications with Python can be surprisingly challenging. With PySpark library evolving fast many of the best practices aren't well-known, especially among Data Scientists. They tend to stick to the methods and workflows they know from more known libraries like Pandas, treating Spark code similarly - which often ends up being inefficient. Moreover, with most tutorials focusing on the basic features of the PySpark library, there is little knowledge about new concepts that make our lives easier and our data pipelines faster. This talk will help you progress beyond .groupBys and .counts. Watch the entire speech ➡️ https://lnkd.in/dJ_6UsxC

6

🎄📈 Elevate Your PySpark Testing Game 🐍💡 Discover the power of dynamically loading unit tests in your PySpark notebooks. This allows you to activate them on demand, so you can run them during development or deployment but disable them daily. Our latest tip breaks down the process step by step, complete with code samples and practical tips. Level up your testing skills and streamline your Databricks workflow today! 💪 🔗 https://lnkd.in/eKj7pu4r #PySpark #Databricks #SoftwareTesting #CodeQuality

2

Gülçin Yıldırım Jelínek: Maintaining Postgres for Modern Workloads 🚀 A Geeky Rendezvous with PostgreSQL: Modern Workloads Edition 🚀 Hey PostgreSQL aficionados! 🐘 Ever wonder what makes software not just functional, but outright maintainable? I had the pleasure of diving into this very topic with the inimitable Gülçin Yıldırım Jelínek on the Maintainable Podcast, hosted by Robby Russell. Imagine juggling modern workloads with the grace of a seasoned circus performer using Postgres as your trusty, reliable, and slightly quirky assistant. Spoiler alert: it involves more than just a well-timed Ctrl+C. We delved into the mysterious art of keeping our beloved databases in prime working condition, even when they grumble under the weight of today's demands. Think of it as giving your beloved vintage car a tune-up before a road trip, except this car handles all your data and sometimes cries for a VACUUM. For those who missed it, the podcast features secrets and snippets you won't find in any manual

1

Data optimization is calling you! 📣 Tomasz Kostyrka will explore everything from B-Tree indexes for relational databases to cutting-edge techniques like Z-Order, Liquid Clustering, and Microsoft's V-Order for speeding up queries in Direct Lake mode. 🎯 What will you learn: 👉 Key differences between partial and linear ordering in table sorting 👉 How Morton and Hilbert's curves are used for optimizing "Data Skipping" 👉 The inner workings of Parquet files, Predicate Pushdown, and how Z-Order and V-Order complement each other And as a bonus, have you ever wondered how many dwarves can fit in a hotel with infinite rooms? Tomek will answer that, too! 😉 🗓 When: 17th October, 6:00 PM GMT+2 📍 Cracow ⛓️‍💥https://lnkd.in/dQqjiPEP #getindata #data #microsoftazure

22

2 komentarze

Being a data engineer is both an art and a science—a blend of creativity, problem-solving, and technical wizardry. Having spent many years in the field, Andrey Torchilo knows firsthand what a typical day in the life of a data engineer looks like, and he shares his experience in his latest article. Read it here: https://hubs.ly/Q02Qn-n70

12

How can Snowflake power your data projects end-to-end? Watch our latest session with Data Engineer Grzegorz (Greg) Rybak to see how Snowflake supports data storage, processing, notebook-based analysis, and even in-platform ML. Get a glimpse of how Snowflake can be the backbone of your data strategy: https://lnkd.in/dJQAxv58 #Snowflake #DataEngineering #AI #ML #DataOps #TechInsights

8

1 komentarz

If you're running Apache Spark, you're wasting resources at the application level. And that means you’re wasting money 😫 Read this blog to learn how Pepperdata eliminates that waste on top of all the infrastructure-level optimizations that you may have already implemented 💰 https://lnkd.in/g7qT_EHU #capacityoptimization | #amazonemr | #amazoneks | #kubernetes | #k8s | #finops | #apachespark | AWS Partners

13

Hey there! Join me for my session at Data Science Summit 2024 in Warsaw on November 22, where we’ll dive into topics in the areas of #data #engineering and #dataops. We’ll start with a quick recap on Delta Lake open table format, take a look under the hood at how the Databricks shallow clone mechanism works, and, most importantly, see how using it smartly can simplify our dev workflows and make safe #continuous #testing on big datasets possible. PS If you don’t have your ticket yet, here’s a 20% discount code!

34

Do you want to get coding in Scala, but don't know where to start? 🤷‍♂️ Well, the Scala Matters library is a great place to start 📖 You can find everything from the OG red Scala book, to Gabriel Volpe's Functional Event Driven Architecture, Michał Płachta's Grokking Functional Programming and so much more! 🚀 All the books are simply linked to the author's chosen point of sale. We don't take a cut, we just want to make great Scala content more accessible 🫡 👉 Here is a link to the full collection: https://lnkd.in/dMrtgknD Are there any books missing from our library that you think should be added?

22

4 komentarze

Are you trying to navigate the complex landscape of data management? This detailed comparison between Data Fabric and Data Mesh explores their architectures, applications, and benefits. Discover how these frameworks address centralized vs. decentralized data platforms, and learn which approach might be best for your organization! https://lnkd.in/gUuhvpfn #datafabric #datamesh #datamanagement #bigdata #cloudcomputing #AI #dataarchitecture #analytics #dataintegration #digitaltransformation

3

Can You Trust Those Charts? Răzvan Ipate, our data analyst, not only unveils lies behind the reports but also spots them and even prevents them as data experts. Learn how to be a more critical viewer of data visualizations in this new episode of DevCast. 👇👇👇 Watch the full episode https://lnkd.in/d_nnuUqG #leadingintech #datavisualization #levininers #dataanalytics

12

Interested in sharpening your #NoSQL data modeling skills? Join Guy Shtub, Head of Training, for a free learning session that covers topics like: - Choosing a Primary Key - Collections, Counters, UDT and TTL - Common pitfalls and how to diagnose Save your spot here: https://ow.ly/xPLL50R4S61 #ScyllaDB #techtips

17

Time is money, right? You can now save more time by analyzing your financial data from Snowflake, MySQL, or SQLite directly on the Terminal Pro. This means you can combine the power of our functionalities with the value of your data. What native integrations would you like to see next on Terminal Pro? 🤔 You can try our Terminal Pro in a 3-week free trial 🦋 Check out the comments to sign up.

33

1 komentarz

Now there is nothing around PostgreSQL that we cannot do, including complicated migrations from Oracle to PostgreSQL. Stormatics and Cloudway help you to migrate to PostgreSQL and be free forever! Cloudway S.A. PostgreSQL Global Development Group #postgreSQL

15

🚀 Transform Your ML Workflows with MLOps! Our Data Science Leader, Jakub Dubowik, and Senior Data Science Engineer, Radek Suwała, are ready to share their expertise in leveraging MLOps to solve real-world business challenges. 📈 Join our free webinar on December 11 to learn how to streamline your machine learning workflows and implement cloud-agnostic solutions across #GCP, #Azure, and #AWS. We’ll showcase practical strategies, best practices, and real-world case studies to help you overcome ML deployment challenges. 📅 Don’t miss out! Register here and take your ML workflows to the next level! 👉 https://buff.ly/3Zl4fK7 #Webinar #MLOps #MachineLearning #AI #Workflow

1

Learn about the intricacies of Adobe's migration from a NoSQL database to #DeltaLake, the custom solutions the Adobe Real-Time Customer Data Platform (CDP) team implemented, and the surprising performance benefits they gained along the way. Read here 👉 https://lnkd.in/emhpGzg8 cc Yeshwanth Vijayakumar, Director of Engineering at Adobe #opensource #oss #lfaidata #adobe

165

3 komentarze

🧊 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻 𝗶𝗻 𝗔𝗽𝗮𝗰𝗵𝗲 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 If you’re working with large-scale data ingestion, especially in a lakehouse format like Apache Iceberg, you've probably heard about compaction. Why Compaction Matters: When data streams into a lakehouse, it often arrives in many small files. This is especially true for real-time data sources, which tend to generate hundreds or thousands of tiny files every hour. While each file is packed with valuable data, too many of them can lead to serious performance issues. Here’s why: 1. Query Slowdowns 🚀: Each file in a query adds overhead, which makes your compute engine work harder and take longer to get results. 2. Higher Storage Costs 💰: Small files create storage inefficiencies that add up over time. 3. Increased Metadata Load 📊: Tracking each tiny file stresses your metadata layer, making it harder for engines to efficiently manage large datasets. How Compaction Solves This: Compaction is the process of merging smaller files into larger, optimized ones. In Apache Iceberg, for example, this happens behind the scenes through automatic compaction. It groups smaller files together at regular intervals, helping to reduce the number of files and make queries faster. With fewer, larger files, you get: 1. Better Query Performance 🏎️: Your compute engine spends less time opening files and more time processing data. 2. Lower Costs 🛠️: By eliminating excess storage from small files, compaction reduces your data lake’s footprint. 3. Cleaner Metadata Management 📂: Fewer files means your metadata system is leaner, leading to faster operations.

13

1 komentarz

Wondering what makes a good data analytical engineer? 🤔 Our #MelonHead Erzana Spaho Bojko reveals some of the biggest challenges she faces as a QA and data analytical engineer. 👩‍💻 ⤵️ It’s all about a mix of essential skills and tackling challenges head-on! For me, it starts with ensuring the highest data quality, integrating diverse data sources effectively, and handling scalability issues. In the world of QA, we face unique challenges like developing comprehensive test scenarios, debugging complex data pipelines, and ensuring smooth and secure data migrations. Keeping strict records of data security and adapting to rapid technological changes are also key. Staying current with industry trends and collaborating with cross-functional teams are crucial for success. It’s a continuous journey of learning and problem-solving that drives excellence in data analytical engineering. Every day brings new opportunities to innovate and improve, and navigating these challenges makes the role both rewarding and dynamic. Embracing these obstacles with a proactive mindset ensures we deliver impactful and reliable data solutions. #DataEngineering #QualityAssurance #TechChallenges #Innovation #DataQuality #OrangeMelons #Melonhead

35

1 komentarz