Daniel Sahal

Gülçin Yıldırım Jelínek: Maintaining Postgres for Modern Workloads 🚀 A Geeky Rendezvous with PostgreSQL: Modern Workloads Edition 🚀 Hey PostgreSQL aficionados! 🐘 Ever wonder what makes software not just functional, but outright maintainable? I had the pleasure of diving into this very topic with the inimitable Gülçin Yıldırım Jelínek on the Maintainable Podcast, hosted by Robby Russell. Imagine juggling modern workloads with the grace of a seasoned circus performer using Postgres as your trusty, reliable, and slightly quirky assistant. Spoiler alert: it involves more than just a well-timed Ctrl+C. We delved into the mysterious art of keeping our beloved databases in prime working condition, even when they grumble under the weight of today's demands. Think of it as giving your beloved vintage car a tune-up before a road trip, except this car handles all your data and sometimes cries for a VACUUM. For those who missed it, the podcast features secrets and snippets you won't find in any manual

1

Hey there! Join me for my session at Data Science Summit 2024 in Warsaw on November 22, where we’ll dive into topics in the areas of #data #engineering and #dataops. We’ll start with a quick recap on Delta Lake open table format, take a look under the hood at how the Databricks shallow clone mechanism works, and, most importantly, see how using it smartly can simplify our dev workflows and make safe #continuous #testing on big datasets possible. PS If you don’t have your ticket yet, here’s a 20% discount code!

34

Data Engineering 2024 Per cdo aspiring data engineer, tech stack gjeneral ne 2024 eshte: Gjuhet: SQL Python apo Scala Advanced analytics lakehouse platforms, cloud data warehouses: Databricks ose Snowflake (Redshift, bigQuery) Distributed computing engines per procesim te te dhenave: Apache Spark apo Dask Pipeline orchestration dhe monitoring tools: Airflow ose (Dagster, Prefect) Cloud service providers: AWS ose Azure Infrastrukture: Docker Kubernetes Terraform Let me know if I forgot to add anything? #dataengineering #bigdata #databricks

41

1 komentarz

Now there is nothing around PostgreSQL that we cannot do, including complicated migrations from Oracle to PostgreSQL. Stormatics and Cloudway help you to migrate to PostgreSQL and be free forever! Cloudway S.A. PostgreSQL Global Development Group #postgreSQL

15

Recently Marko Topolnik made length(varchar) SQL function in #QuestDB faster with the following SWAR code. Previously it was using a much slower UTF-8 parsing routine.

34

4 komentarze

Nie musisz już komentować kodu 😁 Jeśli używasz notatników w Databricks Asystent zrobi to za ciebie i nie tylko to Prompt text / What Assistant does / ➡ Displays common commands /doc ➡ Comments the code in a diff view /explain ➡ Provides an explanation of the code in a cell /fix ➡ Proposes a fix to any code errors in a diff view /findTables ➡ Searches for relevant tables based on Unity Catalog metadata. /findQueries ➡ Searches for relevant queries based on Unity Catalog metadata. /optimize ➡ Improves inefficient SQL queries. /prettify ➡ Formats code for readability. /rename ➡ Suggests updated names to notebook cells and other elements, depending on the context. /settings ➡Adjusts your notebook settings directly from Assistant.

34

2 komentarze

If you somehow haven't noticed, tomorrow is the Google Cloud Summit in Prague 2024. Today, at "partners only" dinner I realized that Google might be too "cool" for "standard" users to comprehend. While every Data Scientist will be excited about #Gemini 1.5 Pro coming with context window of up to 1 million tokens (because like what the hell) and that the new Arm-based CPU Axion is 30% better performing than previous models, what does it actually mean for us? The simplest translation: cheaper and efficient cloud services. And there is another new thingy I liked: Gemini for databases - AI that reads, analyse and simplifies database code - like your old procedures, triggers, and functions and convert them to a PostgreSQL-compatible dialect. ➡ Again in simple terms: it has never been easier to migrate your old / on-premise data warehouse to the Cloud. With that said, I strongly recommend attending the Google Cloud Summit 2024 tomorrow. And of course, Revolt BI will be on-site as well. For the curious, links to the features mentioned above: 🧠 Gemini 1.5 Pro and a milion tokens context window - https://lnkd.in/eSRwGTSU 📟 Arm-based CPU that will deploy BigQuery, Google Earth Engine or even YouTube Ads - https://lnkd.in/eVSEFKu3 🏭 Gemini for databases or "what the hell did they mean by this code from 2015" - https://lnkd.in/erq99W3q Thank you Google Cloud team for inviting us! Boris Georgiev Ruxandra Galbura Nicolae Ciobanu Paulina Koriat #GoogleCloudSummit2024

26

3 komentarze

I submitted the following papers to #apexconn25: - Fighting Bad PL/SQL & SQL with VS Code - The Trivadis PL/SQL & SQL Coding Guidelines are Dead – What Now? Since I feel bad submitting only in German, I tried something new. I submitted the "same" abstract in both English and German.

17

Spark Performance: Boost Efficiency with Kryo Serialization If you're working with Apache Spark, you know that performance is key, especially when dealing with large datasets. One of the simplest yet most impactful optimizations you can make is switching from the default Java serialization to Kryo serialization. Why Kryo? Java serialization is the default in Spark, but it’s slow and produces large serialized objects, which can significantly drag down your job’s performance. Kryo, on the other hand, offers clear advantages: 🔹 Speed: Kryo serialization is significantly faster than Java serialization. This speed improvement comes from Kryo's more efficient algorithm for converting objects to a binary format. Faster serialization directly translates to reduced execution time, particularly in jobs that involve extensive shuffling of data between nodes. 🔹 Efficiency: Kryo produces smaller serialized objects, reducing the amount of data that needs to be shuffled across the network or stored in memory. By minimizing the size of the serialized objects, Kryo reduces memory usage and improves network efficiency, which is crucial in distributed systems where network and memory are often the bottlenecks. 🔹 Simple to Implement: Switching to Kryo serialization in Spark is straightforward. It requires minimal code changes but can lead to significant performance improvements. Given that serialization is a fundamental part of data processing in Spark, optimizing this aspect has a positive effect on overall job performance. How to Enable Kryo Serialization in Spark: Add this to your Spark configuration: sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") You can also register your commonly used classes to avoid the overhead of dynamic class registration. By pre-registering classes, Kryo can skip the runtime reflection that would otherwise slow down the process: val conf = new SparkConf().setAppName("MyApp").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") conf.registerKryoClasses(Array(classOf[MyClass1], classOf[MyClass2])) Impact By enabling Kryo serialization, you’re not just tweaking your job, you’re unlocking significant performance gains, especially for jobs that involve a lot of serialization and deserialization, like those involving shuffling large datasets. This is because Kryo reduces both the computational cost of serialization and the overhead associated with data transfer across the network. In distributed computing, these optimizations can lead to faster job completion times and lower resource consumption, ultimately saving both time and money. If you're looking to optimize your Spark jobs, this is one of the easiest yet most effective places to start. #softwareengineering #dataengineering #bigdata #spark #performance #distributedsystem #parallelcomputing

20

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 komentarzy

Second day of AWS for Data Summer Camp Budapest 🏕️ After learning how to build a serverless data lake on day one, today we dove into putting data to work with machine learning and BI tools. Krisztián Szulák gave an overview of Amazon SageMaker Canvas and how it enables building ML models to predict business outcomes. All without writing a single line of code! Arpad Csoke then demonstrated visualizing ML predictions using Amazon QuickSight dashboards and reports. We have one more day remaining at the camp. Looking forward to tomorrow's sessions on getting more out of QuickSight and exploring generative AI. Day 3: https://lnkd.in/eG8Q9XRA #aws #AmazonSageMaker #AWSSageMaker #ML #MachineLearning #Quicksight #AmazonQuicksight #GenAI #Bedrock #AmazonBedrock #Analytics #AnalyticsOnAWS

20

🚀 Data Warehouse vs. Data Lake: Which One Should You Choose? 🚀 When it comes to modern data management, Data Warehouses and Data Lakes are two of the most powerful and popular approaches. But how do you choose the right one for your business? 🤔 Here's a quick breakdown: 🔹 Data Warehouse: Ideal for structured, processed data, tailored for Business Intelligence (BI) users and reporting. It supports enterprise-wide decision-making, but comes with a longer implementation time due to its complexity. 🔹 Data Lake: Best suited for Data Engineers and Data Scientists who work with raw, unstructured data. It offers more flexibility, enabling broader analysis at scale, and it's perfect for machine learning and advanced analytics. 📊 Key Differences: Scope: Data Warehouse = organization-wide | Data Lake = specific departments Data: Data Warehouse handles massive, processed data | Data Lake stores raw data Users: BI users vs. Data Engineers & Data Scientists Which one fits your needs? Let me know your thoughts in the comments below! 👇 #DataScience #BigData #BusinessIntelligence #DataAnalytics #CloudComputing #DataLakes #DataWarehousing #MachineLearning #Snowflake

3

1 komentarz

I have gotten so many messages about the steps I have taken to become a Databricks MVP, that I thought why not share some. So here we go:   ➡️ First started working with Databricks in 2017, didn't have a clue what Apache Spark was. It was almost used nowhere here in The Netherlands. Data volume was low and Databricks was the only way to run Python code in Azure. I come from an on-prem MSSQL and Azure background. ➡️ Have about 5 serious Databricks enterprise implementations to my name together with Databricks and their (delivery) solutions architects. I know a lot of them in person and have learned so much from them. Fun fact, the architecture of one of my previous clients was initially set up together with Simon Whiteley. ➡️ Actively contributed (and still do) on Databricks community forums and reddit. ➡️ Took and passed the associate and professional Data Engineering certifications. ➡️ Got in touch with other MVPs like Josue A. Bogran, who also nominated me. ➡️ As of this year, actively post on LinkedIn.   That's it, there are more experienced MVPs than me and less experienced. But I believe anyone with the right mindset and discipline can become an MVP.   #databricks #MVP

240

12 komentarzy

https://lnkd.in/dqY75rSb - This is a really nice article but I would to add a few words of advice from my own experience: - Clear separation of your environments and avoiding monorepo terraform monolith are the best ways to go - I recommend isolating you Unity Catalog metastore, workspaces, networking and VPCs, extra cloud resources, and groups / service principals that going be reused across workspaces - all into separate terraform projects with isolated environments (state files) - you don’t want to ruin your infrastructure with a single bad deploy - Plus place all your IaC code into reusable modules to make things DRY and benefit from for-each loops on module level, remember to use module inputs/outputs for managing cross modules deployment dependencies. - Terraform docs pre-commit hook will help you document your code (but you need to type all your variables and provide human-friendly descriptions) - Also use terraform objects maps/lists for configs, add pre-commit hooks for code checks and formatting - Finally, CICD everything with GitHub Actions or your favorite deployment tool :) - Seriously, be prepared to spend hundreds of hours with Databricks documentation, which is great but could be also quite challenging sometimes 😅 #Databricks #IaC #LakehouseDeployment #CICD #Terraform #EngineeringInfrastructure

7

To all my fellow SQL lovers and gurus, Here's a useful tip for working with Databricks SQL: you can select all columns while excluding specific ones you don't want. Use the following syntax: Select * except (columnName1, columnName2) from Tablename https://lnkd.in/g8uR3RJ6 #dataengineering #sparksql

193

15 komentarzy

In Spark, usually it doesn't make any difference in performance if you write your code in Scala, Python or R but it's not the same case when creating an UDF. When we create an UDF in scala, it runs within the JVM and there's no additional overhead as Spark engine primarily runs on the JVM but when you create an UDF in Python, something quite different happens. Python UDFs run in a separate interpreter, outside of JVM. When a PySpark UDF is called, data has to be serialized back to the JVM. This process adds substantial overhead of inter process communication, making python UDFs generally slower than Scala UDFs. 𝐂𝐨𝐧𝐜𝐥𝐮𝐬𝐢𝐨𝐧: It's always recommended to write your UDFs in Scala or Java—the small amount of time it should take you to write the function in Scala will always yield significant speed ups, and on top of that, you can still use the scala function from Python by loading it into a jar. That's the beauty of the spark. #Spark #UDF #Python #Pyspark

6

💡 Some very useful #dbt packages part 2. In this post, I want to highlight some packages developed by the community. Let me know if you use some of them or others. - dbt_snowflake_monitoring: Awesome package to monitor snowflake performance and costs by SELECT. You can monitor costs over time for service, warehouse, database, query, etc. If you like this package, check their website where they have more advanced products. https://lnkd.in/d5UDEeuq - dbt_artifacts: A must-have package for every dbt project. It is like having the dbt artifacts in the format of tables and keeping your project history. It creates objects in your database containing information about your project, resources, and executions. https://lnkd.in/dKXp3_S8 Brooklyn Data Co. (a Velir company) - automate_dv: This package is your best friend if you are implementing a data vault modeling. It contains macros to help you build links, hubs, satellites and more. https://lnkd.in/d9RH6psH AutomateDV (formerly known as dbtvault) - elementary: Great package for observability. It has metadata tables just like dbt_artifacts, but comes also with an incredible UI where you can easily check your whole project's health and behavior. It also adds some anomaly detection tests. https://lnkd.in/dNH8KRuE Elementary - dbt_constraints: If you use dbt-snowflake, this package helps you implement constraints to your models. https://lnkd.in/dSAM3FMS Snowflake -- dbt-checkpoint: Not exactly a dbt package, but I treat it like one 😆. Very helpful to enforce dbt patterns and best practices in your project. You define some project rules and every time you make a commit, it will check if you are following those rules. https://lnkd.in/d2-kXfbS Datacoves Datatonic -- data-diff: No longer supported. But it was a great opens-source tool for data reliability and migration data tests. Datafold still offers the paid version of this product and other solutions, you should definitely check them out. -- Check all available dbt packages here: https://hub.getdbt.com/ Follow me for daily dbt content 🔶 #dataengineering #analyticsengineering

102

7 komentarzy