Martin Goebbels

How to Talk to SQL Data with Function Calling - https://lnkd.in/eXXh885N Join us on learn How to Talk to SQL Data with Function Calling. This session will explore how to leverage function calling in SQL to interact more effectively with your data, enabling you to perform complex operations, streamline queries, and enhance your database's capabilities. Whether you're a developer, data engineer, or database administrator, this webinar will provide you with the skills and knowledge to harness the full power of SQL function calling. Expect practical examples, best practices, and a live demonstration. You’ll learn: - The fundamentals of SQL function calling and its applications. - How to perform complex operations and streamline queries with function calling. - Best practices for optimizing SQL function calls in your database environment. - Real-world use cases where function calling has significantly improved data interactions. - A live demonstration of using SQL function calling to interact with data. This hands-on workshop is for developers and AI professionals, featuring state-of-the-art technology, case studies, code-share, and live demos. Recording and GitHub material will be available to registrants who cannot attend the free 60-min session.

30

2 Comments

Working with JSON in Snowflake and need to validate your data? Meet CHECK_JSON, your data quality checker! CHECK_JSON is a useful function that helps to check the validity of JSON. How does it work? - When you pass a valid JSON or NULL → Returns NULL (indicating all is good!) - When you pass invalid JSON → Returns an error message telling you exactly what's wrong and where Why is it useful? - Helps catch JSON formatting issues early - Provides clear error messages with position indicators - Perfect for data validation before processing #SnowFlake #sql #dataengineering

8

1 Comment

Thanks Renee Ryland for a great review of my book - DATA QUALITY: EMPOWERING BUSINESSES WITH ANALYTICS AND AI . Ths book published by Wiley discusses the 16 most common issues of data quality degradation and offers 10 practical solutions for remediating and sustaining data quality based on the DARS framework i.e. Define -Analyze-Realize-Sustain which I have implemented in many projects. So, how can you implement the DARS framework to improve #dataquality in your organization for your operations, complaince and #analytics? Here are 3 options for you. 1. Get my book in print or audio formats to learn more about the DARS framework. Link to the book in comments. 2. Attend my 8-hour training - DATA PREPARATION AND QUALITY FOR AI AND ANALYTICS. This training is delivered in 4 modules over 2 days. 3. Engage me as an Advisor on your Data Quality improvement initiatives. Remember poor data quality impacts up to 12% of your firm's revenues, and if you don't consciously manage data in your firm, the data quality degrades 2-7% every month. #datagovernance CC: Steve Rosvold Gary Cokins Michael Stratta

18

3 Comments

You need technical skills to get a data job, but you need business skills to keep one... Data people generally know they need to learn those technical skills themselves. But often don't have the same expectations for their business skills. In my experience, passively learning business skills will not get you where you need quick enough. I've spent A LOT of time and money learning all kinds of different business skills, and it's made a huge difference in my career. Senior and mid-level data folks especially, can move up based on their data skills, then really struggle because they didn't sufficiently develop on the business side as they moved up. This is one of the main reasons I launched a professional development program. Instead of just creating content and sending it into the void, I thought it would be great to also spend some time helping people actually implement the lessons in their jobs. If you're a data person who wants to be more impactful at your job with all the benefits that come along with that, check out Top Data People. And regardless of HOW you learn, developing your business skills will have a massive ROI for your data career.

32

5 Comments

👨‍💻 5 Lessons from 5 Data Teams (Lessons for Data Engineers) 💡 Article by: Tim Webster ➡ Link to Article: https://lnkd.in/eK7zq745 - - - - - - - - - - - - - - - - - - - - - - - - - - - 💡 Follow for more great content! ✅ ✍️ Want to contribute? Connect with us. 🔗 Let's create together! ✅ - - - - - - - - - - - - - - - - - - - - - - - - - - - #dataengineering #datascience #careeradvice #dataanalytics

2

Modelling is a common activity and the outcome of the processes results in many different artefacts that are called 'a model'. They are not all interchangeable, however. There are many types of models that serve diverse purposes and each have their strengths and limitations. In this talk, Maria Keet will compare four types of declarative modelling approaches, including mind maps, models in biology, conceptual data models, and ontologies. With each successive approach addressing limitations of the preceding one, we arrive at both a feature-based comparison and a task-based comparison, that, in turn can assist in choosing the right approach for the task at hand. The talk will close with the relatively recent notion of potential biases in such models. Join DAMA Southern Africa and Maria Keet for Strengths and limitations of different types of declarative modelling languages on 5 August at 16:00 SAST to learn more. Get 20% discount off her new book by joining the discussion! #datamodeling #damasa #datamanagement

17

2 Comments

🚀 DE Interview Experience 📢 Apache Spark ⌛ 1-Min-Read How to use write with partitionBy in Spark for efficient data storage and querying. 👉🏽 🧔🏽‍♂️Interviewer: Can you explain how you use the write method with partitionBy in Spark to optimize data storage? 👨‍🦰Candidate: Using the write method with partitionBy in Spark allows you to partition the output data based on the values of one or more columns. This improves query performance and makes data storage more efficient. 🧔🏽‍♂️Interviewer: Can you walk me through how you typically use partitionBy? 👨‍🦰Candidate: Let's say we have a DataFrame df containing sales data, and we want to partition the data by the region column before writing it to disk. df.write.partitionBy("region").parquet("output_path") In this example, the data is written to the output_path directory in Parquet format, partitioned by the region column. This means that the data will be organized into separate directories for each unique value in the region column. 🧔🏽‍♂️Interviewer: How does this improve performance? 👨‍🦰Candidate: Partitioning data by a specific column improves performance in several ways: Efficient Data Retrieval: When querying the data, Spark can skip reading entire partitions that do not match the query criteria, thus reducing the amount of data read from disk. Improved Compression: Partitioning can lead to better compression ratios since data within each partition tends to be more homogeneous. Parallel Processing: Each partition can be processed independently, enabling better parallelism and distributed processing. 🧔🏽‍♂️Interviewer: Can you provide a real-world example where you used partitionBy? 👨‍🦰Candidate: Absolutely. In a project where we needed to store large amounts of user activity data, we partitioned the data by date and user_id. user_activity_df.write.partitionBy("date", user_id").parquet("s3://bucket/user_activity") 🧔🏽‍♂️Interviewer: That’s an excellent example! Are there any other benefits to using partitionBy? 👨‍🦰Candidate: Yes, in addition to performance improvements, using partitionBy helps in data organization and management. It makes it easier to manage large datasets by breaking them down into more manageable chunks. This is particularly useful for archiving old data or performing maintenance tasks on specific partitions without affecting the entire dataset. #DataEngineering #ApacheSpark #DataPartitioning #BigData #DataStorage #DataOptimization #CareerGrowth #OneStepAnalytics

17

How does Data Controller for SAS® defend #DataQuality for #DataWorkflows deriving from Business Inputs? ⚔️ 1st Line of Defence - Data Model ⚔️ The interface will only accept inputs conforming to this model (columns, types, lengths, indexes, constraints etc). The schema determines the behaviour - eg, a date format results in a date picker. ⚔️ 2nd Line of Defence - Data Permissions ⚔️ Changes are made using a SAS System Account (eg `sassrv`). This means you can safely DENY write-access to data for business users, preventing un-controlled #DataIngestion. ⚔️ 3rd Line of Defence - Validation checks ⚔️ Additional validation checks (eg value ranges, specific patterns) can be configured to run at the point of data capture, prior to #SAS upload. The docs for these are here: https://lnkd.in/djtaGPsr ⚔️ 4th Line of Defence - Post Edit Hook ⚔️ Complex / customer specific validation can be deployed as SAS code, to run after every EDIT (prior to APPROVAL). A failure here results in immediate user feedback / change rejection. ⚔️ 5th Line of Defence - Approval Step(s) ⚔️ Changes are reviewed with one or more approvals BEFORE being applied to the target database. A full audit trail is also maintained. All functionality is ZERO-CODE, works on #sasViya / EBI / SASjs Server, applies to any database you have an ACCESS engine for, and - best of all - FREE for 5 users. #datagovernance #sasadmin #saspartners

18

𝐇𝐨𝐰 𝐭𝐨 𝐀𝐭𝐭𝐫𝐚𝐜𝐭 𝐓𝐨𝐩 𝐃𝐚𝐭𝐚 𝐓𝐚𝐥𝐞𝐧𝐭 𝐖𝐢𝐭𝐡𝐨𝐮𝐭 𝐁𝐫𝐞𝐚𝐤𝐢𝐧𝐠 𝐭𝐡𝐞 𝐁𝐚𝐧𝐤: So, you’re on a quest to hire the crème de la crème of data talent, but your budget is tighter than a data engineer's code on a Monday morning. Fear not! Here’s a guide to charm those data wizards without emptying your coffers: 1️⃣ 𝐒𝐭𝐞𝐩 𝟏: 𝐂𝐫𝐚𝐟𝐭 𝐭𝐡𝐞 𝐏𝐞𝐫𝐟𝐞𝐜𝐭 𝐉𝐨𝐛 𝐀𝐝 Start with a job ad that’s full of buzzwords and cutting-edge tools. It doesn’t matter if you’ve never heard of half of them or if your tech stack is older than a dial-up modem. This is a “visionary roadmap,” after all. Mention quantum computing, digital immune systems, and sprinkle in some “AI revolution” for good measure. It’s like a wishlist for Santa, but for tech. 🎁 2️⃣ 𝐒𝐭𝐞𝐩 𝟐: 𝐁𝐞 𝐕𝐚𝐠𝐮𝐞 𝐀𝐛𝐨𝐮𝐭 𝐒𝐚𝐥𝐚𝐫𝐲 You don’t want to scare off the top talent with something as tacky as a salary figure! Use the classic "DoE” to keep them guessing. If they push for details, just remind them you’re guarding against those “money motivated” types. It’s not about the paycheck, it’s about the passion for your dream, right? 💰 3️⃣ 𝐒𝐭𝐞𝐩 𝟑: 𝐒𝐞𝐥𝐥 𝐭𝐡𝐞 𝐃𝐫𝐞𝐚𝐦 Now that you’ve piqued their interest, it’s time to hype up the role. Make it sound like they’re the key to unlocking your company’s hidden potential. Emphasise your company’s grand ambitions for a data-driven utopia. You know, the kind where everyone rides unicorns and dashboards update themselves. 🦄 4️⃣ 𝐒𝐭𝐞𝐩 𝟒: 𝐓𝐨𝐮𝐭 𝐘𝐨𝐮𝐫 𝐂𝐮𝐥𝐭𝐮𝐫𝐞 𝐏𝐞𝐫𝐤𝐬 Here’s where you really hook them: your company culture. Highlight the perks that set you apart, like free office parking, Flexi-Work Fridays, and the infamous “optional” evening/weekend socials. And don’t forget the most irresistible perk of all – the chance to work with you! ✨ 5️⃣ 𝐒𝐭𝐞𝐩 𝟓: 𝐓𝐡𝐞 𝐎𝐟𝐟𝐞𝐫 𝐒𝐭𝐚𝐠𝐞 Congratulations! You’ve snagged their interest. Now, let them know they have incredible potential but aren’t quite there yet. Drop the bomb that the role will require a “slight” pay cut. But reassure them that it’s an opportunity of a lifetime. After all, who needs to pay rent when they’re working on something this groundbreaking? 🌱 6️⃣ 𝐒𝐭𝐞𝐩 𝟔: 𝐂𝐥𝐨𝐬𝐞 𝐭𝐡𝐞 𝐃𝐞𝐚𝐥 By now, your candidate is either fully invested in your vision or too exhausted to say no. This is your moment. They’ll say “YES” because, really, who could resist this dazzling opportunity? 🤩 --- 𝑇ℎ𝑒 𝑅𝑒𝑎𝑙 𝐷𝑒𝑎𝑙 Jokes aside, transparency is key. A mismatch between interview promises and reality on things like capabilities, salary, or culture can be a disaster. Be clear about what you offer and the challenges involved. After all, honesty not only saves everyone time but also builds trust – and that’s the real way to attract the best talent. #Recruitment #DataCulture --- Feel free to connect with me Sam Chaher; I’d love to hear your thoughts on these, and other recruiting strategies you’d rather were left behind!

11

1 Comment

Here is my 6 month study plan.. Python Pipelines Docker, Kubernetes Prepare for Snowflake Certification I'm a huge Udemy fan, so I start with a course - and at the same time get free trials on the tech to practice with. Python Pipelines - I've set up many Python Linux Cron job pipelines. It's time to go deeper... Docker/Kubernetes - I need to nail down my skills on these. I'm really impressed by Snowflake - everything just seems to work, and you just need to know SQL. Anyone, like to share their study plans? #snowflake #docker #Kubernetes #python

12

2 Comments

How to create a radar chart in Tableau (without any complicated calculations) Follow the step-by-step tutorial by Ellen Blackburn. https://lnkd.in/ezFFQeYz My viz based on her tutorial (I used parallel coordinates calcs instead of Distance from Center) https://lnkd.in/ew3ppq4a 𝗖𝗿𝗲𝗮𝘁𝗲 𝘁𝗵𝗲 𝟰 𝗰𝗮𝗹𝗰𝘂𝗹𝗮𝘁𝗶𝗼𝗻𝘀 //Angle RUNNING_SUM((2*PI())/MIN({COUNTD([Stat])}))+(PI()/2) //Distance from Center AVG([Value]) //X [Distance from Center]*COS([Angle]) //Y [Distance from Center]*SIN([Angle]) 𝗕𝘂𝗶𝗹𝗱 𝘁𝗵𝗲 𝘃𝗶𝘇 1. X on Columns, Y on Rows 2. Change Mark Type to Polygon 3. Stat on Detail 4. Angle on Line 5. Team on Color 6. Value to Detail 𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗲 𝘁𝗵𝗲 𝗫 & 𝗬 𝘁𝗮𝗯𝗹𝗲 𝗰𝗮𝗹𝗰𝘀 1. Set Angle to Compute Using Stat 𝗙𝗼𝗿𝗺𝗮𝘁𝘁𝗶𝗻𝗴 1. Clean up Tooltips 2. Clean up chart 𝗔𝗱𝗱 𝘁𝗲𝘅𝘁 𝘁𝗼 𝘁𝗵𝗲 𝗼𝘂𝘁𝗺𝗼𝘀𝘁 𝗺𝗮𝗿𝗸𝘀 1. Duplicate X on Columns 2. Make it a dual-axis and synchronize 3. Change to Text 4. Create Labels Calc (get max value for each Stat and multiple by 1.1) 5. Add to text on second shelf

169

27 Comments

Want to label only the first and last marks in your view? Here’s how to do it for a variety of chart types! 🙌 https://lnkd.in/dfJRsrct

5

In this article, we are going to focus on, 1. What are materialized views? 2. Requirements 3. Create a materialized view 4. Load data from external systems 5. Refresh a materialized view 6. How are Databricks SQL materialized views refreshed? 7. Schedule materialized view refreshes 8. View the status of a materialized view refresh 9. View the refresh status in the Delta Live Tables UI 10. Stop an active refresh 11. Update the definition of a materialized view 12. Drop a materialized view 13. Describe a materialized view 14. Change the owner of a materialized view 15. Control access to materialized views 16. Enable change data feed 17. View the refresh history for a materialized view 18. Determining if an incremental or full refresh is used 19. Limitations https://lnkd.in/dEFyjNnx

2

Creating a heatmap for a single month in Tableau is pretty straightforward. But a whole year? Here's how I do it. In this tip, I show you, step-by-step, how to create a full year calendar heatmap in Tableau. https://lnkd.in/eZVdxM38 What do you think of this technique? Leave a comment below.

147

10 Comments

When it comes to optimizing Databricks job performance, one effective strategy is updating to the latest Databricks Runtime versions. For instance, I achieved a 20% improvement by upgrading from DBR 12.2 to DBR 14.3 LTS. This is because "Photon is no longer required for MERGE operations to leverage deletion vector optimizations." Deletion vectors are particularly beneficial for pipelines involving frequent updates and deletes using merge commands. #dataengineering #databricks

17

Eager to strengthen your Apache Spark and Databricks skills? Then you should... ...definitely check out this Udemy Data Engineering course I was part of: "Prophecy for Data Engineering: Low-code Data Transformation"! Click here to enroll now 👉 https://bit.ly/3LUwgC4 As one of the course hosts, I had the opportunity of creating the "Introduction to Spark and Databricks" section with 7 lectures where you will learn about ✅ Apache Spark architecture & concepts ✅ Spark language & tooling ✅ difference between Spark & Databricks ✅ data lakehouse, unity catalog, optimization & security ✅ working with Spark best practices ✅ Spark & Databricks tips & tricks But there is much more on the course agenda! You will also ✅ learn & design the data lakehouse paradigm for an e-commerce company ✅ implement & deploy a medallion architecture using Prophecy running on Databricks ✅ share & extend pipeline components with data practitioners & analysts ✅ utilize version control & change management in Data Engineering ✅ understand Apache Spark & its best practices with real-life use cases ✅ deploy pipelines to production & CI/CD & best practices ✅ deploy data quality checks & unit tests ➕ hands-on lab environment is provided with this course Whether you're a Data Engineering professional looking to upskill or a newcomer eager to start off his journey, this Prophecy course is for you. #dataengineering #dataengineer #datascience #bigdata #learndataengineering #prophecy #apachespark #databricks #udemycourse

7

PySpark is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python. It also provides a PySpark shell for interactively analyzing your data. PySpark combines Python’s learnability and ease of use with the power of Apache Spark to enable processing and analysis of data at any size for everyone familiar with Python. PySpark supports all of Spark’s features such as Spark SQL, DataFrames, Structured Streaming, Machine Learning (MLlib) and Spark Core. Overcoming Challenges: Connecting PySpark to a SQL Database Engine for Massive Data Processing In the world of big data, integrating various technologies seamlessly is often a daunting task. One of the significant challenges I faced was connecting from a PySpark session to a SQL database engine and efficiently distributing the memory and CPU cache for massive data processing. Here’s how I tackled this complex problem. 1.Establishing a reliable connection between PySpark and a SQL database engine. 2.Ensuring the efficient distribution of memory and CPU resources. 3.Handling massive datasets without compromising performance. Pyspark Scala

5

Data and Analytics Training to open doors. Looking to gain new skills? Time for a Career Change? Sign up for Newtyne’s introductory training courses in the Language of SAS. Fundamentals – the Language of SAS - July 8th 2024 Intermediate -   the Language of SAS – July 15th 2024 With a blended learning approach providing self-study, online ‘live’ classroom and masterclasses. There are mini quizzes and comprehensive exercises throughout to help assess and reinforce your learning with the opportunity to join an end of day Q&A session(s) with the instructor. To book your place go the Courses Tab on our website.   #newskills #SAStraining #dataanalytics #personaldevelopment #sasprogramming #data #analytics 

6

The most common failure I've seen in very senior data people is ignoring their network. This goes both for data leaders as well as very senior data ICs. To become very senior, you generally have to figure out both the business and the technical side of data work. So, you've got great skills. But at the highest level, a good chunk of your impact will be through your network. Internally, you'll need a strong network to get things done inside your company. When you become very senior in data, it's because you can get complex organizational things done, not just because you have strong technical skills. Having a good external network allows you to see what's working and not working at other companies, which is enormously valuable. Lastly, it's incredibly difficult to get very senior jobs outside your network. This is one of the biggest reasons I added a community element to the Top Data Person development program. Everyone is better off by developing relationships with other data people looking to grow. This is one of those "important but not urgent" things that too many data people don't think about. But if you do a bit of work each month to meet new data people and keep up with your data friends, it really snowballs over the years. I've got professional relationships that have lasted almost 20 years. Don't ignore this for a decade - start today!

86

17 Comments