Michael Johnson

Restore a balance between a code-first approach which results in poor data quality and unproductive rework, and too much data modeling which gets in the way of getting things done. Join DAMA Southern Africa & Pascal Desmarets on 22 July to learn more! RSVP link in the comments below. #datamodeling #damasa #dataquality #datamanagement

37

3 Comments

I've been learning about Azure Data Engineering services this week. Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure Data Factory et al. They're not that different to the AWS tools I've used in the past. And I maintain that the basis of using any of these data tools is the same: a fundamental understanding of what data is used for. Oh, and SQL. Naturally. This is partly why it frustrates me when open roles ask for experience with specific providers versions of these tools rather than experience with the general class of tools. Anyway, frustrations aside, my impression is that most of these platforms are easy to use but tying them together with zero-trust security is another thing altogether. It's no wonder that cloud security architects and engineers are entire disciplines of their own. If nothing else, it's shedding new light on why Microsoft Fabric is such an important development. Are you using Microsoft Learn for your learning goals? What are you studying? https://lnkd.in/gZ_RrQry

24

4 Comments

Recently tackled a SQLAlchemy optimisation challenge that might resonate with fellow developers: Scenario: Normalising a 30,000-row table. Script ran in minutes locally but took over an hour on Azure. The culprit? Each row triggered a separate query. Solution: Implemented batch processing. Results: - Before: ~150,000 individual queries - After: 30 batch queries - Query reduction: 99.98% - Runtime: From hours to minutes Key learnings: 1. Use SQLAlchemy's bulk_save_objects() for efficient batch inserts 2. Consider network latency in cloud environments 3. Choose appropriate batch size (I used 5000) 4. Maintain data integrity with proper foreign key handling 5. Think in sets, not individual records, when using ORMs This experience highlighted the crucial role of optimisation in database operations, especially at scale and in cloud environments. Have you faced similar challenges? What strategies worked for you? #SQLAlchemy #DatabaseOptimisation #CloudComputing #PythonDev #Azure

16

Generative AI Fundamentals #databricks #AI #GenerativeAI

8

Note that the Enterprise Big Data Engineer Text Book is now available free of charge (until September 30). Link available below: https://lnkd.in/gn2hnfHw #datascience #dataengineer #bigdataframework #certification #dataengineering #dataframeworks

22

2 Comments

So my current coding issue is Date Conversion from Excel Spreadsheet to C# DateTime object ... I am getting a cannot convert double to DateTime / I think that EEPlus update might have caused this as it has been working in the past. Investigating :)

6

9 Comments

For addicts of {gtsummary} R package like me, please try the new version before Daniel Sjoberg send it to CRAN. Thank you Daniel Sjoberg for the great work. This package have saved the life of several Biostatisticial R Programmers out there 😀. Health Data Acumen #RProgramming #BiomedicalResearch

14

Another great new feature went live in Data Graphs today - Domain Model annotations. You can add post-it notes to your ontology model diagram, show and hide them, and export model images with or without the notes. It really is the domain modeller's dream tool now. Domain model directly on the canvas and export the ontology as RDF at any time. Try it out for free here: https://lnkd.in/e7QuNGR4 #knowledgegraph #domaindrivendesign #graphdatabase

62

1 Comment

People, if your data pipelines are as precarious as this then spend some time tackling those, they will come back and haunt you. #dataengineer #datawarehouse #datapipeline #dataleaders

17

5 Comments

What's happening next week with Modelware Systems and DAMA Southern Africa? We have something for you to learn more about the fascinating world of Data Management. Strengths and limitations of different types of declarative modelling languages - Maria Keet - 5 August In this talk, Maria will compare four types of declarative modelling approaches, including mind maps, models in biology, conceptual data models, and ontologies. With each successive approach addressing limitations of the preceding one, we arrive at both a feature-based comparison and a task-based comparison, that, in turn can assist in choosing the right approach for the task at hand. The talk will close with the relatively recent notion of potential biases in such models. CASE STUDY: Creating a Data Driven Culture at TD Ameritrade - Derek Strauss - 7 August 2024 (16:00 SAST) My 5-year journey as the first Chief Data Officer of TD Ameritrade. What worked and what didn’t. I’ll talk about the lessons learned, including the need for a formal Organisational Change Management program, and the need to holistically address a cluster of digital asset problems—namely: ownership, security, trust and control. The Conceptual Modelling Journey Map - 8 August 2024 (10:00 SAST) We will start out webinar series on Conceptual Modeling in the viewpoint of the Data Manager as well and the overview on conceptual modeling RSVP link in the comments below. #damasa #modelwaresystems #datamodeling #conceptualmodeling #datamanagement #datadriven

7

2 Comments

#pandas on Snowflake lets you run your pandas code in a #distributed manner directly on your #data in Snowflake (Feature now GA in latest release) #analytics #datascience #textminning #snowflake ANZ Tech Talks Further details -

11

INSERT INTO vs SELECT INTO - Which is better? I answered this timeless question on StackOverflow: https://lnkd.in/ePinr2vb In terms of performance for temp tables, there are advantages to using INSERT INTO. When you use INSERT INTO #table SELECT * FROM table1, you create #table before hand. So, you can also create indexes, keys or constraints inline with the #table definition and take advantage of caching that #table. Then this #table can be used again from cache if stored procedure executes again. Caching temp tables helps reduce metadata contention i.e. PAGELATCH_EX on system objects.   But when you do SELECT * INTO #table FROM table1, and if you need to create indexes or keys on #table, then you have to alter the DDL of #table which will not cache #table. When stored procedure executes again, there will not be any table metadata in cache for stored procedure to use from and temp table will have to be created again. This causes more PAGELATCH_EX to be generated which can cause metadata contention on tempdb, if there are too many concurrent operations. So, I always recommend creating #table first and then inserting data into it, meaning INSERT INTO is better than SELECT * INTO, in case of temp tables.   Note: 1. Cache of temp table does not store the name of the temp table but only the metadata, and is very helpful when SPs using #temp with similar structure execute multiple times during the day.   2. Caching of temp tables can be done only if temp tables are wrapped inside a stored procedure. So this use case helps only for temp tables in stored procedure and not for adhoc statements.   Key points for caching: Do not alter the DDL of the temp table. Create indexes inline with the table definition, introduced in SQL2014. Wrap temp tables inside the stored procedure or trigger or function.   https://lnkd.in/eGibn3AN

80

5 Comments

Hi Folks - Another question today,sorry for many questions because I want to rap up the questions on Adf. Imagine a scenario where you have to pull out the 10 tables which contain Streaming data into Azure Alds. The HWM is a table which contains recent date time stamps. How would break the glass for this problem Note ; Client wants data quality with out any data loss. Follow Sumanth Bodduluri for more content on Data Engineering.

8

2 Comments

😎 😎 Looping in Low Code Plugins are you freaking kidding me? James Williams took me through some incredibly elegant #powerfx on my YouTube channel last week. For those of you who may be time poor, here's the companion guide. James solved a number of issues with my original Working Days demo and inadvertently solved looping! Read on for all the details! #powerplatform #powerplatformdeveloper #powerapps #powerappsdeveloper #dataverse #lowcodenocode #microsoft #microsoftmvp #communityrocks

42

Feature Alert! Awesome new Snowflake #data & #ML #lineage feature went into Public Preview today. You now have access to a new tab called "LINEAGE" which shows End2End lineage of standard tables, views, ML #models & #featurestores. You get a few ways to use this feature. 1. via Snowsight UI: https://lnkd.in/eHx-9uXK 2. via SQL GET_LINEAGE() function: https://lnkd.in/embuUeXw 3. ML Lineage API https://lnkd.in/esRzfU_6 Sample ML Lineage Notebook https://lnkd.in/eHcqkd2f

547

30 Comments

Today Connor Deasey and I looked into the Data Row Limit setting in Power Apps. Did you know that this setting is ignored by SQL Stored Procedures? Loading 10000 records is now so much easier and faster! Read the details in the post below. https://lnkd.in/e3bmsMFc

14