Daniel Rothamel 📊🔧🚀

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

How to load data from S3 into Snowflake using Snowflake Copy Command Hands on Labs Labs https://lnkd.in/eYXd_d8g

21

Two video tutorials, carefully prepared by Bob DuCharme, explore two #GraphDB features that allow the users to treat a relational database like #RDF & an RDF repository like #relationaldata. https://hubs.la/Q02yByW10

9

I wrote a thing.... It's about How to Build Data Visualization Dashboards with Apache Superset and Pinot Check it out. #apachepinot #apachesuperset https://lnkd.in/e46uRmuQ

5

Excited to share Medium post: "PG Internals, PostgreSQL Vaccum of A to the Z"!   💡Ever wondered how PostgreSQL maintains its performance and storage efficiency? This comprehensive guide explores the vacuum process in PostgreSQL, detailing how it reclaims storage, prevents transaction ID wraparound, and ensures the database runs smoothly.   📘 Read the full article here: https://lnkd.in/g3gHcaQg   This article is packed with valuable insights to help you master PostgreSQL's vacuum process.   #PostgreSQL #DatabaseManagement #OpenSource #TechInsights #DataOptimization #TechCommunity #DatabaseInternals #MediumArticle #KnowledgeSharing #AGEDB

10

A fair bit of what we’re starting to call “metadata” is just. Actual. Data. Just because most orgs manage it poorly doesn’t make it “metadata”, it’s usually just poorly managed Enterprise Master data, with the ensuing lack of linkages and updates in appropriate places. Keep you feet firmly planted, model the data you need, proceed, repeat. #meatbasedengineer

125

31 Comments

Let's say, theoretically, that someone you know has been procrastinating on backing up your old Universal Analytics data (which starts going away in a month)... If that person is looking for an easy way to do a very basic backup, One option is importing the data into Plausible.io. I imported 1.2B pageviews this week. It took about 2-3 hours to run and it didn't cost me anything since I already had an account (otherwise it'd be $9/mo. for up to 10 sites). To be clear, this is a very basic backup! There's no ability to filter, there's no ability to use goals or funnels, etc. If you use the Plausible demo site at https://lnkd.in/g7SPfiYS -- it's like just being able to use that one page and time is the only dimension you can drill down into. More details on the import are here: https://lnkd.in/guRqACMu Pros: - It couldn't be any easier to run the backup. - It's a backup with an easy-to-use interface that can be shared with non-technical users (as compared to a BigQuery backup). - It's cheap. - It can go back 10 years and get as much volume of data as you have (limited by your UA data retention window). Cons: - You have to keep an active Plausible account to continue to view the data (i.e. $9/mo. in perpetuity). - The data really is very limited! - Not being able to filter is a drag. - At some point in the future the data could be subject to Plausible's own data retention rules (3 years on the $9/mo. plan). If you want something more sophisticated, I also continue to recommend Analytics Canvas' backup service: https://lnkd.in/gFsDUnby which backs up much more of the data and is built explicitly for this time & purpose!

9

2 Comments

Data Modeling for Azure Cosmos DB for the Relational Mind In this episode, Mark Brown is joined by Sarah Dutkiewicz (@sadukie) who will show how the relational mind can get into the nonrelational side by looking at the differences in data modeling. https://lnkd.in/eDD-C6TK

1

Today let's talk about SNOWFLAKE TASKS Snowflake Tasks are an incredible addition to the Cloud Data Warehouse. These allow users to chain together series of user-defined functions in Snowflake, that can execute plain old SQL queries or slightly fancier stored procedures. Leveraging these alongside traditional orchestration frameworks comes at a cost - VISIBILITY AND ALERTING. While easy to configure, monitoring is notoriously difficult. See how Orchestra makes this a breeze below 🕊 https://lnkd.in/dQg6cCFi #dataengineering #snowflake #snowflaketasks #clouddatawarehouse

7

1 Comment

Those of you wondering what's the difference between Azure Data Bricks vs Data Bricks, Brenna Buuck does an excellent job providing a full analysis of these two tools and also touches on #AzureSynapse #AzureDataFactory #DataBricks #Snowflake #DataLakeHouse #AzureDataBricks

2

It's no secret that PostgreSQL has been a popular topic of discussion. Why? Because of its flexibility, scalability, and advantages over commercial databases. So, what can Quest Software offer you for PostgreSQL? 🔸Foglight for PostgreSQL (monitoring & performance management) 🔸SharePlex (data replication & integration) 🔸Benchmark Factory (database performance testing) 🔸and more! #postgresql #dba #developer #database

18

Discover with Cedrik Neumann how to harness Apache Airflow plugins to run tasks within a profiler and access custom metrics through operator links and custom views. Get inspired by ideas for making task metrics more accessible in Airflow's future! 🚀 https://airflowsummit.org/ #airflowtask #airflowmetrics #dataorchestration #batchprocessing #dataprocessing #datapipelines #data #datasience #opensource

10

With #IcebergSummit happening tomorrow online, I took a trip down memory lane by listening to the #DataFemme episode featuring Alex Merced of Dremio. This episode is packed full with information, not just about #Apache Iceberg, but about the best ways to get started as a contributor to #opensource projects. Alex is an expert content creator with extensive knowledge about Apache products and their role in technology. This is a must listen for everyone heading towards Iceberg Summit tomorrow! https://lnkd.in/guqYdzG2 Tabular Amazon Web Services (AWS) bodo.ai Cloudera Confluent PuppyGraph Snowflake Starburst Upsolver

8

Despite the very large number of data lineage solutions out there, getting *full* visibility to your data pipelines remains a common challenge for growing data platforms. And if you have some Spark-based pipelines in your data stack, whether it's Databricks, Amazon EMR or others, you should prepare to work a lot harder... ! However, there's a class of new solutions, such as OpenLineage and Foundational that go beyond traditional SQL parsing. Read more on our blog: https://lnkd.in/gCEdAXSE #datalineage #spark #datagovernance

12