Founder @ okube.ai | Fractional Data Platform Engineer | Open-source Developer | Databricks Partner
Modern data platforms offer conveniences that database admins of the past could only dream of. But the biggest changes are still ahead, likely in the form of paradigm shifts. Dimensional modeling was once essential for data warehouses with limited resources. But with today's infinitely scalable solutions, alternative approaches like the One Big Table model often offer simpler and more performant datasets. I recently wrote an article on this shift, which you can read here: https://lnkd.in/egZBcrXx Another example is how these modern platforms have paved the way for new engines like Spark. While SQL used to be the only game in town, cloud-based solutions that support Spark are opening up a whole new world of possibilities for both data engineers and developers.
I am constantly reminded how solutions like Snowflake, BigQuery and Databricks make my life easier as a data engineer. If you didn't start on SQL Server, Postgres, or Oracle like Jeff Skoldberg and Ryan H. who pointed out some of the ways CDWs make data professional lives easier, then you might be unaware of some of the limitations. So here are a few of the limitations or admin tasks you used to have to perform. 1. Limited Storage - With most cloud solutions you have unlimited storage(which sure, comes with an unlimited bill) but you never have to sit there and wonder if you have a temp table somewhere that is causing storage issues or just generally need to wonder if you need to migrate hardware... 2. Limited compute - This also goes for compute. If you've never had to open up a database activity monitor to see what query is holding up all your other queries, do you even DBA(just kidding nowadays you gotta worry about an accidental 10k query)? 3. General Admin - Ryan Howe covered some of this but he recently came up against having to deal with trying to release space on his database, yet after he released it it wasn't fixed. You can read more about it in the comments below. 4. Query History - Jeff Skoldberg referenced this one. Technically you can find your query history often buried in the sys tables on traditional DBS but cloud data warehouses make it so easy. You can easily find query history as well as metadata about the query, how long it took, its query profile, etc. Now I am sure there are other benefits, which I'd love to hear in the comments below, but I am also sure there are people out there who still prefer using solutions like Postgres for their DW(which I'd also like to hear about)!
