Sumit P.

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 Comments

JSON Handling and Type Management with Large Datasets in Go -- Ep. 4 is out now! In this episode, Miki Tebeka tackles the complexities of working with JSON data, especially when dealing with extensive datasets and optimizing type management. Here, Miki introduces key strategies for managing large JSON documents, avoiding unnecessary type definitions, and using Go’s encoding/json package to handle custom serialization needs effectively. Things you will learn in this video: ✅ Streamlining JSON Handling: How to avoid excessive type declarations by using anonymous structures and selective decoding for large JSON datasets. ✅Implementing Custom Serialization: Techniques for defining custom JSON serialization in Go with the Marshaler and Unmarshaler interfaces. ✅Managing Mixed-Type Arrays: Strategies to work effectively with JSON arrays containing various data types, ensuring smooth integration with Go’s type system. 📰Watch Now: https://lnkd.in/ghjz58pC Stay tuned for more episodes from Miki Tebeka in this mini series, highlighting Intro to #JSON for Engineers & Managers.🤓 #golang

23

Unlock the Power of Data Lake Table Formats with Alex Merced, Developer Advocate at Dremio! 📊 Join Alex Merced as he delves into "The Who, What, and Why of Data Lake Table Formats." 📅 Save the Date: https://lnkd.in/eFZ6m5HV 🔍 Key Learnings: - Explore how data lake table formats handle large datasets. - Learn how these formats optimize query performance on extensive data. - Understand the importance of data durability and recoverability. - Discover how these formats ensure data is auditable and compliant. Don't miss out on this opportunity to level up your data analytics game! 📈 #DataAnalytics #DataLake #Dremio #TechTalk

4

The classic data lake architecture limits to data silos, poor query performance, and limited real-time capabilities. We just published a new blog on #datalakes On Medium 🚀 "Navigating the Power of Real-Time Analytics with Apica InstaStore" Head over here to learn more: https://lnkd.in/g7izubRN #blog #data #cloud #apica #Medium #instastore #articles #datamanagement #observability

13

Data Engineers -> Kimball Data Consumers -> Metric Trees A simple way to think about metric trees: they have the potential to revolutionize data consumption, similar to how modeling frameworks like Kimball transformed the work of data engineers and BI developers. Read more about this analogy here: https://lnkd.in/eE5KaH9m

1

🐬 Dremio users deal with a lot of different data file and table formats. ⚒️ So Dipankar Mazumdar, M.Sc 🥑 of Onehouse and Alex Merced of Dremio worked together to create a new blog post. 🕵️ In the post, they show how you might use Dremio to smoothly do lakehouse analytics on Apache Hudi and Apache Iceberg data using Apache XTable (Incubating). 0️⃣1️⃣ Check it out - and borrow and revise the code as needed. https://lnkd.in/g-jBvJ55 #apachehudi #XTable #datalakehouse #dataengineering #onehouse #opensource #nolockin

28

3 Comments

So you're thinking of getting into data diffing? You might want to read this first. In our newest blog post, Insung Ko and Elliot G. break down data diffing best practices in Datafold. Learn how data engineers are using Datafold's: - Sampling - Filtering - Monitors as code - Efficient hashing algorithm to manage data quality at scale. https://lnkd.in/gskhZcmQ

12

Back in March, #Databricks launched #DBRX—a super-advanced state-of-the-art open language model that's truly next-level. This new model marks a major step forward for #OpenSource #LLMs. What’s really cool is that DBRX outperforms nearly all established open source models and even rivals #ClosedSource giants like #GPT4, #Gemini, and #Mistral across various benchmarks. Want to know what makes #DatabricksDBRX so awesome? We've got you covered! Check out the full article below to learn everything you need to know about it. https://lnkd.in/g3w845G7

2

PRINCIPAL COMPONENTS ANALYSIS (PCA) Curious about how to make sense of complex datasets? Principal Components Analysis (PCA) is your key to uncovering hidden patterns and reducing data complexity! 📊✨ Discover how PCA can enhance your data analysis and drive smarter decisions. 👉 CLICK LINK IN BIO to dive deeper into PCA and transform your data strategy! #DataScience #PCA #PrincipalComponentsAnalysis #DataReduction #MPRW #KEYS #DataVisualization #BigData #Analytics #DataAnalytics #DataDriven #TechInnovation #Efficiency #DecisionMaking #DataSimplification #DataManagement #DataInsights #MachineLearning #AI #TechSolutions #BusinessGrowth #DataStrategy #WRIRK #DataOptimization #DataPatterns #DataTech #InformedDecisions #RESEARCH

5

Have you ever wondered about the differences between Datafold's Data Diff and #dbt tests? Do data teams really need both? In this post, Elliot G. and Leo Folsom explain: ❓ Why dbt tests prevent some data quality issues, but not all 🤔 How the two tests answer fundamentally different data quality questions 🤝 And why having both in your CI pipeline is essential for complete data quality coverage https://lnkd.in/ggmC97hn

13

1 Comment

🚀 Star Schema vs. Snowflake Schema: Which One Should You Pick? When it comes to designing your data warehouse, your schema choice can really shape how things work. Do you go with the straightforward, fast Star Schema or the more efficient (but complex) Snowflake Schema?  I’ve broken down the differences and when each one makes sense. Trust me, it’s not as complicated as it sounds!  Curious to learn more? Head over to the full blog for all the details.  🔗Check it out here: https://lnkd.in/eQE4Ac2Z #DataWarehousing #StarSchema #SnowflakeSchema #TechInsights #MyceliumWeb  

1

❄ Data Modeling with Snowflake: A concise critical review 🖋️ Author: Chad Isenberg 🔗 Read the article here: https://lnkd.in/eWbD85pk ------------------------------------------- ✅ Follow Data Engineer Things for more insights and updates. 💬 Hit the 'Like' button if you enjoyed the article. ------------------------------------------- #dataengineering #datamodeling #snowflake #data

17

A common misconception with AI and machine learning: “all of the resources, infrastructure and technology" it takes to get started. With Databricks ML Flow, that’s simply not the case. In the video ⬇️ Kenny Shaevel breaks down how Databricks makes it easier to manage and track your models while ensuring full transparency, traceability, and governance—without the headache. Kenny covers everything from: - Simplifying Machine Learning Implementation - Utilizing AutoML for Quick Data Evaluation - The Importance of Feature Engineering - Leveraging Feature Store for Reusability - Governance and Traceability with the Model Registry - Seamless Integration and Application of Models Excited for more? Kenny will be diving even deeper into Databricks in our upcoming live stream series. Keep an eye out for details coming soon! 👀 #machinelearning #databricks #dataanalytics #MLFlow #AutoML #featurestore #dataengineering #techinnovation

5