Greg Kroleski’s Post

I like Databricks' dynamic resource allocation; I think it is great. However, I would be cautious when a developer sets the maximum worker nodes to 20. You risk wasting resources and spending a lot of money because the allocation and deallocation of resources are not simple processes behind the scenes. Often, it takes a while to deallocate the newly added executors, leading to overpayment for low utilization. A quick look at the Metrics UI can provide insight into this issue. Instead, try to resize your cluster compute to a larger size, ensuring you do not exceed a maximum of 8 worker nodes in your upper limit setting. #databricks #dataengineer

Criticism is hard to give, and even harder to receive... 1) Shout-out to Chao Cai and his team at Databricks. Last week, I wrote a post praising other Databricks elements while criticizing the current state of "Lakeview Dashboards" on Databricks. I was blunt in my post and he reached out wanting to hear my feedback. Hopped on a call with his team today, and I gave even more blunt feedback. His team was very receptive of my feedback, and Chao's personal message to me after the our call was very special: "My overarching goal is to get us to the point where you’d consider dashboards to be as good as the rest of Databricks". That right there is pride of ownership, and look forward to seeing his team continue to develop the tool more and more. This type of response has been consistent across the Databricks product team, and one of the reasons why I am very confident on their overall product. 2) About a week ago, I gave my boss Matt Ryan some constructive criticism. Instead of being defensive, he told me that my feedback was well-received by him, and something that he is personally working on to empower our team Kythera Labs. This willingness to receive feedback has been the theme at Kythera Labs, and one of the reasons why I am also so confident in the value that we deliver to our clients.

One key insight I gained today about Databricks FinOps is that while we focus on optimizing existing pipelines and reducing costs, we also need to consider whether some pipelines should even be running at all. Business needs change, and that complex pipeline you built two years ago, refreshing data every hour, might no longer be relevant. In some cases, no one is consuming that data anymore, or maybe it’s only needed once a week. This presents an opportunity to either deactivate those pipelines or adjust their schedule to a more appropriate frequency. In Databricks, you can easily identify these scenarios by querying system tables, like system.query.history, which provides valuable insights into pipeline usage. For more details, check out the documentation https://lnkd.in/gxX6mc6n

Breaking records! 🚀 Databricks secures $5 billion in funding, skyrocketing its valuation to $55 billion. This move aims to enhance employee liquidity and fuel innovation in the data-driven world. 💡📊 Databricks Raises $5 Billion in Funding Round Valued at $55 Billion to Provide Employee Liquidity View full news : https://bit.ly/3Z47eqi #Databricks #FundingRound #TechInnovation #DataAnalytics #Valuation #TechNews #BusinessGrowth #EmployeeBenefits

Databricks recently introduced a feature that I've been eagerly awaiting - ForEach loops in a jobs, opening the door to creating metadata-driven pipelines. Imagine having a single generic Databricks notebook, shared across multiple pipelines. With For Each Loops and parametrized job tasks, you can now control your operations simply by modifying your configuration. No more adding new tasks and redeploying your job whenever you need to add a new entity. That said, I'm looking forward to even more - an ability to set up multiple tasks within a For Each loop, as currently it supports only one. Check out the image below to see how it works in action! #databricks #dataengineering #databricksjobs #databricksnews

🚀 Excited to share that I recently had the opportunity to present to my cross-functional and engineering teams on how we've revolutionized our data infrastructure! 🚀 In my presentation, I walked through our journey of streamlining all our data workflows from GCP to Databricks. This integration has significantly enhanced our workflow, bringing efficiency, reliability, and scalability to our data processes. Key highlights of the presentation included: 1. Handling Data Infrastructure: Our approach to managing data infrastructure ensures seamless data flow and robust performance. We've built workflows that efficiently process and analyze large datasets, enabling data-driven decision-making across the organization. 2. Building Workflows: We demonstrated how we design and build workflows that are both efficient and scalable. By leveraging Databricks and CI/CD practices, we've created an environment where data processing is automated and reliable. 3. Ensuring Platform Uptime: Implementing best software practices within our data team has been crucial in keeping our entire data platform up and running. From rigorous testing to continuous monitoring, we've prioritized stability and performance. 4. Strategic Migration Benefits:The migration to Databricks has brought numerous benefits, including improved collaboration, faster processing times, and enhanced data security. 5. CI/CD Setup for Databricks Notebooks:Our CI/CD setup for Databricks notebooks using Databricks Asset Bundles and Github Actions has transformed our development and deployment cycles, ensuring smooth and error-free updates. A big thank you to everyone involved in making this transition a success! Your collaboration and dedication have been instrumental in achieving this milestone. If anyone is interested in discussing how we've built our infrastructure, let's grab a coffee and chat! Feel free to DM me. ☕️ Looking forward to many more innovations and successes together! #DataInfrastructure #Databricks #DataPipeline #TechInnovation #Teamwork

Are you starting a new project on #Databricks? You should not just open a Databricks notebook and do all the development there... 💰 it is a well-known way to burn money without good reason. 🐞 You can not debug in a Databricks notebook as you can in an IDE. 💊 Databricks notebooks do not make it easy to follow best practices: write modules, package your code, and write tests. Instead, develop locally (with data pulled locally) in an IDE! A couple of considerations: ➡ You will need a docker environment with Delta Lake and Spark to make it work. ➡ Use open-source Delta Lake and Spark features. Do not use dbutils. The less proprietary features you use, the more control you have to develop your pipelines using "normal" practices. ➡ You need an environment that 100% mimics the production environment to test your pipelines on Databricks. I came across an awesome article in the Seattle Data Guy newsletter, where Daniel Beach, a guest writer, explains this approach. I agree 100% with everything he says! Give it a read: https://lnkd.in/eUpDBrdX #dataengineering #devops #mlops #machinelearning

Databricks Assistant Databricks is at the core of most of my projects. For my team and me, working with Databricks feels more like a hobby than a job. One of the most amazing possibilities and features is the Databricks Assistant. It helps you become more productive in less time. Recently, after submitting a project, the customer told us, "I am absolutely amazed at how you delivered the project in such a short time!" Well, shout out to Databricks Assistant for making it possible! https://lnkd.in/e44DHGAC #databricks #dataai #advisory #datafoundation #dataarchitecture

With great power ( of the Cloud) comes responsibility !! Great blog from my team on cost and compute management on Databricks. If you have challenges or just getting started on the journey, please DM me or anyone on my team. Great work team Santiago Carrera Allen M. Becker, MBA, GB Rong Z. Jeronimo Acosta Josue A. Bogran on the collaboration on this blog https://lnkd.in/g77VunmZ #Databricks #CostOptimization #DataAISummit #DataEngineering