BI3 Technologies’ Post

In the fast-paced world of #dataanalytics and #machinelearning, the ability to process and extract insights from massive #datasets has become a critical necessity. Traditional methods often fall short when dealing with large-scale #data, leading to bottlenecks and inefficiencies.  Cick on this link to learn how Apache Spark and Databricks can help you tackle this issue. https://lnkd.in/ggaU3KDk Databricks Apache Spark William Rathinasamy Sekhar Reddy Anuj Kumar Sen Lawrance Amburose Brindha Sendhil Praveen Kumar C Rashika S Parthiban Raja Rajesh S H

Do you know Tabular (now part of Databricks) company? Founded in 2021, that company of about ~30 employees has been acquired last June by Databricks. #Tabular is a data platform that manages and optimizes Apache Iceberg open table format. Its founders are also co-creators of #Iceberg which quickly became an open standard in the fragmented world of #data #lakes. That table format enables companies to move toward an #open #data #Lakehouse strategy in the cloud at a lower cost. #Iceberg allows #data #engineers & #data #scientists to work seamlessly, in multiple process engines (#Spark, #Hive, #Flink...) on a single and same copy of a dataset. Qlik Talend #DataFabric delivers native capabilities with Apache Iceberg on #Spark and #Hive. With Qlik, perform data integration, data quality and analyze your data for critical business needs. For more information go to https://www.qlik.com/us or reach out to start a conversation. #etl #elt #qlik #talend #iceberg #parquet #lakehouse #bigdata #ai #ml https://lnkd.in/eNBYMi5t

Last Friday, it became official.  Databricks acquired Tabular. Tabular is the company behind the popular open-source project Apache Iceberg. The founders, Ryan Blue, Dan Weeks and Jason Reid originally built Apache Iceberg while at Netflix.  They open-sourced the project in 2018 and started Tabular in 2021. You may be asking…”Didn’t Databricks create Delta? Isn’t that a competing data format to Iceberg?” Yes, they did. Delta was created and open-sourced to the community in 2019. So why purchase Tabular? In short, to end the “format wars.” For the last three years, the data engineering world debated the merits of each data format and which one was set to “win it all.” While all three are excellent choices for building a transactional layer on top of your lakehouse, there was a growing fragmentation in the market... Ecosystems were getting built around each format type. And, interoperability was suffering as teams were forced to "place their bets." In reality, for the Lakehouse Architecture to work, it needs to open and interoperable. In fact, that is the founding principle of The Lakehouse and it emerged as a reaction to "walled garden" past most companies find themselves in today. So, last year, Databricks released Delta UniForm. The goal was simple. Write once to Delta, create the metadata for Iceberg and Hudi as well. This allows complete interoperability between all three formats. The Tabular acquisition now gives Databricks the smartest Iceberg team on the planet. Their goal is to make Delta UniForm so good that your data format becomes an abstraction that you no longer spend brain cycles on. Databricks founding team comes from academia. Their background is open-source. Spark, ML Flow, Delta, DBRX, etc are all wildly popular open-source projects getting tens of millions of downloads a month.  And, their goal with Delta UniForm is to keep the community’s trust. Apache Iceberg will continue to be an awesome open-source project and customers can rest easy knowing their data is not "locked up". In fact, you can hear from Ryan Blue this week on Thursday during the keynote at the Data & AI Summit. He will be on stage with Ali Ghodsi, explaining why Tabular decided a partnership with Databricks was the right move.

Choosing the right compute cluster in Databricks   In the world of data engineering and data science, selecting the right compute cluster is a critical decision that can significantly impact performance, cost-efficiency, and the overall success of your projects. Databricks offers several cluster options to facilitate various workloads. Here’s how to make the best choice:   Understand Your Workload Different workloads have distinct requirements: - ETL Jobs: Need robust CPU and memory. - Machine Learning: Benefit from GPU support. - Real-Time Analytics: Require low-latency, high-throughput clusters.   Compute size considerations Selecting the optimal compute cluster in Databricks involves more than just deciding the number of workers. Key factors include: - Total Executor Cores: Determines maximum parallelism. - Total Executor Memory: Dictates in-memory data capacity before disk spilling (we talked about this in a previous post). - Executor Local Storage: Used during shuffles and caching in case of spills   Additional considerations include worker instance type and size. Key questions to guide your configuration: - Data consumption by workload - Computational complexity - Data source and partitioning - Required parallelism   Some cluster types to choose from: - All Purpose Compute: Used for analysis and ad-hoc data engineering and data science development; typically shared clusters, best separated by team or workload, and tend to be more expensive. - Jobs Compute: Runs on ephemeral clusters that terminate upon completion, can be pre-scheduled or submitted via API, designed for single-user use, ideal for isolation and debugging, and generally lower in cost. Recommended for ETL jobs. - SQL Warehouse: Suitable for high-concurrency ad-hoc SQL analytics and BI serving, includes built-in Photon, and offers a serverless option for instant startup. - Instance Pools: Provide ready-to-use instances that reduce start and autoscaling time but should never be used for drivers.   Best practices in general: - Use the latest LTS Databricks Runtime when possible - Use Photon - Use the latest gen VM, start with general purpose then move to more specific options - Enable spot instance on worker nodes   Monitoring after creation: Regularly monitor clusters to identify underutilized resources, bottlenecks, and opportunities for cost savings. Not always we can predefine the right settings for the cluster so fine-tuning after some initial runs is all good.     #Databricks #Data #DataEngineering #ETL #AI #CloudComputing

Databricks just added new capabilities to Databricks Workflows, making it even easier for data engineers to monitor and diagnose issues with their jobs.  The latest enhancements include a Timeline view for job runs, a run events feature to visualize job progress, and integration with #DatabricksAssistant, the AI-powered Data Intelligence Engine.

Databricks just added new capabilities to Databricks Workflows, making it even easier for data engineers to monitor and diagnose issues with their jobs.  The latest enhancements include a Timeline view for job runs, a run events feature to visualize job progress, and integration with #DatabricksAssistant, the AI-powered Data Intelligence Engine.

Databricks just added new capabilities to Databricks Workflows, making it even easier for data engineers to monitor and diagnose issues with their jobs.  The latest enhancements include a Timeline view for job runs, a run events feature to visualize job progress, and integration with #DatabricksAssistant, the AI-powered Data Intelligence Engine.

Databricks just added new capabilities to Databricks Workflows, making it even easier for data engineers to monitor and diagnose issues with their jobs.  The latest enhancements include a Timeline view for job runs, a run events feature to visualize job progress, and integration with #DatabricksAssistant, the AI-powered Data Intelligence Engine.

Databricks just added new capabilities to Databricks Workflows, making it even easier for data engineers to monitor and diagnose issues with their jobs.  The latest enhancements include a Timeline view for job runs, a run events feature to visualize job progress, and integration with #DatabricksAssistant, the AI-powered Data Intelligence Engine.

Databricks just added new capabilities to Databricks Workflows, making it even easier for data engineers to monitor and diagnose issues with their jobs.  The latest enhancements include a Timeline view for job runs, a run events feature to visualize job progress, and integration with #DatabricksAssistant, the AI-powered Data Intelligence Engine.