John Ryan

👏 Opendatasoft joins Databricks partner program 🤝 Opendatasoft today announced its partnership with Databricks, the data and AI company, becoming a Validated Databricks Technical Partner, enabling it to deliver innovative new ways to access data, and help organisations maximise value from their data programs. 🔄️ As a validated Databricks Technical Partner, Opendatasoft has developed an integration connector to enable its customers to seamlessly access and reuse data assets. ""We are focused on accelerating data use to increase productivity, enable better decision-making and reduce risk. This partnership offer users innovative new ways to access and harness information"" - Jean-Marc Lazard, CEO and co-founder, Opendatasoft. 🔎 Read the full press release here: https://lnkd.in/ey2nQP5F #data #dataportal #datademocratization

10

Snowflake released the #SnowflakeArctic embed family of models and made them open source under an Apache 2.0 license. The Massive Text Embedding Benchmark (MTEB) Retrieval Leaderboard shows that the Arctic embed model with only 334 million parameters is the only one to perform better than the average retrieval performance of 55.9. The five models, ranging from x-small (xs) to large (l), can be used right away on Hugging Face. Companies use private datasets along with #LLMs as part of a Retrieval Augmented Generation (#RAG) or #semanticsearch service. The impressive embedding models put into practice the technical know-how, search capabilities, and research and development that Snowflake got from Neeva in May. Snowflake Copilot is integrated into Snowflake Cortex, their #AI service, and can be utilized for a variety of tasks such as question answering, and summarization. With these announcements, Snowflake is breaking ground on open LLMs and showing it is continuously innovating for their customers on the AI journey.  

8

Lakehouse Table Formats interoperability in Databricks. 🎯Scenario: Users in a Databricks environment uses Delta Lake to build ML workflows. However, there are certain scenarios when they need to access datasets stored in other formats like Apache Hudi & Apache Iceberg used by other teams to build robust ML models. So they should have an easy way to read any table format data without having to configure other formats or any other dependencies. ✅ This is where Apache XTable (Incubating) comes into the picture. XTable does a lightweight metadata translation that makes reading any table format as if they were Delta Lake tables. Like: spark.read.format(“delta”).load(“/mnt/mydata/<table_name>”) Read all about the use case and how-to in my blog (link in comments) #dataengineering #softwareengineering

79

5 Comments

Could you please provide some key considerations and best practices to improve streaming jobs performance? This is a super simple question from a learner of my Apache Spark Performance Tuning course (Visit my website link at the top). I have created this course with more than 50+ performance tuning scenarios with TB data volume so my students can master the performance tuning skills. However, I still keep answering many questions from my students because they ask good and interesting questions. Here is my answer to the above question. Here is a list of some critical things for Streaming job optimization. 1. Use Rocks DB as your state store provider. 2. Configure streaming queries to execute in separate scheduler pools. 3. Apply watermarks to limit the state store size. 4. Drop duplicates within the watermark. 5. Set spark.sql.streaming.noDataMicroBatches.enabled configuration to false to avoid running empty microbatches without data. 6. Set shuffle partitions to 2 times the number of cores and enable AQE. 7. Chain multiple stateful operations for a better execution plan. 8. Limit the input rate with maxFilesPerTrigger, maxBytesPerTrigger or maxOffsetsPerTrigger for predictive batch size. 9. Do not use autoscaling for streaming jobs. 10. Prefer DLT for streaming jobs if you need autoscaling and use enhanced autoscaling for DLT. Looking forward to more suggestions in the comment. PS: Master the most critical skill of Spark Performance Tuning with my advanced course. Visit my website for more details.

72

4 Comments

New Enterprise Big Data Engineer official guide is OUT!!! Don't missed it for the world if you are serious about BIG DATA! https://lnkd.in/gn2hnfHw #EnterpriseBigDataFramework #BigData #BigDataEngineer #Data #DataEngineer #EBDF #FreeGuide #Free #EnterpriseData

8

📊 Hear from one of our Open-Source Engineers on unlocking valuable insights and cheap metrics from datasets! 📊 Dive into the world of data efficiency with our latest article on observing Spark aggregates. Discover powerful techniques to optimise your data processes and drive smarter decision-making. https://lnkd.in/eJKbdgnS #OpenSource #DataScience #Spark #Analytics

17

🚀 Top 9 new just out Databricks of March 2024 🚀 🌟 DBRX Base and DBRX Instruct are now available in Model Serving Databricks Model Serving now provides support for DBRX Base and DBRX Instruct, state-of-the-art mixture of experts (MoE) language models developed by Databricks. Both models are included in Foundation Model APIs, where DBRX Instruct is a fine-tuned model accessible in pay-per-token serving endpoint regions and DBRX Base is a pretrained model accessible in limited provisioned throughput serving endpoint regions. For more details, refer to the Use Foundation Model APIs section. 🌟 Provisioned throughput in Foundation Model APIs is GA and HIPAA compliant Databricks Foundation Model APIs now provide general availability for provisioned throughput model serving. Within this release, Foundation Model APIs with provisioned throughput workloads comply with HIPAA regulations. Check out Provisioned throughput Foundation Model APIs. 🌟 The Jobs UI is updated to better manage jobs deployed by Databricks Asset Bundles Because changes to Databricks jobs deployed by Databricks Asset Bundles should only be made by updating the bundle configuration, these jobs are initially read-only when viewed in the Jobs UI. In the past, these jobs could be modified in the UI by default, potentially creating discrepancies between the UI configuration and the bundle configuration. An option is available for emergency changes to a job. Refer to "View and run a job created with a Databricks Asset Bundle" for more information. 🌟 Google Cloud Vertex AI supported as model provider for external models External models supported in Databricks Model Serving now include models from Google Cloud Vertex AI. Check out Model providers for external models. 🌟 Access resources from serving endpoints using instance profiles is GA Add an instance profile to your model serving endpoint to enable access to AWS resources permitted by the instance profile. See how to Add an instance profile to a model serving endpoint. 🌟 Interactive notebook debugging Databricks now offers interactive Python debugging directly within the notebook for clusters in Single User or No Isolation Shared Access mode. Step through code line by line and view variable values to pinpoint and resolve errors. For more details, refer to the Databricks interactive debugger guide. For a comprehensive overview of all the new features and enhancements, please visit this link https://lnkd.in/gEBimsFA Kindly visit our website for more details: www.bi3technologies.com #bi3technologies #bi3 #insight #impact #innovation #dataanalytics #datainsights #powerbi #databricks #snowflakes #microsoft #microsoftpartner #azurecloud #ai #powerapps #dataanalysis #enhancing #collaboration #transformation #datatechnology #Azure #LargeLanguageModels #powerbiupdate #march2024

11

With a rapidly evolving feature set, the popularity of PostgreSQL has been rising steadily over the years. As more and more enterprises adopt PostgreSQL, seasoned Oracle experts are keen to add PostgreSQL to their skill set and explore its potential as a powerful open-source alternative. However, I feel that there aren't enough resources that outline the core differences between Oracle and PostgreSQL to make cross-training easier. That is why my upcoming webinar, "From Oracle to PostgreSQL: Understanding Core Differences for a Successful Transition," dives into exactly this. I plan to tackle the following areas of key differences between the two databases: 1️⃣ Schema 2️⃣ Tablespaces 3️⃣ Procedural Language 4️⃣ Partitioning 5️⃣ Indexing 6️⃣ Concurrency Control 7️⃣ Roles and Privileges If you are an Oracle DBA or developer and want to learn about PostgreSQL, register for this webinar: https://lnkd.in/dXq6P7UX

30

With a rapidly evolving feature set, the popularity of PostgreSQL has been rising steadily over the years. As more and more enterprises adopt PostgreSQL, seasoned Oracle experts are keen to add PostgreSQL to their skill set and explore its potential as a powerful open-source alternative. However, I feel that there aren't enough resources that outline the core differences between Oracle and PostgreSQL to make cross-training easier. That is why my upcoming webinar, "From Oracle to PostgreSQL: Understanding Core Differences for a Successful Transition," dives into exactly this. I plan to tackle the following areas of key differences between the two databases: 1️⃣ Schema 2️⃣ Tablespaces 3️⃣ Procedural Language 4️⃣ Partitioning 5️⃣ Indexing 6️⃣ Concurrency Control 7️⃣ Roles and Privileges If you are an Oracle DBA or developer and want to learn about PostgreSQL, register for this webinar: https://lnkd.in/dXq6P7UX

28

Snowflake Cortex became generally available today, and that deserves a moment, not just for technical reasons. ➡️ It's a set of machine learning algorithms that are directly addressable from within SQL statements. Meaning that we don't need to manage infrastructure and we can orchestrate ML pipelines via dbt for certain common use cases. AI projects all of a sudden become far more achievable than ever. ➡️ It's also the fastest public-preview-to-general-availability cycle I've ever seen at Snowflake. Sridhar - the new CEO - managed to push this through within the first 60 days of his appointment. The announcement follows only days after the release of Arctic - their open-source LLM meeting and exceeding the performance of big-name LLM's on enterprise tasks. I remember the appointment of Frank Slootman as the previous CEO. Almost overnight, the company expanded from "the data warehouse for the cloud" into the "data cloud". What we're seeing now is at least comparable in terms of magnitude. The cautious acquisitions and seemingly incoherent feature releases over the past year, seem to allude to an era where both structured and unstructured search and analysis blend together. And it is still built on the same foundations from 7 years ago. Meaning that we don't need 15 skills to load our petabytes of pdf's, extract text, and use an LLM to have a dialogue and blend text retrieval with answers from structured queries. It just takes SQL and Python. Can't wait to discuss these and other use cases at the Snowflake Summit in SanFran in just a month! #SnowflakeCortex

290

3 Comments

From ETL to ELT - A journey through data integration evolution! This insightful document explores: - The rise and fall of traditional ETL - Modern cloud data warehouses - ELT for successful data integration - Why ELT outperforms ETL - Practical differences between ETL and ELT Gain valuable insights for data professionals and decision-makers alike with this document! _____________ If you want to learn data analytics, consider enrolling in my Business Intelligence course at https://bom.so/oAKSqA or contact via Zalo at 096 148 6648 #DataIntegration #ETLvsELT #CloudDataWarehouse #DataStrategy #BusinessIntelligence #Course

15

‘Indexing' in Lakehouse Table formats. Let's touch upon on one of the capabilities that is core to Apache Hudi i.e. Index. One of the main design principles of #ApacheHudi is faster UPSERTs on #datalake. Now to achieve so, we need to have a sense of where a particular record is. If we don’t, we have to scan through all the records & then make updates, which can be extremely expensive (esp at scale). That’s where Indexing comes in and helps in locating the records. How Hudi & Index works is: ✅ the very 1st time a record comes in Hudi → it is assigned to a File Group (logical grouping of files) & that assignment never changes. ✅ Hudi maintains a mapping between an incoming record's key and the File Group ✅ Index is responsible for finding the record based on the File Group mapping. Therefore, any indexing technique will tell whether a record exist or not based on the mapping knowledge. If it exists, it will tell which File Group it exactly belongs to & that way we achieve faster UPSERTs. Takeaways: 👉 This design enables Hudi to limit the number of records that must be merged with each individual base file. 👉 So, a particular base file only needs to be merged with updates related to the records it contains. 👉 Table formats lacking an indexing component may require merging all base files with every incoming update or deletion record, which would be much less efficient. Here is a simple illustration that explains the Total Merge Cost for four 100 MB #Parquet files with & without indexing. Detailed read in comments. #dataengineering #softwareengineering

72

1 Comment

In today's data-driven world, the key to unlocking your business's full potential lies in smarter data management. At DB Results, our Data Management Services are designed to empower organisations with solutions that maximise the value of their data. Our proven framework ensures that your data strategy aligns perfectly with your business objectives: Discovery: We start by assessing your current data performance and exposure, crafting a tailored plan for efficient data management. Planning: Our experts work with you to develop a comprehensive strategy, ensuring that every byte of your data is utilised to its fullest potential, while maintaining robust governance practices. Implementation: From technical capabilities to UX/UI design, we deliver end-to-end solutions tailored to your business needs. Support: Our commitment doesn't end with implementation. We provide ongoing support, operations, and enhancement services. Ready to harness the power of smarter data management? Learn more at https://lnkd.in/gPhhkrEJ #DataManagement #BusinessTransformation #DBResults #BCorp

6

Metadata Indexing in Apache Hudi for faster performance. Data skipping is one of the common techniques used with large volumes of data to achieve better query performances. The whole idea is simple - read as less data as possible! What that means is that as a compute engine paired with a lakehouse platform like Apache Hudi it should read only the data files that are needed to satisfy the query from the storage. Data skipping not only reduces the volume of data that needs to be scanned & processed, but it can also lead to substantial improvements in execution time. Of course this is made possible because of the metadata information provided by file formats like #Parquet. Basically, each Parquet file contains the min/max values of each column along with other useful info such as: num of NULL values. These min/max values are 'column statistics'. Now, although, we could directly leverage these stats for data skipping, this could affect the query performance because the engine still has to go through each file to read the footer. ❌ This process can be very time taking specifically with large data volumes. What's Hudi's approach? ✅ Hudi adds a next level of pruning in this case. It basically takes all these column statistics & collates the info in the form of an INDEX. ✅ Indexes such as this (& more) are incorporated into Hudi's internal metadata table so engines can just get the files where the data is stored. Therefore, instead of reading individual Parquet footers, compute engines can directly go to the metadata table's index & fetch the required files. Way faster. For e.g. in this image, there are 2 parquet files that has well-defined value ranges for each of its columns. - File1.parquet contains 'Salary' ranges in $10000-40000 - File2.parquet contains 'Salary' ranges in $45000-90000 Now, if we run a query to fetch all the records where Salary=30000, the engine (Spark) can fetch the records from only File1 as it has values within the range & skip the other one. Imagine doing this at scale (with large number of files)! #dataengineering #softwareengineering

88

2 Comments

Unity Catalog DataBricks!! 1. Centralized Governance : It provides a unified governance solution across all data and assets in a Databricks environment, ensuring consistent data access policies. 2. Fine-grained Access Control : Unity Catalog allows you to apply permissions at the table, view, and column level, enabling more precise control over who can access specific data. 3. Simplified Data Lineage: It automatically tracks data lineage across tables, columns, and notebooks, which helps in auditing and understanding data flow. 4. Cross-Workspace Data Sharing : It allows seamless data sharing between different Databricks workspaces, making collaboration easier without compromising security. 5. Support for Multi-cloud : Unity Catalog works across multiple cloud platforms, providing flexibility in managing data across cloud providers. 6. Integration with Lakehouse Architecture : It enhances the Databricks Lakehouse platform by unifying data management and governance across data lakes and data warehouses. These features help streamline data governance and compliance while enhancing productivity for data teams.

12

https://zurl.co/0anq Understanding Index Selection Mechanics in PostgreSQL: How the Query Planner Optimizes Performance #PostgreSQLTips #DBATips #SQLTips #OpenSource

5