Jackson Isaac

Previously we introduced Getting started with Azure Cosmos Database (A Deep Dive) blog which is an end-to-end introduction of Azure Cosmos DB. In this blog we are going to talk about one of the Azure Cosmos Database API known as Apache Gremlin.  Graph databases are a powerful way to model and query complex relationships in your data. Let’s dive into working with Azure Cosmos DB for Apache Gremlin, which provides a graph database service.    What's covered in this blog - Need for a Graph database - Challenge: Mapping Relationships and Insights - What is Apache Gremlin?  - Azure Cosmos DB for Apache Gremlin - Benefits for Gremlin API - Common scenarios for Gremlin API - Introduction to graph databases - provisioning Azure Cosmos DB for Apache Gremlin via Azure portal  - Modeling a graph Database https://rb.gy/ngzmhj #AzureCosmosDB #AzureCosmosDBGremlinAPI #GraphDB #MLSA #NoSQLDB Computer Society Of Kirinyaga

17

Trying out the PostgreSQL pg_tde Tech Preview Release "Exciting news for all my fellow DB geeks out there – the tech preview version of PostgreSQL pg_tde is finally here! 🎉 With tons of improvements and bug fixes, this release is definitely worth trying out. But wait, there's more – the new pg_tde supports multi-tenancy and key rotation with a simplified configuration mechanism. Talk about taking database security to the next level! 🔒 Just a heads up, this Tech Preview release is not recommended for production (sorry to rain on your parade 🌧️), but it's a great opportunity to test out the new features and provide feedback to the original author, Percona. So, dear PostgreSQL and MySQL gurus, let's dive in and see what this bad boy can do! 💪 And don't forget to share your thoughts and findings with the community. Happy testing! 🚀 #PostgreSQL #pg_tde #databasegeek #techpreview #keyrotation" https://lnkd.in/eJQqzMnr

6

1 commentaar

Speeding up index creation Let's talk about speeding up index creation in PostgreSQL. If you've ever had the pleasure of waiting for hours while your database churns away at creating indexes on a large dataset, you know the feeling. Fortunately, Hans-Jürgen Schönig has some insights that may help you cut that time significantly. Indexes are crucial for optimal query performance. But creating them can feel like watching paint dry. Thankfully, there are techniques floating around that can make the process a bit less torturous. Speaking of insights, PGConf.dev 2024 had a lineup of talks worth tuning into if you missed it. Topics ranged from autovacuum mishaps to scaling databases at petabyte levels. Definitely a treasure trove for any PostgreSQL enthusiast. Also, if you fancy learning some magic tricks for psql, Elizabeth Christensen shares some nifty settings and presets that can make your life easier. So, whether you're indexing or attending conferences, the PostgreSQL community is here to keep you entertained and informed. Cheers to faster queries and smarter database management! Source: https://lnkd.in/e8SSrvJd

2

🛠️ Build powerful applications with the integration of #AzureCosmosDB NoSQL and #OpenAI! Sandeep Nair demonstrates how to leverage vector embeddings for seamless multi-modal data operations. Watch now: https://lnkd.in/ezx6s8yu

2

When we talk about #DataIntegration there are multiple patterns In this video, Jeroen van Disseldorp explains what patterns you can leverage. Watch the full webinar here: https://lnkd.in/eTDpRE7P #EventDrivenArchitecture #ApacheKafka #Axual

7

𝐃𝐨 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐮𝐧𝐥𝐨𝐜𝐤 𝐭𝐡𝐞 𝐟𝐮𝐥𝐥 𝐩𝐨𝐰𝐞𝐫 𝐨𝐟 Databricks 𝐟𝐨𝐫 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚? Then check out our new “Databricks in a Week” workshop! In this 5 day workshop we will (optionally together with you): ✅ rapidly prototype and validate your data-driven product or solution ✅ showcase the Databricks technology hands-on with your data in your Databricks account ✅ demonstrate feasibility and value to your stakeholders Sounds good? Then head on over to our website and get in touch with our team! 👉 https://lnkd.in/dxHyFgfK What is your biggest challenge with Databricks at the moment? Let us know in the comments and let’s discuss! #databricks #data #DataIntelligence #DatabricksInAWeek

36

1 commentaar

Exciting News! 🚀 I'm happy to announce that Data Catering supports the Open Data Contract Standard (Bitol) for testing data pipelines! Why is this important? 🤔 Implementing the Open Data Contract Standard ensures that data pipelines are robust, transparent, and interoperable. This standard, in combination with Data Caterer, empowers testers, developers, data engineers, and organizations to: 🔹 Simplify Testing: Use contract metadata to generate and validate data 🔹 Boost Efficiency: Streamline testing and validation processes, reducing development cycles 🔹 Improve Reliability: Detect issues before they reach production 🔹 Be Secure: No production data or connections required, fully metadata-driven At Data Catering, we believe in the power of open standards to drive innovation and collaboration. By supporting this initiative, we're not only enhancing our own services but also contributing to a more open and reliable data ecosystem. Join us in embracing this standard and take your data pipeline testing to the next level. https://lnkd.in/gQFJG4RQ

33

3 commentaren

🚀 DE Interview 📢 Cloud Storage Optimization ⌛ 1-Min-Read Optimizing Cloud Storage Using Spark With Small Files Fix Script 👉🏽 🧔🏽‍♂️Interviewer: Can you explain how to optimize cloud storage using Spark with a small files fix script? 👨‍💻Candidate: 1️⃣ In cloud environments, handling a large number of small files can lead to inefficiencies, such as increased storage costs and degraded processing performance. This issue is common in big data processing pipelines where data is frequently ingested in small increments. 2️⃣ To address this, I developed a Spark script that consolidates these small files into larger, more manageable ones. The process starts by identifying the small files in the cloud storage, usually on platforms like AWS S3. 3️⃣ The script then reads these files and combines them using Spark's distributed processing capabilities. The merged files are written back to cloud storage, significantly reducing the number of small files. 4️⃣ This optimization not only cuts down storage costs but also improves the performance of downstream data processing jobs, as larger files are more efficient to read and write. 🧔🏽‍♂️Interviewer: Why is reducing the number of small files important? 👨‍💻Candidate: 1️⃣ Small files increase storage overhead because each file has metadata that consumes space. Over time, the accumulation of small files can lead to higher costs. 2️⃣ Additionally, when processing large datasets, small files can cause inefficiencies because distributed processing frameworks like Spark are optimized for larger file sizes. This can lead to slower job execution and higher resource consumption. 🧔🏽‍♂️Interviewer: How did you implement this in a cloud environment? 👨‍💻Candidate: 1️⃣ I implemented the solution in AWS S3, using Spark running on an EMR cluster. The script was automated to run periodically, ensuring that small files were regularly consolidated. 2️⃣ Versioning in S3 was enabled, so original files were kept intact, ensuring data safety while optimizing storage. 3️⃣ This approach led to a significant reduction in storage costs and improved the efficiency of our data pipelines. Summary: 1️⃣ Problem: Inefficiencies and high costs due to a large number of small files in cloud storage. 2️⃣ Solution: Developed a Spark script to consolidate small files, reducing their number and optimizing storage. 3️⃣ Impact: Lower storage costs and improved performance of data processing jobs. #DataEngineering #CloudOptimization #Spark #AWS #S3 #BigData #CostEfficiency #TechInnovation #CloudStorage #DataProcessing

20

2 commentaren

🤖 Should you use Databricks for ML? We take a look in our new Medium blog where we created a proof of concept integrating Databricks with GoodData's semantic layer for machine learning. See what we learned: https://hubs.ly/Q02PQ0f40

3

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 commentaren

Here's a blog about using the Extract Data data source in #lookerstudio. An interesting exercise in data refinement. https://lnkd.in/e-X4KMQt

6

Take that Azure Cosmos DB powered app to prod with the help from Sandeep and Saranya! Watch the three part video series: "Mastering #AzureCosmosDB: A Comprehensive Guide from Prototype to Production." https://lnkd.in/eEMt5TFG

1

Let's transform raw page view data from Apache Kafka into aggregated insights directly loaded into Databricks Delta tables. 🧱 We explore real-time data aggregation, demonstrating how to downsample page view data across a 10-second time window using just a few SQL queries. By processing data before it reaches Databricks, you can dramatically reduce storage and compute costs while minimizing latency. It's all here 👇 https://lnkd.in/g7QwbSUD

3

"Evaluating LLM frameworks" by Ennia Suijkerbuijk. Nuggets: It is very important to establish a framework for evaluating your RAG applications. The framework should evaluate your solution on metrics relevant to your use case. Relevance, Faithfulness, Semantic Similarity are some metrics that can be used. RAG solutions are better if model agnostic.

6

A cool use of the “contribution analysis” ML model to answer the “why” kind of business questions. Combined with Gen AI, it enables quick insights for any enterprise.

7

🚀 Optimize your #AzureCosmosDB performance by distributing throughput across partitions. Check out this detailed guide to master the process. Dive in here: https://lnkd.in/d-jKrWcW #Azure #Cloud

3

New Blog: Using Apache Hudi & Iceberg tables in Databricks with Apache XTable (Incubating) 🎉 In this hands-on blog, I show how we can use XTable to interoperate among the 3 open table formats within Databricks environment. I explore a use case where a team of data scientists run ML experiments using MLflow on top of Hudi/Iceberg tables by consuming these tables as Delta Lake. Some key points to consider & how XTable helps: ✅ While predominantly focused on Delta tables, this team also would like to access Apache Hudi/Iceberg datasets from other teams to enhance their ML models. ✅ In the past, the data platform team has either duplicated these datasets to rewrite them as Delta Lake or conducted one-off migrations (costly). ✅ Data Platform team's goal: the ability to read table formats other than Delta, should be seamless & dependency-free, without the need for additional configurations. ✅ Irrespective of whether the source table is Hudi, Delta Lake or #ApacheIceberg, users should just be able to read the dataset as if it were a Delta Lake table using spark.read.format(“delta”).load(“table_name”) Link in comments. #dataengineering #softwareengineering

123

6 commentaren

Our featured article this week is a brilliant insight into microbatching in dbt by Christophe O. Give it a read, and check out some of the awesome other content 🔗 https://lnkd.in/eeTQaFg7 #dataengineering #analytics #orchestra

6

Is there maybe a similar article, but about how to transition from data engineering to software engineering? Because I have a strong feeling that I have written more YAML/SQL than Python/Scala/Java in the last year... Maybe one finds DE interesting, but for me writing all these YAMLs/JSONs/SQLs is so boring. The idea of "let's speed up your DE development with just-another-low-code-tool" may sound cool for the business, but not for me. I still like programming, not YAMLing :) https://lnkd.in/dEjeNtgY

26