Girish Venugopalan Nair

Avoid VPA Pitfalls and Optimize Kubernetes Resource Allocation 📉 VPA is powerful, but it has limitations with garbage-collected languages like JVM, stateful applications, and specific workload types. In Daniele Polencic's white paper 'A Practical Guide to the VPA', you'll discover: - The art of setting optimal CPU and Memory requests/limits - Overcoming challenges with DaemonSets and StatefulSets - Configuring VPA modes: Recommender vs. Updater Download your copy today! https://hubs.la/Q02xWY2g0 #Kubernetes #k8s #VPA #HPA #devopstools #devops #learnk8s

9

Our expert Niklas Siltakorpi has been working with #Kubernetes and #Openshift for 7 years already. In this article, he shares his personal views on what you should consider when thinking about moving to Kubernetes. https://lnkd.in/dS3fik5M #platformengineering

7

Kafka Architecture Explained In A Nutshell 𝐃𝐞𝐯𝐎𝐩𝐬 𝐅𝐑𝐄𝐄 𝐖𝐞𝐛𝐢𝐧𝐚𝐫 & 𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬 https://lnkd.in/dmB4EgdR 🔹 Producers: These are the data sources sending logs, events, and other data to Kafka topics. They act as the entry point into the Kafka ecosystem. 🔹 Topics: Think of topics as channels or categories where data streams are stored and published. Each topic can have multiple partitions to enhance scalability. 🔹 Consumers: These components subscribe to topics and process incoming data. They can handle real-time data streams to drive insights and actions. 🔹 Partitions: Each topic is divided into partitions, which are units of scalability and parallelism. This allows for concurrent data processing and better load balancing across consumers. 🔹 Brokers: Brokers are the servers that store data and deliver messages. They manage the data distribution across the Kafka cluster, ensuring high availability and fault tolerance. DevOps FREE Webinar & Resources https://lnkd.in/gqQtqnMQ 🔹 Replicas: To prevent data loss, Kafka maintains replicas of each partition. These backups are crucial for ensuring data durability and consistency in case of a server failure. 🔹 Zookeeper: Zookeeper is the manager of the Kafka cluster. It keeps track of the state of nodes, topics, and partitions, ensuring everything runs smoothly and efficiently. 🔹 Push/Pull: Producers push data into Kafka topics, while consumers pull data at their own pace. This decoupled model allows for flexible and efficient data processing. credit: Omkar Srivastava #devops #engineering #softwareengineer #kubernetes #docker #Kafka

56

2 kommentarer

Watch "Getting Started with Apache Paimon" and unlock the potential of #ApachePaimon! Paimon is an open-source streaming lakehouse platform that is a powerful tool for building real-time data pipelines and applications. It is a good choice for use cases that require high-speed data ingestion, change data tracking, and efficient real-time analytics. Benefits of using Paimon include: ✔️Reduced complexity: Paimon provides a unified platform for batch and streaming data processing, which can reduce the complexity of your data pipelines. ✔️Real-time insights: Paimon can be used to build real-time data pipelines that provide insights into your data as soon as it is generated. ✔️Scalability: Paimon is a scalable platform that can handle large volumes of data. Watch the video now to learn more! https://bit.ly/4bxs3Px

14

Watch "Getting Started with Apache Paimon" and unlock the potential of #ApachePaimon! Paimon is an open-source streaming lakehouse platform that is a powerful tool for building real-time data pipelines and applications. It is a good choice for use cases that require high-speed data ingestion, change data tracking, and efficient real-time analytics. Benefits of using Paimon include: ✔️Reduced complexity: Paimon provides a unified platform for batch and streaming data processing, which can reduce the complexity of your data pipelines. ✔️Real-time insights: Paimon can be used to build real-time data pipelines that provide insights into your data as soon as it is generated. ✔️Scalability: Paimon is a scalable platform that can handle large volumes of data. Watch the video now to learn more! https://bit.ly/3V6BtLk

4

This article explores the Kubernetes API groups and versions, highlighting the role of core and named groups in cluster management. More: https://lnkd.in/dXXHaUU6

1

Discover the Cutting-Edge Consensus Algorithms in Apache IoTDB! In the last part of the #distributed architecture trilogy, we explore: 🔍 The unique unified #consensus protocol framework of IoTDB. ⚙️ How users can flexibly choose different consensus algorithms based on their needs. 💡 The innovative IoTConsensus algorithm tailored for IoT scenarios, offering high performance and availability. Compared to other #timeseries #database, Apache IoTDB is the first and only system to propose and apply a unified consensus protocol framework. Users can flexibly choose different consensus algorithms based on performance, #availability, #consistency, and storage cost requirements. We also introduce the upcoming FastIoTConsensus, which enhances performance and availability by leveraging Apache TsFile for data synchronization. 👉 Read the full article here: https://lnkd.in/eHuc-gY2 🚀 Stay tuned for the release of FastIoTConsensus and more exciting developments! #timeseries #database #iot #opensource #distributedsystems #consensusalgorithms

4

Webinar: Redis Alternatives Compared 😎 🔗 https://lnkd.in/ebNTKA3z Peter Zaitsev will provide analysis of: 🔸 Emerging Redis forks like Valley and Redix 🔸 Established contenders like DragonflyDB and KeyDB 🔸 Unique options like Microsoft Garnet and Redka Also, this deep dive covers: 🔸 Licensing models and implications 🔸 Feature sets and functionality comparisons 🔸 Governance and community support structures 🔸 Performance considerations for various use cases

2

How does Apache XTable Works? Apache XTable (Incubating) provides an abstraction to translate metadata for different #lakehouse table formats. But how does it work internally? ✅ XTable utilizes a ‘standard model’ for table representation via lightweight abstraction layers. ✅ It reads a source table metadata, extracts it into a universal format that can be synchronized with one or more target table formats. ✅ This integrated model has the capability to understand & translate various elements, including schema details, partitioning information, & file metadata, such as statistics at the column level, row count, and size. ✅ After translation, the metadata is stored within a directory located in the base path of your table (_delta_log for #Deltalake, metadata for #ApacheIceberg, and .hoodie for #ApacheHudi). This enables you to use the desired query engine for your workloads. For e.g. Hudi users can use Databricks Photon or query Iceberg tables with Snowflake. It is important to note that there is no data rewrite involved in the process. The translation processes only touches the metadata. We will break down the inner workings of XTable at the upcoming Data+AI Summit 2024. #dataengineering #softwareengineering

161

3 kommentarer

Learn how to use common table expressions, subqueries and joins to perform complex SQL and vector search queries in MyScaleDB. #Database #DataEngineering #SQL by Jianmei Zhang thanks to MyScale

1

1 kommentar

Like yesterday, we've gained a lot of inspiration from the talks at Kubernetes Community Days (KCD) Denmark today, and we're ready to fuel future initiatives within our Contain services. The topics today have also centered around Cloud Native and Kubernetes, and Jonas, a consultant in our Cloud Native team, has shared his reflections on the second day of KCD: "𝘞𝘩𝘢𝘵 𝘐 𝘧𝘪𝘯𝘥 𝘱𝘢𝘳𝘵𝘪𝘤𝘶𝘭𝘢𝘳𝘭𝘺 𝘧𝘢𝘴𝘤𝘪𝘯𝘢𝘵𝘪𝘯𝘨 𝘢𝘣𝘰𝘶𝘵 𝘊𝘳𝘰𝘴𝘴𝘱𝘭𝘢𝘯𝘦 𝘪𝘴 𝘪𝘵𝘴 𝘢𝘣𝘪𝘭𝘪𝘵𝘺 𝘵𝘰 𝘭𝘦𝘵 𝘶𝘴 𝘶𝘴𝘦 𝘰𝘶𝘳 𝘒𝘶𝘣𝘦𝘳𝘯𝘦𝘵𝘦𝘴 𝘤𝘭𝘶𝘴𝘵𝘦𝘳 𝘵𝘰 𝘤𝘰𝘯𝘧𝘪𝘨𝘶𝘳𝘦 𝘴𝘶𝘳𝘳𝘰𝘶𝘯𝘥𝘪𝘯𝘨 𝘪𝘯𝘧𝘳𝘢𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦 𝘥𝘪𝘳𝘦𝘤𝘵𝘭𝘺 𝘸𝘪𝘵𝘩𝘪𝘯 𝘒𝘶𝘣𝘦𝘳𝘯𝘦𝘵𝘦𝘴. 𝘛𝘩𝘪𝘴 𝘮𝘦𝘢𝘯𝘴 𝘸𝘦 𝘤𝘢𝘯 𝘭𝘦𝘷𝘦𝘳𝘢𝘨𝘦 𝘎𝘪𝘵𝘖𝘱𝘴 𝘵𝘰 𝘥𝘦𝘱𝘭𝘰𝘺 𝘤𝘰𝘮𝘱𝘭𝘦𝘵𝘦 𝘴𝘦𝘵𝘶𝘱𝘴 - 𝘧𝘳𝘰𝘮 𝘴𝘦𝘳𝘷𝘪𝘤𝘦𝘴 𝘸𝘪𝘵𝘩𝘪𝘯 𝘒𝘶𝘣𝘦𝘳𝘯𝘦𝘵𝘦𝘴 𝘵𝘰 𝘯𝘦𝘵𝘸𝘰𝘳𝘬 𝘤𝘰𝘯𝘧𝘪𝘨𝘶𝘳𝘢𝘵𝘪𝘰𝘯𝘴 𝘪𝘯 𝘤𝘭𝘰𝘶𝘥 𝘱𝘳𝘰𝘷𝘪𝘥𝘦𝘳𝘴. 𝘞𝘪𝘵𝘩 𝘵𝘩𝘪𝘴 𝘤𝘢𝘱𝘢𝘣𝘪𝘭𝘪𝘵𝘺, 𝘥𝘦𝘱𝘭𝘰𝘺𝘪𝘯𝘨 𝘦𝘯𝘷𝘪𝘳𝘰𝘯𝘮𝘦𝘯𝘵𝘴 𝘧𝘰𝘳 𝘤𝘶𝘴𝘵𝘰𝘮𝘦𝘳𝘴 𝘣𝘦𝘤𝘰𝘮𝘦𝘴 𝘢𝘴 𝘴𝘵𝘳𝘢𝘪𝘨𝘩𝘵𝘧𝘰𝘳𝘸𝘢𝘳𝘥 𝘢𝘴 𝘴𝘦𝘭𝘦𝘤𝘵𝘪𝘯𝘨 𝘵𝘩𝘦 𝘳𝘪𝘨𝘩𝘵 "𝘣𝘶𝘪𝘭𝘥𝘪𝘯𝘨 𝘣𝘭𝘰𝘤𝘬𝘴." 𝘐𝘵 𝘴𝘵𝘳𝘦𝘢𝘮𝘭𝘪𝘯𝘦𝘴 𝘵𝘩𝘦 𝘱𝘳𝘰𝘤𝘦𝘴𝘴, 𝘮𝘢𝘬𝘪𝘯𝘨 𝘪𝘯𝘧𝘳𝘢𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦 𝘮𝘢𝘯𝘢𝘨𝘦𝘮𝘦𝘯𝘵 𝘮𝘰𝘳𝘦 𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘢𝘯𝘥 𝘢𝘤𝘤𝘦𝘴𝘴𝘪𝘣𝘭𝘦". Thanks for two incredible KCD-days! 👏 Reach out to our dedicated Contain-team if you want help to optimise your Cloud Native setup: https://lnkd.in/gezaB3t2 #TriforkFamily #kcd #kcddenmark #kcddenmark2024 #cncf #CloudNativeNordics #contain #Kubernetes #CloudNative

16

We're sharing our journey implementing StringView in Rust for Apache Arrow and integrating it into Apache DataFusion. The result? String-intensive queries are now 20%-200% faster in the ClickBench benchmark! ⚡ https://bit.ly/4cYNOI9 #ApacheArrow #DataFusion #Rustlang #InfluxDB

9

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 kommentarer

I always follow most of the episodes released by Lukas Fittl as part of "5 mins of Postgres". It's great to have an episode featuring one of my blogs, which was the result of performance improvements recommended in the #PostgreSQL Slack channel. If you are a database or performance enthusiast, do check out the episode explaining the overall problems with more depth and explanation.

23

Kubernetes + Postgres = CloudnativePG <3

3