Saiyam Pathak

What is coming to PostgreSQL? According to this interesting article, these two big things. - Separation of Compute and Storage Meaning you will able to indepentily scale computing by adding new compute nodes, independely of the storage. Compute part is that logic that you definitely shouldn't be using in database /s - Pluggable Storage and Improved Table Engines You will be able just to plugin different storage types per table. For example column storage, graph storage, etc... It isn't farfetched to say that PostgreSQL could be the database that will end all other databases. At least it has that potential. So, get on board while you still can.

45

10 Comments

#InnoDB, #MySQL's default storage engine, enhances performance on multi-CPU systems by supporting parallel query execution. This feature splits queries into sub-tasks, processes them concurrently across threads, and combines results for faster execution. #MySQLPerformance @MinervaDB #dba https://zurl.co/1txh

1

Great article regarding Batch Processing Gateway (https://lnkd.in/gGjVCWWe) which we built previously: https://lnkd.in/gBAskpNC

34

2 Comments

How we solved Docker rate limiting issues on EKS and saved thousands of dollars in NAT Gateway costs ? Recently, we faced a challenge where we were running into two major issues for a batch processing workload for one of our clients. Here is an explanation of the scenario. We had to spin up hundreds of containers on demand on EKS for batch processing. 1. Hundreds of jobs were started at the same time. 2. The image used for batch processing was large in size ~ about 2.5 GB.  3. Dockerhub was used to store the images. 4. This activity of batch processing happens hundreds of times a day resulting in hundreds of containers being spun up every day. Here are the two challenges we were facing: 1. Dockerhub Rate Limits - We were hitting rate limits on pulling images from Dockerhub. Dockerhub allows 200 pulls per 6 hour period for Authenticated Users.  2. High NAT Gateway costs - It was causing a very high amount of traffic ingress through the NAT gateway which was causing huge NAT gateway bills. Even though AWS and GCP charge only for data transfer out, a NAT gateway charges for data flow in either direction. We had to stay on Dockerhub for the following reasons: 1. We had to support customers in multiple regions.  2. We also had to spin up the containers on Google Kubernetes Engine (GCP). Hence it made sense for our CI/CD to push images in one place. AWS ECR Pull through cache to the rescue AWS ECR Pull Through Cache is a feature provided by ECR that allows you to pull images from Dockerhub via ECR. It caches the images once the image is pulled from Dockerhub and continues to serve it from there. Lets say you have an image in Dockerhub that lives in the `<organization/repository>`. To leverage the ECR Cache, the only change that you need to do is 1. Create a Pull through cache rule in ECR.  2. Instead of pulling images from `<organization/repository>`, you need to start pulling images from `<accountnumber>.dkr.ecr.<region>.amazonaws.com/docker-hub/<organization>/<repository>` . Here are a few points: 1. In case the image does not exist in ECR, It will be pulled from Dockerhub and stored in ECR the first time the image is requested for.  2. The second time onwards, the image will be served from ECR.  3. The repository does not need to pre exist in ECR. It is created on the fly as the image is pulled. 4. You can store your Dockerhub credentials in AWS secretsmanager to be able to pull private images from Dockerhub.  5. Data transferred between ECR and other services in the same region is free of charge. Using ECR Pull through cache 1. Our NAT Gateway bills dropped instantly and it was visible in the cost explorer dashboard from the next day.  2. We were able to by pass docker rate limits and spin up hundreds of containers every day. PS: If you are looking for help in solving such challenges in your cloud infrastructure, let's connect. #kubernetes #aws #costoptimization

21

2 Comments

Tools like #Kubernetes can incur significant costs if not properly configured or managed. Harness CEO Jyoti Bansal has a three-prong approach for effectively managing costs: 1️⃣: Visibility 2️⃣: Optimization 3️⃣: Cost governance See more of Jyoti's thoughts on managing rising costs in this Runtime news roundtable below.

21

1 Comment

Tools like #Kubernetes can incur significant costs if not properly configured or managed. Harness CEO Jyoti Bansal has a three-prong approach for effectively managing costs: 1️⃣: Visibility 2️⃣: Optimization 3️⃣: Cost governance See more of Jyoti's thoughts on managing rising costs in this Runtime news roundtable below.

4

1. 🔐 While microservices architecture enhances team autonomy, it also makes enforcing security standards more challenging. 2. 🌐 Graph databases are ideal for managing cloud security as they clearly represent complex relationships between entities. 3. 🗂️ When building a graph database, consider long-term storage and technologies like JanusGraph, Neo4j, and Neptune. 4. 💾 Using Postgres as the underlying database, Postgraphile provides a graph API, handling transactions and concurrency better. 5. 📊 Security teams need to push scan results data to the graph database via REST API for analysis and management. #Microservices #Security #CloudSecurity #GraphDB #Database #TechChoices #Postgres #API #CyberSecurity #DataAnalysis

1

⏳ Just a Few Hours Left! Don’t miss AntStack’s "Introducing AWS SAM Template" webinar today! Join Prashanth HN and Akshatha Laxmi at 10:00 AM ET to learn how you can streamline your serverless development. 🔔 Tap 'Notify Me' to get a reminder 👇 #AWS #Serverless #AWSSAM #CloudComputing #DevOps #AntStack #CloudDevelopment #TechWebinar #ServerlessArchitecture #AWSTools #Webinar #FridayTech #TechNews

10

🚀  Tamizharasan R Kubernetes Scaling Strategies➡️ Horizontal Scaling (Scaling Out)🔄 Adjusts the number of pods to match varying workload requirements. 💡 Ideal for preventing resource shortages by scaling pods instead of directly adjusting resources. ➡️ Vertical Scaling (Scaling Up)📈 Dynamically allocates resources like RAM or CPU to cluster nodes based on application demands. 🎯 Achieved by modifying pod resource requests in response to workload metrics. ➡️ Cluster Autoscaling🌐 Increases or decreases the number of nodes in the cluster based on node utilization and pending pod status. ☁️ Interfaces with the cloud provider to request or release nodes as needed. ➡️ Manual Scaling🛠️ Adjusts the number of nodes or allocated resources in the cluster manually. 🔄 Involves adding or removing nodes, adjusting resource requests, and optimizing workload distribution. ➡️ Predictive Scaling in Kubernetes📊 Utilizes data analysis and machine learning to predict future workload demands. 🚀 Enhances efficiency by proactively adjusting resources to meet upcoming needs, rather than reacting to current demands. #devops DevOps and Cloud Labs AWS Cloud Certified Google Cloud Security Microsoft Azure Kubernetes Aditya Jaiswal Cloud Computing Learner Community DevOps Learner Community Amazon Web Services (AWS) #DevOps #CloudComputing #Kubernetes #Docker #AWSCloud #AzureCloud #GoogleCloud #Terraform #Ansible #Jenkins #GitLab #Prometheus #Grafana #Elasticsearch #Kibana #Logstash #Splunk #Nagios #Zabbix #Datadog #NewRelic #Dynatrace #Monitoring #Observability #Automation #InfrastructureAsCode #CI #CD #MicroServices #Serverless #CloudNative #DevSecOps #SRE #Agile #DevOpsTools #CloudArchitecture #CloudSecurity #CloudScaling #CloudAutomation #CloudMigration #CloudOptimization #CloudGovernance #CloudOperations #CloudInfrastructure #CloudPlatforms #CloudServices #CloudProviders #CloudSolutions #CloudStrategies #CloudTechnology #CloudTransformation #DevOpsEngineer #CloudEngineer #SoftwareDevelopment #WebDevelopment #MobileApps #DataEngineering #DataScience #MachineLearning #AI #IoT #EdgeComputing #BigData #Analytics #Networking #Storage #Virtualization #OpenSource #TechCommunity #TechInnovation #TechTrends #CareerGrowth #SkillDevelopment #TechCareers #TechJobs #TechRecruitment #TechTalent #TechHiring #TechEmployment

33

1 Comment

🌟 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 101: 𝗧𝗵𝗲 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗕𝗹𝗼𝗰𝗸𝘀 𝗼𝗳 𝗠𝗼𝗱𝗲𝗿𝗻 𝗖𝗹𝗼𝘂𝗱 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 🌟 💡 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗶𝘀 𝘁𝗵𝗲 𝗱𝗿𝗶𝘃𝗶𝗻𝗴 𝗳𝗼𝗿𝗰𝗲 𝗯𝗲𝗵𝗶𝗻𝗱 𝗺𝗼𝗱𝗲𝗿𝗻 𝘀𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗮𝗻𝗱 𝗿𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲. 𝗛𝗲𝗿𝗲’𝘀 𝗮 𝗾𝘂𝗶𝗰𝗸 𝗯𝗿𝗲𝗮𝗸𝗱𝗼𝘄𝗻 𝗼𝗳 𝗛𝗼𝘄 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗪𝗼𝗿𝗸𝘀: 1️⃣ 𝗖𝗹𝘂𝘀𝘁𝗲𝗿: The Kubernetes cluster is a set of worker machines, called nodes, that run containerized applications. 2️⃣ 𝗡𝗼𝗱𝗲𝘀: Each node is a unit of computing that runs containerized apps, ensuring scalability. Every Kubernetes cluster has at least one worker node. 3️⃣ 𝗣𝗼𝗱𝘀: Worker nodes host pods, the smallest deployable units, which contain your application’s containers. 4️⃣ 𝗖𝗼𝗻𝘁𝗿𝗼𝗹 𝗣𝗹𝗮𝗻𝗲: This is the brain behind Kubernetes. It manages the worker nodes and pods via the API server, ensuring your application stays in its desired state. 🌐 Whether you’re 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀, 𝗱𝗲𝗽𝗹𝗼𝘆𝗶𝗻𝗴 𝗻𝗲𝘄 𝗺𝗶𝗰𝗿𝗼𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀, 𝗼𝗿 𝗿𝘂𝗻𝗻𝗶𝗻𝗴 𝗺𝘂𝗹𝘁𝗶-𝗰𝗹𝗼𝘂𝗱 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀, Kubernetes is essential for managing the complexity of containerized applications with ease. #CloudNative #Kubernetes #DevOps #Containers #Microservices #CloudComputing #CloudArchitecture #KubernetesCluster #ITInfrastructure #TechTransformation Azure DevOps AWS DevOps KodeKloud

18

For CKA Exam aspirants, There’s some good news and some bad news. The bad news is, The CKA exam is updating after November 25, 2024. New topics will be added, focusing more on real-world Kubernetes skills like the Gateway API, Helm, Kustomize, CRDs, and Operators. The good news is, These updates are great for those who want to truly master Kubernetes and become real-world CKA experts. I’ve written a blog comparing the old and new syllabus and outlining the additions. 𝗖𝗞𝗔 𝗨𝗽𝗱𝗮𝘁𝗲 𝗕𝗹𝗼𝗴: https://lnkd.in/gsqUc5TG 𝗡𝗼𝘁𝗲: Any CKA exam taken after the updated release will test on the new set of Domains and Competencies I personally wanted the syllabus to change because Kubernetes administration has evolved a lot in the real world. These updates will help engineers with the concepts needed for real-world setups. ♻️ PS: Dont forget to Repost and share this news with the community. #DevOps #Kubernetes #CKA

680

22 Comments

Elastic 𝐎𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐃𝐚𝐲 - 𝐁𝐞𝐧𝐠𝐚𝐥𝐮𝐫𝐮 (𝐆𝐞𝐧𝐀𝐈, 𝐋𝐋𝐌𝐬 𝐦𝐨𝐧𝐢𝐭𝐨𝐫𝐢𝐧𝐠, 𝐎𝐩𝐞𝐧𝐓𝐞𝐥𝐞𝐦𝐞𝐭𝐫𝐲, 𝐀𝐧𝐨𝐦𝐚𝐥𝐲 𝐝𝐞𝐭𝐞𝐜𝐭𝐢𝐨𝐧) RSVP - https://lnkd.in/d33xC-bG 𝐀𝐠𝐞𝐧𝐝𝐚 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟏: 𝐄𝐥𝐚𝐬𝐭𝐢𝐜 𝐎𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 & 𝐆𝐞𝐧 𝐀𝐈 Speaker: Rishikesh Radhakrishnan, Principal Solutions Architect @ Elastic Time: 10:45 - 11:25 Description: Combine Gen AI with Elastic Observability for a context-aware interactive chat experience based on your proprietary data and runbooks. 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟐: 𝐇𝐨𝐰 𝐭𝐨 𝐝𝐞𝐭𝐞𝐜𝐭 𝐚𝐧𝐨𝐦𝐚𝐥𝐢𝐞𝐬 𝐢𝐧 𝐥𝐨𝐠𝐬, 𝐦𝐞𝐭𝐫𝐢𝐜𝐬, 𝐚𝐧𝐝 𝐭𝐫𝐚𝐜𝐞𝐬 𝐰𝐢𝐭𝐡 𝐄𝐥𝐚𝐬𝐭𝐢𝐜 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 Speaker: Ranjini Ganeshan, Senior Devops Engineer @ Trellix Time: 11:25 - 12:05 Description: Discover methods for correlating telemetry data from APM agents and detecting anomalies in a specific application. 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟑: 𝐁𝐨𝐨𝐬𝐭 𝐀𝐈 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭: 𝐓𝐡𝐞 𝐏𝐨𝐰𝐞𝐫 𝐨𝐟 𝐋𝐋𝐌 𝐎𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 Speaker: Jayesh Sharma, Jr. Platform Engineer @ ZenML Time: 12:15 - 12:50 Description: LLM applications are rapidly transforming the industry. To ship ideas effectively, it's crucial to iterate quickly. This talk delves into the power of observability for gaining insights into LLM workflows, enhancing product quality and developer experience. Key topics include versioning pipelines and models, artifact tracking, effective logging, and performance insights using open-source tools like ZenML and integrations with MLflow, Weights & Biases, and Comet. 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟒: 𝐎𝐩𝐞𝐧𝐓𝐞𝐥𝐞𝐦𝐞𝐭𝐫𝐲 𝐰𝐢𝐭𝐡 𝐄𝐥𝐚𝐬𝐭𝐢𝐜 𝐎𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 Speaker: Rahul Ranjan, Senior Support Engineer @ Elastic Time: 12:50 - 13:35 Description: A session on setting up OTEL with Elastic Observability which includes a demo on end-to-end setup and then looking through available metrics in the observability page of kibana and analyzing the available data. Followed by open discussions over snacks :) See you there.

133

7 Comments

Adding one more session from the CRED. Now its Observability and Security Day 🙂. Calling all Security enthusiasts. Topic - SIEM, SOAR, SOC, Threat detection. Session: Live Forensics Using Elastic-Agents Speaker: Saksham Tushar (Head of Threat Detection Engineering & SecOps, CRED) and Monty Shyama (Threat Detection Engineer, CRED) Time: 10:45 - 11:25 Description: This session on Live Forensics provides an introduction to enabling live forensics features via Elastic's unified elastic-agent. Participants will learn to set up and configure Elastic Agents with Osquery, observe real-time data collection, and perform forensic investigations. The session includes a live demonstration ensuring practical understanding and application of the concepts.

29

Efficient pagination in #MariaDB relies on optimized queries, proper indexing, and performance monitoring. Keyset pagination is ideal for large datasets, while tools like EXPLAIN and transactional controls ensure consistency and reliability. #MariaDBPerformance @MinervaDB #dba https://zurl.co/fpc9

10

1 Comment

Tools like #Kubernetes can incur significant costs if not properly configured or managed. Harness CEO Jyoti Bansal has a three-prong approach for effectively managing costs: 1️⃣: Visibility 2️⃣: Optimization 3️⃣: Cost governance See more of Jyoti's thoughts on managing rising costs in this Runtime news roundtable below.

5

Excited to share the latest episode of "Quality Conversations" with you all! I had the pleasure of chatting with Adnan Rahić, Staff Developer Advocate at Tracetest, about the transformative power of OpenTelemetry and trace-based testing in software quality assurance. Adnan's journey from startups to Developer Relations is fascinating, and he provides valuable insights into the role of OpenTelemetry in standardizing telemetry data across the industry. But that's not all! We dive deep into the impact of OpenTelemetry on QA practices, exploring how it enhances testing with end-to-end visibility. Get ready for live demos of Tracetest and its seamless integration with Playwright, showcasing trace-based testing in action. By the end of this episode, you'll understand why starting with OpenTelemetry and leveraging trace-based testing is can be an awesome opportunity for efficient QA. Watch the full episode now and let me know your thoughts in the comments below. Like, share, and subscribe for more QA insights! #QualityAssurance #SoftwareTesting #OpenTelemetry #TraceTest #Playwright #QualityConversations #Podcast

8

1 Comment