Sumeet Ghosh

Log file synchronization is key to the performance and durability of #MySQL’s InnoDB engine, influencing database throughput, latency, and reliability. This guide explores its impact on InnoDB efficiency and data integrity. #MySQLPerformance @MinervaDB #dba https://zurl.co/cfyb

7

Conducting extensive and diverse performance and scalability testing ensures that YugabyteDB remains a reliable and high-performance distributed PostgreSQL database, ideal for handling mission-critical, global-scale, applications.🚀 This new blog from Yugabyte expert Shaharuk Shaikh gives a deep dive into our comprehensive testing framework.⬇️ https://hubs.la/Q02J_Hjj0 #YugabyteDB #HA #databasetesting #PostgreSQL #distributeddatabase #scalability

15

We're proud to announce that Arya K., Senior Principal Engineer at ShareChat, will be speaking at Rootconf Mini 2024 on November 22nd. In his talk, "𝗠𝗶𝗹𝗹𝗶𝗼𝗻𝘀 𝗦𝗮𝘃𝗲𝗱, 𝗟𝗲𝘀𝘀𝗼𝗻𝘀 𝗟𝗲𝗮𝗿𝗻𝗲𝗱: 𝗧𝗵𝗲 𝗖𝗹𝗼𝘂𝗱 𝗖𝗼𝘀𝘁 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗕𝗹𝘂𝗲𝗽𝗿𝗶𝗻𝘁," Arya will share how ShareChat reduced the cloud infrastructure costs to one-fifth of the prior costs, overcoming organizational, cultural, and complex technical challenges and saving tens of millions of dollars. Key takeaways from his session: - Balancing growth initiatives with cost optimization - Fostering a culture of ownership and accountability - Designing effective cost controls Join Arya at Rootconf Mini 2024 and learn how to optimize cloud costs without compromising growth! 📅: November 22 (Friday) 📍 :Bangalore International Centre, Domlur 🔗 : Register here: https://lnkd.in/gCU2YAQi Hasgeek #ScaleIdeasToReality #ShareChat #moj #RootconfMini #CloudCostOptimization #EngineeringLeadership #CloudComputing

85

🚀 Reflecting on Kafka Summit: Transforming Data Streams into Business Value Kafka Summit Bangalore 2024 was not only enlightening but also provided a delightful reunion with Jay Kreps, co-founder and CEO at Confluent, and my former colleague at NexTag. It's inspiring to see how Confluent is shaping the future of Streaming with Apache Kafka & Apache Flink Unified for the Data Streaming Era (Two(2) big communities coming together) Here are some pivotal learnings from the summit that resonate deeply with our objectives at MakeMyTrip and the broader trends within the tech industry: 1.  𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐀𝐜𝐜𝐞𝐥𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: Confluent is focused on making it easier for users to process data streams more quickly and efficiently than before, emphasizing the importance of moving beyond mere data movement to achieve transformative business impacts while Addressing Data Infrastructure Complexity. 2. 𝐔𝐧𝐢𝐯𝐞𝐫𝐬𝐚𝐥 𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐬: Confluent aims to transition from a model where data consumers pull data from systems, to a model where producers present well-formed data that multiple downstream subscribers can use. This approach is embodied in their concept of "data products," which are well-curated, reusable data sets with a clear ownership and governance structure. 3. 𝐒𝐭𝐫𝐞𝐚𝐦𝐢𝐧𝐠 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 𝐄𝐧𝐡𝐚𝐧𝐜𝐞𝐦𝐞𝐧𝐭𝐬: The new features in Confluent’s streaming platform are designed to federate data effectively across operational and analytical domains. The platform’s capabilities are extended to ensure data is not only real-time but also reusable, governed, and securely connected across all systems. 4. 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐚𝐧𝐝 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐚𝐥 𝐃𝐚𝐭𝐚 𝐄𝐬𝐭𝐚𝐭𝐞𝐬: Confluent team introduced a significant development called "TableFlow," which seamlessly integrates streaming data with analytical systems. This integration aims to make real-time data available in data lakes or warehouses instantly, supporting both operational responsiveness and analytical depth. Confluent added support for Iceberg to Confluent Cloud, allowing customers to convert Kafka topics, schemas, and metadata into Iceberg tables seamlessly. 5. 𝐅𝐮𝐭𝐮𝐫𝐞 𝐕𝐢𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐃𝐚𝐭𝐚 𝐒𝐭𝐫𝐞𝐚𝐦𝐢𝐧𝐠: The overarching theme was the strategic shift towards a holistic view of data management, where streaming is not just a method of transporting data but a fundamental aspect of how data is structured, processed, and consumed across businesses. While they might not be explicitly calling it Kappa architecture, their focus on data streams suggests they're on a similar path. 📸 Here’s a snapshot of Jay Kreps, me, Aditya & Confluent team at the event. Looking forward to many more such reunions and fruitful collaborations! #KafkaSummit #DataStreaming #Personalization #Confluent #MakeMyTrip

193

1 Comment

🚀 Get a walkthrough of data filtering on zero-ETL integrations 📜 https://go.aws/4dWaQBt In this blog, Jyoti Aggarwal, Gokul Soundararajan & Sean Beath show us how to bring in selective data from the database instance on zero-ETL integrations between Amazon Aurora MySQL & Amazon Redshift. Learn about data filtering use cases like: 🔒 Improve data security by excluding tables containing PII data from replication 💰 Save on storage costs & manage analytics workloads by replicating tables required for specific use cases 🦾 Improve individual workload performance by load balancing replication processes 📝 & applying data filters to your use case! #AWS #AmazonRedshift #ZeroETL

183

12 Comments

🚧 Downtime Series: Technology learnings from how Reddit, Inc. went down, not entirely, for ~5 hours in March '23 operating at over 1 Billion MAU (Pi day outage). What was the cause? While the tech team was upgrading a particular Kubernetes cluster, a well planned large-ticket item, it went south. The engineers tried multiple fixes - like restarting services, attempting a forward fix etc. but nothing worked. Ultimately team started the backup process which brought the house to order after 5 hrs. Upon further investigation, they discovered that the root cause of the outage was a change in the label names of the Kubernetes nodes between releases!! What's the major learning? Standardise setup: Inconsistent infrastructure can be a frequent source of outages. It might work a few times but backfires hard when it happens. Always have clear SOP across pods. Official article: https://lnkd.in/gt8bPxYj #technology #learnings #systemdesign #downtime #engineering #hiring #hr #reddit #outage #downtime #startups

32

#InnoDB, #MySQL's default storage engine, enhances performance on multi-CPU systems by supporting parallel query execution. This feature splits queries into sub-tasks, processes them concurrently across threads, and combines results for faster execution. #MySQLPerformance @MinervaDB #dba https://zurl.co/1txh

1

🚀 Excited to share our latest blog post! "PostgreSQL Automatic failover with repmgr" In this blog , Bhupathi Shameer Kumar explores the detailed process of automating failover with repmgr and retain the old primary back. Here is the blog https://lnkd.in/gwKPCQE3 🚀 Aiming New Postgres Horizons 🚀 Stay tuned !! For more info, reach us at OpenSource DB or https://opensource-db.com/ #repmgr #OSDB #OpenSourceDB #Failover #Retain #Primary #PostgreSQL

18

Postgres Women India Thank you Bhupathi Shameer Kumar for sharing the blog "PostgreSQL Automatic failover with repmgr". This blog covers: Seamless promotion of standby nodes Step by step failover process Ensuring minimal downtime #postgresql #womenintech #databasemanagement #opensource #failover #repmgr #careers #dba #highavailability

8

Watch Kiran Chavala's session from the CloudStack India User Group 2024 'CloudStack Tooling Ecosystem'. In this session, Kiran gives a talk about the rich ecosystem of tools (cmk, CAPC, Terraform, Ansible, Packer, csbench, mbx), that support CloudStack. Find out how the various tools work and how easy it is to integrate with Apache CloudStack. This session provides a great way to speed up CloudStack adoption and improve performance by saving valuable time. https://lnkd.in/giUpf-YX

11

1 Comment

Axis Bank with Redis ⏱ Decreased user wait-time from 170 to 40 milliseconds 💰 Saved $82,000 by reducing the number of records it needed to store 🏃‍♂️ Made in-person account changes reflect in the mobile app in real time 🖥 Improved CPU usage with fewer errors during peak times so the app is more reliable

10

In banking, fast matters. Using Redis Data Integration to sync data between Redis and its primary database, Axis Bank revamped its mobile app, boosting its speed by 76% and eliminating customer complaints about UX. Using RDI, Axis also: ⏱ Decreased user wait-time from 170 to 40 milliseconds 💰 Saved $82,000 by reducing the number of records it needed to store 🏃‍♂️ Made in-person account changes reflect in the mobile app in real time 🖥 Improved CPU usage with fewer errors during peak times so the app is more reliable Learn more about RDI and Axis Bank: https://lnkd.in/guasTuK7

77

https://lnkd.in/gJVYpD9b I don't think Indian entrepreneurs quite get it. There is no point in competing with Microsoft Azure, AWS or GCP cloud platform. They don't understand AI and it's infrastructure, it's already an multi billion infrastructure like 10's of billion which is changing rapidly to expand. Unless you have massive compute which only 5 or 6 companies in the world, would be able to build AI infrastructure.

3

3 Comments

𝐓𝐡𝐞 𝐇𝐢𝐝𝐝𝐞𝐧 𝐂𝐨𝐬𝐭𝐬 𝐨𝐟 𝐂𝐥𝐨𝐮𝐝 𝐒𝐞𝐫𝐯𝐢𝐜𝐞𝐬 - 𝐚𝐧𝐝 𝐇𝐨𝐰 𝐭𝐨 𝐀𝐯𝐨𝐢𝐝 𝐓𝐡𝐞𝐦🤔 Cloud services offer flexibility, but hidden costs can quietly inflate your budget. Here’s how to avoid common pitfalls and keep your spending under control : ✅𝐎𝐯𝐞𝐫𝐩𝐫𝐨𝐯𝐢𝐬𝐢𝐨𝐧𝐢𝐧𝐠 𝑻𝒉𝒆 𝑷𝒓𝒐𝒃𝒍𝒆𝒎: Idle servers, oversized VMs, and unused storage waste money. 𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏: Use tools like AWS Compute Optimizer to right-size resources and Spot Instances for non-critical workloads. →𝑻𝒊𝒑: Schedule scaling based on demand to avoid idle capacity. ✅𝐈𝐠𝐧𝐨𝐫𝐢𝐧𝐠 𝐀𝐮𝐭𝐨𝐬𝐜𝐚𝐥𝐢𝐧𝐠 𝑻𝒉𝒆 𝑷𝒓𝒐𝒃𝒍𝒆𝒎: Fixed resource allocations lead to overpaying during low usage and bottlenecks during high demand. 𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏: Enable autoscaling and predictive scaling on platforms like AWS or GCP to match resources with workload needs. →𝑻𝒊𝒑: Combine autoscaling with load balancers for efficient traffic management. ✅𝐃𝐚𝐭𝐚 𝐄𝐠𝐫𝐞𝐬𝐬 𝐂𝐨𝐬𝐭𝐬 𝑻𝒉𝒆 𝑷𝒓𝒐𝒃𝒍𝒆𝒎: Transferring data between regions or clouds incurs high fees. 𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏: Minimize inter-region traffic and use cloud-native CDNs like AWS CloudFront. →𝑻𝒊𝒑: Regularly review transfer logs to detect and reduce costly patterns. ✅𝐋𝐚𝐜𝐤 𝐨𝐟 𝐂𝐨𝐬𝐭 𝐕𝐢𝐬𝐢𝐛𝐢𝐥𝐢𝐭𝐲 𝑻𝒉𝒆 𝑷𝒓𝒐𝒃𝒍𝒆𝒎: Untracked usage leads to unexpected expenses. 𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏: Adopt FinOps principles and tools like AWS Cost Explorer or CloudZero for real-time expense tracking. 𝑻𝒊𝒑: Set budget alerts and enforce spending limits to avoid surprises. ✅𝐏𝐨𝐨𝐫 𝐁𝐚𝐜𝐤𝐮𝐩 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬 𝑻𝒉𝒆 𝑷𝒓𝒐𝒃𝒍𝒆𝒎: Obsolete snapshots and backups inflate storage costs. 𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏: Automate retention policies and archive long-term backups to low-cost storage like Amazon Glacier. →𝑻𝒊𝒑: Audit storage regularly for unused backups. ✅𝐎𝐯𝐞𝐫𝐥𝐨𝐨𝐤𝐢𝐧𝐠 𝐑𝐞𝐬𝐞𝐫𝐯𝐞𝐝 𝐈𝐧𝐬𝐭𝐚𝐧𝐜𝐞𝐬 𝑻𝒉𝒆 𝑷𝒓𝒐𝒃𝒍𝒆𝒎: Predictable workloads running on-demand waste money. 𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏: Use Reserved Instances or Savings Plans for steady workloads and combine with Spot Instances for flexibility. →𝑻𝒊𝒑: Regularly re-evaluate reserved capacity to ensure alignment with current needs. What’s your top tip for cutting cloud costs? Share in the comments! #CloudOptimization #FinOps #DevOpsTips #CostEfficiency

9

𝗕𝗼𝗼𝘀𝘁𝗶𝗻𝗴 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝘃𝗶𝘁𝘆 𝗮𝗻𝗱 𝗦𝗶𝗺𝗽𝗹𝗶𝗳𝘆𝗶𝗻𝗴 𝗪𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 𝗦𝗤𝗟! At #ServerlessDaysBLR2024, Kushal N., Data Solution Architect at EPAM Systems, and Abishek Subramanian, Senior Platform Solutions Engineer at Databricks, highlighted how Databricks Serverless SQL is transforming data workloads. 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: ✅ 𝗦𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗲𝗱 𝗪𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀:Simplify infrastructure management and focus on insights. ✅ 𝗘𝗻𝗵𝗮𝗻𝗰𝗲𝗱 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝘃𝗶𝘁𝘆:Spend less time on overhead, more on innovation. ✅ 𝗖𝗼𝘀𝘁 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆: Optimize resources with serverless capabilities. 𝗗𝗲𝗺𝗼 & 𝗖𝗼𝗺𝗽𝗮𝗿𝗶𝘀𝗼𝗻: The session included a live demo contrasting Databricks Serverless SQL with traditional SQL warehouses, shedding light on preferred use cases and practical insights. 𝗔 𝗺𝘂𝘀𝘁-𝘄𝗮𝘁𝗰𝗵 𝗳𝗼𝗿 𝗱𝗮𝘁𝗮 𝗽𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹𝘀 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝘁𝗼 𝗺𝗮𝘅𝗶𝗺𝗶𝘇𝗲 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 𝗮𝗻𝗱 𝘃𝗮𝗹𝘂𝗲 𝗶𝗻 𝘁𝗵𝗲𝗶𝗿 𝗮𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 𝗷𝗼𝘂𝗿𝗻𝗲𝘆! 🔔 𝗦𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲 𝘁𝗼 𝗔𝗻𝘁𝗦𝘁𝗮𝗰𝗸 𝗳𝗼𝗿 𝗺𝗼𝗿𝗲 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀: https://bit.ly/47fDoCQ #Serverless #Databricks #CloudInnovation #DataSolutions #DataAnalytics #DataWorkloads #CloudComputing #ServerlessTechnology #ProductivityHacks #TechInnovation #BigData #SQLSolutions #CostEfficiency #ModernDataArchitecture

8

We are excited to announce reports-server, a new #Kyverno project which aims to improve scalability of reporting in large clusters giving you all the benefits of visibility without compromising on performance. In this blog post, Vishal Choudhary introduces reports server, discusses its architecture, and provides steps on how to get started. https://hubs.ly/Q02B0QJV0 #kubernetes #devops #devsecops

9

😉 Eager to unlock the secrets of efficient data sharding and load balancing in Apache IoTDB? In this blog, "Apache IoTDB Distributed Architecture Trilogy: Part 2 - Data Sharding and Load Balancing," we uncover: 🔍 The innovative sharding techniques used by IoTDB. ⚖️ How load balancing ensures smooth operations across clusters. 💡 Practical insights and real-world examples to enhance your IoTDB deployment. 👉 https://lnkd.in/eTUuxPNj #TimeSeries #Database #OpenSource #IoT #BigData #DistributedComputing #DataSharding #LoadBalancing

2