TiDB, powered by PingCAP’s Post

🎥 Why Plaid migrated from hosted MySQL to TiDB: https://hubs.ly/Q02X2xPc0 Meet Zander Hill, Principal Engineer at Plaid. One of Zander’s most important projects for the past 1.5 years was working on adopting TiDB within the company. Get to know Zander and Plaid in the below clip! #TiDB #MySQL #ZeroDowntime #CloudComputing #DistributedSQL

Last year, Stripe processed over $1 trillion, equivalent to 1% of global GDP. 🤯 To balance strict regulatory compliance with keeping its developers nimble, Stripe operates Apache Airflow at massive scale: 😲 Airflow processes multiple petabytes of data every day 📶 Airflow orchestrates 250 complex pipelines comprising 150,000 tasks that connect Stripe’s operational and analytical systems ➡️ Airflow manages workflows across Apache Spark with data transformed into the Apache Iceberg table format and queried via Presto for 500 different teams Learn more about Airflow's criticality within Stripe: https://bit.ly/4eMXk1X

🎥 Why Plaid migrated from hosted MySQL to TiDB: https://hubs.ly/Q02X2xPc0 🎥 Meet Zander Hill, Principal Engineer at Plaid. One of Zander’s most important projects for the past 1.5 years was working on the adoption of TiDB within the company. Get to know Zander and Plaid in the below clip! #TiDB #MySQL #ZeroDowntime #CloudComputing #DistributedSQL

The issue of Postgres' inadequate or difficult scalability is surfacing more frequently these days. While Postgres is quite scalable, companies like Uber and Figma have reached a level where hardly any open-source technology can meet their scalability requirements out of the box. First, Uber highlighted this problem, and now Figma has mentioned it. Check out this video for more information.

Check out this blog post to see Amazon CodeWhisperer in action 😱 Walk through common scenarios and learn how to get the most out of #CodeWhisperer in IDE 🔗 https://t.co/OYLMm73B1B via Twitter https://lnkd.in/dGcaTecY

#Day7 - Database Scaling: Vertical vs Horizontal Approaches System design basics 𝐃𝐚𝐭𝐚𝐛𝐚𝐬𝐞 𝐬𝐜𝐚𝐥𝐢𝐧𝐠 There are two broad approaches for database scaling: vertical scaling and horizontal scaling. Vertical scaling, also referred to as scaling up, involves enhancing the power of an existing machine by adding more resources such as CPU, RAM, and disk space. There exist highly robust database servers, like those mentioned by Amazon Relational Database Service (RDS), offering configurations with up to 24 TB of RAM. Such formidable database servers have the capacity to manage extensive amounts of data efficiently. For instance, despite stackoverflow.com experiencing over 10 million monthly unique visitors in 2013, it operated with just one master database. Nevertheless, vertical scaling carries notable drawbacks: Although additional resources like CPU and RAM can be incorporated into the database server, there are inherent hardware limitations. Consequently, a single server may not suffice for handling a large user base. There is an increased risk of encountering a single point of failure. Vertical scaling tends to incur significant costs, as powerful servers come with considerably higher price tags. Horizontal scaling, also termed sharding, involves the addition of more servers to a system. Sharding divides large databases into smaller, more manageable parts known as shards. Each shard maintains the same schema, though the data within each shard is unique. For instance, in a sharded database setup depicted in Figure 1-21, user data is distributed across database servers based on user IDs, utilizing a hash function for shard allocation. The hash function, such as user_id % 4 in this example, determines which shard stores and retrieves the data based on the hash result. The critical aspect in implementing a sharding strategy is the selection of the sharding key, also known as a partition key. This key, illustrated in Figure 1-22 as "user_id," dictates data distribution and enables efficient retrieval and modification of data by directing database queries to the appropriate database. It's essential to choose a sharding key that evenly distributes data. While sharding is an effective technique for database scalability, it brings forth complexities and new challenges to the system. #systemdesign #engineering #learning

I was recently browsing an ecom site and i saw the number of concurrent viewers of that particular product.This is something new which i have never seen before and this shows the current demand of the product.I guess Amazon is missing this wonderful feature. I'm wondering how is data stored as it involves very frequent writes and reads(Might be Redis but i/o operations at such rate can affect the whole redis cluster) and it keeps changing for seconds.Such an innovative system.Reason why software engineers can learn everyday! #softwareengineering #softwaredeveloper

Amazon Q Developer Agent has new super powers! Previously when you wanted it to write code, the agent would generate a plan, present the plan for human approval, and then attempt to work on the plan exactly as proposed. We have modified how the agent works. Amazon Q Developer Agent now begins working on your prompt immediately. It plans and works iteratively, allowing the agent to adapt on the fly, adjust it's plan, and autonomously solve coding problems with less direct human guidance. You'll even see Q Developer Agent produce a stream of "thoughts" as it works. I put this through a test, seeing how well it could upgrade some old Apache Airflow DAGs that I wanted to run on my Airflow 2.9.2 environment. This is a common issue I hear stopping Airflow users upgrading to newer versions of Apache Airflow, so if we can make that task easier, then Airflow developers will be able to take advantage of the latest features that Airflow provides. Check out my video on how easy this was. Now there is no reason to stay on older versions of Apache Airflow...

How does Uber deliver ETAs at 500,000+ requests per second? Behind every "Your ride will arrive in 4 minutes" notification lies a marvel of engineering—distributed systems, real-time traffic data, and advanced algorithms. From microservices and Cassandra to Apache Kafka, Uber’s tech stack ensures scalability, speed, and precision at a global scale. Swipe through the carousel to uncover the secrets behind Uber’s real-time ETA computation. What part of this system do you find most fascinating? Share your thoughts below! #DataEngineering #DistributedSystems #UberTech #RealTimeData

Are you tired of spending too much time on code debugging and maintenance? 🚀 🔎 Check out this blog on leveraging Amazon Q Developer for efficient code debugging and maintenance: https://lnkd.in/d7Z6s4KF