Salvatore Sanfilippo’s Post

Originally posted on X: https://lnkd.in/gQtSimy9 starbeam.one Daily Status - I was able to clean up and complete the API access story yesterday and started to investigate Redis, which will allow us to store data in a distributed manner, allowing us to scale appropriately. Well, in theory at least. What this really does is reduce the pressure on the Sql Server as data is stored in a cache and pulled from there for subsequent calls. There is the catch, however, that it's a bit more expensive to the hosting costs, starting at $15/mo. Worth it? We will see. 🤔 We're currently doing caching for some of our calls but it's only local caching (via IMemoryCache) to the server instance. If we scale out each server instance will have its own IMemoryCache and has to populate it accordingly, leading to the same calls to the database for every instance. With using a distributed cache (Redis), the calls to the database happen once no matter how many server instances exist. Kind of a cool concept. 🤓 During my investigation yesterday there appears to be several approaches that can be used to implement Redis. I will hopefully get that resolved today along with an implementation. 🤞👍 #StarbeamOne #sb1 #DailyDeveloper #Startup #Status #azure #blazor #csharp #dotnet #buildinpublic #buildinginpublic

🚨 Why Redlock in Redis Might Just Be Your Database’s Bouncer 🚨 Monday morning, and the chaos begins: traffic jams, coffee spills, and, in our database world, racing processes fighting over the same resource! 🏎️💥 Enter Redlock in Redis—the cool-headed bouncer that makes sure only the VIPs (the right processes) get access at the right time. 🕶️💼 So, what’s Redlock? 🔐 Redlock is Redis’s distributed locking algorithm, designed to prevent those frustrating resource conflicts in a distributed system. Imagine it as the guy holding the clipboard, saying, “One at a time, please!”—and backing it up with a solid, efficient protocol. Here’s how it works: 1️⃣ Tries to get a lock from N different Redis nodes (typically five) 2️⃣ Sets a lock if the majority says, “Yep, you’re in.” 3️⃣ Works on its task without interruption, knowing it has VIP access 4️⃣ Releases the lock on the way out, so the next in line can enter without a brawl! 👊 Why use Redlock? It’s safe, ensuring only one process holds the lock even across different servers. It’s reliable, making your distributed app more consistent. It’s fair, enforcing first-come-first-served, so everyone gets a chance! Sure, Monday can be chaotic, but with Redlock in Redis, you can at least keep your database drama-free. ✌️ #Redis #Redlock #DistributedSystems #BackendEngineering #MondayMotivation

The best way to implement a job/task scheduling flow in backend is BullMQ with Redis and Redis-Persists. Redis store key-value pair to Memory. That is why it is important to use redis-persist to backup all those data to permanent storage regularly.

GhostBin is a lightweight, high-performance pastebin built with Go and Redis. It's designed with simplicity and speed in mind, offering a fast and efficient platform for sharing text snippets effortlessly. Plus, it's open-source! * Basic usage. ------------- $ curl -F "[email protected]" gbin.me * Pipe output of a command. -------------------------- $ cat file | curl -F "f=@-" gbin.me $ find /var/log/nginx -name "*.log" | curl -F "f=@-" gbin.me GitHub: https://lnkd.in/gqY_VaB2

𝐈'𝐦 𝐪𝐮𝐢𝐭𝐞 𝐩𝐫𝐨𝐮𝐝 𝐨𝐟 𝐭𝐡𝐢𝐬 𝐨𝐧𝐞! 𝐓𝐡𝐞 𝐚𝐭𝐭𝐚𝐜𝐡𝐞𝐝 𝐢𝐦𝐚𝐠𝐞 𝐬𝐡𝐨𝐮𝐥𝐝 𝐬𝐩𝐚𝐫𝐤𝐥𝐞 𝐲𝐨𝐮𝐫 𝐢𝐧𝐭𝐞𝐫𝐞𝐬𝐭 :) Since Redis 7.2.5, we've introduced significant latency reduction and throughput increment for many common commands and usage patterns. We see performance improvements in over 20 commands of Redis, some of which with a broad usage and impact. 𝐖𝐡𝐞𝐫𝐞 𝐢𝐬 𝐭𝐡𝐢𝐬 𝐜𝐨𝐦𝐢𝐧𝐠 𝐟𝐫𝐨𝐦? There are 18 performance optimizations on M01 and M02 that strictly focus on reducing CPU cycles, reducing memory allocations, and optimizing data access patterns of commonly used code. As an example, if you're using ZADD or ZRANGE commands you can expect at least 30% median and tail latency drops, much of it related to the adoption by Redis of the fast_float library (https://lnkd.in/d3TpnQa2) by Daniel Lemire. And the interesting bit, is that while we were adding this to Redis we found an issue on fast_float that we've solved (https://lnkd.in/d66d3mUW). For me, this represents our way of working - always striving to leave every project better than we found it. In short, Redis is now more CPU efficient and cache friendly. 𝐀𝐧𝐝 𝐰𝐞'𝐫𝐞 𝐧𝐨𝐭 𝐬𝐭𝐨𝐩𝐩𝐢𝐧𝐠! 𝐄𝐱𝐩𝐞𝐜𝐭 𝐦𝐨𝐫𝐞 𝐟𝐫𝐨𝐦 𝐮𝐬 𝐢𝐧 𝐭𝐡𝐞 𝐜𝐨𝐦𝐢𝐧𝐠 𝐦𝐨𝐧𝐭𝐡𝐬! Read more about it in https://lnkd.in/dvAvZGzu, and download to give it a try. #redis #performance #latency

How Redis Boosted Our Leaderboard Performance 10X in One Line of Code 🤯 Load times were taking their toll on engagement and retention. Especially leaderboards hitting 30 seconds, our PHP-only approach couldn't cut it. data straight in code was a struggle. In a moment of brilliance/panic, I discovered Redis sorted sets. Redis has an article on their official website about use cases and specifically about leaderboards. Skeptical it could help much, migrating leaderboard data from PHP to a caching layer was actually a genius move. Instantly, loads dropped from 10+ seconds to under 1. It's practically magic and it’s almost real-time now! No more waiting - users spend more time navigating. Redis delivered the speedup we so needed. Sometimes the simplest optimization has the biggest impact You can find the article mentioned through this link https://lnkd.in/drjrFbPR

Today I Learned – Redis connections growing unbounded. In our production deployments, we utilize gunicorn and gevent with great success. Our benchmarking showed this setup was once the fastest many years ago. The Problem: The increase in active Redis connections may not be in your Celery workers or Celery Beat, but rather in your WSGI (gunicorn) process. Despite gevent providing green threads, Celery can create a new connection pool for each task, leading to a rise in connection count. The Solution: To address this issue, ensure Celery recognizes the thread-safe environment by setting result_backend_thread_safe = True. For Django users, add CELERY_RESULT_BACKEND_THREAD_SAFE = True in settings.py. This adjustment should enable proper re-use of Redis connections, resolving the problem. Hope this insight proves beneficial!

Redis is a high performance key-value data store, despite being single threaded. This articles explains how redis does it. #distributedsystems #distributedstorage

I once brought down a production server with a single Redis command. Lesson learned: With great power comes great responsibility. Here's my 60-second guide to Redis mastery: 1. Start with strings and hashes 2. Master lists for queues 3. Use sets for unique collections 4. Leverage sorted sets for rankings 5. Employ pub/sub for real-time features 6. Persist data with RDB and AOF 7. Scale with Redis Cluster 8. Extend with modules like RedisJSON Redis empowers developers to build blazing-fast applications. But remember: Always benchmark and plan for scale. What's your biggest Redis challenge? Let's solve it together! #DatabaseOptimization #SoftwareEngineering #PerformanceTuning

#Leetcode Solved Count Complete Tree Nodes and also learned about Redis: Redis (Remote Dictionary Server) is an open-source, in-memory data structure store used as a database, cache, and message broker. It supports various data structures such as strings, hashes, lists, sets, sorted sets, bitmaps, hyper loglogs, and geospatial indexes. Redis is known for its high performance, scalability, and versatility, making it popular among developers for a wide range of use cases.