Mladen Trampic

The latest AWS open source newsletter, #196 is out. As always, more great new projects are featured in this edition of the newsletter, including a link to the Valkey repo, a nice GUI based project to help you build orchestration workflows that uses Apache Airflow under the covers, a tool to help you find signals through the noise of your security logs, a project to help you run serverless tasks in a cron like fashion, a command line runner for Amazon CodeCatalyst, a tool to help you simplify the deployment of Cruise Control on Amazon MSK, a nice Mac client for experimenting with Amazon Bedrock, and some really cool demo apps, the pick of which (for me) is a nice way of surfacing up your Amazon Bedrock models in a way that existing applications that expect an API key can use. Also featured in this weeks edition is content on Valkey, Apache Airflow, OpenSearch, LangChain, PostgreSQL, WordPress, RAGmap, RAGxplorer, Cedar, AWS CDK, Lambda Web Adapter, Postfix, Spring Boot, Amazon Corretto, Kubernetes, Karpenter, KEDA, Prometheus, OPA, Amazon EMR, PySpark, MySQL, Open JD, AWS Amplify, GraphQL, AWS PDK, Apache Livy, and Nodestream. Make sure you check out the section at the end, where I highlight all the amazon open source Builders that create the content I feature, which this week includes  Zied Ben Tahar, Vadym Kazulkin, Vivek Gite, Johannes Konings, Daniel Aniszkiewicz, Rio Astamal, João Galego, Ricardo Ferreira, Abishek Gupta, Rehan van der Merwe, David Nalley, Swapnil Bhartiya, Amo Moloko, Hernan Garcia, Jeetendra Vaidya, Sriharsh Adar, Mark Stephens, Mark Wiebe, Daniel Neilson, Pooja Singh, Rekha Reddy Anupati, Aditya Samant, Jayesh Shinde, Harshad Yeola, Sameer Malik, Baji Shaik, Anil Maktala, Arundeep Nagaraj, Bukhtawar Khan, Gaurav Bafna, Sachin Kale, Ranjith Ramachandra, Rohin Bhargava, Matthew Welborn, Paul Whittemore, Alex Iankoulski, Riccardo Freschi, Shubir Kapoor, Mihir Surani, Piyush Mattoo, Hans Nesbitt, Siva Guruvareddiar, Imaya Kumar Jagannathan, Oleg Mironov, Dmitry Shurupov, and Gary Stafford. #opensource #oss #aws Go enjoy -> https://lnkd.in/eWjA9xA9

58

3 Kommentare

Really excited about the 'Hudi-rs' release 🎉 Not every data workload requires the need for distributed processing and to setup Spark & JVM to execute them. In fact, there has been an increasing trend in terms of using Rust & Python-based compute engines for a range of workloads. The hudi-rs project is a native Rust library for Apache Hudi, with bindings into Python. This opens up an exciting new world where users can read Hudi tables using non-JVM compute engines. Under the hood, hudi-rs uses #ApacheArrow. So, it scans the Hudi table and reads out records as Arrow RecordBatch. RecordBatch is a table-like data structure that is semantically a sequence of fields, each a contiguous Arrow Array. Once the records are in an Arrow Table, you can easily use it with Arrow-native and integrated tools such as DuckDB, Polars, Daft etc. for further analysis. This is an exciting new direction for open lakehouse platforms like Hudi that brings differentiating capabilities such as: incremental processing, multi-modal indexing subsystem, open table services, among others. We would also love to have any sort of community contributions & PRs. Check out the contributing link & repo in comments. #dataengineering #softwareengineering

127

3 Kommentare

Don't Cry Mr. Donkersgoed = My posts come with quirky headlines, here is a recent example: You Want To Be A Cloud Engineer? Leave Cloud ASAP. lnkd.in/g9gbZPQF This post headline is also strange: Don't Cry Mr. Donkersgoed What is going on?? I claim that this post (Don't Cry Mr. Donkersgoed) and the earlier post (You Want To Be A Cloud Engineer? Leave Cloud ASAP.) are the same even though post headlines are widely different. Please read on to find out. = Luc van Donkersgoed published a heartfelt note: Dear Amazon Web Services (AWS), please let me be a cloud engineer again lnkd.in/gFKVPcZH I published my earlier post as a reaction to this cry for restoring sanity in Amazon Web Services (AWS) Product Management. In this post I am encouraging Mr. Donkersgoed to not cry but take some alternative positive steps. What are these alternative positive steps? Here: 1) Build open source stacks 2) Use open source stacks I have a very good recent example: Bengaluru-based Moving Tech has raised $11 million in a pre-Series A funding round co-led by Blume Ventures and Antler, the startup said. Google, which has pledged to invest $10 billion in India, participated in the round. Namma Yatri works atop the Open Network For Digital Commerce (ONDC), an interoperable scheme backed by the Indian government that is aiming to democratize e-commerce in the country. Namma Yatri’s app connects customers with auto-rickshaw and cab drivers without charging either party for rides. Instead, the startup collects a small monthly fee from its driver partners. Uber and Ola, in comparison, charge their driver partners as much as 25%-30% of the ride cost, and have refused to join the ONDC network for their core mobility offerings. Moving Tech’s co-founders, Magizhan Selvan and Shan M S, told TechCrunch that they identified an opportunity after they found out how frustrated drivers were with their treatment in the existing system. “There was a lack of differentiated approach,” Shan said, reflecting on the decade-long duopoly that Uber and Ola have enjoyed unchallenged in India. Moving Tech doesn’t offer customer discounts or driver incentives, and it is banking on providing a service that people find genuinely useful, he added. See what I mean by: 1) Build open source stacks 2) Use open source stacks = India Stack refers to the project of creating a unified software platform to bring India's population into the digital age. Its website describes its mission as follows: "India Stack is a set of open APIs that allows governments, businesses, startups and developers to utilize a unique digital Infrastructure to solve India’s hard problems towards presence-less, paperless, and cashless service delivery" India Stack is the largest open API in the world. Since its deployment, India has been organizing hackathons to develop applications for the APIs. Don't Cry Mr. Donkersgoed - Think Different.

12

✋ Using Kubernetes early on will sink your startup ☠️ What do I mean? I am not against Kubernetes, but engineering focused teams can sometimes overcomplicate solutions. It is important to think carefully about the architectural decisions you make upfront to optimize for speed and customer needs. But as technologists, it's easy to fall in love with the tech, but that is very rarely why customers chose your startup's solution! You need to focus on delivering for the customer. That is what should dictate your architectural decisions, not what is a coolest tech. For this last Founders Mistakes video for the season, we discuss the dangers of overbuilding and why you should simplify your tech early on. Are there any tech choices you would change for your startup based on what you know now? Do share in the comments below! #startups #cloudcomputing #softwareengineering #FoundersintheCloud ***** Stay on top of all my startup lessons, stories, mistakes, and community updates by subscribing to the Founders in the Cloud newsletter! ➡️ Go to my profile ➡️ Click the button to View My Blog ➡️ Add your email and click subscribe ➡️ Receive a weekly email with practical startup tips

34

8 Kommentare

Will Amazon succeed bringing developers back to the office? YES and NO. At the same time. Landing an IT job is getting harder and harder these days. So some people would abide of fear losing their job. Others will join because they value f2f work. Secondly and this is also influences the first point: Amazon won’t need that many developers in the future because of the accelerating industry breakthrough with LLM and genAI. The prediction now is that by 2026-2027 we will see a major shift in AI capabilities. So NO: Amazon don’t need to bring *all* the developers to the office — only the ones who they want to stay. And YES: they will eventually bring this people to the office. I wish more companies were this bold.

4

2 Kommentare

Because so many have asked: a great project when it comes to dynamic load balancing in #Proxmox. Hopefully soon also directly integrated, but until then an excellent solution.

15

1 Kommentar

Remember 5 years ago when no one could stop talking about Kubernetes and microservices? Well, they won the day! Result: the efficiency enterprises gained building software has made data-driven decision making HARDER... Once upon a time, enterprises: > Built/bought large “monolithic” applications > Stored data in central database Today: > Build microservices & buy SaaS apps  > Store data in a bunch of different places... This has led to new architectures & patterns to gather & analyze data > Modern Data Stack, Lakehouse, Data Mesh, Data Fabric... We can solve the problem of moving data, but it's the CONTEXT that remains challenging: > Sharing context is MUCH HARDER when the data originated from a gazillion different sources. > e.g., column headers in a Gold Data Table aren’t nearly as obvious as a data engineer might think. This is where Dataiku can help: “enable data experts embedded [in] the business to collaborate with subject matter experts on data products” Dataiku offers: > standard connectors to all of your enterprise data sources > ability to customize connectors if you need to connect to something we don't yet support Benefit: analysts can work with domain experts in one place, looking at the same data and sharing context on what it means, which makes data-driven decision making possible.

11

4 Kommentare

I have changed my opinion on Kubernetes in the last years. It might be just the right level of complexitiy for infrastructure at scale. I used to disregard it a lot and start with simpler cloud setups on AWS/GCP/Azure. However, I noticed that you always run into the same problems with scale and solving these problems on the cloud providers nativly is not easier. Doing platform engineering is hard everywhere. In that way, I am starting to like the idea of using full-fledged Kubernetes. Especially because the ecosystem is so rich and you can decide where to buy compute.

4

Without Etcd properly running in your k8s cluster, it’s useless. Etcd is where anything state related is stored for your cluster. It’s the Kubernetes database (well, technically a key store, but it’s the k8s database). Because of that, you’re going to want to not only secure it as much as possible, but ensure you can always restore it. A quick way is with the Etcd command-line tool. ‘Etcdctl snapshot save’ #kubernetes #devops #platformengineering

73

3 Kommentare

There is a big pricing problem on cloud storage ! Why ? 12€/month for 1 To for an S3 in a cloud provider. 9.99€/month for 2 To in Google One. Nobody can do better than this without losing money, and this kill newcomer. This look like there is an antitrust problem here to me, CNIL - Commission Nationale de l'Informatique et des Libertés could you take a look ? or someone else ? (don't know if it's Google that kills the market by setting a not fair price or other that are too expensive) If I want to provide a service like that, I'm at least twice more expensive than Google without margin... The only possibility is with credit from the cloud provider which seems wrong to me, you can't do this because it's too expensive so we are going to help you financially and so all your infrastructure is at almost no cost🤦 why ? Why overpricing everyone to give credit to some instead of just ask for what it cost plus a margin ? Or maybe we need big tech to stop screwing the market ? Is this something normal ? cc Scaleway

3

4 Kommentare

𝐂𝐨𝐧𝐭𝐫𝐨𝐥𝐥𝐢𝐧𝐠 𝐟𝐢𝐧𝐞-𝐠𝐫𝐚𝐢𝐧𝐞𝐝 𝐂𝐨𝐧𝐭𝐚𝐢𝐧𝐞𝐫 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 𝐯𝐢𝐚 𝐊𝐮𝐛𝐞𝐫𝐧𝐞𝐭𝐞𝐬 𝐇𝐏𝐀  🤔 The current implementation of #Kubernetes Horizontal Pod Autoscaling (HPA) takes into account only the total values of the containers to calculate cumulative Pod resource usage, thus ignoring metrics at the container level. This has a drawback, for those workloads where ❗ Container usage is not closely related or does not change at the same rate ❗ Pod with sidecar containers, handling logs might not consume significant resources, while the main blog application container handles most of the workload The Container resource metrics feature graduated as stable in v1.30 now allows HPA to target individual container metrics within a pod, allowing to define the HPA to scale based on the resource utilization (CPU, memory, etc.) of a specific container within the pod. ✅ The feature helps allocating resources efficiently, avoiding unnecessary scaling due to high pod utilization triggered by non-critical containers ✅ Allows #devops / #sre to focus on real load utilization on application workload ✅ Provides better decision-making regarding scaling, preventing performance bottlenecks type: ContainerResource containerResource: name: cpu container: application target: type: Utilization averageUtilization: 60 Ref:- https://lnkd.in/gRZqxxCe #performance #benchmarking #platform #consulting #architect #scaling #sysadmin #microservices #monitoring

135

11 Kommentare

☁️ Exciting developments in cloud infrastructure are on the horizon. These fundamentally industry-altering design patterns, open-source tools, and architectures are set to address the complexity and scaling challenges of Kubernetes and evolve cloud infrastructure as we know it today. Very happy to share my first-ever feature on InfoWorld today, which seeks to answer 'what's next' for K8s and cloud infrastructure. In short, we can expect a reduction of complexity in using Kubernetes, more growth in the mass enterprise market, better cross-cloud visibility with OpenTelemetry, heightened platform abstractions, inner-K8s virtualization, new modes of AI orchestration, a move toward the edge, the rise of WebAssembly and more. As an industry, we need to get out of the weeds of YAML and “climb the ladder of abstraction,” Jonas Bonér, CTO of Lightbend, says. “The next generation of serverless is that you don’t see infrastructure at all.” Thanks to everyone who chimed in on this one! This article features, in order of appearance: - James Watters, director of research and development, VMware Tanzu Labs Division at Broadcom - Thomas Graf, co-creator of Cilium, VP of cloud networking and security of Isovalent at Cisco - Sid Nag, VP Analyst at Gartner - Ellen Chisa, partner at boldstart ventures - Jonas Bonér, CTO of Lightbend, Inc. - Guillermo Rauch, CEO of Vercel - Liam Randall ☁️🚀, CEO of Cosmonic - Lukas Gentele, CEO and co-founder of Loft Labs - Rajiv Thakkar, director of product marketing, Portworx by Pure Storage - Liz Warner, CTO of Percona - Omer Hamerman, principal engineer at Zesty Also, a big thank you to Doug Dineley for your editorial guidance. We've got some other stories in the pipeline, which I'm looking forward to sharing soon. For now, check out the piece on InfoWorld, and feel free to share what YOU think is the next big thing in cloud infra. https://lnkd.in/ekjFS3ej #kubernetes #cloudnative #cncf #cloud #cloudinfrastructure #softwareinfrastructure

57

7 Kommentare

🔐💻 How to securely share your AWS S3 objects without making them public? Let’s talk about Pre-Signed URLs! 🚀 1) Generate a temporary URL to grant time-limited access. 2) Maintain full control over permissions and expiration time. 3) Perfect for secure file sharing and upload access. How are you leveraging Pre-Signed URLs in your AWS setup? Let’s discuss!👇 Read my latest blog post on Hashnode and also give your valuable feedback on this. #AWS #CloudComputing #CyberSecurity #TechTips #CloudSecurity #DataProtection #S3 #DevOps #TechInnovation #AWSCommunity

17

2 Kommentare

Successfully Installed JFrog on Ubuntu 22.04! I've just navigated through setting up JFrog Artifactory on Ubuntu 22.04, and what a smooth journey it was! 🚀 The installation process was straightforward, involving updating the system, installing Java (a prerequisite for running JFrog), downloading the JFrog Artifactory package, and finally, launching the service. The clarity of steps and the seamless execution was a testament to JFrog's user-friendly approach to DevOps tools. JFrog Artifactory is now up and running, ready to revolutionize how we manage our software artifacts and binaries. Its powerful features promise to enhance our development workflow, improve productivity, and ensure a secure repository for our projects. 🛠️ For anyone on the fence about integrating JFrog into their development environment, I highly recommend giving it a try. The benefits of streamlined project management and enhanced security are too good to pass up! #JFrog #Ubuntu #DevOps #SoftwareDevelopment #Innovation

63

2 Kommentare

Controversal Opinion: Stop Using kubectl No matter how you choose to pronounce it, stop using kubectl outside of isolated lab environments (e.g., local kind) and break-glass scenarios (e.g., outages). Instead, build clusters as immutable infrastructure using IaC and GitOps to maintain a known, auditable state protected by configuration management. Both prod and pre-prod.

183

64 Kommentare

About Malte Ubl (Vercel's CTO) talk on "Framework Defined Infrastructure (FDI) – an Evolution of Infrastructure as Code (IaC)". Some thoughts: - FDI seems to be a specialized approach than a direct evolution of IaC. - The core idea behind FDI - allowing devs to work within familiar framework contexts - highlights the growing demand for more abstracted infrastructure interfaces. - FDI and similar concepts demonstrate the need for IDPs that can sit on top of IaC, providing a more developer-friendly experience without replacing the underlying IaC principles. While I'm not convinced FDI is the next step after IaC, it reflects the ongoing efforts to make infrastructure management more accessible to developers and obvious demand for IDP experience. 🔗ref https://lnkd.in/gU-9wT5J #InfrastructureAsCode #DeveloperExperience #FDI #DevOps #IDP

5