Fabian Valle

These are very interesting insights (and also highlight the importance of not having a separation agreement)! It reminds me of Ron Westrum's theory of the Technological Maestro - https://lnkd.in/eeu9iyWQ The leader of the organization must be an expert in the field (it doesn't mean an expert coder, an expert in the industry). I believe Serverless is a game-changer in Cloud (as the next generation of Compute). This shift is not happening overnight. But I would hate to think that a large Cloud Provider is losing a competitive advantage in next generation Compute. Tactics will come and go, but Strategies can be decades long. Let's hope Compute remains a strategy - regardless of technology, Compute and Storage are not going away. GenAi is a tactic that may evolve into a Strategy, but don't let a new stream eat the existing value stream. Interesting times ahead.

17

🔧 Transform Your DevOps with Generative AI! 🔧 Don't miss our upcoming webinar featuring Patrick Debois, co-author of the DevOps Handbook. Learn how to integrate genAI into your DevOps workflows, tackle common misconceptions, and scale your initiatives for sustainable success. Ideal for DevOps, infrastructure engineers, and continuous delivery innovators. Register now! https://hubs.la/Q02xXvpt0 Can't make the time? Feel free to sign up anyway and we'll send you a recording after the session.  #kubernetes #devops #genAI K8s

7

AWS support is the worst service I’ve come across in years. It’s a perfect Kafkaesque doom loop where everything takes you back to a page to submit a case request with the cheery message “Hello we’re here to help” Except they’re not - as unless you’re a root user it won’t let you submit a case request - which if the problem is an issue with the root user account is a complete Catch-22 #aws #amazonwebservices #support #goingroundincircles

2

1 Comment

With #IcebergSummit happening tomorrow online, I took a trip down memory lane by listening to the #DataFemme episode featuring Alex Merced of Dremio. This episode is packed full with information, not just about #Apache Iceberg, but about the best ways to get started as a contributor to #opensource projects. Alex is an expert content creator with extensive knowledge about Apache products and their role in technology. This is a must listen for everyone heading towards Iceberg Summit tomorrow! https://lnkd.in/guqYdzG2 Tabular Amazon Web Services (AWS) bodo.ai Cloudera Confluent PuppyGraph Snowflake Starburst Upsolver

8

Hey there #linkedinfam! Another Friday means another This Week in #AWSCommunity. I am giving you all of my #examprep resources to passing the AI Practitioner beta exam. Enjoy! https://buff.ly/3TKloeg #aws #awscloud #awstraining #amazonaws #amazonwebservices #amazonwebservicesaws #awscommunitybuilder #awscommunitybuilders #certificationtraining #trainingdevelopment #traininganddevelopment #exampreparation

9

1 Comment

i wrote a doc called "llm-for-sre" over a year ago. what was i missing? what's actually working today? --- Can sort of do today: - Generate code - terraform - yaml for k8s (deploy app, create clusters, add healthchecks and limits) - exponential backoff + jitter to http endpoint (go, python, java, js) - Deploy configs - write a canary deployment strategy using any CD tool you want - "now use circleci instead" - Observability - instrument code - create prometheus rules - create SLO and burn rate alerts - General advice - how to set up a multicluster app - what should i do when something breaks? Imaginary wishlist: - Extracting patterns - watch all my logs, find patterns, what should i fix next? - connect to SLOs, alerts - dependency analysis, finding temporal + topological causality - all things "AIOps" etc - Discovering dependencies? - Finding predictive signals? - Look through old postmortems, find themes or common mitigations, recurring patterns. - extract summaries for training - Incident Response - collect my logs during an event - calculate impact  - integrate under a curve for me?   - do SOAP:  - Subjective - SLOs - "it hurts when i do this" - customer impact - Objective - add to context all my yaml etc  - Analyze - think through every model, context - Plan - which generic mitigation should i perform - Systems Control - watch SLO, error rate, take action! - rollback, drain, other generic mitigations - Forecasting - predict traffic, quota, usage, cost Oncall people stuff - handoff analysis, "oncall logbook" ingestion, handoff generation, "situational awareness" assistant - find similar pages/alerts going on across the organization - impact assessment (see: calculate impact above) Next Experiments: - Toil reduction / Automation - transform folder of bash scripts and docs into $other code + scheduler + database? - Architecture: can an LLM read diagrams?  other architecture docs?  can it make suggestions, see gaps?   - Incident analysis: can an LLM read postmortems / incident reports?

74

10 Comments

Logs Rehydration from S3: Data Resurrection ⚡️ Last9 What it does: - Revives cold S3 logs - Merges with hot data - Unified querying across all logs Why you'll love it: - Cheap storage, on-demand analysis - Historical data at your fingertips - Lightning-fast restoration How it works: - Archive → Point → Query. Simple as that. Yes. Done. Use cases: - Time-travel security audits - Trend analysis across months - Solving old, mysterious bugs Bring your logs back from the dead!

18

Join me next week for a discussion about table partitioning at the inaugural 🐘 Postgres Meetup for All 🐘 virtual meetup. We'll talk about partitioning basics, the why and how, and even a few things that recent versions of Postgres have improved. https://buff.ly/4eRUJUU The goal for "Postgres Meetup for All" is to provide not only a variety of speakers and topics, but time for discussion and interaction. I'm planning on a round of introductions from attendees, a short presentation (20-30 minutes), and then discussion time. If you're thinking about table partitioning, working through issues, or even have some tips of your own to share, join us for this discussion.

17

Learn about the evolution of the Elasticsearch .NET client and how it will gradually phase out the old NEST library: https://gag.gl/tCCrQX #ElasticSearchLabs

1

Absolutely love this! Please watch if you have time. I think this short video does a wonderful job briefly explaining the importance of the Cloud Native Computing Foundation (CNCF), which Taylor D. does a wonderful job describing. I also think it helps shed light on how community is likely to play an increasingly important role in (1) software development and (2) ecosystem development for the future, *especially* in the era of Cloud and AI!

9

If it's Friday, you know it's another This Week in #AWSCommunity. This week I'm covering the top takeaways from #AWSReInvent. https://buff.ly/49tshYi #aws #awscloud #awstraining #awseducate #amazonaws #amazonwebservices #amazonwebservicesaws #awscommunitybuilder #awscommunitybuilders #awspartner #buywithaws #awsmarketplace

1

The Medallion architecture can be a good conceptual tool when building data ETL pipelines. The bronze level contains raw data, the silver level has cleaned/joined datasets, and the gold level has business level aggregates. Delta Lake is a open table format for the Medallion architecture because it allows for reliable transactions, deduplication, small file compaction, and performance optimizations (Liquid clustering, deletion vectors), among many other features. Check out this blog post if you're interested in learning more: https://lnkd.in/eQjGiGbc The blog also has an excellent link to a video by Simon Whiteley on why the Medallion architecture should not be rigidly applied in organizations. The Medallion architecture is a great conceptual framework sometimes, but other data modeling frameworks should be used other times.

150

6 Comments

Customers with existing data and documents currently managed on #AWS can now easily set up a #RAG application using SearchBlox. Setting up a RAG solution for documents stored within S3 buckets or filesystems, etc., requires developers to go through multiple steps: #Data crawling and ingestion, processing the suitable chunks for embedding and storing within a vector database, converting the user query into a vector search against stored chunks and then retrieving the most relevant chunks of information for augmenting a personalized response with a large language model (#LLM). https://lnkd.in/esjaGgWA

1

1 Comment

Ever inherited a Kafka cluster that’s a total mystery? No idea who built it or how it’s supposed to work? We get it. That’s why we offer a 5-day cluster health check. We’ll examine everything from server setups to app integrations, security, and capacity. We’ll untangle the mess and give you a clear picture of what you’re dealing with. So you can rest easy, knowing your Kafka cluster is in good hands!

82

2 Comments

I appreciate there are only a few Organizations with Data Infrastructure spanning across 2100 Clusters having 16k nodes and three regions. But still, if that’s your thing, and especially on MySQL – Uber recently shared their in-place MySQL 5.7 to version 8.0 upgrade (https://lnkd.in/gav6QFAm). It’s a great resource and calls out pre-process-post steps along with learnings and outcomes. Also, there are some learnings which could vary or be re-purposed for any such initiatives: 1. Cost to Value: Overall upgrade spanned a year. With 16000 nodes and 2100 clusters, that’s about ~1334 nodes or 175 clusters upgraded every month. Even with automation, that’s a considerable cost with testing and ongoing maintenance. Though with this, Uber realized 29% and 33% improvement in write and read ops. Considering consumption of 3mil queries per second – that’s a substantial saving. So, whilst multiplier effect lands a greater value in Uber’s case – lack of scale could outweigh the advantage and leave with higher costs. 2. Delayed version upgrade: For Prod, I have generally seen Org’s staying 2 steps behind latest release. Considering that, and even with MySQL debate with version 6 and 7, I would have preferred to upgrade earlier than v5.7 reaching EOL. 3. Client Library Incompatibility: For a rollout of this scale – its imperative to plan across the ecosystem, including client end consumption (or connect libraries). A vendor would almost always test only the core operation and functionality, interoperability or connect libraries often leading to interesting challenges. Plan for the core change, but then also look at the edges and end-to-end flow. #engineering #upgrade #mysql #uber #learnings

7

The blog post below illustrates how a small company transitioned from an $11k monthly spend (it's implied but not explicitly stated the spend is monthly) to only $2k by switching their hosted architecture model. It also illustrates the costs of not doing your homework up front on a scalable model. An AWS consultant could have advised the pitfalls of using an RDS model for the kind of workload and probably charged a few grand for the advice, saving potentially tens of thousands of dollars if it was done during the design phase. The best time to contact an expert is before you have a problem, so they can help you avoid it. The second-best time is right now, so they can help guide you out of the problems before they become worse. If you'd like some solid Microsoft SQL Server advice, including advice on AWS RDS, Azure SQL DB, and other cloud options, message me here or visit my website sqltailor.com to schedule a consultation. You'll be surprised by the ROI of a 30-minute call.

5

Some early next-day thoughts about #DASH2024 in New York City this year: 👌 Logs Workspaces look super promising ☀️ Flex Logs go GA (can't wait for Flex Traces 👀) 💥 On-call (👋 Opsgenie & PagerDuty) 🙈 GitHub is on Datadog (and OpenTelemetry!) 💸 Paypal is on Datadog too! 💜 OpenTelemetry + Datadog are getting closer than ever 🎉 Serverless improvements 🥂 Product Analytics revamp, becoming a proper competitor with products like Amplitude 🤖 Bits AI is being used in more and more products 👏 App Builder (custom UI components you can build yourself into the Datadog UI) is better than ever. 🤯 Live Debugger - merges the concepts of debugging and tracing 🔐 Datadog secures the entire software development cycle My biggest takeaway though is that Datadog is bigger than ever. But they are still developing products at an incredible pace and quality. From a Product perspective, they are hitting the nail on the head over and over again. Datadog releases things as early as possible as "Beta", "Private Beta" etc. This may frustrate users (as early products are immature), but it actually provides incredible opportunities for feedback. A bit of an observation: There are a lot more instances where Datadog goes past this "understanding platform" idea. It's now moving to be able to take action. This can be a slippery slope but I do think in many cases it makes loads of sense. A roundup from Datadog themselves of the keynote:

25

4 Comments

Check out this blog to learn how to perform ES|QL queries with the Java client: https://gag.gl/DAzQia #ElasticSearchLabs #Java

1