Data platform folks need to get hip to open-telemetry and observability. Lineage is useful, and interesting, but when data teams really learn to use software-grade observability capabilities, they make a leap forward in system reliability and data trust. Look at your existing tooling and start experimenting with how you can switch to using an OTEL-based stack. It’s a lot of what you’ve been wanting. #meatbasedengineer
Words like "master class" and "experts guide" get thrown around a lot, but this piece by Jeremy Morrell may be the single best thing I've ever read on instrumenting your code the modern way, using wide, traceable log events. It's also a terrific vendor-neutral overview of your options in the observability 2.0 space, including open source and hybrid options. Plus code samples. You'll want to bookmark this puppy. Pretty sure I'll be referring back to it for years to come. https://lnkd.in/gURyetg6
Does open-telemetry make sense for dataset observably? The pipelines, services, tools that make up a platform yes. But I’m wondering about the actual datasets that data platform teams support? Is a log/trace the best format for following what’s happening there?
All things Data | Chief Problem Solver | Hired Gun
1moBrian Greene Kinda sorta. Your stack still falls short. While it may address data flow, it misses on some of the EDC things. Structure is transient. Motion is constant. Observability covers a portion but still misses the mark. Also its a bit more complicated that what people think. ( What works for you may not work for others.) Sorry 🤐