Diagnosing Production Akka.NET Problems With OpenTelemetry

We recently leveraged OpenTelemetry, Phobos, semantic logging, and more to resolve several high-impact, difficult bugs in Akka.NET itself. We want to share the techniques, best practices, methodologies, and tooling we used to do that - so in this webinar you will learn how to:

Collect useful traces, logs, and metrics using a combination of built-in and custom OpenTelemetry instrumentation;
Create useful alerts and dashboards using OpenTelemetry metrics;
Leverage semantic logging and tracing data in platforms like Seq to get behavioral questions answered;
Eliminate wasteful noise and cost using Phobos’ built-in trace filtration system; and
How to configure and deploy the OpenTelemetry agent to collect logs, traces, and metrics and ship them to APM destinations like .NET Aspire, Grafana, Prometheus, DataDog, and so on.

This webinar will take about an hour and everything you learn in it can be easily applied locally on your developers’ machines, in shared environments, or in productions. Everything you learn here will be highly portable and a great tool to keep in your belt.

View the slides from this webinar: https://www.slideshare.net/slideshow/...

Stephen Cleary's Local Telemetry Stack: https://github.com/stephencleary/loca...

Learn more about Phobos: https://phobos.petabridge.com/

Buy Phobos for your organization: http://phobos.petabridge.com/buy.html




0:00 - Intro
0:40 - Table of Contents / Webinar Overview
5:24 - OpenTelemetry at a Glance
8:06 - OTEL Configuration as Code
11:57 - OTEL Logging Configuration
13:15 - OTLP: OTEL Line Protocol
19:30 - Local vs. Production Telemetry Stacks
19:50 - Local Telemetry Stack
22:31 - Production Telemetry Stack
27:14 - About Phobos
30:02 - Metrics: What Data Does Phobos Capture?
31:19 - Demo: Built-in & Custom Dashboards
41:02 - Tracing: What Data Does Phobos Capture?
42:45 - Demo: Tracing Data
48:54 - Generally Useful Phobos Metrics Signals
52:05 - Tracing Examples
55:27 - Phobos Noise Control
58:24 - OTEL Filtering
1:00:17 - Custom OpenTelemetry
1:02:12: Actually Doing Things with OTEL: Metrics
1:06:41 - Actually Doing Things with OTEL: Traces
1:07:32 - Crafting Useful Alerts
1:07:42 - Alert Sources
1:10:45 - Useful Akka.NET Alerts
1:11:35 - Demo: Seq & Grafana Alerts
1:16:38 - Conclusion