The first of the Application Observability Code Challenges is now available. A simple Spring Boot application with a REST endpoint implemented in Jersey/JAX-RS This application has some unexpected behaviour. Can you find out what is happening based on the observability data? Can you improve the observability to prove that a fix really fixed the problems?

Application Observability Code Challenges are a way to challenge yourself to improve the observability of a sample application. Of course, this application has some surprises that you probably did not expect. These challenges will help you develop an observability mindset and become more familiar with your observability tools.

Your local development of a Grafana plugin or dashboard may lack representative data. Connecting to production data sources may be impossible, but there are other ways to connect. Read more to find out how to use Grafana as a proxy to access the data.

The power of observability becomes visible when you combine signals and put it all together. Checks performed by Checkly can provide a piece of the puzzle that gives you a better view of what is happening. It would be really nice if you could combine the data from Checkly in the same Grafana dashboards as your other observability data. This post describes how to bring it together and how to add Grafana annotations to your dashboard based on Checkly results.

Updates from Checkly Based on my previous post I have been in contact with Checkly about two parts that were not working as expected. I have some updates on these topics today. Sending traces from Checkly to Tempo In the previous post I mentioned that the traces in my Grafana environment showed root span not yet received. The root span was not available. This requires Checkly to send these spans to my environment as well.

Checkly This week I came across Checkly, a code-first synthetic monitoring solution that can do API checks and browser checks using PlayWright. It would have saved me a lot of time if this had been around 8 years ago. At that time I was working on a project where we were doing some external checks with a vendor like Checkly, but it was all manual configuration in a web interface. And internally we had a checker tool running that did all sorts of internal checks, both HTTP checks and database checks.

An announcement was made at GrafanaCON. Alloy is introduced in the family of Grafana tools. Alloy is an open source distribution of the OpenTelemetry Collector, but is will also replace Promtail. In the Observability Toolkit I use both Promtail and OpenTelemetry Collector, so it makes sense to merge them. In this blog post I will replace Promtail with Alloy. In another post I will see how replacing of OpenTelemetry Collector with Alloy will look like.

Grafana Scenes can enhance your experience with Grafana dashboards by bringing observability data together and guiding users to the right data. Combining metrics, traces and logs helps to understand the actual behavior of a system. That is the goal of observability. If a dashboard or an app created with scenes does not give the right insights, then still the explore feature of Grafana is available to dive into all data available searching for the unknown-unknown.

Deep understanding of the actual behavior of a system is key for me. I like to know all the details because it helps me to see the whole picture. I am visually oriented, so tools like Grafana really help me in these situations. Observability is a topic that has come up in the last few years, but I have actually been doing this for years. But the good thing is that it is more standardized now.

One of the great things about OpenTelemetry is the standardisation of span attributes and resource attributes. An example of this is deployment.environment. Attribute Type Description Examples Requirement Level deployment.environment string Name of the deployment environment (aka deployment tier). staging; production Recommended Source: https://opentelemetry.io/docs/specs/semconv/resource/deployment-environment/ With Grafana Faro you can configure the environment, as I explained in my blog post about the setup of Grafana Faro. This adds environment to the setup. In my technical deep dive blog post I explained that the data is sent as logs and as traces.