The Dynatrace Go team is proud to announce full code-level visibility into the performance of your Golang-based applications, without requiring any changes to your code or your application (requires OneAgent v1.145 or above).

The analogy to ultrasound examinations is quite striking. Dynatrace OneAgent interferes as little as possible with the applications it monitors while exploring the performance of those applications at the deepest possible level. While our top priority is preserving the integrity of the applications we monitor, OneAgent has a largely diametrically opposed purpose—to reveal all details of application processing.

Go applications are unique in that they compile to binary executables and are executed on bare metal by the CPU. Go lacks the abstracting layer of a virtual machine with defined monitoring hooks, like Java JVM or .NET CLR. Therefore, Dynatrace OneAgent needs to operate on the CPU instruction level to intercept Go function calls. 

This is what makes the Go team most proud of our Golang offspring—knowing that Dynatrace OneAgent for Go is unique in its capabilities. It’s the only solution in existence for monitoring Go applications without source code changes or even application recompilation.

We’ve selected Cloud Foundry Diego’s BBS (Bulletin Board System) to showcase the new OneAgent capabilities. BBS is the central data store and orchestrator of a Diego cluster that uses protocol buffer transporting HTTP requests to communicate with other Cloud Foundry instances.

The process view example below represents a single BBS service running in a Cloud Foundry deployment. Note the new HTTP metrics.

The default web request service summary page is shown below.

Within the default web request service execution, Dynatrace has identified two hotspots. Let’s first drill down to the high service resource consumption hotspot.

Hotspots view unfolds the critical code execution path for entry point /v1/actual_lrps/start to the executing Go handler function StartActualLRP. The acronym “LRP” stands for Long Running Process. Drilling further down into the StartActualLRP execution path reveals requests to the ActualLRP MySQL database.

Next, we investigate the service response time hotspot that Dynatrace has identified. Entry point /v1/desired_lrp/desire.r2 triggers an outgoing request to the Cloud Foundry auctioneer component. This client web request is contributing most of the request response time.

Expanding the stack reveals the composition of the LRP auction request towards auctioneer. The various wrap and record functions indicate usage of a frequently used pattern for wrapping request handlers in Lambda functions to execute code before and after invoking the handler function.

The final stage of our journey into the processing details of BBS features the Dynatrace CPU profiler. The CPU Profiler enables you to investigate code that’s executed outside the context of a web request (for example, background activities and schedulers).

Dynatrace OneAgent understands how much of your code is executed and the amount of CPU that’s consumed by each web request. Background thread CPU and Service request CPU are tracked separately. For example, the Go net/http package (see below) spawns a new Goroutine for each incoming web request (more precisely, each new connection). Thus, the Goroutine executing the web request handler function is tagged as service-related. Other Goroutines, which are started in advance of processing the web request, are also considered to be part of the service execution and are therefore also tagged. On the other hand, processing done by Goroutines that don’t have the service related tag are accounted for as background activity.

The result of this categorization can be seen in the background activity Hotspots view below.

The function names of most of the call trees listed indicate that they do indeed execute background tasks. We can identify the call tree of the BBS Converger process, a worker pool that efficiently executes work items in a set of pooled Goroutines, the socket level connection part of web request processing, and a call tree of ifrit, a process management package hosted on github.com.

If you have any questions concerning Dynatrace, please write us at dynatrace@bakotech.com.