🗺 Distributed Tracing

Distributed Tracing enables tracking of requests as they pass through the microservices in a system. Anteon captures Traces and Spans out-of-the-box on your Kubernetes Cluster without modifying your code or restarting your services. Anteon uses a lightweight eBPF Agent (Alaz) to generate Distributed Tracing data and enable auto context propagation.

To do so, on the Anteon Platform, click "Clusters" under the "Observability" tab first.

After you click on a cluster you want to observe on the clusters page, click on the distributed tracing tab. This will take you to the distributed tracing page. You will see this:

Here, you can filter and view traces in chart and list form.

Filtering​

You can apply the following filtering:

• Operations (trace names)
• Trace count
• Tags
• Min/max duration (in ms)
• Min/max span count
• Time interval

To apply filtering, click the "Filters" button on the upper right.

Operations​

Operations filtering allows you to select traces with specific names. It allows multi-select so that you can look for multiple names.

Max Results​

This limits the number of traces shown.

Tags​

Tags allow you to filter traces by the tags in their spans. Tag filtering accepts the logfmt format.

Let's see how we can query traces by their tags. Let this be an example tags for a span:

{
    "path": "/", 
    "to_ip": "192.168.20.73", 
    "method": "GET", 
    "to_uid": "d0210c8f-fe44-472c-bbb2-39d64973e5a4", 
    "from_ip": "192.168.30.206", 
    "to_name": "kube-state-metrics-shard", 
    "to_port": "8081", 
    "to_type": "daemonset", 
    "from_uid": "2148b55d-cab8-41c6-ab5e-b12d9630ba0c", 
    "protocol": "HTTP", 
    "from_name": "aws-node", 
    "from_port": "41736", 
    "from_type": "daemonset", 
    "status_code": "200"
}

An example tag filter could look like this:

method=GET protocol=HTTP from_port=41736

While filtering, it is enough for a single span of a trace to satisfy this filter. As a result, the traces with at least one span matching this filter are retrieved.

Min/Max Duration​

You can filter the traces by the total time they took in ms.

Min/Max Span Count​

You can also filter the traces by their span count.

Time Interval​

To apply a time interval, click the "Last 5 min" button next to the "Filters" button. You will see this:

Here, you can specify your desired time interval. You can specify your start-end times yourself or pick predefined time intervals from the "Quick Selections" section.

Trace Chart​

The trace chart allows you to view the distribution of your filtered traces by their dates and durations. While the x-axis is the date, the y-axis is the duration.

To view more details (start time, name, duration, span count) about a trace, you can hover on it:

Trace List​

Under the trace chart, you can view the filtered traces in a listed format. You can delete them here as well. You can also use the page numbers below to see the remaining traces.

Trace Details​

After you click on a trace (either on the chart or the list), you can view its details (including its call hierarchy and time).

Each row represents a trace, whereas the columns represent the time.

No matter how long your trace took, the whole hierarchy is always fitted inside your view, so you would not need scrolling/zooming to understand the trace.

You can expand/collapse spans by clicking on their "Name" column.

You can also hover over a span to see its details, such as:

• Source/target ports
• Path (endpoint)
• Status code
• Request method
• Start/end time

For more details, you can click on a span:

Here, you can view the span's details, including:

• From/to resources
• From/to IPs
• From/to port numbers
• Start/end times
• Duration
• Path
• Method
• Protocol
• Status code