Unable to receive metrics in OpenTelemetryCollector sink

[AxelEkdahlWcar]

Hi,

I've started to try out the Promitor scraper. I've successfully managed to configure the scraper so it can fetch metrics and I would now like to ship these metrics to my OpenTelemetryCollector which is deployed in the same namespace. However, it seems like the scraper is not executing the logic for sending the metrics.
As per the implementation I should be able to see trace logs verifying that the metrics should've been sent but I see no such logs.

The validation of the scraper succeeds and it seems to find my OpenTelemetryCollector configuration since it prints the configuration at boot up.

Is there any way of troubleshooting why this is?

Please see my configuration below.

metricSinks:
  openTelemetryCollector:
     enabled: true
     collectorUri: "http://otel-gateway-collector:4317"
telemetry:
   defaultLogLevel: "Trace"
   applicationInsights:
     enabled: false
     key: ""
     logLevel: ""
   containerLogs:
     enabled: true
     logLevel: "Trace"

The only logs my scraper prints are the following

[12:05:00 INF] Timeout after 90
[12:05:00 INF] Init timeout token
[12:05:00 INF] Scraping for resource type PostgreSql.
[12:05:00 INF] Parsed SDK Config True
[12:05:02 INF] Found value 7.4 for metric with aggregation interval 00:01:00
[12:05:02 WRN] release lock

I've tried with 2.12.0 and 2.14.2 versions of the promitor agent scraper

Many thanks for any help in the right direction

[hkfgo]

Do you see error logs like the one listed here: #2665?

I think the OTEL Collector sink needs a fix.

[AxelEkdahlWcar]

Hi thanks for replying. No actually I don't get any log line at all.

Here's an extraction of the logs I do get.

[04:50:00 INF] Scraping PostGreSQL for resource type PostgreSql. 
[04:50:00 INF] Parsed SDK Config True 
[04:50:00 INF] Found value 6.88 for metric PostGreSQL with aggregation interval 00:01:00 
[04:50:01 INF] Request starting HTTP/1.1 GET http://10.67.133.214:5000/api/v1/health?includeDependencies=false - null null 
[04:50:01 VRB] Cannot determine whether or not the endpoint contains the 'ExcludeRequestTrackingAttribute' because the 
endpoint tracking (`IApplicationBuilder.UseRouting()` or `.UseEndpointRouting()`) was not activated before the request tracking 
middleware; or the route was not found 
[04:50:01 VRB] Correlation transaction ID '8015a5d416f0b9a717510367fe4bd982' generated for incoming HTTP request 
[04:50:01 VRB] Correlation operation ID '837542187b52c0f6' generated for incoming HTTP request 
[04:50:01 VRB] Prepare for the operation ID to be included in the response... 
[04:50:01 VRB] Prepare for the operation parent ID to be included in the response... 
[04:50:01 VRB] Prepare for the transactional correlation ID to be included in the response... 
[04:50:01 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature 
System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.Scraper.Controllers.v1.HealthController (Promitor.Agents.Scraper). 
[04:50:01 VRB] Setting correlation response header 'X-Transaction-ID' to '8015a5d416f0b9a717510367fe4bd982' 
[04:50:01 VRB] No response header was added given no operation parent ID was found 
[04:50:01 VRB] Setting correlation response header 'X-Operation-ID' to '837542187b52c0f6' 
[04:50:01 INF] Executed action Promitor.Agents.Scraper.Controllers.v1.HealthController.Get (Promitor.Agents.Scraper) in 0.1728ms 

[hkfgo]

Are there no further log messages? I can't tell if anything's wrong from the existing log messages. And are you sure failure occurs on OTEL Collector sink exclusively? Does it work fine if you use Prometheus

[AxelEkdahlWcar]

So from looking through the implementation I would expect the following log line to be printed

Metric {MetricName} with value {MetricValue} and labels {Labels} was pushed to OpenTelemetry Collector

as soon a metrics has been sent to the OTEL collector, or simply an error log. However, as you say your self, no such log lines do exist which to me seem like the collected metrics are not sent from the scraper at all.

However, we have now abandoned this project due to this. We managed to solve it in another way through other tooling.