Scrapper fails silently and does not recover

[jtexeira]

Report

When scrapping azure metrics fails, after a while the scrapper just gives up but it continues to report everything is healthy and continues to run

Expected Behavior

The scrapper exits on fail or the the health check reports it is unhealthy

Actual Behavior

The scrapper continues to run and never recovers.

Steps to Reproduce the Problem

1. Start Scrapper
2. Remove internet connection

Component

Scraper

Version

2.14.0

Configuration

No response

Logs

No response

Platform

Microsoft Azure

Contact Details

jmt@eurotux.com

[github-actions]

Thank you for opening an issue! We rely on the community to maintain Promitor. (Learn more)

Is this something you want to contribute?

[JohnFrampton]

We encounter that, too. Unfortunately sometimes there is also no log entry hinting that promitor stopped publishing metrics.

[hkfgo]

@jtexeira @JohnFrampton, I believe I've encountered the same issue. This PR. I wrote a while back should at least make sure the scraper doesn't fail silently. However, based on my observation, the scraper still won't recover by itself.

The fix is simple though - the agent just needs a restart. This restart also can be automated based on some criteria(e.g. absence of any scrape success metric). I'm working on this self-healing mechanism for the scraper currently.

Let me create an issue in the public repo as well, and share my progress there.

[hkfgo]

#2688

[hkfgo]

#2694 combined with liveness check against dependencies should result in self-healing through an eventual container restart. Length of time depends on how you configure the liveness check. This is what I used:

        health:
          liveness:
            verifyDependencies: true
            delay: 5
            interval: 60
            timeoutSeconds: 5
            thresholds:
              failure: 3
              success: 1

[hkfgo]

@tomkerkhove any plans to cut a release soon? Would love to roll this out in our production envs

[tomkerkhove]

Will try to cut one ASAP