Clarifications on KEDA scaler behavior with multiple triggers and with/withoutfallback options

[SomaSharathSurya]

I have a couple of questions regarding the behavior of the KEDA scaler in the following scenarios:

Scenario 1: ScaledObject with azure-log-analytics and CPU Triggers, with fallbackReplicas configured

1. What happens when one of the triggers fails to retrieve the metric? Does it fallback to the fallbackReplicas?
2. If the answer to question 1 is yes, is there an option to fallback to the CPU trigger instead of directly falling back to the fallbackReplicas?

log analytics queries are sometimes timing out in our case and wanted to understand whats the best way to control the behaviour in this instance

Scenario 2: ScaledObject with azure-log-analytics and CPU Triggers, without fallbackReplicas configured

1. What happens when one of the triggers fails to retrieve the metric? Does it fallback to the CPU-based scaler, or does it continue running with the current number of replicas?
2. What will be the behavior if both triggers fail?

Couldn't find any detailed documentation on how the behaviour will be in different scenarios, so thought of reaching out for help here. Thanks

[rickbrouwer]

Scenario 1:

1. Yes, it does fallback to the fallbackReplicas
2. No, there is no option for this use case

Scenario 2:

1. It continue running with the current number of replicas
2. Also it continue running with the current number of replicas

Maybe worth mentioning is that in the upcoming release (2.17) we will get an extra option 'behavior' at fallback:

https://github.com/kedacore/keda-docs/pull/1519/files

[mchahar]

Hello @rickbrouwer,

I have tested a ScaledObject with Prometheus and CPU triggers, without fallbackReplicas configured.

1. When one of the triggers fails to retrieve the metric, does it fallback to the CPU-based scaler, or does it continue running with the current number of replicas?
   -- In my testing, when Prometheus was disabled, the application was still able to scale up based on CPU usage. However, it did not scale down, which I was expecting. Based on your comments above, I wanted to confirm if this is the intended behaviour.

[rickbrouwer]

Yes, that is intended behavior.

[SomaSharathSurya]

Thanks @rickbrouwer for sharing the details.

In our environment, we configured maxReplicas to equal fallbackReplicas to ensure that there is no impact on consumers in the event of a failure. However, we have observed that the LAW scaler often fails to retrieve the metric due to query timeouts or network issues, resulting in unnecessary scaling. We have tried adjusting failureThreshold, pollingInterval, and enableCache settings, but the issue still happens sometimes

I am wondering what's the best way to address these types of problems. Any thoughts / suggestions from your experience ?

[SomaSharathSurya]

@zroubalik Yes, for log analytics. It would be greatly appreciated if this could also be done for Datadog and Prometheus scalers.

[zroubalik]

Azure Log Analytics and Prometheus added here: #6607

Datadog is on hold, we need to merge this refactoring first: #6552

[SomaSharathSurya]

@zroubalik That's great, thank you. Appreciate the prompt response and actioning on this.

Just wondering will this be included in v2.17 that is schedule soon, or this will be part of a later release ?

[zroubalik]

There's a high chance it will be in 2.17, hopefully :)

[alt-dima]

@zroubalik Hello! Do you think my idea/PR with "cached value" option makes sense and maybe merged to Keda? Or there are some other plans/ideas? (if yes, I will resume work on the PR)

this actually will fix/solve/implement the idea about two providers (datadog + internal cpu) and continue to scale on internal cpu if datadog failing

#7317

[alt-dima]

So after I learned about #6603 (reply in thread)

I think we just need to disable/remove fallback and we will get scaleUP working by second (CPU) trigger

So I will close my #7317