Don't wait for services which are not expected to restart

[davisagli]

Some services are expected to run once after deployment, then stop trying to restart. (for example: a service to run a migration script) The action currently does not correctly detect when these have converged, and gets stuck waiting.

I'm talking about a service with a deploy configuration like this:

      restart_policy:
        condition: on-failure

[fredvd]

I have experienced maybe related issues with services thar are to be started by our cronjob service (swarm-cronjob) or only on demand when scales up manually.

The state of the service in the swarm state seems to be incomplete or inconsistent. When I try to scale the servicr manually, it hangs on scaling up.

The only workaround I have found so far, is to scale the service fist to zero (scale=0). Then a onetime scale to 1 is properly executed.

I was checking this because of the pack service issues we had a few months ago, where I wanted to check if and how they were actually running.

[davisagli]

@fredvd That's a different consequence of the same root cause, which is that when a service is configured to run once and then not restart, Docker still tracks that it was scaled to 1 replica. So setting it again to 1 changes nothing, and the service does not start again. I don't have a better workaround than the one you found.

By the way, my workaround for this issue on deployment was:

• set the service to 0 replicas so it is not part of the deployment
• use the scale-after option in the docker-stack-deploy action to scale it up after the deployment has converged

[davisagli]

I looked at this again.

The current checks are based on docker service ls. It's hard to distinguish there between a service that ran to completion and a service that failed to start (both show `"Replicas": "0/1").

I just found out about mode: replicated-job which we could use for this kind of service. Then we end up with "Replicas":"0/1 (1/1 completed)". So we could modify the wait script to understand that format too.

Note: switching an existing service to mode: replicated-job requires removing and re-deploying it.