Context
We are using Consul Template to manage dynamically the configuration of our apps.
Depending of the application we might have 3 behaviours:
- The configuration is reloaded automatically by the process when it change (e.g watch in PM2)
- The process MUST be reloaded running a command (e.g
nginx -s reload)
- The process MUST be restarted to take the changes in consideration
For the 2 last cases, we want to reload / restart services one by one in order to prevent service interruption.
See this issue: hashicorp/consul-template#340
Implementation
Consul already provides a lock mechanism to create a distributed semaphore based on Consul Key: https://www.consul.io/docs/commands/lock.html
We want to implement this mechanism wrapping the complexity to configure consul-template and consul lock command in a script ct-supervise :
ct-supervise -n 1 'nginx -s reload'
Options
These options might be passed to the ct-supervise script
monitor-retry - Retry up to this number of times if Consul returns a 500 error while monitoring the lock. This allows riding out brief periods of unavailability without causing leader elections, but increases the amount of time required to detect a lost lock in some cases. Defaults to 3, with a 1s wait between retries. Set to 0 to disable.
--n: Optional, limit of lock holders. Defaults to 1. The underlying implementation switches from a lock to a semaphore when increased past one. All locks on the same prefix must use the same value.
--name: Optional name to associate with the underlying session. If not provided, one is generated based on the child command.
--pass-stdin: Pass stdin to child process.
-
-try: Attempt to acquire the lock up to the given timeout. The timeout is a positive decimal number, with unit suffix, such as "500ms". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-verbose: Enables verbose output.
The only required argument is the command to run:
ct-supervise 'nginx -s reload'
Context
We are using Consul Template to manage dynamically the configuration of our apps.
Depending of the application we might have 3 behaviours:
nginx -s reload)For the 2 last cases, we want to reload / restart services one by one in order to prevent service interruption.
See this issue: hashicorp/consul-template#340
Implementation
Consul already provides a lock mechanism to create a distributed semaphore based on Consul Key: https://www.consul.io/docs/commands/lock.html
We want to implement this mechanism wrapping the complexity to configure
consul-templateandconsul lockcommand in a scriptct-supervise:Options
These options might be passed to the
ct-supervisescriptmonitor-retry- Retry up to this number of times if Consul returns a 500 error while monitoring the lock. This allows riding out brief periods of unavailability without causing leader elections, but increases the amount of time required to detect a lost lock in some cases. Defaults to 3, with a 1s wait between retries. Set to 0 to disable.-
-n: Optional, limit of lock holders. Defaults to 1. The underlying implementation switches from a lock to a semaphore when increased past one. All locks on the same prefix must use the same value.-
-name: Optional name to associate with the underlying session. If not provided, one is generated based on the child command.-
-pass-stdin: Pass stdin to child process.-try: Attempt to acquire the lock up to the given timeout. The timeout is a positive decimal number, with unit suffix, such as "500ms". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".-verbose: Enables verbose output.The only required argument is the command to run: