What's up docker? How to keep your containers up to date

Share your Projects!
,
1

Hi,

As everyone using Docker for all my home ecosystem, I was looking for a simple solution to keep all my Docker containers up-to-date.

I tried Watchtower and Ouroboros so far but I experienced some problems:

  • Auto updates caused issues because of unplanned service downtimes
  • Image versioning strategy is not consistent across the different images (no semver…) so false positive updates occured
  • using latest versions was even worse because all control was lost regarding update strategy (broken service because of major upgrade…)

I concluded that what I really wanted was:

  • Be automatically notified as soon as an update is available (instead of manually crawling the Docker hub to see if there are updates)
  • Manually apply the update (to review the release notes, decide when to update…)

So I coded my own solution which is running fine at home for more than 6 months: What’s up Docker?

  • Github

  • Documentation

  • Docker image

  • Watchers monitor Docker hosts to check updates (versioning strategy can be fine tuned per image using regular expresssions to avoid false positive matches)

  • Triggers are executed when updates are detected (email, slack, mqtt, webhook…)

Feel free to test!
Feedback and contributions are welcome!

10 Likes
2

What issues die you face with watchtower? That can also just notify without updating.

Without updating containers

Will only monitor for new images, not update the containers.

            Argument: --monitor-only
Environment Variable: WATCHTOWER_MONITOR_ONLY
                Type: Boolean
             Default: false
3

that’s very interesting. Quick question, how do you go by setting up the triggers via MQTT?. I saw the example, but I’m not sure where to configure it.

4

Hi @mistrovly ,

All settings are applied using environment variables on the whats-up-docker container.
To configure an mqtt trigger, you just need to set 1 env var (+ 2 other ones if authentication is enabled on your broker).
A typical example to monitor the local Docker host and publish updates to an mqtt topic could look like to:

docker run -d --name wud \
-e "WUD_TRIGGER_MQTT_HASS_URL=mqtt://hass:1883" \ 
-v "/var/run/docker.sock:/var/run/docker.sock" \
fmartinou/whats-up-docker:1.0.0

The updates will pe published, by default, to a topic named /wud/image.

5

thank you very much for taking the time to reply. I’ll give a try

6

Hi @PostSven,

Thank you for your feedback.
When I started with my own project, it wasn’t possible with Watchtower.
There was an issue opened in 2017 which wasn’t implemented at that time.

Furthermore Watchtower and What's up Docker? don’t work the same way.

Watchtower detects updates by regularly pulling the used tags & compare the layers.
Except for few images where the tags are well named (latest = 1 = 1.2 = 1.2.3, etc), the only way to use Watchtower is to use latest tags.

My need is to use immutable tags (e.g. 1.2.3) and be notified when a new tag is pushed to the registry (e.g. 1.3.0).
This feature is missing on Watchtower (see this opened issue).

What's up Docker? doesn’t pull anything, it only uses the Docker Registry REST API to find the Image tags & metadatas and check if something new has been pushed.

1 Like
7

I am going to give it a go! :pray:t3:

1 Like
8

I’ve installed everything correctly as far as I know and I see one “set” of logs in the terminal and one line of those same logs on the /store/ directory i created.

but the logs say it can’t find any watcher.

And even tho I’m (no longer…) getting any errors in the logs about connecting to my MQTT broker (so I think it connected OK…maybe…?) I’m not seeing any mqtt messages on my selected topic. I’ve set the cron for every minute just to make sure I know everything is working before I set it back to once a day.

Here is my docker run command:

sudo docker run -d --name wud -e WUD_LOG_LEVEL="DEBUG" -v "/var/run/docker.sock:/var/run/docker.sock" -v /home/finity/docker/wud/store:/store -e WUD_WATCHER_DOCKER_LOCAL_CRON="1 * * * *" -e WUD_TRIGGER_MQTT_MOSQUITTO_URL="mqtt://192.168.1.11:1883" -e WUD_TRIGGER_MQTT_MOSQUITTO_USER="myuser" -e WUD_TRIGGER_MQTT_MOSQUITTO_PASSWORD="mypwd" -e WUD_TRIGGER_MQTT_MOSQUITTO_TOPIC="wud/image" fmartinou/whats-up-docker

Here is the log:

finity@NUC:~$ sudo docker container logs -f wud
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"What's up, docker? is starting","time":"2020-02-03T22:29:40.090Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"Init DB (/store/wud.json)","time":"2020-02-03T22:29:40.092Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"Register all components for provider mqtt","time":"2020-02-03T22:29:40.100Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"Register component Mosquitto of type Mqtt with configuration {\"url\":\"mqtt://192.168.1.11:1883\",\"user\":\"myuser\",\"topic\":\"wud/image\",\"password\":\"mypwd\"}","time":"2020-02-03T22:29:40.198Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"No Watcher found => Init a default Docker one","time":"2020-02-03T22:29:40.208Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"Register component Local of type Docker with configuration {}","time":"2020-02-03T22:29:40.513Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":30,"msg":"Schedule runner (0 * * * *)","time":"2020-02-03T22:29:40.516Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/fmartinou/whats-up-docker:latest","time":"2020-02-03T22:29:40.604Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/library/influxdb:latest","time":"2020-02-03T22:29:40.611Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/raymondmm/tasmoadmin:latest","time":"2020-02-03T22:29:40.611Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/esphome/esphome:1.13.6","time":"2020-02-03T22:29:40.612Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/linuxserver/code-server:latest","time":"2020-02-03T22:29:40.612Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/nodered/node-red:latest","time":"2020-02-03T22:29:40.613Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/homeassistant/home-assistant:latest","time":"2020-02-03T22:29:40.613Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/philhawthorne/ha-dockermon:latest","time":"2020-02-03T22:29:40.614Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/linuxserver/openvpn-as:latest","time":"2020-02-03T22:29:40.614Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/nekmo/amazon-dash:latest","time":"2020-02-03T22:29:40.615Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/acockburn/appdaemon:latest","time":"2020-02-03T22:29:40.615Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/library/eclipse-mosquitto:latest","time":"2020-02-03T22:29:40.615Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/linuxserver/syncthing:latest","time":"2020-02-03T22:29:40.616Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/linuxserver/letsencrypt:latest","time":"2020-02-03T22:29:40.616Z","v":0}
{"name":"whats-up-doc","hostname":"380f02898055","pid":1,"level":20,"msg":"Check https://hub.docker.com/portainer/portainer:latest","time":"2020-02-03T22:29:40.616Z","v":0}

Anything that you can see I’m doing wrong?

9

Hi @finity,

Thank you for your interest!

About the watcher part

the logs say it can’t find any watcher

When no watcher is configured, by default, WUD creates a default one watching the local Docker host (via the /var/run/docker.sock).

In the logs, I can see that your containers are discovered as expected so everything is fine for this part :slight_smile:

Then, when new versions are found, in DEBUG mode, you should see logs like:

{"name":"whats-up-doc","hostname":"2c47d9698d13","pid":1,"level":20,"msg":"New image version found ({\"registry\":\"https://hub.docker.com\",\"organization\":\"wiremind\",\"image\":\"omnidb\",\"version\":\"latest\",\"date\":\"2018-06-27T14:05:41.004866597Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"size\":194418493,\"isSemver\":false,\"result\":{\"newVersion\":\"latest\",\"newVersionDate\":\"2018-06-27T14:06:18.022056Z\"}})","time":"2020-02-04T09:43:25.793Z","v":0}

Apparently, it’s not the case in your logs.
My guess is that you did several tests and at least one of them found the new versions, stored the information in the /store folder and since that, WUD doesn’t publish new mqtt packets because it considers it was already done.
To validate my assumption, you can open the the /home/finity/docker/wud/store/wud.json file and check if you see data like

result":{ 
  "newVersion":"x.y.z",
  "newVersionDate":"2020-02-04T01:25:22.373367Z"
}

If so, it means that new versions were found but trigger didn’t work as expected.

I advise you to delete the /home/finity/docker/wud/store/wud.json file every time you restart/recreate the container to avoid any caching side effect during your tests.

About the trigger part,
In the logs, I can see that your mqtt trigger is configured and activated.
However I think that your mosquitto URL is not ok ; is the IP 192.168.1.11 reachable from WUD?
If its’ the IP of your host machine, it would work only if you run WUD on the host network

docker run --net host ... fmartinou/whats-up-docker

(If you don’t want to run it on the host network, you need to run WUD & mosquitto on the same network)

I’m going to find some time to add better logs/errors to help with troubleshooting…

10

Thanks for the support!

I added the --net=host setting to the above run command and I think that’s working now. I see the mqtt messages in my broker now.

I’m still confused about why the watcher I created doesn’t seem to work. It’s still saying “no watcher found”. Did I do something wrong there?

11

I’m glad to see that it’s working for you!
About your watcher, I found the issue ; it’s now fixed in the latest version (1.0.1)

12

seems to be working now.

Thanks!

Do you have an example of a working HA notification from MQTT by chance?

I’m sure I could suss one out but if you already have one then why re-invent the wheel? :wink:

13

Sorry, I don’t have any ready-to-go example to share… I haven’t integrated WUD to HASS yet (I’m using the WUD email trigger to be notified).

14

No problem. I just figured I’d ask first before digging in.

15

A very interesting project.

I do agree with your drawbacks regarding Watchtower and Ouroboros. Maybe we should mention Diun too as an alternative. Indeed I believe an automated upgrade is in general not something you should want, and therefore I concentrate on the notification options of all different systems. My main use case that i would like to implement is creating sensors per container and manually press a button when i would like to update the container, after a backup, and a verification of that updated system. Updating itself is not a complex operation, so no direct need to have that in the monitoring software, but it can be an extra advantage.

For Watchtower the most suitable option for that seems to be the lifecycle configurations that calls plain sh scripts. I could however not find any working samples of that, because the sh command is not available in the standard docker container (which uses ash).

Ouroboros uses apprise as notification system, which has a lot of notification options, even native Home Assistant notifications, although I did not look into detail how to configure that. It has some dry run option, that may be used for notification only.

Then both Diun and Whats’up Docker have Webhook or MQTT notifications that could do the job. My main issue with the way both are implemented is that a generic notification is sent and so address a single sensor. So for example when two containers have newer versions and when I configured for example MQTT, my sensor would shortly give the first container outdated, but the second container will remain on that MQTT sensor and first is overwritten with the last update message.

Because I prefer a separate sensor for each container, I don’t see a way how to implement this with this single topic or webhook notification. What should be the approach to have separate sensors. Or should I raise a feature request to sent notifications based on the container name to different MQTT topics or Webhooks?

Thanks.

16

Thank you for your interest!

Just to clarify; I didn’t mention Diun at that time because it didn’t exist while I’d started my project.

But you’re right: Diun and Wud shares the same philosophy.

Regarding your issue, your proposal to separate notifications per image is very interesting.

For MQTT, I could change from a single wud/image topic
to wud/image/$organization/$image/$version topics

e.g.
wud/image/homeassistant/home-assistant/2021.4.3

(for backward compatibility, current users will have to subscribe to wud/image/# to get all notifications at once as now)

I’ m not sure if I just enforce the topic naming or if I offer end-users the possibility to write their own patterns (using a simple placeholder syntax).
e.g.
env WUD_TRIGGER_MQTT_MOSQUITTO_TOPIC=wud/#organization_#image_#version
which will result as notifications sent to a wud/homeassistant_home-assistant_2021.4.3 topic
…

Regarding the webhook part, I don’t really get your issue because I don’t understand how you consider integrating WUD to HASS.

Do you want WUD to call HASS using HTTP to notify of new image versions?
If so, the WUD Http trigger should do the trick, isn’t it?
Even if there is a single URL, the Http Request contains the info about which image is involved (info present in a query param or in the request body).

But for that one too, I could imagine a solution where end-users can write their own URL patterns (using a simple placeholder syntax).
e.g.
env WUD_TRIGGER_HTTP_MYREMOTEHOST_URL=https://myhook/wud/#organization/#image/#version
which will result as notifications sent to a https://myhook/wud/homeassistant/home-assistant/2021.4.3 url

What do you think?
I’d be really happy to fulfill your request; can you open an issue on Github to continue the discussion and find the best solution?

Thanks!

17

Thanks for your answer. I will continue on GitHub.

18

@fmartinou I’m also interested in container specific notifications.

#version in the topic might be painful so maybe just: wud/$image ($serviceName would be even better)?

In order to make a binary sensor easy to create, ideally there would be a Bool in there somewhere as well such as: “updateAvailable”:“true”

19

Thanks for your recommendations!
Now I need to find some spare time to start implementing a solution.
In the meantime, feel free to add comments on GitHub to share your thoughts.
I let you know when I get something ready to test.

20

Hi @eiri @leighcurran ,

A first implementation is available!
Please find all the details on Github.

Any feedback will be much appreciated!