From a6f6d493249797be7dbf476366c49148abb30ea2 Mon Sep 17 00:00:00 2001 From: Ognjen Vukovic Date: Sat, 9 May 2020 23:38:45 +0200 Subject: [PATCH] Prepare for version 0.2: crontab file mount and ssh support --- Dockerfile | 1 + README.md | 120 ++++++++++++++++++++++++++++++----------------------- rsync.sh | 38 ++++++++--------- 3 files changed, 88 insertions(+), 71 deletions(-) diff --git a/Dockerfile b/Dockerfile index 288235b..cc274f2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,6 +2,7 @@ FROM alpine RUN apk add --update-cache \ rsync \ + openssh-client \ tzdata \ && rm -rf /var/cache/apk/* diff --git a/README.md b/README.md index d7ff6a2..8129d5f 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,12 @@ The provided image is open-source and built from scratch with the goal to enable you to run a stateless and an immutable container, according to the best practices. +With this image you can: + +* run a simple one-time rsync job for local drives +* run a rsync job for remote drives using ssh (included in the image) +* run scheduled rsync jobs using cron (included in the image) + Supported architectures: * the image supports multiple architectures: `x86-64` and `arm32` @@ -22,9 +28,17 @@ Supported architectures: | `:x64` | targeted to the `x64` architecture | | `:arm32v7` | targeted to the `arm32v7` architecture | +The image is based on `alpine` image and it includes: + +* `rsync` +* `openssh-client` for remote sync over ssh +* `tzdata` for easier setting up of local timezone (for file timestamps) +* `cron` (included in the alpine) for scheduling regular back-ups +* `rsync.sh` script that prepares the cron job and starts the cron daemon + ## Usage -### Quick Start: one time run, local, no logs +### Quick Start: one time run and sync of local folder ```bash docker run --rm \ @@ -40,64 +54,66 @@ Replace: * `/path/to/destination/data` with the destination folder * `[OPTIONS]` with desired rsync optional arguments -### Start with All Options Explained +### Use with cron or ssh -1. _First time only_: Prepare the setup - * create a folder, for example `~/rsync`, that will be later mounted in the container - * this folder is intended to hold supporting files such as log files, a list of files to rsync, a file with exclude patterns, etc. - * the subfolder `logs`, for example `~/rsync/logs` is intended for the log files - * important note: for Docker Swarm, this directory **needs to be available on all nodes in Docker swarm**, e.g., via network shared storage +#### Step 1. Prepare the setup (_First time only_) - ```bash - mkdir -p ~/rsync/logs - ``` +* create a folder, for example `~/rsync`, that will be later mounted in the container + * this folder is intended to hold supporting files such as log files, crontab file, and any supporting rsync files (e.g., list of files to rsync) + * the subfolder `logs`, for example `~/rsync/logs` is suggested for the log files + * important note: for Docker Swarm, this directory **needs to be available on all nodes in Docker swarm**, e.g., via network shared storage - * replace `~/rsync` with any other desired location + ```bash + mkdir -p ~/rsync/logs + ``` -2. Start - * as a container: + * you can replace `~/rsync` with any other desired location - ```bash - docker run --rm \ - --name=rsync \ - -e TZ="Europe/Zurich" \ - -e RSYNC_LOG="rsync" \ - -e RSYNC_CRON="* 5 * * *" \ - --volume ~/rsync:/rsync - --volume /path/to/source/data:/data/src \ - --volume /path/to/destination/data:/data/dst \ - ogivuk/rsync [OPTIONS] /data/src/ /data/dst/ - ``` +#### Step 2. Run - * Replace: - * `~/rsync/` with a folder created in the step 1. (if another is chosen) - * `/path/to/source/data` with the source folder to be copied or backed-up - * `/path/to/destination/data` with the destination folder - * `[OPTIONS]` with desired rsync optional arguments - * as a swarm service: +* run as a container: - ```bash - docker service create \ - --name=rsync \ - -e TZ="Europe/Zurich" \ - -e RSYNC_LOG="rsync" \ - -e RSYNC_CRON="0 5 * * *" \ - --mount type=bind,src=~/rsync,dst=/rsync \ - --mount type=bind,src=/path/to/source/data,dst=/data/src \ - --mount type=bind,src=/path/to/destination/data,dst=/data/dst \ - ogivuk/rsync [OPTIONS] /data/src/ /data/dst/ - ``` + ```bash + docker run --rm \ + --name=rsync \ + --env TZ="Europe/Zurich" \ + --env RSYNC_CRONTAB="crontab" \ + --volume ~/rsync:/rsync + --volume /path/to/source/data:/data/src \ + --volume /path/to/destination/data:/data/dst \ + ogivuk/rsync + ``` - * Replace: - * `~/rsync/` with a folder created in the step 1. (if another is chosen) - * `/path/to/source/data` with the source folder to be copied or backed-up - * `/path/to/destination/data` with the destination folder - * `[OPTIONS]` with desired rsync optional arguments +* run as a swarm service: -Parameters + ```bash + docker service create \ + --name=rsync \ + --env TZ="Europe/Zurich" \ + --env RSYNC_CRONTAB="crontab" \ + --mount type=bind,src=~/rsync,dst=/rsync \ + --mount type=bind,src=/path/to/source/data,dst=/data/src \ + --mount type=bind,src=/path/to/destination/data,dst=/data/dst \ + ogivuk/rsync + ``` -| Parameter | Function | -| --- | ------- | -| `-e TZ="Europe/Zurich"` | Sets the timezone in the container, which is important for the correct timestamping of logs. Replace `Europe/Zurich` with your own timezone from the list of available [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | -| `-e RSYNC_LOG="rsync"` | Enables the logging or the rsync output. The provided value will be the prefix for the log file name which is appended with the date and time of the rsync execution, e.g., `rsync.20200101-12:34.log`. | -| `-e RSYNC_CRON=1` | Specifies that the rsync is to be run as a cron job. The provided value needs to be a cron expression describing when to run the rsync job, and this expression will be appended to the crontab. | +| Parameter | Explanation | When to Use | +| :-------- | :---------- | :---------- | +| `--env TZ="Europe/Zurich"` | Sets the timezone in the container, which is important for the correct timestamping of logs. Replace `Europe/Zurich` with your own timezone from the list of available [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | Always | +| `--env RSYNC_CRONTAB="crontab"` | Specifies that the rsync is to be run as one or multiple cron jobs, and that the jobs are defined in the `crontab` file located in the mount-binded `~/rsync` folder. The rsync parameters used in the crontab must be mindful of the data locations in the container. | When using cron for regular rsync jobs | +| `~/rsync` | Specifies the local folder `~/rsync` that is mounted to the container at `/rsync`. Change `~/rsync` if another location is chosen in Step 1. | When using cron or ssh | +| `/path/to/source/data` | Specifies the source folder for sync and is mounted to the container at `/data/src`. Change to the appropriate folder. Multiple folders can be mounted in this way. | If any source is local | +| `/path/to/destination/data` | Specifies the destination folder for sync and is mounted to the container at `/data/src`. Change to the appropriate folder. Multiple folders can be mounted in this way. | If any destination is local | +| `--env RSYNC_UID=$(id -u)` | Provides the UID of the user starting the container so that the ownership of the files that rsync copies belong to that user. | If the rsync option for preserving ownership is not selected | +| `--env RSYNC_GID=$(id -g)` | Provides the GID of the user starting the container so that the ownership of the files that rsync copies belong to that user group. | If the rsync option for preserving ownership is not selected | + +Remarks: + +* **rsync will not be run by default**, you need to be specify the rsync command with all its arguments in the crontab, or in a script called in the crontab +* **any later changes to the crontab file require the service to be restarted**, and that's why consider to define the rsync job in a script that is called in the crontab +* when defining the rsync arguments, including source and destination, do that from the perspective of the container (/data/src, /data/dst) +* more volumes can be mount binded if needed +* the ssh client is included in the image in case your source or destination is a remote host + * ssh required files (private key, known_hosts, ssh_config) needs to be stored in a folder mounted to the container, for example in `~/rsync/ssh/` + * you can define the ssh connection in a `ssh_config` file + * rsync option `-e "ssh -F /rsync/ssh/ssh_config"` instructs rsync to use the ssh with the `ssh_config` for the remote sync diff --git a/rsync.sh b/rsync.sh index b6a4425..20a367b 100755 --- a/rsync.sh +++ b/rsync.sh @@ -6,27 +6,27 @@ echo "Preparation steps started." -# should logs be created? -if [ "$RSYNC_LOG" != "" ]; then - # logging enabled - echo "Logging selected." - # the specific name is to be used - LOG_COMMAND=" >> /rsync/logs/$RSYNC_LOG"'.$(date +%Y%m%d-%H%M).log 2>&1' - echo "Logging command is $LOG_COMMAND" +# Create an account for rsync +## this is recommended when rsync is not asked to preserve file ownership +if [ "$RSYNC_UID" != "" ] && [ "$RSYNC_GID" != "" ]; then + # UID and GID provided, create user + echo "UID and GID provided: $RSYNC_UID and $RSYNC_GID. Creating the user" + adduser -D -u $RSYNC_UID -g $RSYNC_GID rsyncuser + RSYNC_USER=rsyncuser else - # no logs - echo "Logging NOT selected." - LOG_COMMAND="" + # UID and GID not provided + echo "UID and GID are NOT provided. Proceeding as the root user." + RSYNC_USER=root fi # run as a cron job? -if [ "$RSYNC_CRON" != "" ]; then - # yes, run it as a cron job - echo "Running rsync as a cron job selected." - # make a crontab entry - #shift - echo "$RSYNC_CRON rsync $@$LOG_COMMAND" >> /etc/crontabs/root - echo "Entry created in the crontab" +if [ "$RSYNC_CRONTAB" != "" ]; then + # using cron + echo "Running rsync with cron and with a provided crontab selected." + echo "The crontab file is expected to be at /rsync/$RSYNC_CRONTAB ." + echo "Any provided rsync arguments will be ignored, specify them in the crontab file instead" + # define the crontab file location + crontab -u $RSYNC_USER /rsync/$RSYNC_CRONTAB else # no cron job echo "Running rsync as a cron job NOT selected." @@ -36,12 +36,12 @@ echo "Preparation steps completed." ### EXECUTION -if [ "$RSYNC_CRON" != "" ]; then +if [ "$RSYNC_CRONTAB" != "" ]; then # run as a cron job, start the cron daemon echo "Starting the cron daemon..." crond -f else # one time run echo "Executing rsync as an one time run..." - eval rsync $@"$LOG_COMMAND" + eval rsync $@ fi