vincent/rsync-docker
Archived
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Prepare for version 0.2: crontab file mount and ssh support

Browse Source
This commit is contained in:
Ognjen Vukovic 2020-05-09 23:38:45 +02:00
parent 5d34f00834
commit a6f6d49324
3 changed files with 88 additions and 71 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Dockerfile
View File

@ -2,6 +2,7 @@ FROM alpine
RUN apk add --update-cache \ RUN apk add --update-cache \
rsync \ rsync \
openssh-client \
tzdata \ tzdata \
&& rm -rf /var/cache/apk/* && rm -rf /var/cache/apk/*

82
README.md
View File

@ -10,6 +10,12 @@ The provided image is open-source and built from scratch with the goal to
enable you to run a stateless and an immutable container, according to the best enable you to run a stateless and an immutable container, according to the best
practices. practices.
With this image you can:
* run a simple one-time rsync job for local drives
* run a rsync job for remote drives using ssh (included in the image)
* run scheduled rsync jobs using cron (included in the image)
Supported architectures: Supported architectures:
* the image supports multiple architectures: `x86-64` and `arm32` * the image supports multiple architectures: `x86-64` and `arm32`
@ -22,9 +28,17 @@ Supported architectures:
| `:x64` | targeted to the `x64` architecture | | `:x64` | targeted to the `x64` architecture |
| `:arm32v7` | targeted to the `arm32v7` architecture | | `:arm32v7` | targeted to the `arm32v7` architecture |
The image is based on `alpine` image and it includes:
* `rsync`
* `openssh-client` for remote sync over ssh
* `tzdata` for easier setting up of local timezone (for file timestamps)
* `cron` (included in the alpine) for scheduling regular back-ups
* `rsync.sh` script that prepares the cron job and starts the cron daemon
## Usage ## Usage
### Quick Start: one time run, local, no logs ### Quick Start: one time run and sync of local folder
```bash ```bash
docker run --rm \ docker run --rm \
@ -40,64 +54,66 @@ Replace:
* `/path/to/destination/data` with the destination folder * `/path/to/destination/data` with the destination folder
* `[OPTIONS]` with desired rsync optional arguments * `[OPTIONS]` with desired rsync optional arguments
### Start with All Options Explained ### Use with cron or ssh
#### Step 1. Prepare the setup (_First time only_)
1. _First time only_: Prepare the setup
* create a folder, for example `~/rsync`, that will be later mounted in the container * create a folder, for example `~/rsync`, that will be later mounted in the container
* this folder is intended to hold supporting files such as log files, a list of files to rsync, a file with exclude patterns, etc. * this folder is intended to hold supporting files such as log files, crontab file, and any supporting rsync files (e.g., list of files to rsync)
* the subfolder `logs`, for example `~/rsync/logs` is intended for the log files * the subfolder `logs`, for example `~/rsync/logs` is suggested for the log files
* important note: for Docker Swarm, this directory **needs to be available on all nodes in Docker swarm**, e.g., via network shared storage * important note: for Docker Swarm, this directory **needs to be available on all nodes in Docker swarm**, e.g., via network shared storage
```bash ```bash
mkdir -p ~/rsync/logs mkdir -p ~/rsync/logs
``` ```
* replace `~/rsync` with any other desired location * you can replace `~/rsync` with any other desired location
2. Start #### Step 2. Run
* as a container:
* run as a container:
```bash ```bash
docker run --rm \ docker run --rm \
--name=rsync \ --name=rsync \
-e TZ="Europe/Zurich" \ --env TZ="Europe/Zurich" \
-e RSYNC_LOG="rsync" \ --env RSYNC_CRONTAB="crontab" \
-e RSYNC_CRON="* 5 * * *" \
--volume ~/rsync:/rsync --volume ~/rsync:/rsync
--volume /path/to/source/data:/data/src \ --volume /path/to/source/data:/data/src \
--volume /path/to/destination/data:/data/dst \ --volume /path/to/destination/data:/data/dst \
ogivuk/rsync [OPTIONS] /data/src/ /data/dst/ ogivuk/rsync
``` ```
* Replace: * run as a swarm service:
* `~/rsync/` with a folder created in the step 1. (if another is chosen)
* `/path/to/source/data` with the source folder to be copied or backed-up
* `/path/to/destination/data` with the destination folder
* `[OPTIONS]` with desired rsync optional arguments
* as a swarm service:
```bash ```bash
docker service create \ docker service create \
--name=rsync \ --name=rsync \
-e TZ="Europe/Zurich" \ --env TZ="Europe/Zurich" \
-e RSYNC_LOG="rsync" \ --env RSYNC_CRONTAB="crontab" \
-e RSYNC_CRON="0 5 * * *" \
--mount type=bind,src=~/rsync,dst=/rsync \ --mount type=bind,src=~/rsync,dst=/rsync \
--mount type=bind,src=/path/to/source/data,dst=/data/src \ --mount type=bind,src=/path/to/source/data,dst=/data/src \
--mount type=bind,src=/path/to/destination/data,dst=/data/dst \ --mount type=bind,src=/path/to/destination/data,dst=/data/dst \
ogivuk/rsync [OPTIONS] /data/src/ /data/dst/ ogivuk/rsync
``` ```
* Replace: | Parameter | Explanation | When to Use |
* `~/rsync/` with a folder created in the step 1. (if another is chosen) | :-------- | :---------- | :---------- |
* `/path/to/source/data` with the source folder to be copied or backed-up | `--env TZ="Europe/Zurich"` | Sets the timezone in the container, which is important for the correct timestamping of logs. Replace `Europe/Zurich` with your own timezone from the list of available [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | Always |
* `/path/to/destination/data` with the destination folder | `--env RSYNC_CRONTAB="crontab"` | Specifies that the rsync is to be run as one or multiple cron jobs, and that the jobs are defined in the `crontab` file located in the mount-binded `~/rsync` folder. The rsync parameters used in the crontab must be mindful of the data locations in the container. | When using cron for regular rsync jobs |
* `[OPTIONS]` with desired rsync optional arguments | `~/rsync` | Specifies the local folder `~/rsync` that is mounted to the container at `/rsync`. Change `~/rsync` if another location is chosen in Step 1. | When using cron or ssh |
| `/path/to/source/data` | Specifies the source folder for sync and is mounted to the container at `/data/src`. Change to the appropriate folder. Multiple folders can be mounted in this way. | If any source is local |
| `/path/to/destination/data` | Specifies the destination folder for sync and is mounted to the container at `/data/src`. Change to the appropriate folder. Multiple folders can be mounted in this way. | If any destination is local |
| `--env RSYNC_UID=$(id -u)` | Provides the UID of the user starting the container so that the ownership of the files that rsync copies belong to that user. | If the rsync option for preserving ownership is not selected |
| `--env RSYNC_GID=$(id -g)` | Provides the GID of the user starting the container so that the ownership of the files that rsync copies belong to that user group. | If the rsync option for preserving ownership is not selected |
Parameters Remarks:
| Parameter | Function | * **rsync will not be run by default**, you need to be specify the rsync command with all its arguments in the crontab, or in a script called in the crontab
| --- | ------- | * **any later changes to the crontab file require the service to be restarted**, and that's why consider to define the rsync job in a script that is called in the crontab
| `-e TZ="Europe/Zurich"` | Sets the timezone in the container, which is important for the correct timestamping of logs. Replace `Europe/Zurich` with your own timezone from the list of available [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | * when defining the rsync arguments, including source and destination, do that from the perspective of the container (/data/src, /data/dst)
| `-e RSYNC_LOG="rsync"` | Enables the logging or the rsync output. The provided value will be the prefix for the log file name which is appended with the date and time of the rsync execution, e.g., `rsync.20200101-12:34.log`. | * more volumes can be mount binded if needed
| `-e RSYNC_CRON=1` | Specifies that the rsync is to be run as a cron job. The provided value needs to be a cron expression describing when to run the rsync job, and this expression will be appended to the crontab. | * the ssh client is included in the image in case your source or destination is a remote host
* ssh required files (private key, known_hosts, ssh_config) needs to be stored in a folder mounted to the container, for example in `~/rsync/ssh/`
* you can define the ssh connection in a `ssh_config` file
* rsync option `-e "ssh -F /rsync/ssh/ssh_config"` instructs rsync to use the ssh with the `ssh_config` for the remote sync

38
rsync.sh
View File

@ -6,27 +6,27 @@
echo "Preparation steps started." echo "Preparation steps started."
# should logs be created? # Create an account for rsync
if [ "$RSYNC_LOG" != "" ]; then ## this is recommended when rsync is not asked to preserve file ownership
# logging enabled if [ "$RSYNC_UID" != "" ] && [ "$RSYNC_GID" != "" ]; then
echo "Logging selected." # UID and GID provided, create user
# the specific name is to be used echo "UID and GID provided: $RSYNC_UID and $RSYNC_GID. Creating the user"
LOG_COMMAND=" >> /rsync/logs/$RSYNC_LOG"'.$(date +%Y%m%d-%H%M).log 2>&1' adduser -D -u $RSYNC_UID -g $RSYNC_GID rsyncuser
echo "Logging command is $LOG_COMMAND" RSYNC_USER=rsyncuser
else else
# no logs # UID and GID not provided
echo "Logging NOT selected." echo "UID and GID are NOT provided. Proceeding as the root user."
LOG_COMMAND="" RSYNC_USER=root
fi fi
# run as a cron job? # run as a cron job?
if [ "$RSYNC_CRON" != "" ]; then if [ "$RSYNC_CRONTAB" != "" ]; then
# yes, run it as a cron job # using cron
echo "Running rsync as a cron job selected." echo "Running rsync with cron and with a provided crontab selected."
# make a crontab entry echo "The crontab file is expected to be at /rsync/$RSYNC_CRONTAB ."
#shift echo "Any provided rsync arguments will be ignored, specify them in the crontab file instead"
echo "$RSYNC_CRON rsync $@$LOG_COMMAND" >> /etc/crontabs/root # define the crontab file location
echo "Entry created in the crontab" crontab -u $RSYNC_USER /rsync/$RSYNC_CRONTAB
else else
# no cron job # no cron job
echo "Running rsync as a cron job NOT selected." echo "Running rsync as a cron job NOT selected."
@ -36,12 +36,12 @@ echo "Preparation steps completed."
### EXECUTION ### EXECUTION
if [ "$RSYNC_CRON" != "" ]; then if [ "$RSYNC_CRONTAB" != "" ]; then
# run as a cron job, start the cron daemon # run as a cron job, start the cron daemon
echo "Starting the cron daemon..." echo "Starting the cron daemon..."
crond -f crond -f
else else
# one time run # one time run
echo "Executing rsync as an one time run..." echo "Executing rsync as an one time run..."
eval rsync $@"$LOG_COMMAND" eval rsync $@
fi fi