vincent/rsync-docker
Archived
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Prepare for version 0.2: crontab file mount and ssh support

Browse Source
This commit is contained in:
Ognjen Vukovic 2020-05-09 23:38:45 +02:00
parent 5d34f00834
commit a6f6d49324
3 changed files with 88 additions and 71 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Dockerfile
View File

@ -2,6 +2,7 @@ FROM alpine
RUN apk add --update-cache \
rsync \
openssh-client \
tzdata \
&& rm -rf /var/cache/apk/*

120
README.md
View File

@ -10,6 +10,12 @@ The provided image is open-source and built from scratch with the goal to
enable you to run a stateless and an immutable container, according to the best
practices.
With this image you can:
* run a simple one-time rsync job for local drives
* run a rsync job for remote drives using ssh (included in the image)
* run scheduled rsync jobs using cron (included in the image)
Supported architectures:
* the image supports multiple architectures: `x86-64` and `arm32`
@ -22,9 +28,17 @@ Supported architectures:
| `:x64` | targeted to the `x64` architecture |
| `:arm32v7` | targeted to the `arm32v7` architecture |
The image is based on `alpine` image and it includes:
* `rsync`
* `openssh-client` for remote sync over ssh
* `tzdata` for easier setting up of local timezone (for file timestamps)
* `cron` (included in the alpine) for scheduling regular back-ups
* `rsync.sh` script that prepares the cron job and starts the cron daemon
## Usage
### Quick Start: one time run, local, no logs
### Quick Start: one time run and sync of local folder
```bash
docker run --rm \
@ -40,64 +54,66 @@ Replace:
* `/path/to/destination/data` with the destination folder
* `[OPTIONS]` with desired rsync optional arguments
### Start with All Options Explained
### Use with cron or ssh
1. _First time only_: Prepare the setup
* create a folder, for example `~/rsync`, that will be later mounted in the container
* this folder is intended to hold supporting files such as log files, a list of files to rsync, a file with exclude patterns, etc.
* the subfolder `logs`, for example `~/rsync/logs` is intended for the log files
* important note: for Docker Swarm, this directory **needs to be available on all nodes in Docker swarm**, e.g., via network shared storage
#### Step 1. Prepare the setup (_First time only_)
```bash
mkdir -p ~/rsync/logs
```
* create a folder, for example `~/rsync`, that will be later mounted in the container
* this folder is intended to hold supporting files such as log files, crontab file, and any supporting rsync files (e.g., list of files to rsync)
* the subfolder `logs`, for example `~/rsync/logs` is suggested for the log files
* important note: for Docker Swarm, this directory **needs to be available on all nodes in Docker swarm**, e.g., via network shared storage
* replace `~/rsync` with any other desired location
```bash
mkdir -p ~/rsync/logs
```
2. Start
* as a container:
* you can replace `~/rsync` with any other desired location
```bash
docker run --rm \
--name=rsync \
-e TZ="Europe/Zurich" \
-e RSYNC_LOG="rsync" \
-e RSYNC_CRON="* 5 * * *" \
--volume ~/rsync:/rsync
--volume /path/to/source/data:/data/src \
--volume /path/to/destination/data:/data/dst \
ogivuk/rsync [OPTIONS] /data/src/ /data/dst/
```
#### Step 2. Run
* Replace:
* `~/rsync/` with a folder created in the step 1. (if another is chosen)
* `/path/to/source/data` with the source folder to be copied or backed-up
* `/path/to/destination/data` with the destination folder
* `[OPTIONS]` with desired rsync optional arguments
* as a swarm service:
* run as a container:
```bash
docker service create \
--name=rsync \
-e TZ="Europe/Zurich" \
-e RSYNC_LOG="rsync" \
-e RSYNC_CRON="0 5 * * *" \
--mount type=bind,src=~/rsync,dst=/rsync \
--mount type=bind,src=/path/to/source/data,dst=/data/src \
--mount type=bind,src=/path/to/destination/data,dst=/data/dst \
ogivuk/rsync [OPTIONS] /data/src/ /data/dst/
```
```bash
docker run --rm \
--name=rsync \
--env TZ="Europe/Zurich" \
--env RSYNC_CRONTAB="crontab" \
--volume ~/rsync:/rsync
--volume /path/to/source/data:/data/src \
--volume /path/to/destination/data:/data/dst \
ogivuk/rsync
```
* Replace:
* `~/rsync/` with a folder created in the step 1. (if another is chosen)
* `/path/to/source/data` with the source folder to be copied or backed-up
* `/path/to/destination/data` with the destination folder
* `[OPTIONS]` with desired rsync optional arguments
* run as a swarm service:
Parameters
```bash
docker service create \
--name=rsync \
--env TZ="Europe/Zurich" \
--env RSYNC_CRONTAB="crontab" \
--mount type=bind,src=~/rsync,dst=/rsync \
--mount type=bind,src=/path/to/source/data,dst=/data/src \
--mount type=bind,src=/path/to/destination/data,dst=/data/dst \
ogivuk/rsync
```
| Parameter | Function |
| --- | ------- |
| `-e TZ="Europe/Zurich"` | Sets the timezone in the container, which is important for the correct timestamping of logs. Replace `Europe/Zurich` with your own timezone from the list of available [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). |
| `-e RSYNC_LOG="rsync"` | Enables the logging or the rsync output. The provided value will be the prefix for the log file name which is appended with the date and time of the rsync execution, e.g., `rsync.20200101-12:34.log`. |
| `-e RSYNC_CRON=1` | Specifies that the rsync is to be run as a cron job. The provided value needs to be a cron expression describing when to run the rsync job, and this expression will be appended to the crontab. |
| Parameter | Explanation | When to Use |
| :-------- | :---------- | :---------- |
| `--env TZ="Europe/Zurich"` | Sets the timezone in the container, which is important for the correct timestamping of logs. Replace `Europe/Zurich` with your own timezone from the list of available [timezones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | Always |
| `--env RSYNC_CRONTAB="crontab"` | Specifies that the rsync is to be run as one or multiple cron jobs, and that the jobs are defined in the `crontab` file located in the mount-binded `~/rsync` folder. The rsync parameters used in the crontab must be mindful of the data locations in the container. | When using cron for regular rsync jobs |
| `~/rsync` | Specifies the local folder `~/rsync` that is mounted to the container at `/rsync`. Change `~/rsync` if another location is chosen in Step 1. | When using cron or ssh |
| `/path/to/source/data` | Specifies the source folder for sync and is mounted to the container at `/data/src`. Change to the appropriate folder. Multiple folders can be mounted in this way. | If any source is local |
| `/path/to/destination/data` | Specifies the destination folder for sync and is mounted to the container at `/data/src`. Change to the appropriate folder. Multiple folders can be mounted in this way. | If any destination is local |
| `--env RSYNC_UID=$(id -u)` | Provides the UID of the user starting the container so that the ownership of the files that rsync copies belong to that user. | If the rsync option for preserving ownership is not selected |
| `--env RSYNC_GID=$(id -g)` | Provides the GID of the user starting the container so that the ownership of the files that rsync copies belong to that user group. | If the rsync option for preserving ownership is not selected |
Remarks:
* **rsync will not be run by default**, you need to be specify the rsync command with all its arguments in the crontab, or in a script called in the crontab
* **any later changes to the crontab file require the service to be restarted**, and that's why consider to define the rsync job in a script that is called in the crontab
* when defining the rsync arguments, including source and destination, do that from the perspective of the container (/data/src, /data/dst)
* more volumes can be mount binded if needed
* the ssh client is included in the image in case your source or destination is a remote host
* ssh required files (private key, known_hosts, ssh_config) needs to be stored in a folder mounted to the container, for example in `~/rsync/ssh/`
* you can define the ssh connection in a `ssh_config` file
* rsync option `-e "ssh -F /rsync/ssh/ssh_config"` instructs rsync to use the ssh with the `ssh_config` for the remote sync

38
rsync.sh
View File

@ -6,27 +6,27 @@
echo "Preparation steps started."
# should logs be created?
if [ "$RSYNC_LOG" != "" ]; then
# logging enabled
echo "Logging selected."
# the specific name is to be used
LOG_COMMAND=" >> /rsync/logs/$RSYNC_LOG"'.$(date +%Y%m%d-%H%M).log 2>&1'
echo "Logging command is $LOG_COMMAND"
# Create an account for rsync
## this is recommended when rsync is not asked to preserve file ownership
if [ "$RSYNC_UID" != "" ] && [ "$RSYNC_GID" != "" ]; then
# UID and GID provided, create user
echo "UID and GID provided: $RSYNC_UID and $RSYNC_GID. Creating the user"
adduser -D -u $RSYNC_UID -g $RSYNC_GID rsyncuser
RSYNC_USER=rsyncuser
else
# no logs
echo "Logging NOT selected."
LOG_COMMAND=""
# UID and GID not provided
echo "UID and GID are NOT provided. Proceeding as the root user."
RSYNC_USER=root
fi
# run as a cron job?
if [ "$RSYNC_CRON" != "" ]; then
# yes, run it as a cron job
echo "Running rsync as a cron job selected."
# make a crontab entry
#shift
echo "$RSYNC_CRON rsync $@$LOG_COMMAND" >> /etc/crontabs/root
echo "Entry created in the crontab"
if [ "$RSYNC_CRONTAB" != "" ]; then
# using cron
echo "Running rsync with cron and with a provided crontab selected."
echo "The crontab file is expected to be at /rsync/$RSYNC_CRONTAB ."
echo "Any provided rsync arguments will be ignored, specify them in the crontab file instead"
# define the crontab file location
crontab -u $RSYNC_USER /rsync/$RSYNC_CRONTAB
else
# no cron job
echo "Running rsync as a cron job NOT selected."
@ -36,12 +36,12 @@ echo "Preparation steps completed."
### EXECUTION
if [ "$RSYNC_CRON" != "" ]; then
if [ "$RSYNC_CRONTAB" != "" ]; then
# run as a cron job, start the cron daemon
echo "Starting the cron daemon..."
crond -f
else
# one time run
echo "Executing rsync as an one time run..."
eval rsync $@"$LOG_COMMAND"
eval rsync $@
fi