Appending timestamp to Dropbox backups

[877dev]

Hi @gcgarner

I think this was discussed over in Discord chat, it would be ideal to have date/time appended to the Dropbox backup file to avoid it being overwritten each time.

PS - I'm sure you have a solution so feel free to ignore the below ramblings:

Surprisingly it raises some issues in my inexperienced brain:

I had a look into docker_backup.sh, it would be simple enough to rename the file before uploading; sudo cp -a ~/IOTstack/backups/docker.tar.gz "backup-$(date +"%Y-%m-%d-%H%M%S").tar.gz"
Or even better create the timestamped name when compressing the files.

But how would the script know the filename to upload as it keeps changing with the time? Maybe just appending the date which would fix this issue. And how to remove the file from the backups folder after uploading, to save space? Maybe some thing like sudo rm backup*

And then on the Dropbox side of things, if doing daily backups this will add up to a lot of space over time.

Leaving it as it is solves all of the above issue, as it just overwrites with the latest copy. But if there is an issue and 11pm swings by, your last good backup is overwritten. However Dropbox does have a 30day 'rewind' feature so that could be used to recover a recent backup.

I'll be interested to see how this could be approached, or if it's not worth implementing.

Thanks!

[gcgarner]

Hey

I think one way to approach it is to change the backup name of the original backup using the formatting you used. Create a logfile and append it with the name of the backup.

You can use the tail command to build an exclude list and remove all backups not on the exclude list. Another possibility is to use the following snippet find /path/to/files* -mtime +5 -exec rm {} \; That removes everything older than 5 days. I think the tail option will work better, tail -n 5 backups/backup.log and somehow pipe that into the rm command

like this https://askubuntu.com/questions/830776/remove-file-but-exclude-all-files-in-a-list

It would be easy enough to delete files from Dropbox because there is a rm command for Dropbox-Uploader. rclone on the other hand would be a little trickier. I'll play around with it and see what i can get working.

[877dev]

Great idea, I did some searching based on the tail command, what do you think about this:

ls -t1 backup* | head -n -5 | xargs rm

• ls -t1 is printing a list, one file/directory per line, of all files in the folder, ordering by the most recent modification date.
• head -n -5 is printing all files except the last 5.
• xargs rm says to delete any file output from the head command.

I tested it and it works for me, keeps the last 5 files starting with "backup".

From this post here

It works for local files, but for Dropbox your backup.log file might be better, In don't know..

[gcgarner]

Great, can you fork the project and push up your changes or submit a pull request?

[877dev]

I would like to help, but I've never done that before. I will read up on how to do that correctly, I wouldn't want to mess something up! I was only running the command ls -t1 backup* | head -n -5 | xargs rm in terminal, not using your script.

I have now edited the script to the best of my knowledge and it works uploading the files, I changed it so it only appends the date, this way it does at least upload to Dropbox. Deleting of the old files does not work properly:

• Issue 1: If there are less than 5 files in the folder it errors with rm: missing operand
• Issue 2: Old files can't be deleted : Permission denied

Here's what I have so far, please let me know how I can help!

docker_backup.sh new script:
Pastebin

Output of my terminal:

pi@raspberrypi:~/IOTstack/scripts $ ./docker_backup.sh
~/IOTstack ~/IOTstack/scripts
Moving old influxdb backups if they exist
backing up influxdb database
influxdb backup complete
compressing stack folders
backup saved to ./backups/backup-2019-11-09.tar.gz
15M     ./backups/backup-2019-11-09.tar.gz
uploading to dropbox
 > Uploading "/home/pi/IOTstack/backups/backup-2019-11-09.tar.gz" to "/IOTstackBU/backup-2019-11-09.tar.gz"... DONE
rm: cannot remove './backups/backup-2019-11-09.tar.gz': Permission denied
rm: cannot remove './backups/backup01.gz': Permission denied
rm: cannot remove './backups/backup02.gz': Permission denied
last five backup files are saved in ~/IOTstack/backups
~/IOTstack/scripts
pi@raspberrypi:~/IOTstack/scripts $

[877dev]

OK I spent an hour learning about github branches, forking and pull requests after I got back from work today. I have forked the project and will have a look at improving things, and submit a pull request.

Please don't get too excited though I am quite new to this ;)

[gcgarner]

Excellent work, I saw the pull request, we can continue the conversation there