##########################################################
#                       hdup FAQ                         #
##########################################################
Questions:
  1. What is hdup?
  2. What do I need for hdup?
  3. How do I install hdup?
  4. Where can I get the latest version?
  5. Where can I find older stable releases?
  6. How do I use hdup?
  7. How do I restore an archive?
  8. How do I restore a single file?
  9. Using the @remotehost syntax in hdup OpenSSH asks for a password?
  10. Can I restrict access even more?
  11. How do I only backup certain files?
  12. How do remote backups work?
  13. How do remote restores work?
  14. How do I use the pre- and postscripts?
  15. When I restore an archive I get all these weird numbered directories?
  16. Can hdup split up archives into CD sized chunks?
  17. How do I use the encryption features of hdup?
  18. A central backup host, what is that?
  19. What is a good way to remove old archives?
  20. Creation of backups works well, restoration fails. How can that be?
  21. File redirection does not work?
  22. Can I list the contents of a split archive?
  23. Where is the hdup executable installed?
  24. Tar fails in some weird way?
  25. Incremental backups are as large as the full ones (using SaMBa)
  26. Current bugs?
  27. How do I build the rpm?
  28. Monthly dump takes too long?

Q1: What is hdup?
A: Hdup is a simple backup utility that backs up whatever you want to a location that you specify. It can do the following:

Q2: What do I need for hdup ?
A: You will need: Furthermore you need data that needs to be backed up, media to backup up your stuff. Hdup cannot write directly to devices, it only works with mounted media. (Ie. it does not work with tapes).

Hdup has been successfully compiled (without any warnings) and used on:


Q3: How do I install hdup?
A: ./configure && make install, installs hdup in /usr/local/bin.

The configuration file is NOT installed per default. You should copy hdup.conf to /etc/hdup/hdup.conf, and tweak the settings in there.

See the dir examples/ for a sample config.


Q4: Where can I get the latest version?
A: The public CVS of hdup was closed, because I don't trust CVS anymore.

The latest stable can be found on www.miek.nl/projects/hdup/hdup.html


Q5: Where can I find older stable releases?
A: Older releases may be found at www.miek.nl/projects/hdup/previous

Q6: How do I use hdup?
A: To create a full backup use 'hdup monthly [hostname]'. A first incremental backup is created with 'hdup weekly [hostname], and the second incremental is made with 'hdup daily [hostname]'. The supplied cron file will do a monthly at the first day of each new month. The weekly's are done on the 1st, 7th, 14th, 21th and 28th day of each month. And the daily are run every day (1-31). This cron file also makes sure that on the 1st day of each month, the monthly backup runs first, then the weekly and after that the daily.

Thus:
Full backup - hdup monthly hostname
Level 1 dump - hdup weekly hostname
Level 2 dump - hdup daily hostname


Q7: How do restore an archive?
A: To restore an archive the following command is used: 
$ hdup restore hostname date targetdir

where:
hostname - is the name of the machine you want to restore
date - everything up to this date is restored
targetdir - the archive is extracted in this directory


Q8: How do I restore a single file?
A: If you want to restore a specific file use -s [path/to/file] 
% hdup -s /home/miekg/lostfile restore hostname date targetdir
Will extract /home/miekg/lostfile from the archive. The leading '/' is chopped, giving the desired result: [target]/home/miekg/'lostfile' is the location of the restored file.

Note:
You must specify the FULL path to a file that is to be restored. Only specifying 'motd' will not work, as GNU tar will not search for matching files in the archive when restoring specific files from an archive. You must specify '-s /etc/motd'.


Q9: Using the @remotehost syntax in hdup OpenSSH asks for a password. How do I fix this so that hdup can run in a cron job?
(Updated by Art Sackett)
A: There are two ways this can be accomplished, a secure way and a somewhat less secure way. There are both listed here.

  1. The secure way
    Use a keychains script. Excellent howto's can be found here:

    The keychain is only broken when the remote machine reboots, but it removes the need for an empty passphrase.


  2. The not so secure way
    On localhost:
    Step 1.
    create a identity file: 
    $ ssh-keygen
    Don't use a passphrase.

    Step 2.
    Copy the contents of the identity file: id_dsa.pub On remotehost:
    Put the contents of id_dsa.pub in the file ~user/.ssh/authorized_keys

    This 'user' is the user under which the archives should be ssh-ed. Thus with: 

    $ hdup monthly localhost @user@remotehost
    will transfer the archive to remote host without asking for a password.

    Note: this is not a safe technique. If possible use technique 1.


Q10: Can I restrict access even more?
 (Contributed by Boris Goldowsky)
A: Yes, you can, SSH allows you to do the following. In your authorized_keys2 file, add the something like this: 
from="IP ADDRESS",command="/usr/local/bin/hdup -c /etc/hdup.conf remote HOST",
no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-rsa 
AAAAB3NzcC1y.....
Where IPADDRESS is the address from which you connect, it can contain wildcards: 10.0.0.* is valid.
And where HOST is the host you're backing up remote.

If your key is now compromised they can only use it to run 'hdup'


Q11: How do I only backup certain files. Suppose I only want to backup *.doc and *.txt files. Is this possible?
A: Yes. With 'include' statement this is possible. If you only want to backup *.txt and *.doc files the following will work: 

dir = /home/miekg
include = *.txt, *.doc

This will backup everything under /home/miekg and it include all files ending in *.txt or *.doc. With the 'tar_patch' hdup will only create full dumps when using this option. Hdup will use 'find' to generate to file list to has to be backed up. Note that the backup is now a two staged process: first the files to backup have to get found, then the backup is run.


Q12: How do remote backups work?
A: Remote backups are very easy in hdup. If you want to run hdup from cron be sure you've setup ssh to connect to the remote server without using a password.
Step 1:
Add the location of the remote hdup program to the local configuration: 
[localhost]
remote hdup = /usr/local/bin/hdup
remote hdup option = -c /usr/local/etc/hdup.conf
Step 2:
On remote host add this to hdup.conf: 
[localhost]
allow remote = yes
Step 3:
Generate a backup on localhost and send it to remotehost: 
$ hdup monthly localhost @user@remotehost

Q13: How do remote restores work?
A: Essentially they work the same as remote backups, add the remote hdup location and set 'allow remote' to yes.

The restoring can now be done via: 

$ hdup restore localhost 2003-03-2003 /tmp/a @user@remotehost
When this is finished there restored archive will reside in /tmp/a on remote host.

Q14: How do I use the pre- and postscripts?
A: With the 'prerun = startscript' and 'postrun = endscript' hdup can be told to run startscript before doing the actual backup. And it runs endscript afterwards. This can be used to mount a SaMBa share - back it up - and then unmounting it.

Whatever the scheme is these scripts are always executed. It is best give the scheme to the scripts and let them decide what to do. I.e. 'prerun = script %s'. And shell script then looks like this: 

#!/bin/sh
if [ "$1" == "restore" ]; then exit 1; fi
echo "rest of the script here"
A list of variables that get expanded is given in hdup.conf(5).

Q15: When I restore an archive I get all the weird numbered directories?
A: There looks to be a problem with incremental tar files when using an old version of tar. If one unpacks a tar file created with --listed-incremental tar will create "checksum directories", like 07510533400/, with parts of the tar file extracted in them.

If you have this problem upgrading to tar version 1.13.25 fixes this. [Confirmed by Petr Divina for Solaris]


Q16: Can hdup split up archives into sized chunks?
A: As of development version 1.6.12, hdup has the ability to split up archives. To use it, you only need to use the "chunk size =" option in the configuration file. For instance you want to backup your harddisk and split the archive into 1.44 M sized chunks to put them on floppy (lots of floppies...). Just specify "chunk size = 1440k" and the chunks get created. When burning the stuff to CD use "chunksize = 640M".

The chunks of the split archive get the suffix: "__split__XX", where X is a letter. The first archive is gets the suffix: "__split__aa", the second "__split__ab", up to "__split__zz". So currently 26*26 = 676 chunks are supported.


Q17: How do I use the encryption features of hdup?
A: The use of encryption is rather trivial in hdup. First you'll need to pick an algorithm which hdup will use for the encryption:
mcrypt --list
...
rijndael-192 (32): cbc cfb ctr ecb ncfb nofb ofb 
saferplus (32): cbc cfb ctr ecb ncfb nofb ofb 
saferplus (32): cbc cfb ctr ecb ncfb nofb ofb 
wake (32): stream 
wake (32): stream 
blowfish-compat (56): cbc cfb ctr ecb ncfb nofb ofb 
...
Only the first word on each line is of importance. Suppose we want to use rijdael-192 (which is NSA approved, so you might not want to use it...) Next you need to make a file which holds the key used to encrypt the archive: 
cat > my_secret_key
this is a very secret key
^D
Next hdup must be told to use this, add the following under the host statement, or under the global statement if you want all hosts to use encryption: 
[myhost]
algoritm = rijndael-192

key = /path/to/my/key/my_secret_key
That is it, hdup will now use mcrypt to encrypt the archives.

Note:
Be aware that encryption is a local operation. Encrypted remote archives are generated locally and then transmitted to the remote host. A restore works in the opposite manner: it is directly transmitted to the client machine which has to decrypt it while restoring. The rational behind this is that you don't want people on the remote machine to be able to look into your archive. A drawback is that you will have to store your secret key somewhere safe (ie. not on the machine that you are backing up - cause that has just crashed).


Q18: A central backup host, what is it?
A: That is something very cool. Currently see the howto at www.miek.nl for more information.

Q19: What is a good way to remove old archives?
A: You could use a cronjob (see examples/) that deletes old archives after 2 months.

Or you could set 'no history = yes' in hdup's config file. This will use a single directory and a uniform archive name for all archives, archives will be called: static/hostname.monthly.static.tar.gz.
As such all new backups overwrite the old ones - saving space on your machine.

But be careful with this option! When a backup fails you don't have the old backup (that was being overwritten) nor the new one (that one failed) leaving you with no backup at all! Look in examples/ for a little postrun script (no-history-post-run.sh) that can be used to copy the archive to a safe place after a backup. Still only use this option when you know what you are doing.


Q20: Creation of backups works well, restoration fails. How can that be?
 (Contributed by: Jiri Bajer)

A: Have you sufficient space in /var/tmp to uncompress the whole host? Mounting a subdirectory of another disk (with plenty of space) to /var/tmp can be used for quick fix.

Q21: File redirection does not work?
A: Hdup sends all logging to stderr. When hdup is doing a remote operation, the archives are send to stdout. To not clutter the archives all other output of hdup is send to stderr. If you want to redirect that use somehting like this: 
hdup 2> bla

Update: As of version 1.6.14 the overview is sent to stdout again. All error messages are still sent to stderr.


Q22: Can I list the contents of a split archive?
A: Yes you can, although not via hdup. Use the following:
Go to the split archive you want to see, type 
cat hostname.monthly.* | tar tvfz -
Ofcourse you have to you the right compression in tar and cat the right archive(s).

Q23: Where is the hdup executable installed?
A: As of version 1.6.11 (when autoconf was used for hdup's compilation), the exe from hdup is installed in /usr/local/bin. Before that is was installed in /usr/local/sbin.

Q24: Tar fails in some weird way?
Seeing stuff like this: 
/usr/local/sbin/hdup: www: STARTING BACKUP.
/usr/bin/tar: can't add file --files-from : No such file or directory
/usr/bin/tar: Removing leading / from absolute path names in the archive.
/usr/bin/tar: can't add file --listed-incremental : No such file or directory
/usr/bin/tar: can't add file /backup/www/etc/inclist.monthly : No such file or directory
/usr/bin/tar: can't add file --exclude-from : No such file or directory
/usr/bin/tar: can't add file --ignore-failed-read : No such file or directory
Than you tar version if either not the GNU version or it is too old. You should upgrade to version 1.3.25 of GNU tar.

Q25: Incremenal backups are as large as the full ones (using SaMBa)
When you are using SaMBa to backup windows files be sure that the underlying filesystem is something better than fat (or vfat). Those ancient filesystems don't have enough support to properly do incremental backups. You will always get full dumps.

Q26: Current bugs?
A: The %a expansion is not always the same in the prerun and postrun scripts (when using encryption).

A listing of bugs in hdup can be found on: hdup's bugzilla

Please also report new bugs in the bugzilla.


Q27: How do I build the rpm? Just use the following: 
wget http://www.miek.nl/projects/hdup16/previous/hdup-1.6.XX.tar.gz
rpm -tb --clean hdup-1.6.XX.tar.gz
rpm -Uvh /usr/src/redhat/RPMS/i386/hdup-1.6.XX-1.i386.rpm
Where XX is the hdup release.

Q28: Monthly dump takes too long? You should take care that your monthly backup, usually started on the 1st every month, is finished before the weekly kicks in. The is no mechanism in hdup to prevent this from happening. Although hdup uses a LOCK files it has not a schedular (we abuse cron for that). So it would be wise to estimate the backup time needed for a full monthly dump, multiply that by two and schedule the weekly dump only after that amount of time following the monthly.

This still is no guerantee that it will not happen. If weekly starts before the monthly is finished cron should mail you a failed backup attempt.

The same applies when the weekly bites the daily.