Recipe To Verify The Integrity of a Backup

From: http://ftimes.sourceforge.net/Files/Recipes/ftimes-map-verify-backup.txt

This recipe demonstrates how to verify the integrity of a backup.
The information presented here is general in the sense that it
could be applied to any type of backup scheme.  It is specific in
the sense that this recipe focuses on one particular archiving
tool, namely tar.  This recipe assumes that 1) all local commands
are in the search path and will be executed from a Bourne shell
and 2) the local operating system is a supported flavor of UNIX.

1. Determine the subject tree that you wish to backup.  To keep
this recipe consistent we will create and populate an example
tree.  This can be done as follows:

— ftimes-make-example-tree.sh —
#!/bin/sh

if [ “$#” != “1” ]; then
echo “”
echo “Usage: `basename $0` scratch-directory”
echo “”
exit 1
else
BASENAME=`basename $1`
DIRNAME=`dirname $1`
ORIGINAL_PARENT=${DIRNAME}/${BASENAME}
TREE=example-tree
fi

umask 22

if [ -d ${ORIGINAL_PARENT}/${TREE} ]; then
rm -rf ${ORIGINAL_PARENT}/${TREE};
fi

mkdir -p ${ORIGINAL_PARENT}/${TREE}
COUNT=0
while [ ${COUNT} -lt 10 ]
do
echo “file-${COUNT}” > ${ORIGINAL_PARENT}/${TREE}/file-${COUNT}
COUNT=`expr ${COUNT} + 1`
done
— ftimes-make-example-tree.sh —

To automatically extract the preceding commands into a shell
script, run the following ‘sed’ command.  Note: This command
expects to find a file called ftimes-map-verify-backup.txt
in the current working directory.

sed -e ‘1,/^— ftimes-make-example-tree.sh —$/d; /^— ftimes-make-example-tree.sh —$/,$d’ ftimes-map-verify-backup.txt > ftimes-make-example-tree.sh

2. Baseline and archive the subject tree.  Here, the ‘tar’ program
will be used to create the archive.  Since file content verification
is the primary objective, a FieldMask of ‘NONE+md5’ will be used
to create the baseline.  If you want to preserve other attributes,
modify the FieldMask as appropriate.  When the baseline is
complete, write down the value for OutFileHash.  You will find
this value in the log output.  Later, it can be used to verify
the integrity of the baseline contained within the tar archive.

Using the ftimes-create-archive.sh script, this step can be
accomplished with the following command:

ftimes-create-archive.sh /tmp/ftimes-original

— ftimes-create-archive.sh —
#!/bin/sh

WORKDIR=/tmp
WORKFILE=${WORKDIR}/ftimes-create-archive.log.$$

if [ “$#” != “1” ]; then
echo “”
echo “Usage: `basename $0` directory”
echo “”
exit 1
else
SUBJECT=$1
BASENAME=`basename ${SUBJECT}`
PARENT=`dirname ${SUBJECT}`
DATE=`date “+%Y-%m-%d”`
ARCHIVE=${PARENT}/${BASENAME}-${DATE}.tar
BASELINE=baseline
FIELDMASK=None+md5
fi

umask 22

(
echo “BaseName=${BASELINE}”
echo “OutDir=${SUBJECT}”
echo “FieldMask=${FIELDMASK}”
echo “Include=${SUBJECT}”
) | ftimes –mapfull – 2> ${WORKFILE}

OUTFILENAME=`grep OutFileName ${WORKFILE} | sed ‘s/^.*OutFileName=\(.*\)$/\1/’`
OUTFILEHASH=`grep OutFileHash ${WORKFILE} | sed ‘s/^.*OutFileHash=\(.*\)$/\1/’`
LOGFILENAME=`echo ${OUTFILENAME} | sed ‘s/\.map/\.log/’`
mv ${OUTFILENAME} ${SUBJECT}/baseline.map
mv ${LOGFILENAME} ${SUBJECT}/baseline.log
tar -C ${PARENT} -cf ${ARCHIVE} ${BASENAME}
rm -f ${WORKFILE}

echo “”
echo “MD5 (${SUBJECT}/baseline.map) = ${OUTFILEHASH}”
echo “”
— ftimes-create-archive.sh —

sed -e ‘1,/^— ftimes-create-archive.sh —$/d; /^— ftimes-create-archive.sh —$/,$d’ ftimes-map-verify-backup.txt > ftimes-create-archive.sh

3. Inspect the baseline data.  You should get something similar to
the following:

cat /tmp/ftimes-original/baseline.log

— baseline.log —
<<<  EXECDATA  >>>|Program=ftimes 3.0.0 ssl –mapfull
<<<  EXECDATA  >>>|SystemOS=i386 FreeBSD 4.2-RELEASE
<<<  EXECDATA  >>>|Hostname=utopia.ir.exodus.net
+++  LANDMARK  +++|Stage1=MapModeInitialize
+++  LANDMARK  +++|Stage2=MapModeCheckDependencies
+++  LANDMARK  +++|Stage3=MapModeFinalize
<<<  PROPERTY  >>>|BaseName=baseline
<<<  PROPERTY  >>>|Compress=N
<<<  PROPERTY  >>>|FieldMask=None+md5
<<<  PROPERTY  >>>|HashDirectories=N
<<<  PROPERTY  >>>|LogDir=/tmp/ftimes-original/example-tree
<<<  PROPERTY  >>>|MapRemoteFiles=N
<<<  PROPERTY  >>>|NewLine=LF
<<<  PROPERTY  >>>|OutDir=/tmp/ftimes-original/example-tree
<<<  PROPERTY  >>>|RequirePrivilege=N
<<<  PROPERTY  >>>|RunType=baseline
<<<  PROPERTY  >>>|URLPutSnapshot=N
<<<  PROPERTY  >>>|URLAuthType=none
<<<  PROPERTY  >>>|SSLVerifyPeerCert=N
<<<  PROPERTY  >>>|SSLUseCertificate=N
<<<  PROPERTY  >>>|Include=/tmp/ftimes-original/example-tree
<<<  PROPERTY  >>>|Exclude=/tmp/ftimes-original/example-tree/baseline_20020307161750.log
<<<  PROPERTY  >>>|Exclude=/tmp/ftimes-original/example-tree/baseline_20020307161750.map
+++  LANDMARK  +++|Stage4=MapModeWorkHorse
+++  LANDMARK  +++|Stage5=MapModeFinishUp
<<<  MODEDATA  >>>|LogFileName=/tmp/ftimes-original/example-tree/baseline_20020307161750.log
<<<  MODEDATA  >>>|OutFileName=/tmp/ftimes-original/example-tree/baseline_20020307161750.map
<<<  MODEDATA  >>>|OutFileHash=8d7601bd431041349c133a7514708a91
<<<  MODEDATA  >>>|DataType=map
<<<  MODEDATA  >>>|DirectoriesEncountered=1
<<<  MODEDATA  >>>|FilesEncountered=10
<<<  MODEDATA  >>>|SpecialsEncountered=0
<<<  MODEDATA  >>>|AnalysisStages=Digest
<<<  MODEDATA  >>>|ObjectsAnalyzed=10
<<<  MODEDATA  >>>|BytesAnalyzed=70
<<<  MODEDATA  >>>|CompleteRecords=11
<<<  MODEDATA  >>>|IncompleteRecords=0
<<<  EXECDATA  >>>|Warnings=0
<<<  EXECDATA  >>>|Failures=0
<<<  EXECDATA  >>>|RunEpoch=2002/03/07 16:17:50 GMT
<<<  EXECDATA  >>>|Duration=0
— baseline.log —

cat /tmp/ftimes-original/baseline.map

— baseline.map —
name|md5
“/tmp/ftimes-original/example-tree/file-0″|8b57a783a6dcf09bc691f6d26ae4f451
“/tmp/ftimes-original/example-tree/file-1″|a23f1c379829d744eb857b5744008a31
“/tmp/ftimes-original/example-tree/file-2″|c33246d3db226129ba39143b6429893d
“/tmp/ftimes-original/example-tree/file-3″|e6a90768221a5739337d3499323cfe67
“/tmp/ftimes-original/example-tree/file-4″|9b92a1b8349534f382e72df7a3fbccb4
“/tmp/ftimes-original/example-tree/file-5″|78ffacd0e4cfcc69bb03937397e578c3
“/tmp/ftimes-original/example-tree/file-6″|1c212908e74cd850dbfd73f62c08f1f7
“/tmp/ftimes-original/example-tree/file-7″|e1a087d7f7fa720aa9ceb6d5d32827e4
“/tmp/ftimes-original/example-tree/file-8″|971cd516a410b2aedc4c3d0d193e9e93
“/tmp/ftimes-original/example-tree/file-9″|71d8dc4aaf73e8eec8960ed1b3f253f1
“/tmp/ftimes-original/example-tree”|DIRECTORY
— baseline.map —

4. Time passes, and now you wish to verify the archive.  This recipe
assumes that the validation process will take place in some
temporary storage area that does not, in general, have the same
path prefix as the original location.  One example of this would
be to verify a backup of /usr/local/bin by extracting the
corresponding archive to /tmp.  Because FTimes always collects
full path information, it is will be necessary to pre-process
the snapshot and rectify prefix differences.

Note: if you simply restore the archived tree back to the its
original location, it would not be necessary to resolve prefix
differences.  However, if the archive is corrupt, there is a
risk that system resources and/or availability could be impaired.
Ultimately, this risk depends on the relationship between the
content of the archive and the subject system.

Therefore, the verification process is as follows:

– Restore the backup to a) a temporary or b) the original location.
– Create a snapshot of the restored tree.
– Resolve the snapshot’s path prefix information, if necessary.
– Compare the snapshot to the restored baseline.

Using the ftimes-verify-archive.sh script, this step can be
accomplished with the following command:

ftimes-verify-archive.sh /tmp/ftimes-original/example-tree-2002-03-07.tar /tmp/restore-area

— ftimes-verify-archive.sh —
#!/bin/sh

WORKDIR=/tmp

if [ “$#” != “2” ]; then
echo “”
echo “Usage: `basename $0` archive restore-area”
echo “”
exit 1
else
ARCHIVE=$1
RESTORE_AREA=$2
RESTORE_AREA_REGEX=`echo ${RESTORE_AREA} | sed ‘s/\//\\\\\//g’`
fi

umask 22

if [ -d ${RESTORE_AREA} ]; then
rm -rf ${RESTORE_AREA};
fi

mkdir -p ${RESTORE_AREA}
tar -C ${RESTORE_AREA} -xf ${ARCHIVE}

BASELINE=baseline
SNAPSHOT=snapshot
SUBJECT=`basename \`tar -tf ${ARCHIVE} | head -1\“

ftimes –mapauto none+md5 ${RESTORE_AREA}/${SUBJECT} > ${RESTORE_AREA}/${SNAPSHOT}.map 2> ${RESTORE_AREA}/${SNAPSHOT}.log

ORIGINAL_PARENT=`dirname \`tail -1 ${RESTORE_AREA}/${SUBJECT}/${BASELINE}.map | sed ‘s/^”\(.*\)”.*$/\1/’\“
ORIGINAL_PARENT_REGEX=`echo ${ORIGINAL_PARENT} | sed ‘s/\//\\\\\//g’`
SED_EXPRESSION=s/^\”${RESTORE_AREA_REGEX}/\”${ORIGINAL_PARENT_REGEX}/

sed ${SED_EXPRESSION} ${RESTORE_AREA}/${SNAPSHOT}.map | ftimes –compare none+md5 ${RESTORE_AREA}/${SUBJECT}/${BASELINE}.map –
— ftimes-verify-archive.sh —

sed -e ‘1,/^— ftimes-verify-archive.sh —$/d; /^— ftimes-verify-archive.sh —$/,$d’ ftimes-map-verify-backup.txt > ftimes-verify-archive.sh

If the archive’s integrity is uncompromised, you should get some
output that looks similar to that shown below.  There should be
two new files: baseline.{map|log} and everything else should be
unchanged.  The reason that baseline.{map|log} show up as new
files is because FTimes excluded them during the baselining
process.  In other words, their content was in flux.

— compare.log —
<<<  EXECDATA  >>>|Program=ftimes 3.0.0 ssl –compare
<<<  EXECDATA  >>>|SystemOS=i386 FreeBSD 4.2-RELEASE
<<<  EXECDATA  >>>|Hostname=utopia.ir.exodus.net
+++  LANDMARK  +++|Stage1=CmpModeInitialize
+++  LANDMARK  +++|Stage2=CmpModeCheckDependencies
+++  LANDMARK  +++|Stage3=CmpModeFinalize
category|name|changed|unknown
<<<  PROPERTY  >>>|FieldMask=none+md5
+++  LANDMARK  +++|Stage4=CmpModeWorkHorse
N|”/tmp/ftimes-original/example-tree/baseline.log”||
N|”/tmp/ftimes-original/example-tree/baseline.map”||
+++  LANDMARK  +++|Stage5=CmpModeFinishUp
<<<  MODEDATA  >>>|LogFileName=stderr
<<<  MODEDATA  >>>|OutFileName=stdout
<<<  MODEDATA  >>>|DataType=cmp
<<<  MODEDATA  >>>|RecordsAnalyzed=11
<<<  MODEDATA  >>>|ChangedCount=0
<<<  MODEDATA  >>>|MissingCount=0
<<<  MODEDATA  >>>|NewCount=2
<<<  MODEDATA  >>>|UnknownCount=0
<<<  EXECDATA  >>>|Warnings=0
<<<  EXECDATA  >>>|Failures=0
<<<  EXECDATA  >>>|RunEpoch=2002/03/07 16:27:16 GMT
<<<  EXECDATA  >>>|Duration=0
— compare.log —

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s