In the config file for the pcapsipdump program, there is a retention option where you are supposed to be able to enter the number of days to keep the directories in your spool folder and it should auto purge out the old directories/files, however, this option either doesn’t appear to be implemented yet or doesn’t work if you run the app as a different user (could be a permissions issue maybe?).
Anyhow, this script is my solution to the problem. It could also very easily be modified and used where someone needs to search and delete directories that are based on YYYYMMDD format with the ability to whitelist or ignore certain directories, hence the alternate title. Yet another alternate title could even be “Bash Script Functions – Convert YYYYMMDD to Unix Time and Vice Versa” but obviously the script would need modified a little because it has been tailored to fit my needs.
When run as a cron job, the script will grab both the spool directory and the retention period from the pcapsipdump config file and will purge out any old folders based on YYYYMMDD format so they aren’t eating up all your valuable disk space.
I also added a couple of noteworthy user-configurable options. The first being a directory whitelist feature (IGNORE_DIRS array), this option can be used in situations where there are directories that you don’t want purged or would like to keep the data indefinitely and that can also be used in conjunction with the PURGE_ALL feature to delete out any subdirectories that aren’t specifically listed in the whitelist if you are a clean freak.
Download it now – pcaprotate.sh
#!/bin/bash
#
# pcapsipdump file rotation
# By Nathan Thomas
# 12/05/2014
#
### VARS ###
# Location of the pcapsipdump config file
CONF_FILE='/etc/default/pcapsipdump';
# Keep the spool folder clean - Delete all folders (except those in the ignored directories array)
# yes - Keep the spool folder free from any directories not in the ignore list
# no - Don't worry about any other random folders in the spool directory
# NOTE: Beware that any folders with names similar to YYYYMMDD format and are integers may possibly
# get deleted from the spool folder if an exception is not placed specifically in the ignore whitelist
PURGE_ALL='no';
# An array of any subdirectories in the spool folder we want to keep no matter what
# If PURGE_ALL='yes' then any subdirectories you want to keep in the spool folder need to be added to this whitelist
# If PURGE_ALL='no' then any subdirectories not in date format will be ignored and you can
# still use this variable to specify any specific folders in date format that you do wish to keep
IGNORE_DIRS=('tools' 'somedir' 'someotherdir');
### CODE ###
# If file doesn't exist quit
if [ ! -e "${CONF_FILE}" ] ; then
echo "The pcapsipdump config file does not exist at '${CONF_FILE}', aborting...";
exit 1;
fi
# Simple var check
case "${PURGE_ALL}" in
"yes"|"no")
;;
*)
echo "The PURGE_ALL variable can only be 'yes' or 'no'.";
exit 1;
;;
esac
# Get retention period and spool directory variables
RETENTION=$(cat "${CONF_FILE}" | grep "RETENTION=" | awk -F '=' '{ print $2 }');
SPOOLDIR=$(cat "${CONF_FILE}" | grep "SPOOLDIR=" | awk -F '=' '{ print $2 }');
# If retention period isn't numeric quit
if [ ! "${RETENTION}" -eq "${RETENTION}" ] > /dev/null 2>&1; then
echo "The retention period in the config file was not numeric, aborting...";
exit 1;
fi
# If spool directory doesn't exist quit
if [ ! -d "${SPOOLDIR}" ] > /dev/null 2>&1; then
echo "The pcapsipdump spool directory does not exist at '${SPOOLDIR}', aborting...";
exit 1;
fi
# Get subdirectory string
SUBDIRS=$(basename -a $(ls -d "${SPOOLDIR}"/*)) > /dev/null 2>&1;
# Set internal field seperator to split on carriage return
IFS=$'\n';
# Create array of subdirectory names
SUBDIRS=($(echo "${SUBDIRS}" | sed ':a;N;$!ba;s/\r\n/ /g')) > /dev/null 2>&1;
# Loop through subdirectory names
for DIRS in "${SUBDIRS[@]}" ; do
i=0
# Loop through the ignored directories array and test for matches
for ITEMS in "${IGNORE_DIRS[@]}" ; do
# if string doesn't match increase counter
if [ ! "${DIRS}" == "${ITEMS}" ] ; then
((i++));
# if counter reaches total number of elements
if [ "${i}" -eq "${#IGNORE_DIRS[@]}" ] ; then
# Push into new array
NEWSUBDIRS=("${NEWSUBDIRS[@]}" "${DIRS}");
i=0; # reset counter
fi
else
# strings match, do nothing (bitchslap)
break;
fi
done
done
# Convert YYYYMMDD to Unix Time
# params: $1=date
function YMD2U () {
local VAR="${1}";
if [ "${VAR}" -eq "${VAR}" ] > /dev/null 2>&1; then # if is numeric
# check var length
if [ ! "${#VAR}" -eq 8 ] ; then # Not a full 8 digits
return 1; # Date not in YYYYMMDD format...probably a subdirectory
fi
# Additional crude/laughable date checks
local YEAR="${VAR:0:4}";
local MONTH="${VAR:4:2}";
local DAY="${VAR:6:2}";
if [ "${YEAR}" -lt 0000 -o "${YEAR}" -gt 3000 ] ; then
return 1; # Invalid year
fi
if [ "${MONTH}" -lt 1 -o "${MONTH}" -gt 12 ] ; then
return 1; # Invalid month
fi
if [ "${DAY}" -lt 1 -o "${DAY}" -gt 31 ] ; then
return 1; # Invalid day
fi
# lastly check validity with date command
local RETVAR="$(date -d ${VAR} +%s)";
echo "${RETVAR}" | grep -q "invalid date";
if [ ! "$?" -eq 0 ] ; then
echo "${RETVAR}";
return 0;
else
return 1; # Date command returned invalid
fi
else
return 1; # Wasn't numeric
fi
}
# Convert Unix Time to YYYYMMDD
# params: $1=date
function U2YMD () {
local VAR="${1}";
if [ "${VAR}" -eq "${VAR}" ] > /dev/null 2>&1; then # if is numeric
# check validity with date command
local RETVAR="$(date +"%Y%m%d" -d @${VAR})";
echo "${RETVAR}" | grep -q "invalid date";
if [ ! "$?" -eq 0 ] ; then
echo "${RETVAR}";
return 0;
else
return 1; # Date command returned invalid
fi
else
return 1;
fi
}
# Convert retention period to seconds
RETSEC=$((${RETENTION}*86400));
for DIR in "${NEWSUBDIRS[@]}" ; do
# Make sure the function runs ok
YMD2U "${DIR}" > /dev/null 2>&1;
RETVAL="$?";
if [ "${RETVAL}" -eq 0 ] ; then
# if utime of dir <= (current utime - retention period)
if [ $(YMD2U "${DIR}") -le $(($(date +%s)-${RETSEC})) ] ; then
rm -rf ${SPOOLDIR}/${DIR} > /dev/null 2>&1;
fi
else
# Function returned 1 and this folder is not numeric or in YYYYMMDD format
if [ "${PURGE_ALL}" == "yes" ] ; then
rm -rf ${SPOOLDIR}/${DIR} > /dev/null 2>&1;
fi
fi
done