Posted on by IG-N88

Bash Script – Move Files into Subdirectories Based on Modified Date – OR – Moving Ntop Netflow Dumps into Subdirectories by Date

In a busy network environment, it is critical to have some form of network monitoring on all your servers and equipment. Network monitoring comes in many different forms and flavors, whether it be to monitor critical system services and applications via SNMP, WMI, or some proprietary third party software, or just generically pinging some devices to make sure they are up.

Nowadays, it is getting more and more necessary to dig deeper to be able to track what end users are doing and what websites they are visiting and this is where the Cisco Netflow comes in handy. Essentially, netflow allows us to peer down into the network traffic streams and give us vital source, destination, and protocol information coming to and from our network hosts but isn’t quite as storage intensive as doing a full fledged pcap dump, which makes historical accounting of this data a whole lot nicer.

Ntop used to be an open source netflow collector and analyzer, that is, until versions higher than 5.0.1. In my world, we make due with what open source utilities we have available at our disposal, so that version of Ntop does have some useful features and can help display some interesting trends, but, from it you can also dump the network flows to disk somewhere for analyzing with other tools such as Nfdump, Nfsight, and Nfsen.

That’s where this little bash script I wrote the other day comes into play. So, in this example here, I have all of netflow data being dumped to a single folder on a big ass local drive somewhere. In doing this, each separate netflow interface we want to monitor inside of Ntop will create it’s own separate subfolder of flows dumped at the specified interval. I believe the flow names are basically “UnixTimestamp.flow”.

Using the script below and running it on a cron job at midnight, we can move each netflow device’s flows for the day into it’s own subdirectory with a date timestamp for the folder name. Later on down the road, you could easily modify this script to tar up or compress the files to save some space. Cheers!

#!/bin/bash
#
# Nightly script to move each days netflow captures for each collector into it's own subdirectory
# Author: Nathan Thomas
# Date: 02/18/2016

### VARS ###
BASEDIR='/Volumes/storage/netflow-dump/interfaces';
FLOWDIRS=('NetflowDevice1' 'NetflowDevice2' 'NetflowDevice3' 'NetflowDevice4');
NEWDIR=$(date --date="yesterday" +"%m-%d-%Y");
SDAY=$(date --date="yesterday" +"%Y-%m-%d");
EDAY=$(date --date="today" +"%Y-%m-%d");

### CODE ###
# Loop through the directory array
for DIRS in "${FLOWDIRS[@]}" ; do
        # Create subdir with yesterday's date
        mkdir ${BASEDIR}/${DIRS}/${NEWDIR} > /dev/null 2>&1;
        if [ -d ${BASEDIR}/${DIRS}/${NEWDIR} ] ; then  # if dir exists
                # Find the flow files with yesterdays modified date and move to the subdir
                find ${BASEDIR}/${DIRS} -maxdepth 1 -newermt "${SDAY}" ! -newermt "${EDAY}" -type f -iname "*.flow" -exec mv {} ${BASEDIR}/${DIRS}/${NEWDIR}/ \; > /dev/null 2>&1;
                # Another lazier method you could use
                # mv ${BASEDIR}/${DIRS}/*.flow ${BASEDIR}/${DIRS}/${NEWDIR}/ > /dev/null 2>&1;
        fi
done

Leave a Reply