Archive for the ‘scripting’ Category

Cleaning up /tmp under HDFS

Monday, October 8th, 2018

The script to wipe out /tmp under HDFS (originally posted here). Could be run in a crontab to periodically delete files older than XXX days.


usage="Usage: [days]"

if [ ! "$1" ]
  echo $usage
  exit 1

now=$(date +%s)

hadoop fs -ls /tmp/hive/hive/ | grep "^d" | while read f;
  dir_date=`echo $f | awk '{print $6}'`
  difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))

if [ $difference -gt $1 ]; 
  hadoop fs -ls `echo $f | awk '{ print $8 }'`;
### hadoop fs -rm -r `echo $f | awk '{ print $8 }'`;


By default the script will be executed in a “dry” mode, listing the files that are older than XXX days. Once you’re comfortable with the output, comment the line containing ‘fs -ls’ and uncomment the one with ‘fs -rm’.

If you get Java memory errors while executing the script, make sure to pass HADOOP_CLIENT_OPTS variable prior to calling the script:

export HADOOP_CLIENT_OPTS="-XX:-UseGCOverheadLimit -Xmx4096m"

validrcptto & merging two different recipient sources

Sunday, May 4th, 2008

Today I’ve been approached by one of our projects with the request to build qmail’s validrcptto.cdb from two different sources.

They use two email servers: the first one, powered by qmail and patched with great patch set by John Simpson, is providing SMTP relay services including virus/spam scanning for, the second one is powered by Exchange within AD. qmail server also serves as a mailing list server hosting subdomain.

The idea is the following. First, get the list of valid recipients from Exchange server by using the script called adexport written by Brian Landers wrapped by John Simpson’s adexport-go script. Then merge it with the local list generated by John Simpson’s mkvalidrcptto script and finally build validrcptto.cdb.


How to change permissions separately for files and directories

Friday, February 8th, 2008

I wanted to change permissions so that all files within /tmp/some/directory will be 644 and all subdirectories 755.

cd /tmp/some/directory
chmod -R 0644 *
find . -type d -exec chmod 0755 {} \;

Happy scripting.