Posts Tagged ‘scripting’

Cleaning up /tmp under HDFS

Monday, October 8th, 2018

The script to wipe out /tmp under HDFS (originally posted here). Could be run in a crontab to periodically delete files older than XXX days.


usage="Usage: [days]"

if [ ! "$1" ]
  echo $usage
  exit 1

now=$(date +%s)

hadoop fs -ls /tmp/hive/hive/ | grep "^d" | while read f;
  dir_date=`echo $f | awk '{print $6}'`
  difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))

if [ $difference -gt $1 ]; 
  hadoop fs -ls `echo $f | awk '{ print $8 }'`;
### hadoop fs -rm -r `echo $f | awk '{ print $8 }'`;


By default the script will be executed in a “dry” mode, listing the files that are older than XXX days. Once you’re comfortable with the output, comment the line containing ‘fs -ls’ and uncomment the one with ‘fs -rm’.

If you get Java memory errors while executing the script, make sure to pass HADOOP_CLIENT_OPTS variable prior to calling the script:

export HADOOP_CLIENT_OPTS="-XX:-UseGCOverheadLimit -Xmx4096m"

http to https redirect under F5

Monday, October 13th, 2014

Say you have a virtual web server, serving, configured on F5 with the following iRule redirecting plain http to https:


	if { [HTTP::host] equals "" } {
		HTTP::redirect "https://[HTTP::host][HTTP::uri]"

You decided to buy another domain ( and want to point it to the same IP where is hosted, with the redirect to Here is how the modified iRule will look like:


	if { [string tolower [HTTP::host]] ends_with "" } { 
		HTTP::redirect "[HTTP::uri]" 
	} elseif { [HTTP::host] equals "" } {
		HTTP::redirect "https://[HTTP::host][HTTP::uri]"

Here is an alternative way to implement redirection by using HTTP Class profile.

Under Local Traffic › Profiles › Protocol › HTTP Class create a new profile:

Parent Profile: httpclass

Hosts: Match all
URI Paths: Match all
Headers: Match all
Hosts: Match all
Cookies: Match all

Send To: Redirect to...
Redirect to Location: https://[getfield [HTTP::host] ":" 1][HTTP::uri]

For redirection, modify it and change Redirect to Location to:

Send To: Redirect to...
Redirect to Location:[HTTP::uri]

Backup of F5 BIG-IP

Monday, April 8th, 2013

An overview of steps to backup configuration of F5 BIG-IP appliance (10.2.X) to the remote server running Linux Ubuntu 12.04.2 LTS (precise). The backup is done over SSH.

List of actions to be done on the Linux server:

1. Create a user (for example f5backup with /home/f5backup as a home directory).

2. Create .ssh directory under /home/f5backup:

mkdir /home/f5backup/.ssh && chown -R f5backup:f5backup /home/f5backup/.ssh

3. Copy /var/ssh/root/ file from the F5 appliance to /home/f5backup/.ssh/authorized_keys file on the Linux server and fix permissions:

cd /home/f5backup/.ssh && chown f5backup:f5backup authorized_keys && chmod 600 authorized_keys

4. Ensure that PubkeyAuthentication is set to yes in /etc/ssh/sshd_config file:

PubkeyAuthentication yes

List of actions to be done on the F5 appliance:

1. Create a directory, for example /root/scripts/.

2. Download the relevant version of backup script from here (you need to register to access F5 DevCentral), name it, place it under /root/scripts/ directory and ‘chmod 755’ it.

3. Create f5archive_config file under /root/.ssh/ directory:

Host *
   User f5backup
   PasswordAuthentication no
   StrictHostKeyChecking yes
   IdentityFile /root/.ssh/f5archive_dsa
   Port 22
   Protocol 2
   Ciphers aes128-cbc,aes192-cbc,aes256-cbc
   UserKnownHostsFile /root/.ssh/f5archive_host

4. Copy /var/ssh/root/identity file into /root/.ssh/f5archive_dsa:

cd /root/.ssh && cp /var/ssh/root/identity f5archive_dsa

5. Now ssh to the Linux server so /var/ssh/root/known_hosts file is updated with the host entry.

6. Copy /var/ssh/root/known_hosts file into /root/.ssh/f5archive_host.

7. Fix permissions:

cd /root/.ssh && chmod 600 f5*

8. Modify /root/scripts/ file to suit your needs, particularly SCP_DESTINATION part:


9. Finally, create a cron task to execute the file on a regular basis (I run it daily at 2am). To do so, create /etc/cron.d/f5backup file with the following content:

0 2 * * * root /bin/bash /root/scripts/ 1>/var/tmp/f5backup.log 2>&1

validrcptto & merging two different recipient sources

Sunday, May 4th, 2008

Today I’ve been approached by one of our projects with the request to build qmail’s validrcptto.cdb from two different sources.

They use two email servers: the first one, powered by qmail and patched with great patch set by John Simpson, is providing SMTP relay services including virus/spam scanning for, the second one is powered by Exchange within AD. qmail server also serves as a mailing list server hosting subdomain.

The idea is the following. First, get the list of valid recipients from Exchange server by using the script called adexport written by Brian Landers wrapped by John Simpson’s adexport-go script. Then merge it with the local list generated by John Simpson’s mkvalidrcptto script and finally build validrcptto.cdb.


How to change permissions separately for files and directories

Friday, February 8th, 2008

I wanted to change permissions so that all files within /tmp/some/directory will be 644 and all subdirectories 755.

cd /tmp/some/directory
chmod -R 0644 *
find . -type d -exec chmod 0755 {} \;

Happy scripting.