Posts Tagged ‘scripting’

Cleaning up /tmp under HDFS

Monday, October 8th, 2018

The script to wipe out /tmp under HDFS (originally posted here). Could be run in a crontab to periodically delete files older than XXX days.

  1. #!/bin/bash
  2.  
  3. usage="Usage: cleanup_tmp.sh [days]"
  4.  
  5. if [ ! "$1" ]
  6.  then
  7.   echo $usage
  8.   exit 1
  9. fi
  10.  
  11. now=$(date +%s)
  12.  
  13. hadoop fs –ls /tmp/hive/hive/ | grep "^d" | while read f;
  14.  do
  15.   dir_date=`echo $f | awk '{print $6}'`
  16.   difference=$(( ( $now – $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))
  17.  
  18. if [ $difference -gt $1 ];
  19.  then
  20.   hadoop fs –ls `echo $f | awk '{ print $8 }'`;
  21. ### hadoop fs -rm -r `echo $f | awk '{ print $8 }'`;
  22. fi
  23.  
  24. done

By default the script will be executed in a “dry” mode, listing the files that are older than XXX days. Once you’re comfortable with the output, comment the line containing ‘fs -ls’ and uncomment the one with ‘fs -rm’.

If you get Java memory errors while executing the script, make sure to pass HADOOP_CLIENT_OPTS variable prior to calling the script:

  1. export HADOOP_CLIENT_OPTS="-XX:-UseGCOverheadLimit -Xmx4096m"

Tags:, , , , ,
Posted in centos, hadoop, scripting, tech | No Comments »

http to https redirect under F5

Monday, October 13th, 2014

Say you have a virtual web server, serving domain1.org, configured on F5 with the following iRule redirecting plain http to https:

  1. when HTTP_REQUEST {
  2.  
  3.  if { [HTTP::host] equals "domain1.org" } {
  4.   HTTP::redirect "https://[HTTP::host][HTTP::uri]"
  5.  }
  6. }

You decided to buy another domain (domain2.org) and want to point it to the same IP where domain1.org is hosted, with the redirect to https://domain1.org. Here is how the modified iRule will look like:

  1. when HTTP_REQUEST {
  2.  
  3.  if { [string tolower [HTTP::host]] ends_with ".domain2.org" } {
  4.   HTTP::redirect "https://domain1.org[HTTP::uri]"
  5.  } elseif { [HTTP::host] equals "domain1.org" } {
  6.   HTTP::redirect "https://[HTTP::host][HTTP::uri]"
  7.  }
  8. }

Here is an alternative way to implement redirection by using HTTP Class profile.

Under Local Traffic › Profiles › Protocol › HTTP Class create a new profile:

  1. Name: HTTP2HTTPS
  2. Parent Profile: httpclass
  3.  
  4. Hosts: Match all
  5. URI Paths: Match all
  6. Headers: Match all
  7. Hosts: Match all
  8. Cookies: Match all
  9.  
  10. Send To: Redirect to…
  11. Redirect to Location: https://[getfield [HTTP::host] ":" 1][HTTP::uri]

For domain2.org redirection, modify it and change Redirect to Location to:

  1. Send To: Redirect to…
  2. Redirect to Location: https://domain1.org/[HTTP::uri]

Tags:, , , , ,
Posted in f5, tech | No Comments »

Backup of F5 BIG-IP

Monday, April 8th, 2013

An overview of steps to backup configuration of F5 BIG-IP appliance (10.2.X) to the remote server running Linux Ubuntu 12.04.2 LTS (precise). The backup is done over SSH.

List of actions to be done on the Linux server:

1. Create a user (for example f5backup with /home/f5backup as a home directory).

2. Create .ssh directory under /home/f5backup:

  1. mkdir /home/f5backup/.ssh && chown -R f5backup:f5backup /home/f5backup/.ssh

3. Copy /var/ssh/root/identity.pub file from the F5 appliance to /home/f5backup/.ssh/authorized_keys file on the Linux server and fix permissions:

  1. cd /home/f5backup/.ssh && chown f5backup:f5backup authorized_keys && chmod 600 authorized_keys

4. Ensure that PubkeyAuthentication is set to yes in /etc/ssh/sshd_config file:

  1. PubkeyAuthentication yes

List of actions to be done on the F5 appliance:

1. Create a directory, for example /root/scripts/.

2. Download the relevant version of backup script from here (you need to register to access F5 DevCentral), name it backup.sh, place it under /root/scripts/ directory and ‘chmod 755’ it.

3. Create f5archive_config file under /root/.ssh/ directory:

  1. Host *
  2.    User f5backup
  3.    PasswordAuthentication no
  4.    StrictHostKeyChecking yes
  5.    IdentityFile /root/.ssh/f5archive_dsa
  6.    Port 22
  7.    Protocol 2
  8.    Ciphers aes128-cbc,aes192-cbc,aes256-cbc
  9.    UserKnownHostsFile /root/.ssh/f5archive_host

4. Copy /var/ssh/root/identity file into /root/.ssh/f5archive_dsa:

  1. cd /root/.ssh && cp /var/ssh/root/identity f5archive_dsa

5. Now ssh to the Linux server so /var/ssh/root/known_hosts file is updated with the host entry.

6. Copy /var/ssh/root/known_hosts file into /root/.ssh/f5archive_host.

7. Fix permissions:

  1. cd /root/.ssh && chmod 600 f5*

8. Modify /root/scripts/backup.sh file to suit your needs, particularly SCP_DESTINATION part:

  1. SCP_DESTINATION="f5backup@192.168.0.11:/home/f5backup"

9. Finally, create a cron task to execute the backup.sh file on a regular basis (I run it daily at 2am). To do so, create /etc/cron.d/f5backup file with the following content:

  1. SHELL=/bin/bash
  2. PATH=/sbin:/bin:/usr/sbin:/usr/bin
  3. HOME=/var/tmp
  4. 0 2 * * * root /bin/bash /root/scripts/backup.sh 1>/var/tmp/f5backup.log 2>&1

Tags:, , , , ,
Posted in f5, tech | No Comments »

validrcptto & merging two different recipient sources

Sunday, May 4th, 2008

Today I’ve been approached by one of our projects with the request to build qmail’s validrcptto.cdb from two different sources.

They use two email servers: the first one, powered by qmail and patched with great patch set by John Simpson, is providing SMTP relay services including virus/spam scanning for domain.org, the second one is powered by Exchange within AD. qmail server also serves as a mailing list server hosting lists.domain.org subdomain.

The idea is the following. First, get the list of valid recipients from Exchange server by using the script called adexport written by Brian Landers wrapped by John Simpson’s adexport-go script. Then merge it with the local list generated by John Simpson’s mkvalidrcptto script and finally build validrcptto.cdb.

(more…)

Tags:, , , ,
Posted in freebsd, qmail, scripting, tech | No Comments »

How to change permissions separately for files and directories

Friday, February 8th, 2008

I wanted to change permissions so that all files within /tmp/some/directory will be 644 and all subdirectories 755.

  1. cd /tmp/some/directory
  2. chmod -R 0644 *
  3. find . –type d –exec chmod 0755 {} \;

Happy scripting.

Tags:,
Posted in freebsd, misc, scripting, tech | No Comments »