Posts Tagged ‘linux’

Cleaning up /tmp under HDFS

Monday, October 8th, 2018

The script to wipe out /tmp under HDFS (originally posted here). Could be run in a crontab to periodically delete files older than XXX days.


usage="Usage: [days]"

if [ ! "$1" ]
  echo $usage
  exit 1

now=$(date +%s)

hadoop fs -ls /tmp/hive/hive/ | grep "^d" | while read f;
  dir_date=`echo $f | awk '{print $6}'`
  difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))

if [ $difference -gt $1 ]; 
  hadoop fs -ls `echo $f | awk '{ print $8 }'`;
### hadoop fs -rm -r `echo $f | awk '{ print $8 }'`;


By default the script will be executed in a “dry” mode, listing the files that are older than XXX days. Once you’re comfortable with the output, comment the line containing ‘fs -ls’ and uncomment the one with ‘fs -rm’.

If you get Java memory errors while executing the script, make sure to pass HADOOP_CLIENT_OPTS variable prior to calling the script:

export HADOOP_CLIENT_OPTS="-XX:-UseGCOverheadLimit -Xmx4096m"

Moving /var/log to a different drive under CentOS 7

Saturday, July 14th, 2018

A quick how-to with the set of instructions to move /var/log partition to a different drive. Done on CentOS 7.5.1804 with all partitions managed by LVM, while /var/log is moved to a USB key.

The tricky part with /var/log is that there is always something being written to it, and although simple archive/restore might work, you risk to lose changes from the moment you create archive till the moment you restore it. Depending on how big /var/log is it could be minutes/hours of data. The procedure below assumes an outage since it will be performed offline, however it assures that no data will be lost.


DRBD with OCFS2 and fstab

Sunday, May 28th, 2017

Two-nodes active/active DRBD cluster implemented on Debian Jessie with OCFS2 on top of it, so the file system can be mounted and accessed on both nodes at the same time. Sounds like a easy-peasy task considering the amount of articles on the web (mainly copy/paste of the same content though).

So, you finish with the setup, everything is synced and shiny, you edit fstab, perform the final reboot, and… oopsie daisy, nothing is mounted. You start digging into _netdev direction, or suspecting that perhaps an order in which drbd and ocfs2 are started is to blame, or putting mount stanza into rc.local — none of this helps. You might even come up with an excuse that you will not reboot those servers often, however, the fact that you need to manually perform some post-reboot actions doesn’t sound promising at all. Particularly if it’s an unexpected reboot over a weekend. Particularly if it happened some years after the installation hence you need to find (and most importantly, keep in mind about) those notes. Particularly if you already quit this job, and there is another poor fella taking care of the servers. And finally, to make things even more complicated, you might have services that actually depend on the availability of the mounted drive after the reboot (Apache or Samba for example).

Obviously, this needs to be fixed once and for all, and I have good news for you. :) If you were vigilant enough during troubleshooting you’d notice that a) if you try to mount the drive through /etc/rc.local there will be a warning thrown at boot time (something about missing device), and b) when you mount drbd drive manually it’s not mounted instantly — there is several seconds delay before the disk is successfully attached. That brought me to the suspicion that perhaps drbd is actually not ready at the time mount in /etc/rc.local is executed, and by deliberately introducing some delay things can be improved. And voila — it really did seem to do the trick!

Here is my /etc/fstab entry:

/dev/drbd0   /var/www   ocfs2   noauto,noatime   0   0

And here is my /etc/rc.local introducing 30 seconds delay prior to mount, to give enough time for DRBD to cool down:

sleep 30
mount /dev/drbd0
exit 0

Now, I’m not sure whether this is by design, since DRBD nodes do have to communicate with each other (initial election and/or sync), and that contributes to the delay in creating /dev/drbd0, OR, my environment is generally slow (everything is virtualized on not-so-super-fast SATA drives), but it works.

Viewing package ChangeLog with rpm

Monday, April 4th, 2016

Here is how to view the ChangeLog of installed package using rpm under CentOS:

rpm -q ----changelog libuuid-2.23.2-26.el7_2.2.x86_64 | more

* Wed Mar 16 2016 Karel Zak  2.23.2-26.el7_2.2
- fix #1317953 - lslogins crash when executed with buggy username

Same applies to the kernel. By adding -p switch you can actually check the rpm file itself without installing it:

rpm -qp ----changelog | more

* Thu Mar 31 2016 Akemi Yagi  []
- Apply debranding changes
- Roll in i686 mods
- Modify config file for x86_64 with extra features turned on including

Unattended installation of CentOS 7 with Kickstart

Sunday, March 13th, 2016

While setting up my first Hadoop cluster I faced with the dilemma of how to perform installations of CentOS 7 on multiple servers at once. If you have 20 data nodes to deploy, anything you chose to automate an installation will greatly reduce the deployment time, but most importantly, it will eliminate the possibility of human error (typo for example).

Initially, I started looking at the disk cloning direction. Since all my data nodes are identical, I was thinking to prepare one data node server, then dd the system drive, place it on a NFS share, boot the server and re-image the system drive using dd image from the share. Clonezilla and DRBL seem to be the perfect pair for a such scenario. And although you will spend some time configuring, testing and tuning it, it was still worth to look into it.

Then I realized that even if I manage to establish the setup above, I’ll still have to deal with manual post-installation tweaks, like regeneration of SSH keys and probably adjusting of MAC addresses. On top of that, to transfer raw dd image (in my case it was ~30GB) might take longer than initial installation itself. Therefore I ended up using Kickstart method. I’m pretty sure there are more efficient solutions and if you happen to know one I’d love to hear your comments.


Configuring shared network with DHCP

Monday, August 20th, 2012

Say you have a DHCP server in the LAN serving /24 network and one day you’re running out of IP addresses. You want to add additional /24 network that should be distributed in the same LAN. Ugly, but what to do.

According to man dhcpd.conf:

The shared-network statement is used to inform the DHCP server that some IP subnets actually share the same physical network. Any subnets in a shared network should be declared within a shared-network statement. Parameters specified in the shared-network statement will be used when booting clients on those subnets unless parameters provided at the subnet or host level override them. If any subnet in a shared network has addresses available for dynamic allocation, those addresses are collected into a common pool for that shared network and assigned to clients as needed. There is no way to distinguish on which subnet of a shared network a client should boot.

Here is how you add additional network to be included into DHCP scope. Done on Ubuntu 9.10 (karmic) and ISC DHCP v3.1.2.

shared-network "officea01" {
  option domain-name "";
  option domain-name-servers;
    subnet netmask {
      option routers;
      allow unknown-clients;
    subnet netmask {
      option routers;
      allow unknown-clients;

Instructions below are not necessary, however I decided to add an alias to the LAN interface so I can see addresses in the ARP table.

ifconfig eth1:0 netmask up

And to make it permanent edit /etc/network/interfaces:

auto eth1:0
iface eth1:0 inet static

Selective NAT with iptables

Thursday, January 6th, 2011

Here is a quick note on how to exclude one particular network from the NAT while the rest to be remained NATed. Done on Ubuntu with iptables running.

iptables -t nat -A POSTROUTING -o eth0 ! -d -j MASQUERADE

In case you need one more network to be excluded:

iptables -t nat -A POSTROUTING -o eth0 -d -j RETURN
iptables -t nat -A POSTROUTING -o eth0 ! -d -j MASQUERADE

Injecting multiple networks into VPN tunnel

Wednesday, January 5th, 2011

We have a number of IPSec tunnels running on Linux Ubuntu with Openswan installed. Here is the typical configuration for a site: