Unattended installation of CentOS 7 with Kickstart

March 13th, 2016

While setting up my first Hadoop cluster I faced with the dilemma of how to perform installations of CentOS 7 on multiple servers at once. If you have 20 data nodes to deploy, anything you chose to automate an installation will greatly reduce the deployment time, but most importantly, it will eliminate the possibility of human error (typo for example).

Initially, I started looking at the disk cloning direction. Since all my data nodes are identical, I was thinking to prepare one data node server, then dd the system drive, place it on a NFS share, boot the server and re-image the system drive using dd image from the share. Clonezilla and DRBL seem to be the perfect pair for a such scenario. And although you will spend some time configuring, testing and tuning it, it was still worth to look into it.

Then I realized that even if I manage to establish the setup above, I’ll still have to deal with manual post-installation tweaks, like regeneration of SSH keys and probably adjusting of MAC addresses. On top of that, to transfer raw dd image (in my case it was ~30GB) might take longer than initial installation itself. Therefore I ended up using Kickstart method. I’m pretty sure there are more efficient solutions and if you happen to know one I’d love to hear your comments.

Read the rest of this entry »

How to configure vLAG on a Brocade VDX 6740T-1G switch to work with SafeNet Network HSM

January 26th, 2016

Caution! I deleted my previous post on how to configure vLAG on Brocade VDX 6740T-1G switch to work with SafeNet Network HSM because actually it didn’t work as it should. If you get a cached version somewhere please disregard it.

I have no idea how I managed to get bonding to operate in round-robin mode on SafeNet Network HSM:

  1. [hsm-node-1] lunash:>network interface bonding show
  2.  
  3. ———————————————————–
  4. Ethernet Channel Bonding Driver: v3.4.0-2 (October 7, 2008)
  5.  
  6. Bonding Mode: load balancing (round-robin)

Because once the appliance was rebooted the bonding mode has changed to active-backup and the whole story with LAGs became irrelevant. The primary interface started flapping again and the only way to stabilize connectivity to HSM was to disable the slave interface.

  1. [hsm-node-1] lunash:>network interface bonding show
  2.  
  3. ———————————————————–
  4. Ethernet Channel Bonding Driver: v3.4.0-2 (October 7, 2008)
  5.  
  6. Bonding Mode: fault-tolerance (active-backup)

So, back to the original subject of the post: how do you configure a LAG on Brocade switch to work with SafeNet Network HSM? The answer is — you don’t. In fault-tolerance bonding mode, when one interface is active and another one is backup (read passive), you don’t create any LAGs on the switch. All you have to do is to bring both interfaces to switchport mode access mode and ensure that VLAN and speed settings are identical. Here is how our switch config looks like:

  1. !
  2. interface TenGigabitEthernet 12/0/2
  3.  speed 1000
  4.  description -=HSM-NODE-1:ETH0=-
  5.  switchport
  6.  switchport mode access
  7.  switchport access vlan 12
  8.  spanning-tree shutdown
  9.  no fabric isl enable
  10.  no fabric trunk enable
  11.  no shutdown
  12. !
  13. interface TenGigabitEthernet 13/0/2
  14.  speed 1000
  15.  description -=HSM-NODE-1:ETH1=-
  16.  switchport
  17.  switchport mode access
  18.  switchport access vlan 12
  19.  spanning-tree shutdown
  20.  no fabric isl enable
  21.  no fabric trunk enable
  22.  no shutdown
  23. !

Now, you certainly lose link aggregation and load balancing functionalities, because only one interface will be passing traffic at a time. The slave interface comes into play only if the primary interface is down. We’re still good though when it comes to redundancy — you can disconnect the cable from ETH0 without any impact on connectivity.

On a HSM side, you don’t have many options so you follow the standard procedure: assign the IP address to the bond (network interface bonding config -ip x.x.x.x -netmask y.y.y.y -gateway z.z.z.z) and bring it up (network interface bonding enable).

To check the status:

  1. [hsm-node-1] lunash:>network interface bonding show
  2.  
  3. ———————————————————–
  4. Ethernet Channel Bonding Driver: v3.4.0-2 (October 7, 2008)
  5.  
  6. Bonding Mode: fault-tolerance (active-backup)
  7. Primary Slave: eth0 (primary_reselect failure)
  8. Currently Active Slave: eth1
  9. MII Status: up
  10. MII Polling Interval (ms): 100
  11. Up Delay (ms): 2000
  12. Down Delay (ms): 0
  13.  
  14. Slave Interface: eth0
  15. MII Status: up
  16. Speed: 1000 Mbps
  17. Duplex: full
  18. Link Failure Count: 0
  19. Permanent HW addr: 00:15:c4:n7:13:06
  20.  
  21. Slave Interface: eth1
  22. MII Status: up
  23. Speed: 1000 Mbps
  24. Duplex: full
  25. Link Failure Count: 0
  26. Permanent HW addr: 00:15:c4:n7:6a:34
  27. ———————————————————–
  28. ———————————————————–
  29. Status for eth0:
  30.         Link detected: yes
  31.  
  32. Status for eth1:
  33.         Link detected: yes
  34. ———————————————————–
  35.  
  36. Command Result : 0 (Success)
  1. [hsm-node-1] lunash:>status interface
  2.  
  3. bond0     Link encap:Ethernet  HWaddr 00:15:C4:N7:13:06
  4.           inet addr:192.168.100.42  Bcast:192.168.100.255  Mask:255.255.255.0
  5.           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
  6.           RX packets:13479 errors:0 dropped:0 overruns:0 frame:0
  7.           TX packets:3183 errors:0 dropped:0 overruns:0 carrier:0
  8.           collisions:0 txqueuelen:0
  9.           RX bytes:1059045 (1.0 MiB)  TX bytes:446623 (436.1 KiB)
  10.  
  11. eth0      Link encap:Ethernet  HWaddr 00:15:C4:N7:13:06
  12.           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
  13.           RX packets:12670 errors:0 dropped:0 overruns:0 frame:0
  14.           TX packets:2082 errors:0 dropped:0 overruns:0 carrier:0
  15.           collisions:0 txqueuelen:1000
  16.           RX bytes:996811 (973.4 KiB)  TX bytes:300205 (293.1 KiB)
  17.           Interrupt:58 Memory:fb4c0000-fb4e0000
  18.  
  19. eth1      Link encap:Ethernet  HWaddr 00:15:C4:N7:6A:34
  20.           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
  21.           RX packets:809 errors:0 dropped:0 overruns:0 frame:0
  22.           TX packets:1101 errors:0 dropped:0 overruns:0 carrier:0
  23.           collisions:0 txqueuelen:1000
  24.           RX bytes:62234 (60.7 KiB)  TX bytes:146418 (142.9 KiB)
  25.           Interrupt:169 Memory:fb6e0000-fb700000
  26.  
  27. Command Result : 0 (Success)

How to configure SNMP on a Brocade VDX 6740T-1G switch

January 25th, 2016

Below is a snippet of the config that worked for me to allow SNMP v1 polling of a Brocade VDX 6740T-1G switch. Nothing fancy, I just wanted to enable read-only, SNMP v1 access to the switch to start capturing the load of the interfaces. Note that the NOS version is 6.0.2.

  1. snmp-server contact "Your network crew"
  2. snmp-server location "DC A"
  3. snmp-server sys-descr "Brocade VDX 6740T-1G"
  4. snmp-server community XXXXX groupname monitor
  5. snmp-server view monitor 1.3.6 included
  6. snmp-server group monitor v1 read monitor

The first three lines are not interesting. The forth and the last one will enable SNMP v1 read-only access. Note that you have to specify a groupname. You can name it whatever you like but it has to be consistent.

Finally, without ‘snmp-server view monitor 1.3.6 included’ line you will be able to poll the switch but no data will be returned. Perhaps it could be useful if you have multiple teams and you want to separate who can monitor what, but since I don’t need it I allowed access to the whole MIB.

How to add a license to a Brocade VDX6740T-1G switch

January 24th, 2016

In order to license a particular feature on a Brocade VDX 6740T-1G switch you’ll need:

  • transaction key (22 characters long string received from your Brocade supplier, which is bound to a particular feature, for example BR-VDX6740T-1G-16X10G-COD (to add 16x10GB Capacity on Demand feature) or BR-VDX6740-2X40G-POD (to unlock two remaining QSFP ports));
  • access to the Brocade portal (Software Licensing section);
  • license ID of the switch where the license is going to be attached to.

To get a license ID, log in to the switch and run:

  1. show license id rbridge-id 12
  2.  
  3. ===================================================
  4.   12                    XX:XX:XX:XX:XX:XX:XX:XX

Since all my VDXs are in a VCS Logical Chassis mode, I have to specify the rbridge-id of the member.

Login to the Brocade portal, go to Software Licensing and enter the transaction key. On the next page you’ll be prompted for an email address and the license ID.

Once generated, you’ll receive a XML file with the long string between licKey tags.

Copy it (omit licKey tags) and execute on the switch:

  1. license add rbridge-id 12 licStr "XX XXXXXXXX#"

Make sure to place the license inside quotes, since normally there is a space in the license key.

To check whether the license was deployed run:

  1. show license rbridge-id 12
  2.  
  3. rbridge-id: 12
  4. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  5.        10G Port Upgrade license
  6.        Feature name:PORT_10G_UPGRADE
  7.        License is valid
  8.        Capacity: 16

Booting F5 BIG-IP LTM 3900 from USB

November 8th, 2015

As a side note: should you own BIG-IP LTM 3900 appliance and wonder how to boot it from the USB stick (to reinstall the OS or run the End User Diagnostics software) make sure to use a USB stick which is precisely 1GB in size. Any other USB sticks (2/4/8/16GB) simply won’t work.

Why it can’t be documented somewhere on the F5 website remains a mystery to me.

[20151205] : I stand corrected. It looks like it has nothing to do with the size but with the chipset. I received a Rescue Kit the other day from F5 and they included two USB 32GB sticks — it’s SanDisk Cruzer Glide.

Adding and troubleshooting KMS keys

October 29th, 2015

On the KMS host:

Use Volume Activation Tools to add a new host key or:

  1. cscript.exe c:\windows\system32\slmgr.vbs -ipk XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

To see what keys have been deployed and what the status is:

  1. cscript.exe c:\windows\system32\slmgr.vbs /dlv all

To activate (requires http/s access from the KMS host):

  1. cscript.exe c:\windows\system32\slmgr.vbs /ato activation.ID

where Activation ID is a string you get from running ‘cscript slmgr.vbs /dlv all’.

On the KMS client:

To manually force the activation of Windows OS (requires 1688/tcp access to the KMS host):

  1. cscript.exe c:\windows\system32\slmgr.vbs /ato

To manually force the activation of Windows Office (requires 1688/tcp access to the KMS host):

  1. cscript.exe c:\"program files (x86)"\"microsoft office"\office14\ospp.vbs /act

Deploying wildcard SSL certificate for VMware Horizon 6

February 6th, 2015

Quick notes on how to deploy a wildcard SSL certificate with VMware Horizon 6 setup. In my case there is one Connection server and one Security server, both running Windows 2012 R2 Server OS. We also own a wildcard certificate covering our public domain, say domain.org.

Read the rest of this entry »

Managing multiple FreeBSD servers

November 17th, 2014

If you run multiple installations of FreeBSD sooner or later you will face with the issue of how to update them all in the most efficient and centralized way. Building kernel/world for a FreeBSD server with one CPU and couple of GB of RAM will take hours to complete. Fortunately, there is a way to optimize it.

Read the rest of this entry »

http to https redirect under F5

October 13th, 2014

Say you have a virtual web server, serving domain1.org, configured on F5 with the following iRule redirecting plain http to https:

  1. when HTTP_REQUEST {
  2.  
  3.  if { [HTTP::host] equals "domain1.org" } {
  4.   HTTP::redirect "https://[HTTP::host][HTTP::uri]"
  5.  }
  6. }

You decided to buy another domain (domain2.org) and want to point it to the same IP where domain1.org is hosted, with the redirect to https://domain1.org. Here is how the modified iRule will look like:

  1. when HTTP_REQUEST {
  2.  
  3.  if { [string tolower [HTTP::host]] ends_with ".domain2.org" } {
  4.   HTTP::redirect "https://domain1.org[HTTP::uri]"
  5.  } elseif { [HTTP::host] equals "domain1.org" } {
  6.   HTTP::redirect "https://[HTTP::host][HTTP::uri]"
  7.  }
  8. }

Here is an alternative way to implement redirection by using HTTP Class profile.

Under Local Traffic › Profiles › Protocol › HTTP Class create a new profile:

  1. Name: HTTP2HTTPS
  2. Parent Profile: httpclass
  3.  
  4. Hosts: Match all
  5. URI Paths: Match all
  6. Headers: Match all
  7. Hosts: Match all
  8. Cookies: Match all
  9.  
  10. Send To: Redirect to…
  11. Redirect to Location: https://[getfield [HTTP::host] ":" 1][HTTP::uri]

For domain2.org redirection, modify it and change Redirect to Location to:

  1. Send To: Redirect to…
  2. Redirect to Location: https://domain1.org/[HTTP::uri]

User based access control for Skype with Dante and FreeBSD 10

October 4th, 2014

[20170903/ Yet another one. Since I rarely (read never) use Skype to call, I couldn’t spot it immediately, but all calls were actually dropped after 10 seconds. To fix it, you’ll need to open 3478/udp from your LAN to the outside world. More details are here. Confirmed to work with 7.39.32.102./20170903]

[20170512/ Another update. It actually came out that the latest Skype (confirmed with 7.36.0.101) does support socks, however there is an issue with the password length used to authenticate to the socks instance. Anything greater than 5 (five) characters fails. I don’t know whether it’s done deliberately, or there is some bug that’s never going to be fixed, but anyway, the workaround is either to use some generic account with the five-or-less-characters password, or disable socks authentication altogether./20170512]

[20170312/ Somewhere around first week of March 2017 (or end of February), Skype started dropping connections from versions 7.1.32.xx and below. When you try to log in, you’ll be presented with the message about outdated version. I’m not aware about any versions after 7.1.32 that support Socks. Despite of numerous bugs opened Socks functionality was never fixed, therefore the content of this article is no longer valid, and you won’t be able to use Skype with Socks. The configuration option is still there, but no connection attempts to the Socks server are made. Perhaps this is how they promote the usage of Skype Business Server./20170312]

Today we’re going to configure Dante running on FreeBSD 10.0-STABLE to allow Skype connectivity based on username/password stored in Active Directory. The version of Dante being used is 1.4.1 installed from ports and Active Directory is handled by Windows Server 2008 R2.

Note: as at the time of writing, the version of Dante available in FreeBSD ports collection is 1.4.0 and it’s marked as BROKEN because of the bug 192295. Use this patch to install 1.4.1.

Install required software:

  1. % cd /usr/ports/security/pam_ldap && make install clean
  2. % cd /usr/ports/net/dante && make install clean

Create /usr/local/etc/pam.d/sockd file:

  1. auth       required /usr/local/lib/pam_ldap.so
  2. account    required /usr/local/lib/pam_ldap.so
  3. password   required /usr/local/lib/pam_ldap.so

Create /usr/local/etc/ldap.conf file and fix permissions:

  1. host 10.9.128.1
  2. base OU=Users,DC=int,DC=domain,DC=org
  3. ldap_version 3
  4. binddn CN=socksd,OU=Users,DC=int,DC=domain,DC=org
  5. bindpw xxxxxxx
  6. pam_filter objectclass=user
  7. pam_login_attribute samaccountname
  1. % chmod 600 /usr/local/etc/ldap.conf

Adjust host, base, binddn and bindpw to reflect your environment.

Modify /usr/local/etc/sockd.conf file:

  1. logoutput: stdout /var/log/dante.log
  2. internal: 10.9.36.10 port = 1080
  3. external: 10.9.36.10
  4.  
  5. socksmethod: pam.username none
  6.  
  7. user.privileged: root
  8. user.unprivileged: nobody
  9. user.libwrap: nobody
  10.  
  11. client pass {
  12.         from: 10.9.128.0/24 port 1024-65535 to: 0.0.0.0/0
  13.         log: error connect disconnect
  14. }
  15.  
  16. socks pass {
  17.         from: 10.9.128.0/24 to: 0.0.0.0/0
  18.         command: connect udpassociate
  19.         socksmethod: pam.username
  20.         log: error connect disconnect iooperation
  21. }
  22.  
  23. socks pass {
  24.         from: 0.0.0.0/0 to: 10.9.128.0/24
  25.         command: udpreply
  26.         log: connect error
  27. }

Modify /etc/rc.conf to start Dante at boot:

  1. # enable dante
  2. sockd_enable="YES"

Configure Skype to use Socks with proxy authentication and check the logs of Dante:

  1. Oct  4 16:45:44 (1412433944.407661) sockd[482]: info: pass(1): tcp/accept [: 10.9.128.47.59574 10.9.36.10.1080
  2. Oct  4 16:45:44 (1412433944.525454) sockd[483]: info: pass(1): udp/udpassociate [: pam.username%usera@0.0.0.0.64526 10.9.36.10.60780
  3. Oct  4 16:45:49 (1412433949.196857) sockd[630]: info: pass(1): tcp/accept [: 10.9.128.47.59578 10.9.36.10.1080
  4. Oct  4 16:45:49 (1412433949.378804) sockd[499]: info: pass(1): tcp/connect [: pam.username%usera@10.9.128.47.59578 10.9.36.10.1080 -> 10.9.36.10.59578 64.4.23.143.33033
  5. Oct  4 16:45:49 (1412433949.380925) sockd[499]: info: pass(1): tcp/connect -: pam.username%usera@10.9.128.47.59578 10.9.36.10.1080 -> 10.9.36.10.59578 64.4.23.143.33033 (35)
  6. Oct  4 16:45:49 (1412433949.544730) sockd[499]: info: pass(1): tcp/connect -: 64.4.23.143.33033 10.9.36.10.59578 -> 10.9.36.10.1080 pam.username%usera@10.9.128.47.59578 (63)
  7. Oct  4 16:45:49 (1412433949.548340) sockd[499]: info: pass(1): tcp/connect -: pam.username%usera@10.9.128.47.59578 10.9.36.10.1080 -> 10.9.36.10.59578 64.4.23.143.33033 (40)
  8. Oct  4 16:45:49 (1412433949.706747) sockd[499]: info: pass(1): tcp/connect -: 64.4.23.143.33033 10.9.36.10.59578 -> 10.9.36.10.1080 pam.username%usera@10.9.128.47.59578 (3)
  9. Oct  4 16:45:49 (1412433949.962844) sockd[499]: info: pass(1): tcp/connect -: 64.4.23.143.33033 10.9.36.10.59578 -> 10.9.36.10.1080 pam.username%usera@10.9.128.47.59578 (144)

Since Dante is quite talkative make sure to rotate logs by editing /etc/newsyslog.conf file:

  1. /var/log/dante.log                      640  3     100  *     JB    /var/run/sockd.pid