Category Archives: Infrastructure

Unplugging an LVM partitioned USB drive

Recently I had the heartbreaking experience of having to reboot a Linux server. Normal usage should almost never require you to reboot the OS like you have to so frequently in Windows. In this case I had an external USB drive partitioned with LVM humming along on a Linux server. I needed to pull the drive, so like I’ve done with other drives I unmounted all partitions on the drive. Then proceeded to unplug it from the USB port. All well and good. But when I plugged it back in, the lvs command was showing error messages on the partitions and I was unable to mount them.

Some Google searches later I found that when it comes to LVM partitions the OS keeps references to it unless you explicitly tell it to unhook them. Only then can you tell the OS to hook the LVM partitions back up when you’ve plugged the drive back in. In my case I had to resort to rebooting the server in order for the OS to hook all the pieces together for the LVM partitions. Short of this I would have to manually delete certain files and move things around to get the LVM partitions to work again. So here are the magic incantations that will save you the headache.

Before you unplug an LVM partitioned USB drive, you must run the following commands:

lvchange -an /dev/your_volume_group_name
vgexport -a

Use the man command to explore what these commands do.

Now you should be able to unplug the drive. When you are ready to plug it back in, stick it back in the USB port and run the following commands:

vgimport -a
lvchange -ay /dev/your_volume_group_name

You should now be able to run lvs and see you LVM partitions on the USB drive without any errors and proceed to mount the partitions.

Hope you found this useful. Are there other or different ways of doing this? Please add your comments below and Happy Holidays!

Apache/Tomcat with Failover and Load Balancing in 20 minutes or less…

In order to get this done, you’ll need Apache, Tomcat, and the MOD_JK connector library. I’ve included the Windows binaries below, as I’m writing this blog post from a Windows machine. Feel free to swap out these downloads with whatever RPM, YAST, APT, etc. commands you want. Also, you’ll need to have a Java JDK installed, and your JAVA_HOME pointing to it. Make sure your path contains %JAVA_HOME%/bin.

Download and Apache from here:

Download and extract Tomcat from here:

Download MOD_JK from here:

  1. Install Apache.
  2. Copy the file to your apache/modules directory.
  3. Add the following to your apache/conf/httpd.conf

    LoadModule jk_module modules/
    JkWorkersFile          conf/
    JkLogFile              logs/jk.log
    JkLogLevel             debug
    JkMount                /*            router
    JkMount                /jk_status    status
  4. Create a file in your apache/conf directory. The file should contain the following:

  5. Extract the Tomcat installation ZIP archive to three different directories, as we’re going to load balance three instances of Tomcat. You’ll be replacing the server.xml file in each of the Tomcat conf/ directories with the following:

    <Server port="8100" shutdown="SHUTDOWN">
       <Resource name="UserDatabase" auth="Container"
          description="User database that can be updated and saved"
          pathname="conf/tomcat-users.xml" />
    <Service name="Catalina">
       <Connector port="8180" protocol="HTTP/1.1"
          redirectPort="8443" />
       <!-- Define an AJP 1.3 Connector -->
       <Connector port="8109" protocol="AJP/1.3" redirectPort="8443" />
       <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
          <Realm className="org.apache.catalina.realm.UserDatabaseRealm" 
          <Host name="localhost"  appBase="webapps"
             unpackWARs="true" autoDeploy="true"
             xmlValidation="false" xmlNamespaceAware="false">
             <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                <Manager className="org.apache.catalina.ha.session.DeltaManager"
                <Channel className="">
                   <Membership className="org.apache.catalina.tribes.membership.McastService"
                   <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                   <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                      <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
                   <Interceptor className=""/>
                   <Interceptor className=""/>
                <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
                <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
                <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
                <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>

    Looking into the XML above, there are three ports we’re concerned with: 8100, 8180, and 8109. These are the server, HTTP, and AJP13 ports, respectively. We’ll need each instance of Tomcat to run on it’s own ports. So, you can use this file as is in your first folder containing Tomcat, however, you’ll need to change the port numbers to: 8200, 8280, and 8209 for you 2nd installation. The third installation will use the ports, 8300, 8380, and 8309.

  6. Start (or Restart) Apache
  7. Start each instance of Tomcat (use the startup script in the Tomcat /bin directory) – you should see no errors.
  8. Verify each Tomcat is working by opening a browser window to each Tomcat instance – if you’ve followed my instructions, the links are: http://localhost:8180/examples/servlets/, http://localhost:8280/examples/servlets/, and http://localhost:8380/examples/servlets/.
  9. If Tomcat started correctly, start Apache. You should be able to access the Tomcat example pages via the following URL: http://localhost/examples/servlets/
  10. You’re done. Using my configuration, you can access a page to control the JK connector here: http://localhost/jk_status, I’d recommending hiding and protecting this should you want to put this configuration into production. Try experimenting with the configuration by stopping instance of Tomcat… as long as one instance of Tomcat is running, you should be able to see the examples.


GlusterFS Replication for Clustering

I recently was searching for a way to simulate shared physical storage in a VPS environment for clustering purposes.  In an enterprise data center we can expect some type of SAN available to provide shared physical storage.  GFS is a simple solution in this case to create a shared file system that can be used as a resource in a cluster.  GlusterFS allows us to provide this type of functionality to multiple nodes when we have no means of providing access to the same physical storage.

The gluster community site will be a great resource for anyone wanting to implement the file system and is located at

For the remainder of this post I will be referring to an environment consisting of two CentOS VPS nodes.

Preparing Ext3 File System for Sharing

Gluster will not share raw devices but instead will use an already mounted file system.  I will be assuming the use of a complete ext3 file system on the mount point /replicator.  If you can’t provide a unique storage device for this purpose you can just use a directory on the root file system for testing.

Installing GlusterFS Server and Client

The following commands need to be executed on each node to grab and install the necessary RPMs.

wget -r -l 1
rpm -Uvh glusterfs-*-3.0.0-1.x86_64.rpm

Execute the following on either node to generate the necessary configuration files in the current working directory.  This will create a client configuration along the lines of replicator-tcp.vol.  A server configuration file will be created for each node and begin with the appropriate node hostname.

glusterfs-volgen --name replicator --raid 1 node1:/replicator node2:/replicator

Move the client file to /etc/glusterfs/glusterfs.vol on each node.  Also move the appropriate server file to /etc/glusterfs/glusterfsd.vol for each node.

Mounting GlusterFS Volumes

The simplest way to configure mounting of the volumes is via /etc/fstab.  Place a line in fstab on each node.

/etc/glusterfs/glusterfs.vol    /data   glusterfs   defaults  0 0

This will mount the shared volumes to /data.  Try writing a file to one node and watch it appear on the other!

cd /data
dd if=/dev/zero of=/data/test bs=1M count=32

High Availability Implications

At this point I am still vetting gluster’s reliability as a HA solution.  It will most definitely keep data intact during planned maintenance.  If we properly stop the client/server on any node then changes can continue to occur on the other.  Also we can join a node to active shared storage and synchronization is automatic.

The real test is whether gluster will hold up in the not so routine situations.  Some crude tests involving yanking network connectivity from a node that is replicating changes seems to cause some issues.  For example, if I start the dd operation above on node1 and kill the connection to node2, one way or another, before it finishes then node1 still completes the operation fine.  When I reattach node2 even the active mount on /data seems to synchronize with node1 just fine.  Where some differences start to appear is in the /replicator directory on node2.  It seems that this does get out of whack and neither client pays attention to this server any longer.

If gluster can hold up to software and hardware failures without data corruption it can certainly be used as shared storage for clustering.  I’ll continue to explore these options and report back later.

Online malware scanner

F-secure has the best antivirus/anti-malware software available. I have been using it since rootkits became a threat. F-Secure had a free app that would detect and remove rootkits which is a step above technet’s Rootkit Revealer. F-Secure licensing is inexpensive and user friendly. They also have an online scanner that detects and removes threats and also will send samples back to the vendor.

You can find the online scanner here

Spring LDAP Group Authorization Tip

The folks at Spring have made it extremely easy to allow your application authenticate and authorize users with Spring LDAP. This blog entry explains how to check your directory structure and use some sparsely documented Spring LDAP parameters ({0} and {1}) to get everything working.

In your Spring Security configuration, pointing to your directory is straight forward:

 <ldap-server id="ldapServer" url="ldap://" />

But in configuring the ldap-authentication-provider, you need to know a few things about your directory of course! We recommend using Apache Directory Studio to browse your directory – it’s a fantastic application.

If you’re more of a command-line person, just use ldapsearch (example below):

ldapsearch -H ldap:// -ZZ -x 
-D "cn=AdminUser,dc=yourdomain,dc=com" -W -b "cn=users,ou=groups,dc=yourdomain,dc=com" 
-s base -a always "(objectClass=*)" "*"

Once connected to your directory, you’ll need to figure out how your groups are configured. Specifically, you’ll want to know if your configuration looks like

Example A:

  • dc=yourdomain,dc=com
    • ou=groups
      • cn=users
        • memberUid=USERNAME

or Example B:

  • dc=yourdomain,dc=com
    • ou=groups
      • cn=users
        • memberUid=uid= USERNAME,ou=people,dc= yourdomain,dc=com

If it’s like Example A, you’ll want your config like this:

<ldap-authentication-provider server-ref="ldapServer"  
	role-prefix="ROLE_" />

otherwise, you’ll want this config:

<ldap-authentication-provider server-ref="ldapServer"  
	role-prefix="ROLE_" />

Note the difference in the group-search-filter:

  • {0} contains the username with the entire ldap base.
  • {1} only contains username.

Using Conga Web Configuration with Red Hat Cluster Suite – Part 1


Red Hat Cluster Suite provides high availability and clustered storage for RHEL platforms.  Unfortunately the configuration management for each node can be tedious as the /etc/cluster/cluster.conf file must be copied to each as changes are made.  Conga makes life a little simpler.

Conga provides a client/server architecture for cluster management with the ricci and luci services.  Luci acts as the configuration interface and sends instructions the the ricci client on each server.  Ricci takes instructions from luci and updates cluster.conf as necessary.

Package Installation and Configuration

Install the Cluster Suite with the following package group installation commands:

yum groupinstall "Clustering"
yum groupinstall "Cluster Storage"

Once installed luci must be initalized on one node.  The initialization script will ask for a password for the default admin account.

luci_admin init
/etc/init.d/luci restart

Next start ricci on all nodes that will be joined to the cluster.

/etc/init.d/ricci start

Log In to the Web Interface

Use the url provided by the luci_admin script to login to the web interface.

Luci Login

Luci Login

Coming in Part 2

In the next post I’ll go over the basics of initializing a cluster via Conga.  Seeing as luci and ricci do occasionally get in a fight I will provide some tips on dealing with these situations also.

Open Source Router, Proprietary Cake

Keeping with SAI’s proclivity toward open source software, I present to you Vyatta.  Vyatta is a small company with the goal of taking down Cisco by offering an open source router that can run on standard x86 hardware.  With the prevalence of virtualization, one could realistically open a branch office using just a single x86 server with a T1 card from Vyatta.  The router, firewall, and VPN are covered by Vyatta and the apps could run in a virtualized OS.

Better yet is their current sales promotion.  If Cisco’s gross profit margin is 70%, Vyatta will give you a 30% discount.  As Cisco makes less money, Vyatta gets cheaper.

Lastly, proprietary cake tastes good.  I can prove it, too.

Mark is really excited with the Router Cake at his wedding

Mark is really excited with the Router Cake at his wedding

Open Source Enterprise Search

Has locating information across a multitude of systems on your corporate network finally made you consider an enterprise search appliance?

Our company has a number of systems in place designed to capture corporate knowledge and subject matter expertise.  Once it became too time consuming to find information across these systems (and we struck out with demos of search appliances like SearchBlox), we purchased the entry level Google Mini.  We’ve been happy with the appliance, but wanted to search more information formats (beyond digest authenticated SSL web pages and SMB shares), authenticate to our central authentication system (not LDAP), and introduce additional security levels.  (Word stemming would be nice too!)  To avoid the costs of graduating to Google Search Appliances, (and creating an internal Source Allies project to front end the Google Mini XML responses with some custom XSLT), we looked to open source again.

The trend towards enterprise search consolidation (Autonomy acquiring Interwoven for $775M, Microsoft offering $1.2B for FAST) has been interrupted by strong open source Lucene-based products like Nutch and Solr.  They have broken the enterprise search market segment wide open again.  Nutch provides basic web & file system crawling search appliance functionality and Solr gives us the ability to infused structured data into the same underlying Lucene index.

Lucene Logo Apache SolrApache Nutch

We decided to implement these technologies into our company network.  In our environment, the Nutch and Solr indexes are updated on a regular basis.  We use Nutch to index unstructured data such as our intranet, wikis, blogs and subversion document repositories.  Solr indexes structured data such as our corporate CRM application – a SugarCRM instance.  (Incidentally, we use a separate product called OpenGROK to index our subversion source code repositories).  Because both Nutch and Solr are both open source, it was very simple tie them into our single-sign-on system (front-ending them with our CAS server).  — Stay tuned for a follow-up blog highlighting the technical details of our configuration.

Ultimately, Nutch and Solr are going to provide our company with a more flexible enterprise search solution, but the solution is not without its fair share of Lucene/Nutch/Solr expertise to make it all happen.  Now that we have commodity cloud computing, Hadoop Map/Reduce, structured and unstructured indexing tools on top of Lucene, I’m anxious to see what the open source community will do next in the enterprise search space.   It doesn’t seem to far off to have an appliance that will do the normal Nutch/Goole Mini web and SMB share crawling, but also actively update the index with corporate collaboration (shared email/group chat/social media/RSS/wave protocol/video transcribing/forums/KM systems/custom SQL queries).  Of course all of this is currently possible with Solr/Nutch and even Google Mini’s OneBox modules, but who will be the first to make it really easy to setup?

Internet Routing Tables Reaches 300,000

A few weeks ago the global routing table reached its 300,000th route.  Below is a graph showing the exponential growth over the last 15 years.

BGP Table (Yearly)

BGP Table (Yearly)

Let’s do a little math:

Assuming all 4.3 billion IPv4 addresses are used (which isn’t quite true), each route represents approximately 4,294,967,296 / 300,000 = 14317 addresses.   This is almost equivalent to a /18 (16,384 addresses).  However, there are only 2^18 = 262,144 subnets of this size.

Why are there so many routes in the table?

Because there are BGP Administrators who advertise junk like this.  AT&T WorldNet Services is advertising over 1100 prefixes; most of them are /24s.  Due to their lack of summarization, this one group of routers is responsible for almost .5% of the fluctuation in global routing tables during any given week.  That’s really bad.

Poke around here for some more info on BGP and the global routing table.