How to accomplish load balancing?
Load balancing can be accomplished through software but is usually achieved with the help of a dedicated hardware appliance due to the speed requirements. A load balancer can use many different algorithms. The most basic method is round-robin, which sends a request to each server in the cluster successively. More sophisticated techniques make decisions based on CPU usage, number of requests queued, average response time or even number of lost packets. One problem with load balancing is that a user’s session is likely not replicated across the the whole server cluster so typically a single user’s requests will be “sticky” and always be routed to the same backend server.

Another, more global, method of load balancing is Anycast. Anycast is an addressing strategy often used in DNS where hosts in multiple geographical locations represent the same IP address. This balances workload because any request is routed to the closest host (as determined by the routing protocol). In the event of one of the hosts going offline the request is then routed to the next closest available host. Routing requests to the nearest host has the added benefits of increasing response time and reliability.

Why load balance?
Simply put, work gets done more efficiently when resources are not used to their capacity and as a result user response times are generally better. The best reason, however, is often times failover. Failover refers to the ability of a system to remain operational while one or more components have failed or gone down. Say your clumsy friend trips over a power cord, eliminating one of three nodes in a cluster hosting a website. People opening a new connection to your website will not notice because the load balancer will realize that the one node is not responding and will route all requests to the other two nodes until the first one is responsive again. The only users who might notice would have been sticky to that one node. Failover is also useful for maintenance. If something has to be upgraded across all nodes, then one node can be taken down at a time to be upgraded without any downtime for whatever service the cluster may be hosting.

Load balancing and failover are powerful concepts which allow network administrators to have some peace of mind and users to keep on working.

This blog post lets you manage all of your email in OCD detail, with just (2) keyboard shortcuts: u and s. Explaining how, requires a quick overview of Zimbra’s features:

Overview

Zimbra Filters: The great part of Zimbra Filters is that they may be run on the server. (After spending a couple of years using Apple Mail and MailTags, this feature means that my iPhone, Zimbra Desktop, and Zimbra Web Client are all finally synchronized).

My favorite new feature of Zimbra version 6.0 is the ability to run filters on email that is already in your inbox.

Zimbra Tags: We’re going for simplicity, so I recommend only using a single tag called: Unread (but Filed).

Zimbra Saved Searches:: You can make a complex query and return to it’s search results at any time. Saved searches tie into this blog post as they allow us to view all of our email tagged with “Unread (but Filed)”.

With that overview of a few Zimbra features, here are the 5 simple Setup Steps to managing your email with just two keyboard shortcuts:

1. Create a Zimbra Tag called “Unread (but Filed)”

• Press ‘t’
• Press the ‘New’ button
• Type ‘Unread (but Filed)’
• Pick a color and press ‘OK’

2. Create a Zimbra Saved Search

• Type into the search box ‘in:inbox OR tag:”Unread (but filed)”‘
• Press ‘Save’
• Type ‘Inbox OR Unread (but filed)’
• Press ‘OK’

3. Start creating Zimbra Filters for emails as you receive them

• Right click on the email you want to filter
• Click ‘New Filter’
• On Perform the following actions, use these actions:
• Tag with: ‘Unread (but Filed)’
• File into Folder: YourFolderName

4. Set your Default Mail Search

• 4. Goto Preferences –> Mail
• Set your Default Mail Search to ‘in:inbox OR tag:”Unread (but filed)”‘
• This will make your normal “Inbox” view goto this search even on the Mobile Interface!

5. You’re done! Now just use your keyboard shortcuts!

• Gradually create filters for your email as it arrives (run the filters against your inbox for immediate feedback)
• When new email arrives, it will be automatically filed away, but will still appear in your “inbox” until you remove your custom tag.
• Simply press ’s’ to select a saved search ‘Unread (but Filed)’
• Tip: If you’re using the web interface, simply press ‘=’ as this will refresh your Default Mail Search
• Simply press ‘u’ to remove the tag, when you want to remove the email from your inbox (because it’s already been filed away for you!)

• Linux server with NFS (or compatible)
• TFTP server
• DHCP server
• syslinux / pxelinux files

To simplify these instructions we are going to make the following assumptions.

• DHCP server is 10.0.0.2
• TFTP server is 10.0.0.3
• NFS is a Ubuntu server at 10.0.0.4

In reality it’s likely your TFTP and NFS server are going to be the same server, however because we go by IP in this, it is hopefully easier to understand.

Basic working of PXE

When a client attempts to boot by PXE, it makes a DHCP request. The DHCP server responds with an IP address and the IP of the TFTP server as well as a file to request.

The PXE client connects to the given IP and requests the file (pxelinux)

The system then looks for a configuration file.

Setting up NFS

Installing NFS

 sudo apt-get install nfs-kernel-server nfs-common portmap

Setting up NFS for Ubuntu Live Network Boot

In this we will assume that we are going to mount the NFS at /srv/nfs/ubuntu.
Download Ubuntu ISO
Mount the ISO:

 sudo mount -o loop ubuntu-x,xx-desktop.iso /mnt

Copy the contents of the CD:

 sudo cp -a /mnt /srv/nfs/ubuntu

Edit /etc/exports:

 /srv/nfs/ubuntu 10.0.0.0/255.255.255.0(async,no_root_squash,no_subtree_check,ro)

Restart NFS

 sudo /etc/init.d/nfs-kernel-server restart

Setting up TFTP

There are windows TFTP servers, but for this we are going to use the Ubuntu box.

To Install:

 sudo apt-get install xinetd tftpd tftp

Configuration:
Edit/Create /etc/xinetd.d/tftp to:

service tftp{
 protocol = udp
 port = 69
 socket_type = dgram
 wait = yes
 user = nobody
 server = /usr/sbin/in.tftpd
 server_args = /tftpboot
 disable = no
}

Start TFTP

 sudo /etc/init.d/xinetd start

Setting up TFTP for PXE

Download PXELinux (preferably) or SysLinux.
Extract the contents into the /tftpboot boot directory
Copy Ubuntu’s Kernel & initrd.gz

mkdir /tftpboot/casper
sudo mount 10.0.0.4:/srv/nfs/ubuntu /mnt
sudo cp /mnt/casper/initrd.gz /tftpboot/casper
sudo cp /mnt/casper/vmlinuz /tftpboot/casper

Create the config directory:

mkdir /tftpboot/pxelinux.cfg

Create the default config for PXE

vi /tftpboot/pxelinux.cfg/default

with the contents:

default menu.c32
prompt 0
LABEL ^Boot Live Ubuntu
kernel casper/vmlinuz
append root=/dev/nfs boot=casper netboot=nfs nfsroot=10.0.0.4:/srv/nfs/ubuntu initrd=casper/initrd.gz quiet splash --

Setting up DHCP

Windows Server

Open the DHCP Manager, edit expand your scope
Select Scope Options
Add the following options:

067   Boot File Name                            pxelinux.0
066    Boot Server Hostname             10.0.0.3

Linux

Edit dhcpd.conf to include:

allow booting;
allow bootp;

Add the dhcp.conf group:

next-server 10.0.0.3;
filename "pxelinux.0";

Boot Ubuntu

With a bit of luck you can now boot any PXE compatable PC on the network into a network booted Ubuntu Live CD.

Step 1: Create the Service Manifest

<?xml version='1.0'?>;
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='mynewservices'>
        <service name='application/service1' type='service' version="1">
                <method_context>
                        <method_credential user='root' />
                        <method_environment>;
                                <envvar name="MY_VAR" value="--flag1 --flag2" />
                        </method_environment>
                </method_context>
 
                <exec_method name='start' type='method' exec='/opt/myscript.sh start' timeout_seconds='0' />
 
                <exec_method name='stop' type='method' exec='/opt/myscript.sh stop' timeout_seconds='120' />
        </service>
</service_bundle>

Service Bundle

A service bundle is just a collection of services which can be very useful if you have a suite of applications that you wish to manage within on manifest.xml file.

Service

The attributes you define on your service can be anything and are only useful for when you invoke administrative commands on them ( start, stop, etc ).

Method Context

The method context allows you to define the environment in which the executed methods will be called within. The example above shows the ability to define what user will be used to execute the method and the environment variables available to the script. This can be very handy when you want to leave an installation directory’s bin alone, and configure everything through the service. Keep in mind that what you see is a global method context and will apply to all executed methods. You may also customize the context per method by moving the tags out of service and into the exec_method tags.

Executable Method

Its purpose is self explanatory. Note that in one entry, the timeout is 0 and the other is 120 seconds. A timeout with a value of 0 will wait indefinitely for the script to start-up, however the service manager will always assume that it is online when in fact it could be hanging.

Also, there are three common commands used when invoking the service manager; enable, disable, and restart.

• When enabled, the start method will be executed.
• When disabled, the stop method will be executed.
• When restarted/refreshed, both the stop and the start method will be executed unless there is a method named restart/refresh.

Step 2: Import the Manifest

Once you have your manifest xml file completed and ready to import, all you need to do is run the following commands:

svccfg validate manifest.xml

That will run the configuration through validation and make sure you didn’t make any typo’s.

svcfg import manifest.xml

Step 3: Enable the Service

Now that your manifest has been imported, it is available to be enabled within the service admin.

svcadm enable service1

Step 4: Testing the Service

After the service has started the run, you need to make sure it is executing properly and can handle a machine restart and/or service restart.

svcs -x

That command will tell you which services are currently broken and the log files to view the errors. Make sure that your service is not in that list. You may also do the following:

svcs service1
svcs -x service1

to determine the state of that particular service. If it isn’t working, I suggest taking a look at the log files and fix it

Step 5: Updating the Service

Say that while testing, you noticed that you passed the wrong parameter to your shell script or maybe in the future you need to add additional environment variables to your service, its the same process. Even if you no longer have the manifest xml file, SMF will give you a copy when you ask for it so that you can modify it and re-import.

svccfg export myservices > manifest.xml

After you have made your changes, validate and import the xml file and restart your service.

perl -MCPAN -e "install HTML::Mason"

Generally the first thing I notice is that all (or a large number) of the tests are failing. For example here is the output of one of the tests:

t/20-plugins.......................Can't locate HTML/Entities.pm in @INC (@INC
 contains: /root/.cpan/build/HTML-Mason-1.42/blib/lib /root/.cpan/build/H
TML-Mason-1.42/blib/arch t/lib inc /usr/perl5/5.8.4/lib/i86pc-solaris-64int
 /usr/perl5/5.8.4/lib/i86pc-solaris-64int /usr/perl5/5.8.4/lib /usr/perl5/site_
perl/5.8.4/i86pc-solaris-64int /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int
 /usr/perl5/site_perl/5.8.4 /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int /u
sr/perl5/site_perl/5.8.4 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.8.4/i8
6pc-solaris-64int /usr/perl5/vendor_perl/5.8.4/i86pc-solaris-64int /usr/per
l5/vendor_perl/5.8.4 /usr/perl5/vendor_perl/5.8.4/i86pc-solaris-64int /usr
/perl5/vendor_perl/5.8.4 /usr/perl5/vendor_perl . /usr/perl5/5.8.4/lib/i86pc
-solaris-64int /usr/perl5/5.8.4/lib /usr/perl5/site_perl/5.8.4/i86pc-solaris-6
4int /usr/perl5/site_perl/5.8.4 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.
8.4/i86pc-solaris-64int /usr/perl5/vendor_perl/5.8.4 /usr/perl5/vendor_perl 
.) at /root/.cpan/build/HTML-Mason-1.42/blib/lib/HTML/Mason/Escapes.pm
 line 14.

Hmm, because we’re not installing HTML::Entities this means that it should have been brought in when CPAN.pm was building the dependencies. I trust CPAN.pm got this right, if for no other reason than that I remember it prompting to install that module. After digging around in the sea of output eventually I find that installing Digest::SHA1 failed because of this error:

cc -c   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_TS_ERRNO -xO3 -xspace -xildoff   -DVERSION=\"2.12\" -DXS_VERSION=\"2.12\" -KPIC "-I/usr/perl5/5.8.4/lib/i86pc-solaris-64int/CORE"   SHA1.c
cc: unrecognized option `-KPIC'
cc: language ildoff not recognized
cc: SHA1.c: linker input file unused because linking not done

Our C compiler isn’t happy with how it’s being called. Now we need to find out which compiler we’re using:

 bash-3.2$ which cc
/usr/gnu/bin/cc

While the GCC compiler is a fine fine compiler, the way CPAN.pm tries to build things on OpenSolaris involves params that only SunStudio understands. At this point there are two options. We can either patch the generated Makefile so that it doesn’t include stuff GCC can’t understand or we can switch the compiler to SunStudio. I prefer the latter because it is easiest:

pkg install sunstudio  # Install it if not already done
export PATH="/opt/SUNWspro/bin:$PATH" # Put it first on the path

Now that we’re using the correct compiler the build goes slightly less awful. We get a new error.

/usr/bin/perl "-Iblib/arch" "-Iblib/lib" Build.PL Build
Too early to specify a build action 'Build'.  Do 'Build Build' instead.

This is caused by the ExtUtils::MakeMaker module not being installed. I don’t know how my system got into such a state, but it is easy enough to fix:

perl -MCPAN -e "install 'ExtUtils::MakeMaker' "

Let’s try one more time:

perl -MCPAN -e "install HTML::Mason"
...Huge Snip...
 /usr/gnu/bin/make install  -- OK

Success!

“Port 8080 required by Tomcat v6.0 Server at localhost is already in use. The server may already be running in another process, or a system process may be using the port. To start this server you will need to stop the other process or change the port number(s).”

I have been seeing this error from Tomcat every time I reboot my Windows development box.

It is easy enough to track down the process holding onto the port:

netstat -a -o

Now I know that the process PLEX760… is listening to port 8080. It is trivial to kill the process using Task Manager now that I know the process ID (3832), but I really want to know what this thing is, and why it is getting started. This brings me to my favorite Windows utility program, Process Explorer.

Using Process Explorer I can see where the actual exe lives on the local disk:

Under the services tab I can see that this is associated with a Windows service:

Going to:

Control Panel -> Administrative Tools -> Services

Lets me stop the service, and set it to manual start-up.

This little example does not even begin to scratch the surface in what information Process Explorer can provide about currently running applications. I highly encourage you to try it out next time you are trying to nail down what is going on with  a windows application.

Some Google searches later I found that when it comes to LVM partitions the OS keeps references to it unless you explicitly tell it to unhook them. Only then can you tell the OS to hook the LVM partitions back up when you’ve plugged the drive back in. In my case I had to resort to rebooting the server in order for the OS to hook all the pieces together for the LVM partitions. Short of this I would have to manually delete certain files and move things around to get the LVM partitions to work again. So here are the magic incantations that will save you the headache.

Before you unplug an LVM partitioned USB drive, you must run the following commands:

#!/bin/bash
lvchange -an /dev/your_volume_group_name
vgexport -a

Use the man command to explore what these commands do.

Now you should be able to unplug the drive. When you are ready to plug it back in, stick it back in the USB port and run the following commands:

#!/bin/bash
vgimport -a
lvchange -ay /dev/your_volume_group_name

You should now be able to run lvs and see you LVM partitions on the USB drive without any errors and proceed to mount the partitions.

Hope you found this useful. Are there other or different ways of doing this? Please add your comments below and Happy Holidays!

Download and Apache from here:
http://apache.mirrors.tds.net/httpd/binaries/win32/apache_2.2.14-win32-x86-no_ssl.msi

Download and extract Tomcat from here:
http://mirrors.axint.net/apache/tomcat/tomcat-6/v6.0.20/bin/apache-tomcat-6.0.20.zip

Download MOD_JK from here:
http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/win32/jk-1.2.28/mod_jk-1.2.28-httpd-2.2.3.so

1. Install Apache.
2. Copy the mod_jk-1.2.28-httpd-2.2.3.so file to your apache/modules directory.
3. Add the following to your apache/conf/httpd.conf

   LoadModule jk_module modules/mod_jk-1.2.28-httpd-2.2.3.so
    
   JkWorkersFile          conf/workers.properties
   JkLogFile              logs/jk.log
   JkLogLevel             debug
    
   JkMount                /*            router
   JkMount                /jk_status    status

4. Create a workers.properties file in your apache/conf directory. The file should contain the following:

   worker.list=router,status
    
   worker.worker1.port=8109
   worker.worker1.host=localhost
   worker.worker1.type=ajp13
   worker.worker1.lbfactor=1
   worker.worker1.local_worker=1
   worker.worker1.sticky_session=0
    
   worker.worker2.port=8209
   worker.worker2.host=localhost
   worker.worker2.type=ajp13
   worker.worker2.lbfactor=1
   worker.worker2.local_worker=0
   worker.worker2.sticky_session=0
    
   worker.worker3.port=8309
   worker.worker3.host=localhost
   worker.worker3.type=ajp13
   worker.worker3.lbfactor=1
   worker.worker3.local_worker=0
   worker.worker3.sticky_session=0
    
   worker.router.type=lb
   worker.router.balanced_workers=worker1,worker2,worker3
   worker.router.local_worker_only=1
    
   worker.status.type=status

5. Extract the Tomcat installation ZIP archive to three different directories, as we’re going to load balance three instances of Tomcat. You’ll be replacing the server.xml file in each of the Tomcat conf/ directories with the following:

   <Server port="8100" shutdown="SHUTDOWN">
    
   <GlobalNamingResources>
      <Resource name="UserDatabase" auth="Container"
         type="org.apache.catalina.UserDatabase"
         description="User database that can be updated and saved"
         factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
         pathname="conf/tomcat-users.xml" />
   </GlobalNamingResources>
    
   <Service name="Catalina">
      <Connector port="8180" protocol="HTTP/1.1"
         connectionTimeout="20000"
         redirectPort="8443" />
    
      <!-- Define an AJP 1.3 Connector -->
      <Connector port="8109" protocol="AJP/1.3" redirectPort="8443" />
    
      <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
         <Realm className="org.apache.catalina.realm.UserDatabaseRealm" 
            resourceName="UserDatabase"/>
    
         <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
    
            <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
               channelSendOptions="8">
    
               <Manager className="org.apache.catalina.ha.session.DeltaManager"
                  expireSessionsOnShutdown="false"
                  notifyListenersOnReplication="true"/>
    
               <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                  <Membership className="org.apache.catalina.tribes.membership.McastService"
                     address="228.0.0.4"
                     port="45564"
                     frequency="500"
                     dropTime="3000"/>
    
                  <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                     address="auto"
                     port="4000"
                     autoBind="100"
                     selectorTimeout="5000"
                     maxThreads="6"/>
    
                  <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                     <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
                  </Sender>
    
                  <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
                  <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
               </Channel>
    
               <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
               <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
    
               <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                  tempDir="/tmp/war-temp/"
                  deployDir="/tmp/war-deploy/"
                  watchDir="/tmp/war-listen/"
                  watchEnabled="false"/>
    
               <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
               <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
            </Cluster>
         </Host>
      </Engine>
   </Service>
   </Server>

   Looking into the XML above, there are three ports we’re concerned with: 8100, 8180, and 8109. These are the server, HTTP, and AJP13 ports, respectively. We’ll need each instance of Tomcat to run on it’s own ports. So, you can use this file as is in your first folder containing Tomcat, however, you’ll need to change the port numbers to: 8200, 8280, and 8209 for you 2nd installation. The third installation will use the ports, 8300, 8380, and 8309.

6. Start (or Restart) Apache
7. Start each instance of Tomcat (use the startup script in the Tomcat /bin directory) – you should see no errors.
8. Verify each Tomcat is working by opening a browser window to each Tomcat instance – if you’ve followed my instructions, the links are: http://localhost:8180/examples/servlets/, http://localhost:8280/examples/servlets/, and http://localhost:8380/examples/servlets/.
9. If Tomcat started correctly, start Apache. You should be able to access the Tomcat example pages via the following URL: http://localhost/examples/servlets/
10. You’re done. Using my configuration, you can access a page to control the JK connector here: http://localhost/jk_status, I’d recommending hiding and protecting this should you want to put this configuration into production. Try experimenting with the configuration by stopping instance of Tomcat… as long as one instance of Tomcat is running, you should be able to see the examples.

-Jim

The gluster community site will be a great resource for anyone wanting to implement the file system and is located at http://www.gluster.org.

For the remainder of this post I will be referring to an environment consisting of two CentOS VPS nodes.

Preparing Ext3 File System for Sharing

Gluster will not share raw devices but instead will use an already mounted file system.  I will be assuming the use of a complete ext3 file system on the mount point /replicator.  If you can’t provide a unique storage device for this purpose you can just use a directory on the root file system for testing.

Installing GlusterFS Server and Client

The following commands need to be executed on each node to grab and install the necessary RPMs.

wget -r -l 1 http://ftp.gluster.com/pub/gluster/glusterfs/3.0/3.0.0/CentOS/
cd ftp.gluster.com/pub/gluster/glusterfs/3.0/3.0.0/CentOS/
rpm -Uvh glusterfs-*-3.0.0-1.x86_64.rpm

Execute the following on either node to generate the necessary configuration files in the current working directory.  This will create a client configuration along the lines of replicator-tcp.vol.  A server configuration file will be created for each node and begin with the appropriate node hostname.

glusterfs-volgen --name replicator --raid 1 node1:/replicator node2:/replicator

Move the client file to /etc/glusterfs/glusterfs.vol on each node.  Also move the appropriate server file to /etc/glusterfs/glusterfsd.vol for each node.

Mounting GlusterFS Volumes

The simplest way to configure mounting of the volumes is via /etc/fstab.  Place a line in fstab on each node.

/etc/glusterfs/glusterfs.vol    /data   glusterfs   defaults  0 0

This will mount the shared volumes to /data.  Try writing a file to one node and watch it appear on the other!

cd /data
dd if=/dev/zero of=/data/test bs=1M count=32

High Availability Implications

At this point I am still vetting gluster’s reliability as a HA solution.  It will most definitely keep data intact during planned maintenance.  If we properly stop the client/server on any node then changes can continue to occur on the other.  Also we can join a node to active shared storage and synchronization is automatic.

The real test is whether gluster will hold up in the not so routine situations.  Some crude tests involving yanking network connectivity from a node that is replicating changes seems to cause some issues.  For example, if I start the dd operation above on node1 and kill the connection to node2, one way or another, before it finishes then node1 still completes the operation fine.  When I reattach node2 even the active mount on /data seems to synchronize with node1 just fine.  Where some differences start to appear is in the /replicator directory on node2.  It seems that this does get out of whack and neither client pays attention to this server any longer.

If gluster can hold up to software and hardware failures without data corruption it can certainly be used as shared storage for clustering.  I’ll continue to explore these options and report back later.

You can find the online scanner here