Tomcat and Apache Setup
My example at bottom…
Most Tomcat configurations are a Apache/Tomcat setup, Apache serving up the static content and then passing any JSP to Tomcat to process. Tomcat can be integrated with Apache by using the JK Connector. The JK Connector uses the Apache JSserv Protocol (AJP) for communications between Tomcat and Apache.
The AJP Connector
The AJP protocol is used for communication between Tomcat and Apache, the software modules used on Apache are mod_jk or mod_proxy. Both are native code extension modules written in C/C++, on the Tomcat side the software module is the AJP Connector written in Java.
The below diagram shows how the native code Apache module (mod_jk or mod_proxy) works with Tomcat. Apache will receive the incoming JSP or servlet request and using the Apache module will pass this request via the AJP protocol to Tomcat, the response will also be sent back to the Apache server via the AJP protocol.
The Apache JServ Protocol (AJP) uses a binary format for transmitting data between the Web server and Tomcat, a network socket is used for all communication. The AJP packet consist of a packet header and a payload, below is the structure of the packet
As you can see, the binary packet starts with the sequence 0X1234, this is followed by the packet size (2 bytes) and then the actual payload. On the return path the packets are prefixed by AB (the ASCII codes for A and B), the size of the packet and then the payload.
The major feature of this protocol are
- Good performance on fast networks
- Support for SSL, encryption and client certificate
- Support of clustering by forwarding requests to multiple Tomcat 6 servers
One of the ways the AJP protocol reduces latency is by making the Web server reuse already open TCP-level connections with Tomcat. This saves the overhead of opening a new socket connections for each request, its a bit like a connection pool.
The configuration of a AJP Connector is below
| AJP Connector example | <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> |
A worker represents a running instance of Tomcat, a worker serves the requests for all dynamic web components. However you can run multiple instances of Tomcat in a cluster to implement load balancing or site partitioning. Each worker is identified by a unique hostname or a unique IP address and port number. You may what to implement multiple workers for the following reasons
- When you want different Web application contexts to be served by different Tomcat workers
- When you want different virtual hosts to be served by different Tomcat workers
- When you want to service more requests than the capacity of a single physical server
To let Apache know where the Tomcat servers are a file called workers.properties is created detailing this information. I describe this file next
|
Worker List
|
|
| Attribute | Description |
| work.list |
Describe the workers that are available to Apache via a list
|
|
Worker Types
|
|
| Attribute | Description |
| ajp13 |
This type of worker represents a running Tomcat instance
|
| lb |
used for load balancing
|
| status |
display useful information about how the load among the various Tomcat workers is distributed
|
| jni |
Used in process, this worker handles the forwarding of requests to in-process Tomcat workers using JNDI
|
| ajp12 |
worker that support the AJP 1.2 protocol
|
|
Other Worker Properties
|
|
| Attribute | Description |
| worker.test1.type | Describes the type of worker (see above for types) |
| worker.test1.host |
The host where the worker Tomcat instance resides
|
| worker.test1.port |
The port the AJP 1.3 Connector Tomcat instance is listening on (default 8009)
|
| worker.test1.connection_pool_size |
The number of connections used for this worker to be kept in a connection pool
|
| worker.test1.connection_pool_minsize |
The minimum number of connections kept in a connection pool
|
| worker.test1.connection_pool_timeout |
The number of seconds that connections to this worker should be left in the connection before expiry |
| worker.test1.mount | The contexts paths that are serviced by the worker, you can also use the JkMount directive in the http.conf file |
| worker.test1.retries |
Controls the number of times mod_jk will retry when a worker returns a error
|
| worker.test1.socket_timeout |
controls how long a worker will wait for a response on a socket before indicating an error
|
| worker.test1.socket_keepalive | indicates if the connection to the worker should be subject to keep alive |
| worker.test1.lbfactor |
An integer indicating the local-balance factor used by the load balancer to distribute work between multiple instances of Tomcat.
|
|
Worker Loading Balancing Properties
|
|
| Attribute | Description |
| worker.bal1.balance_workers | A list of workers to load balance between |
| worker.bal1.lock | The type of locking used O (Optimistic) or P (Pessimistic) |
| worker.bal1.method |
can be set to R (Requests), T (Traffic), B (Busy-ness)
R = The worker to use is based on the number of requests forwarded |
| worker.bal1.secret | Sets a default secret password for all workers |
| worker.bal1.sticky_session |
Tells the mod_jk to respect the sessionID in the request and ensures that the same session is always serviced by the same worker instance.
|
| worker.bal1.sticky_session_force | This is used for failover |
|
Example
|
|
| Simple example | worker.list = worker1 worker.worker1.type = ajp13 worker.worker1.host = 192.168.0.1 worker.worker1.port = 9009 worker.worker1.connection_pool_size = 5 worker.worker1.connection_pool_timeout = 300 |
| Load Balancing example |
worker.list = loadbal1,stat1 Note: if one of your servers is a slow server then lower the lbfactor of that server |
There are a number of Apache directives that you can configure in the httpd.conf file
|
Apache mod_jk Directives
|
|||||||||||||||||||||
| Directive | Description | ||||||||||||||||||||
| JkWorkerFile | tells mod_jk where to find the workers property file | ||||||||||||||||||||
| JkLogFile | tells mod_jk where to write its logs | ||||||||||||||||||||
| JkLogLevel | sets the level of logging (info, error or debug) | ||||||||||||||||||||
| JkRequestLogFormat | specifies the log format, below are the options that you can use
|
||||||||||||||||||||
| JkMount | control the URL matching and forwarding to the Tomcat workers | ||||||||||||||||||||
|
Example
|
|||||||||||||||||||||
| JkWorkerFile | JkWorkerFile conf/worker.properties | ||||||||||||||||||||
| JkLogFile | JkLogFile /var/logs/httpd/mod_jk.log | ||||||||||||||||||||
| JkLogLevel | JkLogLevel debug | ||||||||||||||||||||
| JkRequestLogFormat | JkRequestLogFormat "%w %U %T" | ||||||||||||||||||||
| JkMount | JkMount /examples/jsp/* worker1 | ||||||||||||||||||||
SSL provides a secure connection between Tomcat and Apache, the steps involved in getting this working are
- Install OpenSSL on your server
- Check that Apache has mod_ssl support
- Get or generate a SSL certificate and install it into Apache
- Test the SSL-enabled Apache-Tomcat setup
To make sure that you have openssl installed and the mod_ssl modules installed in Apache run the following
| Check foe OpenSSL | # openssl version |
| Check for Apache module mod_ssl | # <apache path>/httpd -D DUMP_MODULES |
If any of these are not installed then I recommend you download the latest version and install as per the Installation guides.
There are a number of steps to generate a test certificate using OpenSSL
- Create the configuration file for generating the certificate
- Create a certificate signing request, this is what you send to the CA if you are buying a certificate
- Remove the passphrase from the private key
- Purchase a certificate from a CA or create a self-signed certificate
- Install the key and certificate to the Apache server
Below are the steps to creating your own cert
| step 1 |
Create a working directory called certs
RANDFILE = ./random.txt Create a random file called random.txt put a large number in it |
| step 2 |
Now create the certificate Two files should have been created server.csr and keyfile.pem |
| step 3 | Now remove the passphrase from the private key # openssl rsa -in keyfile.pem -out server.key |
| step 4 |
Now create a self-signed certificate # openssl X509 -in server.csr -out server.crt -req -signkey server.key -days 365 Note: in a production environment the certificate signing request file generated (server.csr) is sent to a Certificate Authority and a certificated purchased |
| step 5 | Last but not least copy the server.key and server.crt in to the Apache conf directory |
To setup the mod_ssl in Apache you need to perform the following in Apache httpd.conf file
| include the httpd-ssl.conf | include conf/extra/httpd-ssl.conf |
| Load the SSL modules | LoadModule ssl_module modules/mod_ssl.so |
| SSLCertificateKeyFile | set this attribute to the path to the server.key file |
| SSLCertificateFile | set this attribute to the path to the server.crt file |
Once all the above is completed you can now point your browser to the Apache server, hopefully the browser will pop up with a security alert (because of the self-signed certificate).
The only change to make the Apache-Tomcat setup is to change the <VirtualHost> attribute
| <VirtualHost> | <virtualHost _default_:443> …. JkWorkersFile …… JkMount …….. </VirtualHost> |
I will be discussing Tomcat clustering in a later topic will describes a more detailed viewing of load balancing and persistent sessions with-in-memory session replication but for this section I will discuss a basic load balancing solution.
The mod_proxy module can also be used for load balancing but will not be discussed here, the mod_jk module sup[ports load balancing with seamless sessions, it uses a simple round-robin algorithm. Each Tomcat worker is weighted in the worker.properties file which specifies how the request load is distributed between workers.
A seamless session is also known as session affinity or a sticky session. When a request is made any of the Tomcat instances is used, but any subsequent request will be routed to the same Tomcat instance. to keep the same user session.
The following steps are required to set up load balancing in Tomcat
- Change the CATALINA_HOME in the Tomcat startup files to point to different locations for each of the Tomcat instances
- Set different AJP Connector ports for the instances
- Disable the Coyote HTTP/1.1 Connector
- set the jvmroute in the Standalone Engine
- Configure the Tomcat worker in the workers.properties file.
One assumption I will be making here is that all the Tomcat instances will be running on the same server
The first step is to change the CATALINA_HOME variable in each of the startup.bat (Windows) or startup.sh (Unix) instances
| CATALINA_HOME |
set CATALINA_HOME=c:\apps\tomcatA Note: the other Tomcat instances would be tomcatB and tomcatC |
Now in each Tomcat instance we must set a different AJP Connector port number (server.xml)
| AJP Connector port |
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> Note: on the other Tomcat instances use ports 8010 and 8011 |
To avoid startup/shutdown port conflicts we must change each Tomcats worker server port (server.xml)
| server port |
<Server port="8005" shutdown="SHUTDOWN" debug="0"> Note: on the other Tomcat instances use ports 8006 and 8007 |
Because all the Tomcat instances will be running in conjunction with the load-balancer worker, it's possible that someone could directly access any of the available workers vioa the default HTTP Connector, by passing the load-balancer path. To avoid this comment out the HTTP Connector configuration of all the Tomcat instances (server.xml)
| disable HTTP Connector |
<!– Define |
An important step for load balancing is specifying the jvmRoute. The jvmRoute is an attribute of Engine directive that acts as an identifier for that particular Tomcat worker. The attribute must be unique across all Tomcat instances, this unique ID is used in the workers.properties file for identifying each Tomcat worker (server.xml).
| jvmRoute |
<Engine name="Standalone" defaultHost="localhost" jvmRoute="tomcatA" > Note: the other Tomcat instances would be tomcatB and tomcatC |
You will also need to comment out the Catalina Engine directive (server.xml)
| Catalina Engine disable |
<!– Define Note: the other Tomcat instances would be tomcatB and tomcatC |
In Apache's httpd.,conf file you need to add some load balancing directives, also make sure you have the module mod_jk loaded
| httpd.conf directives |
JkWorkersFile conf/worker.properties |
The last thing is to create the workers property file, i have already discuss this file above.
| worker.properties file |
worker.list = loadbal1,stat1 |
To test the load balancer and sticky sessions use the below JSP page (one for each instance), just place it in the webapps/examples/jsp directory.
| jsp test page |
<%@ page language="java" %> |
Use the below URL's for testing, etc. Don't forget to play around with the lbfactor on each Tomcat instance to see what affect it has.
| URL's |
http://local/examples/jsp/index.jsp |
==============================================
Be sure to update sample.com with new loadbalancer modJK
Don't forget to set your jvmRoute on each instance of TomCat within the server.xml
<Engine name="Catalina" defaultHost="localhost">–>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tc8xx">
==============================================
My Example located: vi /etc/libapache2-mod-jk/workers.properties
==============================================
#
# The workers that your plugins should create and work with
#
#worker.list=loadbalancer,status
#worker.list=loadbal1,stat1
worker.list=lb_tc8xx,lb_tc81x,stat1
#
#—— ajp13_worker WORKER DEFINITION ——————————
#———————————————————————
#
#
# Defining a worker named ajp13_worker and of type ajp13
# Note that the name and the type do not have to match.
#
worker.tc811.host=192.168.1.81
worker.tc811.port=8113
worker.tc811.type=ajp13
worker.tc812.host=192.168.1.81
worker.tc812.port=8123
worker.tc812.type=ajp13
worker.tc821.host=192.168.1.82
worker.tc821.port=8213
worker.tc821.type=ajp13
worker.tc822.host=192.168.1.82
worker.tc822.port=8223
worker.tc822.type=ajp13
#
# Specifies the load balance factor when used with
# a load balancing worker.
# Note:
# —-> lbfactor must be > 0
# —-> Low lbfactor means less work done by the worker.
worker.tc811.lbfactor=1
worker.tc812.lbfactor=1
worker.tc821.lbfactor=1
worker.tc822.lbfactor=1
#
# Specify the size of the open connection cache.
#worker.ajp13_worker.cachesize
#
#—— DEFAULT LOAD BALANCER WORKER DEFINITION ———————-
#———————————————————————
#
#
# The loadbalancer (type lb) workers perform wighted round-robin
# load balancing with sticky sessions.
# Note:
# —-> If a worker dies, the load balancer will check its state
# once in a while. Until then all work is redirected to peer
# workers.
# workers.
#Method — can be set to R (Requests), T (Traffic), B (Busy-ness)
# R = The worker to use is based on the number of requests forwarded
# T = The worker to use is based on the traffic that had been sent to the workers
# B = The worker to use is based on the load dividing the number of concurrent requests by the load factor
#setup the load-balancer
worker.lb_tc8xx.type=lb
worker.lb_tc8xx.method=R
worker.lb_tc8xx.balance_workers=tc811,tc812,tc821,tc822
worker.lb_tc8xx.sticky_session=True
###(This will generate a 500 instead of rerouting to a new server)
#worker.loadbalancer.sticky_session_force=True
worker.lb_tc81x.type=lb
worker.lb_tc81x.method=R
worker.lb_tc81x.balance_workers=tc811,tc812
worker.lb_tc81x.sticky_session=True
###(This will generate a 500 instead of rerouting to a new server)
#worker.loadbalancer.sticky_session_force=True
# Status worker for managing load balancer
worker.stat1.type=status