I haven’t had much time to write much over the past few weeks as I’ve spent most of my time writing up https://jazz.net/wiki/bin/view/Main/DeployingCLMClusterWithWASND and a week with customers in Seoul. A bit cold in Seoul, especially going from a warm though wet Sydney, but I love the food in Seoul and managed to keep warm.
Working on the clustering stuff has meant I’ve done very little on the Jazz side and a lot ( and I mean a LOT:-) on the Websphere side. Which reminded me of a conversation I had with Alex (http://alexetestelle.almandra.com/) around simplicity. Alex and Estelle had just completed a cycling tour of New Zealand and were visiting us in Sydney on their way back to France. Life on the road was pretty simple for them: not much more than the clothes on their back, their bicycles and panniers, and an iPhone. Alex is also pretty handy with computers and after overhearing a long conversation I had about troubleshooting an RTC installation on WAS, he shook his head and said “Ziz ees too complex no?” (No, he doesn’t really speak like that but is my rendition of his Gallic accented English.) We went on to talk about whether the IT world is guilty of making things unnecessarily complicated, Websphere being a case in point. I’m all in favor of simplicity or at least a facade of simplicity, taking a cue from what I see in nature: simple and beautiful patterns everywhere hiding incredibly complex systems, which funnily enough, are usually the result of fairly simple interactions. After the last few weeks, I’m craving even more simplicity, though I’ve learnt a ton more about WAS.
As I went through multiple iterations of setting up a CLM cluster, I started to get a pretty severe case of RSI: I had a very slow connection to the Linux servers I was using and working the GUIs on them was (very literally) painful. I first tried X forwarding from the servers to the X server on my Windows laptop but that was excruciatingly sloow. Using VNC proved a better option. After about the second iteration when I got the whole thing working (that was pretty exciting in itself) I began collecting command line versions of various steps including installing WAS ND version 7, creating the profiles and using wsadmin to configure it. WAS is somewhat helpful with the ability to log command assistance commands. So the next iteration I happily copied and pasted away my collected commands into Putty sessions and probably got about 85% done in maybe 60-70% of the time.
I was feeling pretty pleased with myself and decided it was time to up the ante and try it all out again with the CLM M8 build and Websphere ND version 8. Of course, nothing worked! For a start the WAS 8 install/uninstall moved to using IBM Installation Manager rather than ISMP so my wonderful WAS 7 installation response files were all for naught. So the next iteration was really spent working out the WAS 8 versions of commands, though most of the wasadmin related stuff was thankfully still valid. Having to switch between the servers was also driving me insane so one of the major goals I had was to try to do as much as I could on the single (deployment manager) host, which I mostly achieved, except for creating the physical app server profiles and starting the nodeagents.
Here’s a very bare bones command line version of that wiki page, with the headings roughly matching; refer back to that page and the WebSphere Infocentre for more detailed information. Before diving in :
- All “wsadmin” commands used are run on the deployment manager host from /opt/IBM/WebSphere/AppServer/bin/, need “-profileName Dmgr01 -user admin -password admin -lang jython” supplied and can be run as commands from a file with “-f”, interactively or in single command mode with “-c”.
- Unless explicitly specified all “manageprofiles” , “startServer/stopServer” commands used should be run from /opt/IBM/WebSphere/AppServer/bin/.
- The paths, ports, hostnames, usernames and passwords are all mutable. In particular “Dmgr01” represents the deployment manager profile, “admin” is used for the WAS administrative user and password, “dmgrhost” represents the host where the deployment manager and proxy server run, “vm3host” represents the host where the first node in the cluster runs, “vm4host” represents the host where the second node in the cluster runs
- All commands are run as “root”.
Building vm 2 (WAS ND Deployment Manager + Websphere Proxy Server + Websphere Extreme Scale Catalog Server)
Install WAS ND V8
/opt/IBM/InstallationManager/eclipse/tools/imcl listAvailablePackages -repositories /opt/software/was80nd/repository.config
Then install it
/opt/IBM/InstallationManager/eclipse/tools/imcl install com.ibm.websphere.ND.v80_188.8.131.5210503_0200 -repositories /opt/software/was80nd/repository.config -acceptLicense
Install WebSphere Application Server V8 FP 2
/opt/IBM/InstallationManager/eclipse/tools/imcl install com.ibm.websphere.ND.v80_184.108.40.20611202_1708 -acceptLicense -installationDirectory /opt/IBM/WebSphere/AppServer -repositories /opt/software/8.0.0-WS-WAS-FP0000002/repository.config
Create the Deployment Manager Profile
./manageprofiles.sh -create -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/dmgr -profileName Dmgr01 -profilePath /opt/IBM/WebSphere/AppServer/profiles/Dmgr01 -enableAdminSecurity true -adminUserName admin -adminPassword admin
Start the Deployment manager
./startServer.sh dmgr -profileName Dmgr01
Set up WAS Custom User Registry for CLM
./wsadmin.sh -f /opt/software/scripts/createjazzug.py
Configure WAS ND for CLM
Set the generic JVM arguments
./wsadmin.sh -c "AdminTask.setGenericJVMArguments('[-nodeName dmgrhostCellManager01 -serverName dmgr -genericJvmArguments \"-Xgcpolicy:gencon -Xnocompressedrefs\"]')"
Add the virtual host ports for the cluster members, assuming that they listen on secure ports “9444” and “9445” (see Notes)
./wsadmin.sh -profileName Dmgr01 -user admin -password admin -lang jython -c "AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:dmgrhostCell01/VirtualHost:default_host/'), '[[hostname \"*\"] [port \"9444\"]]')" ./wsadmin.sh -profileName Dmgr01 -user admin -password admin -lang jython -c "AdminConfig.create('HostAlias', AdminConfig.getid('/Cell:dmgrhostCell01/VirtualHost:default_host/'), '[[hostname \"*\"] [port \"9445\"]]')" ./wsadmin.sh -profileName Dmgr01 -user admin -password admin -lang jython -c "AdminConfig.save()"
Install WebSphere eXtreme Scale V220.127.116.11
There are 2 ways to do this, the first to install WXS and augment the profile in one step, the second (if WXS is already installed) to augment a profile with WXS.
Method 1 : Install WXS and augment profile in one shot
/opt/software/wxs711/install -options "/opt/software/wxs711/wxsdmgrinstall.rsp" -silent
To monitor installation progress use
tail -f ~/wxs_install_logs/log.txt
Method Two: WXS already installed, augment profile only
./manageprofiles.sh -augment -profileName Dmgr01 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/xs_augment/dmgr
Restart the Deployment manager
./stopServer.sh dmgr -profileName Dmgr01 ./startServer.sh dmgr -profileName Dmgr01
Create and federate WebSphere Proxy Server
./manageprofiles.sh -create -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/managed -profileName proxy_server -profilePath /opt/IBM/WebSphere/AppServer/profiles/proxy_server -enableAdminSecurity true -adminUserName admin -adminPassword admin /opt/IBM/WebSphere/AppServer/profiles/proxy_server/bin/addNode.sh dmgrhost 8879 -username admin -password admin ./wsadmin.sh -c "AdminTask.createProxyServer('dmgrhostNode01', '[-name proxy_server -templateName proxy_server_foundation -genUniquePorts true -selectProtocols [-list [http sip ]]]')" ./wsadmin.sh -c "AdminConfig.save()"
Build vm 3 (clm node 1)
Install WAS ND V8 and related Fix Pack 2
See the previous section.
Create and Federate a custom profile to act as CLM node 1
First run this on VM 3 to create the profile
./manageprofiles.sh -create -profileName Custom01 -enableAdminSecurity true -adminUserName admin -adminPassword admin -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/managed -profilePath /opt/IBM/WebSphere/AppServer/profiles/Custom01 -federateLater true /opt/IBM/WebSphere/AppServer/profiles/Custom01/bin/addNode.sh dmgrhost 8879 -username admin -password admin
Then run this on VM 2 to federate it
./wsadmin.sh -c "AdminTask.createApplicationServer('vm3Node01', '[-name server1 -templateName default -genUniquePorts true ]')" ./wsadmin.sh -c "AdminConfig.save()"
Save and unzip automateWASInstall_v1.1.1.zip (see Notes) to /opt/software on VM 2( dmgrhost), edit /opt/software/automateWASInstall_v1.1/setWASPropertiesOnly.props to modify the cellName, nodeName and serverName properties and then run
./wsadmin.sh -f /opt/software/automateWASInstall_v1.1/configureWASforCLM.py --configFile /opt/software/automateWASInstall_v1.1/setWASPropertiesOnly.props
Install and deploy IBM Rational Solution for CLM v4.0 RC0 on server1
First download and unzip the CLM 4.0 RC0 IM repository and the CLM 4.0 RC0 Web installer to /opt/software on VM 3. Then edit the /opt/software/RTC-Web-Installer-Linux-4.0RC0/im/linux.gtk.x86/silent-install-server2.xml and change
<repository location='../repo' />
<repository location='/opt/software/JTS-CCM-QM-RM-repo-4.0RC0' />
Then run the following (still on VM 3) to install all the CLM apps to /opt/IBM/JazzTeamServer
cd /opt/software/RTC-Web-Installer-Linux-4.0RC0/im/linux.gtk.x86/ ./installc -acceptLicense -showVerboseProgress -input silent-install-server2.xml --launcher.ini silent-install.ini
To deploy the CLM apps to server1 through the deployment manager:
- switch to VM 2
- copy the /opt/IBM/JazzTeamServer/server/webapps directory over from VM 3
- edit /opt/software/automateWASInstall_v1.1/allappsfedrealm.props (from the previously downloaded automateWASInstall_v1.1.1.zip see Notes)to modify the cellName, nodeName and serverName properties and
- run the following:
./wsadmin.sh -f /opt/software/automateWASInstall_v1.1/installCLM.py --configFile /opt/software/automateWASInstall_v1.1/allappsfedrealm.props
Post CLM Application Deployment Configuration for server1
Edit the CCM, QM, JTS teamserver.properties on VM 3 to change the database connection properties.
Add the cluster attributes as described in step 2 of Post CLM Application Deployment Configuration for server1.
Then create the databse tables using
cd /opt/IBM/JazzTeamServer/server ./repotools-jts.sh -createTables noPrompt ./repotools-jts.sh -createWarehouse noPrompt ./repotools-ccm.sh -createTables noPrompt ./repotools-qm.sh -createTables noPrompt
Run the JTS Setup Wizard
Stop all the servers on VM 2 and VM 3
All of this can be done from VM 2 through the deployment manager, provided the nodeagent on VM 3 is running. If the nodeagent on VM 3 is stopped, switch to VM 3 and use stopNode.sh and stopServer.sh.
From VM 2 stop server1 and the nodeagent (not strictly required) on VM 3:
./wsadmin.sh -c "AdminControl.stopServer('server1', 'vm3Node01')" ./wsadmin.sh -c "AdminControl.stopServer('nodeagent', 'vm3Node01')"
Stop the proxy_server, nodeagent and deployment manager on VM 2:
./wsadmin.sh -c "AdminControl.stopServer('proxy_server', 'dmgrhostNode01')" ./wsadmin.sh -c "AdminControl.stopServer('nodeagent', 'dmgrNode01')" ./stopServer.sh dmgr -profileName Dmgr01 -username admin -password admin
Start all the servers on vm 2
./startServer.sh dmgr -profileName Dmgr01 /opt/IBM/WebSphere/AppServer/bin/startOgServer.sh catalogServer -listenerPort 7000 /opt/IBM/WebSphere/AppServer/profiles/proxy_server/bin/startNode.sh ./wsadmin.sh -c "AdminControl.startServer('proxy_server', 'dmgrhostNode01')"
Start all the servers on vm 3
The nodeagent on VM 3 cannot be started from the deployment manager (though it *can* be restarted )so the following needs to be run on VM 3:
While on VM 3 start server1 with
/opt/IBM/WebSphere/AppServer/bin/startServer.sh server1 -profileName Custom01
or switch back to VM 2 and start server1 through the deployment manager:
./wsadmin.sh -c "AdminControl.startServer('server1', 'vm3hostNode01')"
Run through the JTS Setup wizard
CLM 4.0 RC0 has a new feature introduced in M7 that allows running the setup wizard from the command line:
cd /opt/IBM/JazzTeamServer/server ./repotools-jts.sh -setup
Create CLM Server template for clustering
Ignore this step as the template doesn’t get used – I didn’t know then what I know now:-)
Build vm 4 (clm node 2)
Install WAS ND V8 + FP 2
See the previous section.
Create and Federate a custom profile to act as an additional CLM node
First run this on VM 4 to create the profile
./manageprofiles.sh -create -profileName Custom01 -enableAdminSecurity true -adminUserName admin -adminPassword admin -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/managed -profilePath /opt/IBM/WebSphere/AppServer/profiles/Custom01 -federateLater true
Then run this on VM 2 to federate the profile
/opt/IBM/WebSphere/AppServer/profiles/Custom01/bin/addNode.sh dmgrhost 8879 -username admin -password admin
Copy /opt/IBM/JazzTeamServer from VM 3 to exactly the same location on VM 4
I use either an scp or nfs mount and copy.
CLM WAS Cluster setup
Create CLM Cluster
All the following are run on VM 2.
./wsadmin.sh -c "AdminTask.createCluster('[-clusterConfig [-clusterName CLM_Cluster -preferLocal false]]')"
Add the first member:
./wsadmin.sh -c "AdminTask.createClusterMember('[-clusterName CLM_Cluster -memberConfig [-memberNode vm3hostNode01 -memberName CLM_Cluster_node1 -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateServerNode vm3hostNode01 -templateServerName server1 -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope cluster]]')"
Add the second member:
./wsadmin.sh -c "AdminTask.createClusterMember('[-clusterName CLM_Cluster -memberConfig [-memberNode vm4hostNode01 -memberName CLM_Cluster_node2 -memberWeight 2 -genUniquePorts true -replicatorEntry false]]')"
Save the configuration and sync the nodes:
./wsadmin.sh -c "AdminConfig.save()" ./wsadmin.sh -c "AdminNodeManagement.syncActiveNodes()"
Map applications to the cluster
for f in admin jts ccm qm rm converter clmhelp do echo "Processing $f.war" /opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -c "AdminApp.edit('$f_war', '[ -MapModulesToServers [[ $f.war $f.war,WEB-INF/web.xml WebSphere:cell=dmgrhostCell01,cluster=CLM_Cluster ]]]' )" done /opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -c "AdminConfig.save()"
Restart everything as before *except* this time start the CLM_Cluster rather than server1 and all should be well in a clustered world.
To start the cluster from VM 2:
./wsadmin.sh -c "AdminControl.invoke('WebSphere:name=CLM_Cluster,process=dmgr,platform=common,node=dmgrhostCellManager01,version=18.104.22.168,type=Cluster,mbeanIdentifier=CLM_Cluster,cell=dmgrhostCell01,spec=1.0', 'rippleStart')"
- WordPress does not seem to allow attaching arbitrary files so all the attachments here have a “.doc” extension that should be removed before doing anything with them.
- createjazzug.py: is simply a collection of wsadmin AdminTask commands that creates the required Jazz groups in the WAS default repository, creates a whole bunch of users, including those from the MTM Sample and adds them to the various groups.
- Ports for virtual hosts: When using the WAS proxy server, it need to be “told” which ports to redirect requests. The port used is found in the value of “WC_defaulthost_secure” for each server. It is also simple enough to configure and use a single port that the proxy server and all app servers listen on, removing the need to adda new entry to the virtual hosts each time a new node is added. One way is to set genUniquePorts to “false” when creating the application servers or the cluster members.
- wxsdmgrinstall.rsp: This is a response file that will get the installer to install WXS and then augment the Deployment manager profile.
- automateWASInstall_v1.1.1.zip : This contains a couple of excellent property file driven Jython scripts that my friend and colleauge Indradri Basu whipped up. Originally used to configure a base WAS server and deploy and configure the CLM apps to it, I’ve tweaked (hacked:-) it slightly to support this setup. The original version supports an LDAP based configuration with a minor tweak. It’s a testament to his superb development skills that a newbie to Jython (me:-) was able to easily find and modify the bits needed. He’s in the process of writing a whole new set of scripts that will support creating a proper clustered CLM deployment. He’s a better WAS man than I so watch this space…
- All the scripts and files attached to this post are provided completely as-is and are only examples.
Finally: this has been a very interesting Websphere learning experience for me, so if there are better, faster and yes, SIMPLER, ways to do any/all of this, feel free to let me know.