Since I mentioned that I would write further about IBM WebSphere Application Server (WAS), I’ve had several requests. Well, one. But I’m going to write about it anyway.

Working with WAS for several years now (since v4.0), reading about it and now writing about it makes me realise something important: I need to work less on WAS. Continued exposure proves detrimental to one’s joie de vivre, much like administering database systems.

True, it’s not as awful as BroadVision One-To-One 4.0, a pre-standards middleware product based on (I kid you not) server-side JavaScript. (On second thoughts, at least JavaScript on the server afflicts fewer computers than in widespread use on the client. Too late now though.) And WebSphere V5 is much more stable than V4 ever was.

Unfortunately, a day with any version of WebSphere reminds you of the fleeting transience of human mortality. Here’s why:

Java

OK, I realise Java is a fundamental component of a J2EE application server. But it’s the kind of language and framework you get when academic “software engineers” are running the asylum. EJBs, JNDI, JSSE, JMX, JMS, JCA, JAF, JSF, JSP, JARs, WARs and EARs - is there no end to the madness? Why should I care?? The answer is: you shouldn’t, or you’ll never get it running. Abstraction is usually considered a good thing because it “hides the details of the underlying implementation”, but in this case too many levels of abstraction serve only to obscure clarity and obfuscate understanding. Write once, run away.

It’s not just me. Even an extremely competent Java developer and colleague opined that much of the J2EE standard was the emperor’s new clothes writ large. After a concerted attempt at using XML/XSL transforms to generate web pages, separating the presentation logic from the business logic like a good little buzzword-compliant coder, he threw away the lot with all its grim performance problems in favour of plain old JSPs.
WAS is a product that feels like it was developed by people who’ve been smoking the J2EE crack pipe for far too long. For overall integration, it scores highly - too bad that it’s integrated around this dog’s dinner.

Security

Out of the box, WAS is completely unsecured. Anyone can log on to the administration console via a web browser and do what they want. It even installs with global read permission on every file (although that is admittedly an improvement on V4, which had global write permission everywhere). You can firewall it from the Internet (oh, except the admin URI is automatically added to the web server plugin config so you better remember to remove it), but that won’t stop people on your company network - and the odd lucky hacker - from reconfiguring your entire setup. Heck, it’s so complicated, you might as well let someone else have a go.

(Un)Fortunately, WAS has a large, extensive security component that can be enabled with pretty much one click. And then it’s so secure, even you can’t login to it. Although it can authenticate against the local OS, an LDAP server or a custom registry, the former isn’t supported on multi-node setups (precisely where it would be most useful) and the second doesn’t support standard LDAP resilience features like replicated servers - which means another heap of hardware and/or software to make your LDAP server highly available. (IBM claim that the local OS can’t be used in distributed cells because it would require the passwd files to be synchronised on each host, but that’s a lot easier to do than running a wide-open e-commerce platform. In fact, it would let you use the native LDAP/NSS backend support in the local OS, which would actually be a huge improvement on their own inadequate hack.) LDAP implementation isn’t exactly a mature skill yet and, much as their somewhat limited support of it is laudable, it’s a mistake to assume such an infrastructure exists at every site. (Of course, IBM would love to sell you a resilient LDAP solution too.)

Naturally, WAS Security flavours the Java acronym soup - CSI, SASL, LTPA, WTF is all this crap?? Hiding at the bottom of all this is plain ol’ SSL, for which IBM give you another icky Java management utility called IKeyman. Don’t forget to replace the default (‘dummy’) certificates, by the way. Oh, and copy them manually to every node in the cell. Including the web server.

Assuming you have a suitable registry, you can set up WAS users with defined roles and passwords. Once enabled, an authorised account is required for all administrative actions, including running the command scripts - and if you want to run those scripts unattended (e.g. to start WAS during boot), you’ll have to specify the appropriate “secret” password on the command line. So this vast and baroque security edifice is completely undermined by the kind of no-no that all sysadmin neophytes have beaten out of them in their first year.

It would be interesting to know how many WebSphere sites run with Security disabled, either because they don’t know it’s there, nobody had time to figure it out or because they decided it’s a waste of space. At the very least, a simple username/password lock on the admin console and an auto-generated self-signed SSL cert, both created at install time, would stop every Tom, Dick and Harriet from reconfiguring your entire live cluster. (Interestingly, Sun’s JES Application Server, supposedly an inferior product, does this.)

Administration console

IBM replaced the dedicated Java administration client in WAS V4 with a smart new web-based console in V5, which was probably the right move (although installation of the relevant client software was previously at least one barrier to accessing an unsecured WAS cluster). Unfortunately, the interface is clunkier than ever and you’ll expend many mouse miles expanding menus, scrolling long frames of dialogue, clicking on sub-options, then saving your changes and navigating back to the same place again. Also, there’s no way to make an identical change to every server in a cell or cluster simultaneously. V4 at least had some settings that were shared across every cloned server, but V5 removes even this partial support. If you want to tweak the JVM heap size for every app server (a parameter that is buried about three or four levels deep - depending on which route through the menus you take), IBM makes you click and navigate all those dialogues for every single one. The only alternative is to script your changes using the WSADMIN client.

Scripting

WAS admin actions can be scripted in JACL, a Java implementation of TCL based around Managed Beans (MBeans). TCL itself is a simple scripting language with only a few keywords. But the MBean malarky is bolted on like the head of Frankenstein’s monster, only not as prettily.

To perform an operation on some aspect of WAS using JACL, you first figure out what object you need and then fetch a configuration ID for it. Then you call the relevant method with the appropriate arguments. Don’t know the object name, the ID syntax, the method name or the argument usage? Well, tough. There’s a help facility that gives basic one line summaries of each item for those who enjoy cryptic puzzles. There are various commands to list out all the types, attributes and objects in the cell, although the WSADMIN client doesn’t let you page back through the reams of output these produce. Nothing is properly documented and the adventurous WAS administrator is reduced to scratching around for the few inadequate examples tantalisingly revealed by IBM. Throw me a frickin’ bone here, people.

If you don’t like JACL, you can also script in Jython but it’s the same shit in a different syntactic sugar.

IBM

My main struggle with any IBM product is the fact that it was made by IBM. Because there’s everyone else’s way, and there’s the IBM way - and the IBM way is…well, weird frankly. Look at AIX for example. Even their suggested Best Practices for WebSphere leave you thinking, “Hmm, not sure I’d do it that way.”

Add to that, the extra support you constantly need demands navigating and searching the IBM web site, which is the kind of arduous quest that would have even Henry Stanley saying, “Let’s just write ‘Livingstone presumed dead’ and go to the beach instead.” Consistency and effectiveness fly out the window, through the hole made by your monitor. The search function appears to be some kind of advanced anti-Google engine that returns only irrelevant results. Every link takes you somewhere you didn’t quite expect, even the ones that ostensibly lead to the same place: if you go directly to the IBM Redbooks site, there’s a special “WebSphere domain”, but if you click “Redbooks” in the WAS support zone, it returns a badly sorted search result for tangentially related documents. Many of the bug reports (APARs) make no sense in English and searches on symptoms, such as error message texts, usually return bugger all. When you finally cave under the weight of inadequate, badly organised documentation that tells you nothing you want to know, and pay for three days of IBM consultancy, they send someone who repeats exactly what it says in the documentation.

These are just a few of the reasons why WebSphere administration lends your days a Bergmanesque quality (and I don’t mean Ingrid) - thanks, IBM. That’s why a thirty page “quick start” document written by someone who’s been there is necessary. (If it’s any consolation, I’ve heard that BEA WebLogic is just as bad.)

Update, 2006-06-14:

Well, don’t say IBM never listen (um, they don’t but they don’t want to hear anyone saying that, capish?). You can now search their support web site via Google, which seems to be a tacit acknowledgement that their own search engine sucks. They’ve had a go at simplifying scripting in WAS 6.0 by adding some higher level objects, although whether this is sufficient is a moot point (you can’t polish a turd). And some of the security & LDAP madness has been addressed in WAS 6.1, specifically the lack of security out-of-the-box.

Oh, and there’s finally going to be support for Solaris x86. (Don’t laugh - without this, you’ll end up running Linux if you migrate off SPARC.)

Now the downside: WAS 5.1 doesn’t support Solaris 10 and WAS 6.1 doesn’t support Solaris 8. So to go from one extreme to the other in any kind of staged manner, you must perform a completely pointless WAS 6.0 installation inbetween. This simply isn’t sufficient breadth of support for a rapidly changing application, particularly on an OS that carries backwards compatibility guarantees. Hopeless…