java & tuxedo @@

Was facing issue where my java program failed to load the so library from Tuxedo8.

Spent quite much time. Finally solved it today. I am so happy!!! The feeling is the same every time my article got published on newspaper!

I got to decompile the tuxedo.jar, which comes with tuxedo installer. From there i put in as many system out as possible for troubleshooting. The program actually stopped at the part where System.loadLibrary is being called. It failed to load a particular so file even tough physically the file is there.

I started to check if the bit version of java matches the one of so file. My machine is AIX6.1 and i got to know that AIX6.1 only supports 64bit kernel. Hence i decided to run my program using 64bit version5 java.

After running the program using 64bit java, i saw more meaningful error messages this time as compared to the previous non-informative messages. From there i got to know the SO file failed to load its dependencies.

So i started to look for the dependencies, and put them under the same directory where the major SO file is located.

Of course i also set the path of where my SO file is to the variable LIBPATH and also to the -Djava.library.path

That's it :) Of course before writing this down i actually have gone through so many researches and trial and errors. Gosh. Imagine how happy am i now! :)

Oh ya, i dropped the System.loadLibrary, instead i hard coded the path using System.load :)

Some helpful commands along the way of my troubleshooting process

1) java -d64 -version
2) nm -X64

Making the world work better

It was IBM's 100 year old birthday. As an IBM staff i was given a book "Making the world work better".

"Making the world work better", this has inspired me. This is such a great idea. I am so touched by it.

This is what human need, isn't it?

The world is in the mess, we really need people to have a common goal, which is to make this world work better! After all we, human, are far too fragile in the universe.

Honestly speaking, i never have the sense of belonging to IBM since day 1 i joined. My perspective is i work in many projects, close them to make sure clients pay IBM and that's it. Never have i had a single thought that my work actually contributes to the human civilization in a direct manner, just like scientists do.

In fact, some of the projects were actually initiated from some awesome ideas! It will be cool if the consumers are able to enjoy the final result from these projects. The thing is, clients always make sure gaining profit is the priority, and that's the perspective they will look at all the time.

Resource constraint should not be the barrier that stops us to make the world work better. After all, it is such a great honor being part of the members to help the civilization grow.

It's time to move on!

Well, finally i decided to quit from my current company. Last week i even signed the new deal to firm up my mind. Anyway the next job offers lesser income compared to the current one.

Yesterday i tendered my resignation letter to my People Manager. She was quite shocked and was trying to persuade me to stay. She was getting a high profile guy who is leading a new big project with a bank account to check with me if i am interested in joining that project. After a short chat with the guy I still requested for "Separation" from my current company.

Before getting home from office i told my Work Manager my decision to quit. I think at that moment he was about to go home too. Few minutes later another senior colleague, whom i believe was in the same car as my Work Manager, called me to persuade me to move to another department within the organization. Alright this drew my attention as the office as i know is very much near to my house. That's the thing, the next company is actually located near to my house, thus giving me a chance to go back on time for activities such as sports and so on!

So in conclusion, a relocation within the same organization is ok for me.

I have approached the department and i am looking forward to a good news from them.

I am glad and i am indeed touched as these colleagues are really willing to help me to stay in my current company. I really like them. I think IBM is really a place that appreciates human resource. Thanks guys! Thanks all of you!

WMB It's best to set BLOB as the parser for message containing MQRFH2 header

I was struggling for this message with MQRFH2 Header.

I have tried MRM, the drawback is that you got to choose a schema, which makes it less flexible. Also the parsers being selected at runtime (printed in the mqsitrace log) are not consistent! Still puzzling.

Setting parser as BLOB in the MQInput node solves my issue. Without making any further message format conversion, you can read the xml using XMLNSC parser anytime anywhere.

Transfer message with RFH header from one queue to another queue

I have this simple java program transferring message with RFH header from one queue to another queue in bulk.

First time it failed due to the program failed to maintain the RFH header during the transfer. It read the message in STRING format and then put the message back in the destination queue.

After that i changed this by sending the MQMessage directly to the destination queue. This works.

java.sql.Date not working on WPS 6.2?

I had this program reading database table column as java.sql.Data type. Well it failed to read this when running from the WPS AIX server. Funny thing is it was fine when i was testing this program from my local WPS server runtime. In application log the column data type is recognized as java.sql.Timestamp. So i have to change to program to read Timestamp instead of Date.

MQ-related CRON job setting

Recently my colleague was faced with the crontab issue. Below is the description:

1. CRON Tab

We developed a JAR where it connects to the Queue Manger and puts message into the Queue to kick start the WMB flow. We have created a Shell script to send message to the queue. It is successful by executing the Shell script manually where it connects to the Queue Manager and successfully put the message into the Queue.

Unfortunately, the CRON job failed to execute this script. After a few rounds of investigation we found out that this is due to the AIX Environment Setting. We need to set the environment variables at our Shell script as below before connecting to MQ:
. /opt/IBM/mqsi/6.1/bin/mqsiprofile

The default environment setting for CRON does not have the MQ Specific environment.
Therefore, when we are using server binding, we need in include this into our Shell script.
If the Shell is running externally as client, we do not need to include this.

Modify WSDL to get the java client generated right

I have to generate java client for this WSDL. However the bean generated is not what i expect. The reason is the SOAP response format actually looks like this:

<data>
<field id="field1"><field>
<field id="field2"><field>
...
</data>

The bean generated is like this:

public Class Response{
Field[] data;
....
}

I failed to get the value of each . The Field class does not provide a getValue method. Instead it provides getId method which returns the field id.

I had 2 solutions:

1) Modify the generated java web-service client code. (Those serializer, deserializer etc)
2) Modify the WSDL to get the code generated as according to what i want.

I chose the second option.

I commented this segment of the WSDL:

<xsd:element maxOccurs="unbounded" name="field">
<xsd:complexType>
<xsd:sequence maxOccurs="1" minOccurs="1" />
<xsd:attribute name="id" type="xsd:string" />
</xsd:complexType>
</xsd:element>

I replace the above segment with:

<maxoccurs="unbounded" minoccurs="0" name="field" nillable="true" type="xsd:string"/>

Done!

The world is really small

After so many years, me and my sifu(mentor) in ex-company are in the same team again! He left my ex-company in year 2005 and has joined a few companies in the following years. In year 2009 he joined my company and finally recently he officially became my teammate! So it has been 3 years then. When the news was announced i was like "wow, it's happening"

Issue with JMS & MQInput node in WMB

The XML is sent by JMS as below format:

reqMsg.setJMSType("mcd://mrm//" + msgType + "?format=CwXML");

WMB trace file throws the below error:

See the following messages for details of the error.
2010-03-31 16:49:05.949233 13942 ParserException BIP5313E: Message format ''CwXML'' is not defined for message set ''RR_MVNO_Port_In_MessageSet (H74SG78002001)''.
The broker received a message for processing within the MRM domain.
However, the message cannot be processed because the message format specified in the physical format identifier ''CwXML'' has not been defined for message set ''RR_MVNO_Port_In_MessageSet (H74SG78002001)''.
The message format describes the physical representation of the message, and you can specify this format either in the properties on an input node, or dynamically within an MQRFH2 message header.

The solution is to add one more "XML Wire Formats" at the Message Set.

Also, need to make sure the MQInput node's Message Domain & Message Set are set with the correct values.

Another cert issue

We have this standalone java program running on a AIX box calling a web service which is running on https. The web service provider provided us the cert and we imported this cert to the cacerts keystore using IBM Key Management tool(Alternatively you may use the java keytool command).
(Anyway this is the first time i am exposed to this, so please correct me if i am wrong)

At first the web service called failed. We got the below error:

unable to find valid certification path to requested target

We checked the shell script where we started this program. In the shell script, the java command path is specified as /usr/java5_64/bin and the cacerts actually resides at /usr/java5_64/jre/lib/security

Therefore we changed in our script from /usr/java5_64/bin/java to /usr/java5_64/jre/bin/java and now it is working well.

Please find the 2 URLs as below

Link1

Link2

Issue with Java Security Provider

While working on one enhancement. I encountered a weird problem. The program within WBIA framework failed in calling web service. It was throwing me a NullPointerException from Axis library as below:

at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
at org.apache.axis.utils.SessionUtils.generateSessionId(SessionUtils.java:62)
at org.apache.axis.SOAPPart.<init>(SOAPPart.java:164)
at org.apache.axis.Message.setup(Message.java:377)
at org.apache.axis.Message.<init>(Message.java:246)
at org.apache.axis.client.Call.invoke(Call.java:2425)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)

I made the investigation as below:

1) I put a java.security.Security.getProviders() in my program to list all the security providers. I compared the results from successful call (standalone java program running at the same AIX server) and unsuccessful call (program extending from WBIA framework running at the same AIX server).

Here is the result:

//Unsuccessful call.
- IBMJSSE2
- IBMJGSSProvider
- IBMCertPath

//Successful call
- IBMJSSE2
- IBMJCE
- IBMJGSSProvider
- IBMCertPath
- IBMSASL

Obviously some security providers are missing hence caused the unsuccessful call. I double confirmed this by modifying my java.security file at local and tested the same program from my local. Yes the web service call actually failed due to the missing security provider. The missing security provider is actually com.ibm.crypto.provider.IBMJCE

Therefore, i did the below the solve this issue:

1) Set the Java home path to the JRE folder instead of the JDK folder itself.

2) I added the security provider at runtime.
java.security.Security.addProvider(new com.ibm.crypto.provider.IBMJCE());

Finally it worked. One question is, how come it somehow did not refer to the java.security file that i set everything correctly else i do not have to do the 2nd step as mentioned. I still need to find out this.

Load Test

Past few days i encountered one serious production issue, which the web service timeout before the backend process sent back the message to the SOAP response node in WMB flow. This was due to the high load in production that caused the delay in the backend process.

The whole process is: WMB flow extracts SOAP request message, turns it into a MQ Header message, puts the message to Java adaptor's request queue, then Java adaptor processes the content and puts a response message to the response queue, then WMB flow will read the message in the response queue and will turn it into a SOAP response.

In the progress of investigation, we decided to move one database query out from the java adaptor program to the WMB flow. The reason being is sometimes the db connection will just drop and the java adaptor program has to spend some time reinitiating the db connection. In fact according to the time recorded in the adaptor's log the process was still quite fast, it's even within miliseconds. Well we just gave it a try.

After making all the necessary changes we deployed the works to the testing environment. I then performed some load tests to the web service. The result was negative. The tps was around 3.5 but the average time taken was > 10 seconds and we had to achieve < 10 seconds! So still plenty of work to do :(

We performed a few cycles of investigation, trial and error & testing until we found the solution - increasing the WMB flow instance. This is a setting in the WMB Toolkit. So we increased the number of instances to 3. This time we managed to achieve an average time of 3 seconds! We were so happy and excited!

What about the cause? See below:

There were many messages piling at the response queue which is supposed to be read by the WMB flow. These messages were sent in by the Java Adaptor after it finished processing the data. Apparently these messages were not picked up by the WMB flow in time hence causing the timeout error. By increasing the WMB flow instance, concurrently there are 3 identical WMB flows reading the content of this queue, hence problem resolved!

I was really relieved!

One year already!!

Time flies. It has been one year already since i joined this company in KL. Also it is my 1 year in KL.

I have been working on ESB/SOA stuff since day 1. Back then i was really blur of all these queue/web services/broker/adapter stuff as i was from web development background. I am glad that i was given the chance to get exposed to this area. Thanks to the support from my colleagues. They are really great!

Since i am doing integration job, i often have to liaise with a few parties. Hence the interaction between me and them is quite important. The communication between me and them has to be effective to ensure that i am able to deliver the right solution as according to what business users need. This part has troubled me a lot. I think i still have much space for improvement in this area. Perhaps i need to take some psychology course. Hahaha.

I am quite lucky to witness the outsourcing process. My client - a telco company has outsourced its IT department to my company some time back. Things have turned quite messy at the first few months after the outsourcing. For example my project manager has changed 4 times in 3,4 months time and the process is no longer as clear as the one before the outsourcing. Fortunately i can see things are getting stable now.

Just looking forward to learn more things as long as i have not left this company. I hope can get high pay with minimum workload. Hahaha.

NullPointerException in SecureRandom.nextBytes

I was facing this issue where a web service client which had always been running fine suddenly stopped functioning properly due to an error NullPointerException in SecureRandom.nextBytes

AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.lang.NullPointerException
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}stackTrace:java.lang.NullPointerException
at java.security.SecureRandom.nextBytes(SecureRandom.java:406)
at org.apache.axis.utils.SessionUtils.generateSessionId(SessionUtils.java:62)

As you can see the exception was thrown when some class in Axis is calling the nextBytes function from SecureRandom Class. I spent quite some time in this and i think that is a bug for IBM version of JRE 1.4.2. If you want to know more, read here.

Therefore i changed the script which starts my web service client by using the java command from Java 5 instead of java 1.4.2 and finally it worked!

Busy with Web Services on WPS

For the past two weeks i had been very busy with building web services using java 1.4. Bad thing was i had to deploy them on Websphere Process Server 6.0,

I had no problem building the web services using Rational Software Architect(RSA). I tested my work by deploying to Websphere Application Server 6.1 runtime that comes together with the RSA. Everything was working fine.

However i started to have a lot of headaches while trying to deploy these web services by using Websphere Admin Consle to Websphere Process Server 6.0.2 which is running on a remote server. My web service application just could not start!

After trying out several solutions, in the end i migrated my source codes to Websphere Integration Developer 6.0(WID). I regenerated every web service related deployment configuration files and proxy classes by using WID.

This time my web service application started successfully! Well the next problem was it was unable to get database connection! It took me quite some time to figure out what's wrong with it and finally i realized that i had to get the JDBC connection from the WPS itself instead of creating my own OracleDataSource cache.

I created all JDBC connection that the web service applications needs by using the Websphere Admin Console. I also changed my code to get the JDBC connection via JNDI.
The method is simple. Just create a default initialContext, then within the context, get your JDBC connection by name like "JDBC/MyDB" will do.

Run java program at backend on Windows

Normally, if you want to start a Java program to run as back end process, just type the command follow by a '&'. However, how do you do the same on Windows?

Yesterday my colleague was facing this problem and i remembered i have gone through the same problem before in my previous company. After we went through the windows command list i found something, which is the 'start' command. Then i remembered i might have used this command before when i had faced the same problem last time.

So we tried the 'start' command and it was successful.

Why IT Project Fails!

I have come across an article talking about why IT project fails!

Take a look on it!

My experience told me that business users often do not want to spend much time with the developers during the development of a system. They do not realize that it is very very critical and useful to the final result of the system by spending time reviewing developers work to find out whether something is missing at the early phase of the project! Well i do agree that agile methodology approach is very very good provided the business users are willing to allocate resource to examine the system.

Finally i am working on 2nd project!!!

Finally i have a chance to work on 2nd project after joining this company.

The first one has gone for production and it is being supported by the production support team now. Anyway there are several environment issues occurring recently. I want to help them but too bad i am involved in new project already.

To be accurate it is not my 2nd project, but 3rd one. I was involved in a 3 days proof of concept anyway and that gave me a terrible nightmares. It was a typical last minute project which everyone in the team had to work extra time! It is about to build a portal for a telecommunications client.

Now one weird thing of the WBIA running on production is that sometimes the message will be processed twice! This creates a false impression to users of the telco company that the account registration process just fails! The production support team should be quite tension right now of this issue. I think it has something to do with the configuration anyway that why certain XML message is processed twice. Why it is routed back to the adapter again!

Also today i have learned how to upload a WSDL file to DataPower and do a testing today! My teammate said basically we can replace the whole DataPower team after we learned how to configure the WSDL at DataPower. That's funny!

Working on wrong direction

One day the client raised an issue to me. That issue would not happen if the Websphere Message Broker (WMB) flow that i deployed on the server is running fine.

I checked the queue and channel status by using a tool called MQJ explorer and found out both the sender and receiver channels' status were 'inactive'. I then turned it to 'running' again. However their status will still be 'inactive' after some time. I had gone mad about this and spent a whole day trying to find out what went wrong.


In the end i know i went into the wrong direction. It is normal that after some time the status of the channels go to 'inactive'. As long as the transmission trigger is on then there will be no problem. When the next message comes in then the sender and receiver channels will be activated again.

So what actually went wrong. Argh it is because the portal team did not set the correct queue name to the JMS header of the XML message and thus no response from the whole flow!