Load Test

Past few days i encountered one serious production issue, which the web service timeout before the backend process sent back the message to the SOAP response node in WMB flow. This was due to the high load in production that caused the delay in the backend process.

The whole process is: WMB flow extracts SOAP request message, turns it into a MQ Header message, puts the message to Java adaptor's request queue, then Java adaptor processes the content and puts a response message to the response queue, then WMB flow will read the message in the response queue and will turn it into a SOAP response.

In the progress of investigation, we decided to move one database query out from the java adaptor program to the WMB flow. The reason being is sometimes the db connection will just drop and the java adaptor program has to spend some time reinitiating the db connection. In fact according to the time recorded in the adaptor's log the process was still quite fast, it's even within miliseconds. Well we just gave it a try.

After making all the necessary changes we deployed the works to the testing environment. I then performed some load tests to the web service. The result was negative. The tps was around 3.5 but the average time taken was > 10 seconds and we had to achieve < 10 seconds! So still plenty of work to do :(

We performed a few cycles of investigation, trial and error & testing until we found the solution - increasing the WMB flow instance. This is a setting in the WMB Toolkit. So we increased the number of instances to 3. This time we managed to achieve an average time of 3 seconds! We were so happy and excited!

What about the cause? See below:

There were many messages piling at the response queue which is supposed to be read by the WMB flow. These messages were sent in by the Java Adaptor after it finished processing the data. Apparently these messages were not picked up by the WMB flow in time hence causing the timeout error. By increasing the WMB flow instance, concurrently there are 3 identical WMB flows reading the content of this queue, hence problem resolved!

I was really relieved!

又一个日曜日的傍晚

日曜日的傍晚，又是上京时。
喂座驾以足食。
日已落、天凉气爽、犹如雨后、其实无雨。
油站冷清客稀、四周旧楼添诗意。
燕群鸣不已、翼声不绝。举头 - 落寞高楼现，昔日斗气作、今日玄鸟巢。
回忆深深、来年茫茫、不胜唏嘘。
此景焉能永恒？

Classloading in Websphere Administrative Console

I have this web service server application which has been running fine at a Websphere Process Server. One day, my application's log was writing to another application B's log. That is really weird. I checked and found out that this B was recently deployed to the server only. The cause for my application to write to B's log is due to B's log4j jar file was put in the server's common directory ! If we do not change the classloading policy, which is default to PARENT_FIRST, then Wepshere's classloader will load that common directory first only will load the jars in our application's WEB-INF/lib directory.

Hence i logged on to Websphere Administrative Console, went to my application's war file and changed the classloader policy to PARENT_LAST. By doing this Websphere will find and load the neccessary classes from your web module's WEB-INF/lib(web module classloader) first, also you must ensure that the jars you put under your application's WEB-INF/lib folder do not clash with the one at server's common directory, else you will get a java.lang.LinkageError exception due to classloader is trying to load duplicate class.

One year already!!

Time flies. It has been one year already since i joined this company in KL. Also it is my 1 year in KL.

I have been working on ESB/SOA stuff since day 1. Back then i was really blur of all these queue/web services/broker/adapter stuff as i was from web development background. I am glad that i was given the chance to get exposed to this area. Thanks to the support from my colleagues. They are really great!

Since i am doing integration job, i often have to liaise with a few parties. Hence the interaction between me and them is quite important. The communication between me and them has to be effective to ensure that i am able to deliver the right solution as according to what business users need. This part has troubled me a lot. I think i still have much space for improvement in this area. Perhaps i need to take some psychology course. Hahaha.

I am quite lucky to witness the outsourcing process. My client - a telco company has outsourced its IT department to my company some time back. Things have turned quite messy at the first few months after the outsourcing. For example my project manager has changed 4 times in 3,4 months time and the process is no longer as clear as the one before the outsourcing. Fortunately i can see things are getting stable now.

Just looking forward to learn more things as long as i have not left this company. I hope can get high pay with minimum workload. Hahaha.

昔华中学杰出校友

目前为止所知道的杰出校友有数位如下：

1. 黄子。他常在报章电台等媒体发表意见。他在电台上的说话方式非常滑稽，常常反讽时弊。他现时在南洋商报有个专栏。
2. 杨善勇。自多年前就常在各报章看到他的文章。从中得知他的专长乃城市策划。近来他在东方日报有个专栏。
3. 后起之秀彭靖岚彭靖雯两姐妹。他们的部落格/博客/网志在本地非常受落因而累积了一些知名度。2年前我偶然间于凌晨在国营电台听到他们的声音，原来是当天的嘉宾。

就只知道这数位，有待日后补充。

Ai FM's DJ was crying over Yasmin Ahmad's death

I was driving back to KL from my hometown. I was listening to Ai FM(a mandarin radio) as usual. 10.05pm was DJ Ji An's programme. He was talking about a lot of things about Yasmin Ahmad because he invited Yasmin Ahmad a few times to his programme as guest. I think after a few times Yasmin Ahmad was invited they became some sort of close friends. Well at first his tone was still calm but suddenly he burst into tears. I guess that was really a sad moment for him when Kak Yasmin passed away. So from 10.05pm to 12pm DJ Ji An was replaying all the programs that he had invited Yasmin Ahmad as his guest.

The 2 hours was a nice one. Yasmin Ahmad was really anak Malaysia yang betul-betul membuat movie untuk semua Malaysian. It's a great loss for us Malaysian that Yasmin Ahmad could not produce more touching TV advertisements & movies anymore.

NullPointerException in SecureRandom.nextBytes

I was facing this issue where a web service client which had always been running fine suddenly stopped functioning properly due to an error NullPointerException in SecureRandom.nextBytes

AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.lang.NullPointerException
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}stackTrace:java.lang.NullPointerException
at java.security.SecureRandom.nextBytes(SecureRandom.java:406)
at org.apache.axis.utils.SessionUtils.generateSessionId(SessionUtils.java:62)

As you can see the exception was thrown when some class in Axis is calling the nextBytes function from SecureRandom Class. I spent quite some time in this and i think that is a bug for IBM version of JRE 1.4.2. If you want to know more, read here.

Therefore i changed the script which starts my web service client by using the java command from Java 5 instead of java 1.4.2 and finally it worked!