Issue with Java Security Provider

While working on one enhancement. I encountered a weird problem. The program within WBIA framework failed in calling web service. It was throwing me a NullPointerException from Axis library as below:

at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
at org.apache.axis.utils.SessionUtils.generateSessionId(SessionUtils.java:62)
at org.apache.axis.SOAPPart.<init>(SOAPPart.java:164)
at org.apache.axis.Message.setup(Message.java:377)
at org.apache.axis.Message.<init>(Message.java:246)
at org.apache.axis.client.Call.invoke(Call.java:2425)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)

I made the investigation as below:

1) I put a java.security.Security.getProviders() in my program to list all the security providers. I compared the results from successful call (standalone java program running at the same AIX server) and unsuccessful call (program extending from WBIA framework running at the same AIX server).

Here is the result:

//Unsuccessful call.
- IBMJSSE2
- IBMJGSSProvider
- IBMCertPath

//Successful call
- IBMJSSE2
- IBMJCE
- IBMJGSSProvider
- IBMCertPath
- IBMSASL

Obviously some security providers are missing hence caused the unsuccessful call. I double confirmed this by modifying my java.security file at local and tested the same program from my local. Yes the web service call actually failed due to the missing security provider. The missing security provider is actually com.ibm.crypto.provider.IBMJCE

Therefore, i did the below the solve this issue:

1) Set the Java home path to the JRE folder instead of the JDK folder itself.

2) I added the security provider at runtime.
java.security.Security.addProvider(new com.ibm.crypto.provider.IBMJCE());

Finally it worked. One question is, how come it somehow did not refer to the java.security file that i set everything correctly else i do not have to do the 2nd step as mentioned. I still need to find out this.

再临华美 Hua Mui Revisited

既然到了新山，我当然会去华美，吃吃它的面包，喝喝它的咖啡，体验殖民地风情，享受慵懒的午后气氛。
Been to Hua Mui once more! Yummy Yummy Hua Mui


外观
Hua Mui here i come again!


内观


内观


非常殖民地风味
Love the colonization era feel. It reminds me this restaurant is close to 100 years old.


叫了杯茶c
I ordered Teh-C. Good. Very Kow.


叫了面包，好吃好吃。
The toast banyak bagus! Delicious. Enak betul.

访新山华族历史文物馆失败记

上个星期五，心血来潮往新山去。既然到了新山，就想顺道去看看不久前开张的新山华族历史文物馆。找了好久才发现那馆近在尺尺。既然没开！大失所望！

没关系，下次再来。

拍了一张外观照。

永平福州鱼丸 Yong Peng FuZhou Fish Ball

哇，首尝福州鱼丸。
Yummy. First time eating FuZhou fish ball.

以往多次路经永平，对荣凯的西刀鱼丸鱼饼情有独钟，所以无缘尝试其他店的鱼丸。
I love XiDao Fish Ball. I used to eat a lot of XiDao Fish Ball & Fish Cake at a shop called Antony's every time i traveled back to my home town.

这次由于已近晚上十点，荣凯已闭门休息。唯有试试别家鱼丸，找上了这间。
Last Friday night at 10pm, i wanted to have dinner at Antony's, which had already closed. Therefore i had my dinner at this shop (i have forgotten the shop name).

其干面与鱼丸配搭没荣凯出色，其鱼饼似乎与荣凯的同出一源。让我欢喜的是那一晚福州鱼丸。鱼丸里头裹着猪肉，是新发明吧？非同凡响，值得一试！
There is a piece of meat inside the fish ball. New invention? Worth trying!

Load Test

Past few days i encountered one serious production issue, which the web service timeout before the backend process sent back the message to the SOAP response node in WMB flow. This was due to the high load in production that caused the delay in the backend process.

The whole process is: WMB flow extracts SOAP request message, turns it into a MQ Header message, puts the message to Java adaptor's request queue, then Java adaptor processes the content and puts a response message to the response queue, then WMB flow will read the message in the response queue and will turn it into a SOAP response.

In the progress of investigation, we decided to move one database query out from the java adaptor program to the WMB flow. The reason being is sometimes the db connection will just drop and the java adaptor program has to spend some time reinitiating the db connection. In fact according to the time recorded in the adaptor's log the process was still quite fast, it's even within miliseconds. Well we just gave it a try.

After making all the necessary changes we deployed the works to the testing environment. I then performed some load tests to the web service. The result was negative. The tps was around 3.5 but the average time taken was > 10 seconds and we had to achieve < 10 seconds! So still plenty of work to do :(

We performed a few cycles of investigation, trial and error & testing until we found the solution - increasing the WMB flow instance. This is a setting in the WMB Toolkit. So we increased the number of instances to 3. This time we managed to achieve an average time of 3 seconds! We were so happy and excited!

What about the cause? See below:

There were many messages piling at the response queue which is supposed to be read by the WMB flow. These messages were sent in by the Java Adaptor after it finished processing the data. Apparently these messages were not picked up by the WMB flow in time hence causing the timeout error. By increasing the WMB flow instance, concurrently there are 3 identical WMB flows reading the content of this queue, hence problem resolved!

I was really relieved!

又一个日曜日的傍晚

日曜日的傍晚，又是上京时。
喂座驾以足食。
日已落、天凉气爽、犹如雨后、其实无雨。
油站冷清客稀、四周旧楼添诗意。
燕群鸣不已、翼声不绝。举头 - 落寞高楼现，昔日斗气作、今日玄鸟巢。
回忆深深、来年茫茫、不胜唏嘘。
此景焉能永恒？

Classloading in Websphere Administrative Console

I have this web service server application which has been running fine at a Websphere Process Server. One day, my application's log was writing to another application B's log. That is really weird. I checked and found out that this B was recently deployed to the server only. The cause for my application to write to B's log is due to B's log4j jar file was put in the server's common directory ! If we do not change the classloading policy, which is default to PARENT_FIRST, then Wepshere's classloader will load that common directory first only will load the jars in our application's WEB-INF/lib directory.

Hence i logged on to Websphere Administrative Console, went to my application's war file and changed the classloader policy to PARENT_LAST. By doing this Websphere will find and load the neccessary classes from your web module's WEB-INF/lib(web module classloader) first, also you must ensure that the jars you put under your application's WEB-INF/lib folder do not clash with the one at server's common directory, else you will get a java.lang.LinkageError exception due to classloader is trying to load duplicate class.