Friday, May 8, 2009

Perform Static Analysis using Rational Software Analyzer

Static Analysis is a core practice in eXtreme Programming to detect code quality issue at an earlier phase. Rational Software Analyzer is a powerful and flexible tool to perform static analysis, to improve code quality.

Rational Software Analyzer has been integrated into RAD/RSA 7.5 from 2008. You can easily use it to check one code file or a batch of code files. Of course, you can download and install a stand alone version if you don't use RAD/RSA.

1. Start from the Configurations:
   

Click the Software Analyzer Configurations ... to open configuration window. Create a new configuration for your analysis, and the embedded SA version include totally 211 rules for code analysis. As I know the standalone version provided more domains and rules. But I believe you can install the whole set rules yourself if you need. At lease I found the 211 rules are enough for my project. Even I don't need all of them.

In the configuration rules tab, you can customize which rules you need for your analysis work. A nice thing is you can also export your rules set to share with others, which ensure the whole team developers can analysis code using exactly same rules.
   

2. Right click on the file, or package, or whole project which you want to analyze, select "SoftwareAnalyzer - " to start, you can see the analysis result (errors or warnings) in the Java Code Review tab. Fix those issue one by one. Sometime, you can see the "Quick Fix" in right click menu.
   

Best Practice:

But sometimes, we are unable to fix all errors or warnings in the analysis result for various reason. In this case, the developer should provide a proper explanation for the unfixed items.. This is a very good practice I believe. But for those unfixed items, if another developer works on the same code in future, he will meet the same error or warning again. He have to spend time to analyze it again, that's a time waste!

So my suggestion is that the developer must add some javadoc comment in the code where the error or warning is marked and it isn't fixed, then the javadoc comment can save another developer's time in future. For such a case, I think we don't need work out and formal document to record that as the document will never be updated or read by others possiblily.

And for the javadoc comment content, maybe we can include these information:
1. the violated rule name;
2. the explanation why you didn't fix this error or warning;
3. comment date and develper's name, for possible future discussion

A feedback from Tim Shanahan is we can consider to use annotations mechanism instead of javadoc. The advantage an annotation could provide the ability to dynamically decide whether to count or report items that have been documented. Maybe this is a better idea, will try it later.


Wednesday, April 29, 2009

An severe performance impact by subselect in sql "from" clause (DB2)

Started working on the auto-pricing process performance test, but an very strange thing blocks me: a very simple query runs in very very bad performance, only 1 record inserted per second! The database is DB2 8.5 on zLinux server which runs in normal CPU utilization (35% only).

Originally, I thought there must be some problem in the database, so bad performance should not happen even the SQL hasn't been optimized. But the feedback from DBA is that he checked all things in DB, but can not find any abnormal indication. I really felt overwhelmed!

But today, I finally acknowledge that the bad performance is really caused by my query:

// less 1 record inserted per second, 30000 records insertion need 10 hours to complete (timeout...)
INSERT INTO WWPRT.CABLEPRODUCT_PRICETYPE_JOIN_CN (CABLEPRODUCTID, PRICETYPE, SLAVE) (
     SELECT cableproduct.ID, pricetypes.PRICETYPE, 'N'
     FROM WWPRT.CABLE_PRODUCT_JOIN_CN AS cableproduct,
     (
          SELECT DISTINCT usprice.PRICETYPE FROM WWPRT.PRICE_CN usprice
          JOIN WWPRT.CONF_PRICETYPE pt ON pt.ID = usprice.PRICETYPE
          WHERE usprice.CABLEID = 'USCABF3FVT01'
          AND pt.DOMAIN IN ( SELECT DISTINCT DOMAIN FROM WWPRT.CONF_GAP )

     ) AS pricetypes
     WHERE CABLEID = 'GEOANNF3FVT1'
     AND EXISTS (
         ......
     )
)


The arch-criminal is the subselect statement in "from" clause, the subselect query is performed for each record in WWPRT.CABLE_PRODUCT_JOIN_CN table. (awful!)

I changed it to a new way, using temporary table which query is executed only once.

// 30000 records insertion only costs 9 seconds to complete!
WITH US_PRICETYPES (PRICETYPE) AS (
    SELECT DISTINCT usprice.PRICETYPE FROM WWPRT.PRICE_CN usprice
    JOIN WWPRT.CONF_PRICETYPE pt ON pt.ID = usprice.PRICETYPE
    LEFT OUTER JOIN WWPRT.CONF_GAP gap ON pt.DOMAIN = gap.DOMAIN
    WHERE usprice.CABLEID = 'USCABF3FVT01'
    AND gap.DOMAIN IS NOT NULL
)
SELECT COUNT(*) FROM NEW TABLE
(INSERT INTO WWPRT.CABLEPRODUCT_PRICETYPE_JOIN_CN (CABLEPRODUCTID, PRICETYPE, SLAVE) (
     SELECT cableproduct.ID, pricetypes.PRICETYPE, 'N'
     FROM WWPRT.CABLE_PRODUCT_JOIN_CN AS cableproduct,
     US_PRICETYPES AS pricetypes
     WHERE CABLEID = 'GEOANNF3FVT1'
     AND EXISTS (
         ......
     )
))


The improvement is awesome! From 10 hours to 10 seconds! Although it happens in DB2, but I think this case should also exist in other types of database, but not verified.

--------------------------------------------------------------------------------------
Feedback from Daniel, thank you very much:

If you don't care the result of how many records inserted, I think you can use 'FETCH FIRST 1 ROWS ONLY' instead of count(*).

SELECT CABLEPRODUCTID FROM NEW TABLE
(INSERT INTO WWPRT.CABLEPRODUCT_PRICETYPE_JOIN_CN (CABLEPRODUCTID, PRICETYPE, SLAVE) (
     SELECT cableproduct.ID, pricetypes.PRICETYPE, 'N'
     FROM WWPRT.CABLE_PRODUCT_JOIN_CN AS cableproduct,
     US_PRICETYPES AS pricetypes
     WHERE CABLEID = 'GEOANNF3FVT1'
     AND EXISTS (
         ......
     )
)) FETCH FIRST 1 ROWS ONLY


Monday, April 20, 2009

Rock's first outing

Today, I play in the park first time with my daddy and mama.











Wednesday, April 15, 2009

An Derby Properties and Connection URL Example

Have been using Derby for a long time, which is a very good choice as memory database or embedded database, let me backup some useful Derby DB Properties here:

Developer can use a derby.properties file in the application classpath or javadb classpath to configure the database, these are some configuration examples in the file:

---------------------------- Derby derby.properties information ----------------------------
--### Derby System Home Directory
derby.system.home=D:/workspace/Boulevard/BoulevardDB

--### Derby Authentication Configuration and Turn On
derby.connection.requireAuthentication=true
derby.authentication.provider=BUILTIN
derby.database.propertiesOnly=true

--### User/Password Definition (System Level)
derby.user.alfred=alfred

--### User Authorization Definition (System Level)
derby.database.defaultAccessMode=fullAccess
derby.fullAccessUsers=cube

--### Some Others Configuration
derby.infolog.append=true
derby.storage.pageSize=8192
derby.storage.pageReservedSpace=60
derby.database.defaultConnectionMode
derby.locks.deadlockTimeout=20
derby.locks.waitTimeout=30
---------------------------- Derby derby.properties information ----------------------------


You have 2 ways to start the database:

1. Start embedded Derby from java application. The database is active from the first time you connect to it, and become inactive when your java application stop; and any other application outside of the same JVM can not access the database instance.

To connect to embedded Derby by URL:
------------------------------- Derby connection information -------------------------------
-- ### Use ij command to connect embeded database; if not exist, create it.
CONNECT 'jdbc:derby:data;create=true;user=alfred;password=alfred' AS CUBE;
-- ### Use ij command to connect embeded database; if not exist, don't create it, throw out error.
CONNECT 'jdbc:derby:data;user=alfred;password=alfred' AS CUBE;
------------------------------- Derby connection information -------------------------------


2. Start Derby network server. The network server instance can only be started by command "...\javadb\bin\startNetworkServer.bat", and it runs as a server which will not stop when your application stops; and other applications in any JVM can also access the database locally or remotely.

To connect to Derby network server:
------------------------------- Derby connection information -------------------------------
-- ### Use ij command to connect network server database, reading default repository;
-- ### If not exist, create it.
CONNECT 'jdbc:derby://localhost:1527/data;create=true;user=alfred;password=alfred' AS BOULEVARD;
-- ### Use ij command to connect network server database; reading default repository;
-- ### If not exist, don't create it, throw out error.
CONNECT 'jdbc:derby://localhost:1527/data;user=alfred;password=alfred' AS BOULEVARD;
-- ### Use ij command to connect network server database, reading specific repository;
-- ### If not exist, create it.
CONNECT 'jdbc:derby://localhost:1527/D:/workspace/Boulevard/BoulevardDB/data;create=true;user=alfred;password=alfred' AS BOULEVARD;
-- ### Use ij command to connect network server database, reading specific repository;
-- ### If not exist, don't create it, throw out error.
CONNECT 'jdbc:derby://localhost:1527/D:/workspace/Boulevard/BoulevardDB/data;user=alfred;password=alfred' AS BOULEVARD;
------------------------------- Derby connection information -------------------------------



Let's rock without Microsoft

Today, Microsoft formally announced to stop the major technical support for Windows XP, and the extension support will stop before 2014. Are you using Window XP or Vista? or Window 7? Do you feel the Windows is your only choice as the OS on your machine?

I am imaging a world without Microsoft in a short future. At least three major trends can lead to a new world without Microsoft:

1. Cloud computing, Virtual Infrastructure Access service, all these new technologies show an approach to compute and run applications without a complex local OS. More and more applications are deployed on server side, and they can be started and run remotely easily. In a short future, we only need a thin client or just only a browser in computer, cell phone, play station or something other machine like current TV station to run most of useful applications. The OS is not important any longer, while the key things are the applications or services in the network.

2. Many office applications become more and more mature, such as Google Apps including Gmail, Document and Calendar, IBM's Lotus Notes and Symphony, Sun's OpenOffice. Actually these software are powerful enough to totally replace the Microsoft Office now, and these software are more open and flexible through the open API library.

3. Even for the OS itself, the Windows is challenged by many competitors. Ubuntu is very nice and more and more powerful Linux system for personal usage. And these days, you know, most of eyes are focused on some new operation systems in mobile phone, like Google Android, Apple iPhone and Nokia Symbian. This is another new and more important war field than traditional computer desktop, as more and more applications and services are provided through network in future, but Microsoft is a loser in this field.

Less Microsoft, more wonderful world! Let's rock without Microsoft!

For now, I am also using Windows as it's required by company, but I believe when Microsoft stops all the support for Windows XP, I bet that the company will embrace Linux, for example Ubuntu, instead of paying money for next Windows version. These are some major applications for daily work in my computer:
  - Using Firefox as browser, instead of IE; It's faster, and there are many Firefox Add-ons to make the browser powerful, but IE can not;
  - Using Lotus Symphony and OpenOffice to create and edit document, spreadsheet or presentation materials, instead of using Microsoft Office. And these software can also easily create and edit Microsoft Office format document for customer, but Microsoft Office can not support OpenOffice format;
  - Using Lotus Notes (v8) to communicate with others. It integrated To Do List, Calendar, Email, Contacts, Instant Messging, Feed and many other useful functions together. Instead of using the Microsoft Outlook;
  - Using Lotus Sametime for online conference, instead of NetMeeting which function is too limitted and very slow;
  - Using Rational RAD/RSA for software design/development, and RTC for working planning, tracking, and for team communication and collaboration;
  - Using Twitter and Blue Community to build up my social network;
......

And the most important is all above applications can run well in Linux system, even better. If it was not required by company, I would like to use Ubuntu. I installed it which runs very well.