From the Trenches | Patching OSB and SOA Suite to PS5

Recently I was involved in an Oracle Fusion Middleware upgrade from 11g PS2 to 11g PS5 with Jacco Landlust, Aiman Salama, and Jens Peters. The environment that was patched consists of the following domains:

• Java domain running Java/JEE applications;
• IDM domain running identity management components including OID;
• SOA domain running Oracle Service Bus and Oracle SOA Suite. 

The domains consist of several Managed Servers in a single-node cluster configuration. OSB and SOA Suite both run in their own Managed Server.

There are plenty of excellent blogs that discuss the infrastructure and middleware side of such an upgrade. This blogs contains some things we encountered from the application side of things. More specific, the SOA composites and OSB projects.

Rebuild custom Java classes and JAR files in the SOA extension library

You can add custom Java classes and JAR files to SOA Suite that are used by your SOA composites. The SOA extension library for adding extension classes and JARs is available in the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 directory. For example, an extension can be used to add a custom fault handler.

After applying the patchset you need to rebuild and package the JAR files by running Ant in the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 directory:

$ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1>ant
Buildfile: build.xml

create-manifest-jar:
     [echo] Creating oracle.soa.ext at $ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/oracle.soa.ext.jar :$ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/MyCustomFaultHandler.jar;$ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/classes
      [jar] Updating jar: $ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/oracle.soa.ext.jar

BUILD SUCCESSFUL
Total time: 0 seconds 

Reconfiguring OWSM policies

The Oracle Service Bus is used to expose services to the outside world in this environment. These services are secured using Oracle Web Services Manager (OWSM). Services can only be invoked over SSL and the SOAP requests require a WS-Security token for authentication. Among others, these services are invoked by SOA composites that run on SOA Suite and a Silverlight/.NET Portal application. The SOA composites apply the oracle/wss_username_token_over_ssl_client_policy policy to enforce SSL and add the required WSS token.

Due to security integration issues between the Silverlight/.NET Portal and the Oracle Service Bus the service provider doesn't include a timestamp on the transport layer (SSL). The include timestamp setting of the oracle/wss_username_token_over_ssl_client_policy policy was therefore disabled.

After the upgrade we noticed errors in the log files when the SOA composites invoked the Web Services exposed by the OSB: the SSL endpoints of the Web Services could not be reached. Although it seemed the Web Service was actually invoked, and the same SSL endpoint was reachable from a browser. To verify what went wrong we enforced the security/logging assertions of the policy. This causes the outbound SOAP call to be logged to the OWSM logfile: once before applying the policy (without WS-Security tokens) and once after applying the policy (message including the WS-Security tokens). The log showed that the SOAP request contained the correct WSS headers, and that a valid SOAP response was returned by the Web Service. So the Web Service call actually took place. However, the OWSM client policy returned an error that the SSL endpoint was unreachable. It seemed the timestamp setting was activated again by the patch causing a validation error on transport level. Disabling the include timestamp setting fixed the problem.

See Securing Heterogeneous Systems Using Oracle Web Services Manager on Slideshare for more info on this subject.

Using the default SOA Suite XSDs from MDS

SOA Suite supplies out-of-the-box XSDs for interacting with Human Workflow, using Sensors, catch BPEL-specific faults, and so on. These XSDs are preseeded in MDS: both in the MDS schema created by RCU and in JDeveloper after installing the SOA extension in $JDEVELOPER/jdeveloper/integration/seed/soa/shared. Applying the patch will upgrade some of these XSDs. It is therefore important to refer to those XSDs in your SOA composites using MDS. Don't copy and package these XSDs locally inside your composites. Doing so results in errors since SOA Suite detects discrepancies between the packaged SOA Suite XSDs (old patchset), and the same XSDs in MDS (that are updated by applying the patch).

Invoking the TaskService.acquireTask operation 

The Portal application retrieves Human Tasks by invoking the out-of-the-box SOA Suite APIs. When tasks are acquired by users, authentication information needs to be passed to the TaskService.acquireTask operation. A required token is obtained by invoking the TaskQueryService.authenticate operation. The SOAP request that is generated by the Portal includes both the user and onBehalfOfUser as part of the authentication information. While this worked in PS2, in PS5 this resulted in an authentication error indicating that the user had too few privileges to authenticate on behalf of another user. The issue is resolved by removing the onBehalfOfUser element in the SOAP request. Since the SOA Suite APIs are exposed through the OSB in this environment we could alter the request pipeline of the OSB routing flow to delete this element as a temporary solution. A better solution is to upgrade the Portal so the SOAP request doesn't include the onBehalfOfUser element altogether.

Workflow Notification settings

The SOA composites use the Workflow Notification capabilities to send (actionable) emails and email alerts for Human Worfklow. After applying the patch the following error occurs:


<Feb 7, 2013 12:04:38 PM CET> <Warning> <oracle.soa.services.notification> <BEA-000000> <<.> With the current setting, only Email notifications will be sent; Notifications via voice, SMS or IM will not be sent. If you would like to enable them, please configure corresponding sdpmessaging driver. Then modify the accounts and set NotificationMode attribute to ALL in workflow-notification-config.xml> 
<Feb 7, 2013 12:04:39 PM CET> <Error> <oracle.soa.services.notification> <BEA-000000> <<.> Incorrect incoming Notification Configuration.
Incorrect Incoming Configuration : AccountName : Default.
The AccountName in workflow-config.xml should be one of the names used in workflow-notification-config.xml.

ORABPEL-31031

Incorrect incoming Notification Configuration.
Incorrect Incoming Configuration : AccountName : Default.
The AccountName in workflow-config.xml should be one of the names used in workflow-notification-config.xml.

Reconfigure the Workflow Notification settings in Enterprise Manager to solve this issue.

Summary

All in all, the Oracle Fusion Middleware upgrade of all domains was done in a few days. We had some excellent help from Oracle Support for one issue (config.xml with an incorrect Coherence configuration after the patching) that was quickly resolved. Just as any project, good preparation is half the battle. Make sure to read and follow the Oracle patching guides and use the tips & tricks from the available blogs. Be sure to backup the environment, including the OFM database schema's such as MDS and SOAINFRA, and test the upgrade on a production look-a-like test environment. Even if all consoles such as the Service Bus Console and Enterprise Manager work correctly, also inspect the log files for any error. Finally, in case you have longer running SOA composites, make sure you have a testset of open and running composite instance before patching the environment for regression testing.