From the Trenches 3 | Patching OSB and SOA Suite to PS5

In two previous blogs (part I and part II) I talked about a recent upgrade from Oracle Fusion Middleware 11g PS2 to 11g PS5. This blog finishes with a post-installation step that was required to complete the upgrade and a quick note.

All in all, after a good preparation the team was able to patch a production environment to PS5 in almost an afternoon. The environment runs several Fusion Middleware products (OSB, SOA Suite, OID, WebLogic Servers for JEE apps) and spans multiple WebLogic Domains and multiple WebLogic Managed Servers in single-node clusters.

Migrating running composite instances that use While activities

One of the new features in SOA Suite 11g PS3 is that the Enterprise Manager Fusion Middleware Console shows the loop count in the BPEL Audit Trail when using While activities. This is shown in the following figure.

This feature works for new instances that are created after applying the patchset and for running instances that haven't reached the While activity yet when the patchset is applied. However, when you have running instances that are executing a While activity (also when the instance is in the dehydration store), you see the following entry in the SOA Suite log after applying the patchset:

ORABPEL-02118

Variant not found.
The variable "__loopCondition" is not declared in the current scope. All variables must be declared in the scope before being accessed. This was an internal error. The flow was not generated correctly by the BPEL compiler.

The audit trail will show the following message:

The transaction was rolled back. The work performed for bpel instance "xxx" was rolled back to the previous dehydration point, but the audit trail has been saved. You can recover the instance from the recovery console by resubmitting the callback message or activity for execution.

It seems SOA Suite uses an internal variable to count the number of times a loop is executed. Probably, this variable is not initialized when using a SOA Suite 11g version predating PS3 and thus results in the error.

There are two solutions to this:

• Validate there are no running instances that execute a While loop while patching your environment;
• If not, use the patch as indicated in patch reference 1451018.1 at the Oracle Support Site.

After applying the patch and restarting SOA Suite you will notice the following log entry indicating the loop count exception is handled by SOA Suite:

<Warning> <oracle.soa.bpel.engine> <BEA-000000> <Handling Exception on setting __loopCondition. This is expected for in-flight instances created before SOA upgrade from PS3 or older versions. 

Default Human Worklist Application and OWSM

The out-of-the-box Human Worklist Application is configured to use the oracle/no_authentication_service_policy policy of Oracle Web Services Manager (OWSM). This policy became available in later patchsets, but isn't shipped with SOA Suite and OWSM PS2. Without updating your OWSM policy store you'll see the following error when logging into the Human Worklist Application after applying PS5:

An error occurred for port: {http://xmlns.oracle.com/bpel/services/IdentityService} IdentityServicePort: oracle.fabric.common.PolicyEnforcementException: PolicySet Invalid: WSM-06162 PolicyReference The policy referenced by URI "oracle/no_authentication_service_policy" could not be retrieved

This can be fixed by updating the OWSM policies using the upgradeWSMPolicyRepository() command. This is documented in Upgrading the Oracle WSM Policies in the Repository. Make sure the Managed (or Admin) Server on which the OWSM PM application is deployed is running. Otherwise you'll see an error indicating that an MBean that is used for the policy upgrade cannot be reached.

See the Oracle Fusion Middleware Security and Administrator’s Guide for Web Services 11g Release 1 (11.1.1.6) for more information on OWSM and its policies.

From the Trenches | Patching OSB and SOA Suite to PS5

Recently I was involved in an Oracle Fusion Middleware upgrade from 11g PS2 to 11g PS5 with Jacco Landlust, Aiman Salama, and Jens Peters. The environment that was patched consists of the following domains:

• Java domain running Java/JEE applications;
• IDM domain running identity management components including OID;
• SOA domain running Oracle Service Bus and Oracle SOA Suite. 

The domains consist of several Managed Servers in a single-node cluster configuration. OSB and SOA Suite both run in their own Managed Server.

There are plenty of excellent blogs that discuss the infrastructure and middleware side of such an upgrade. This blogs contains some things we encountered from the application side of things. More specific, the SOA composites and OSB projects.

Rebuild custom Java classes and JAR files in the SOA extension library

You can add custom Java classes and JAR files to SOA Suite that are used by your SOA composites. The SOA extension library for adding extension classes and JARs is available in the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 directory. For example, an extension can be used to add a custom fault handler.

After applying the patchset you need to rebuild and package the JAR files by running Ant in the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 directory:

$ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1>ant
Buildfile: build.xml

create-manifest-jar:
     [echo] Creating oracle.soa.ext at $ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/oracle.soa.ext.jar :$ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/MyCustomFaultHandler.jar;$ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/classes
      [jar] Updating jar: $ORACLE_HOME/soa
/modules/oracle.soa.ext_11.1.1/oracle.soa.ext.jar

BUILD SUCCESSFUL
Total time: 0 seconds 

Reconfiguring OWSM policies

The Oracle Service Bus is used to expose services to the outside world in this environment. These services are secured using Oracle Web Services Manager (OWSM). Services can only be invoked over SSL and the SOAP requests require a WS-Security token for authentication. Among others, these services are invoked by SOA composites that run on SOA Suite and a Silverlight/.NET Portal application. The SOA composites apply the oracle/wss_username_token_over_ssl_client_policy policy to enforce SSL and add the required WSS token.

Due to security integration issues between the Silverlight/.NET Portal and the Oracle Service Bus the service provider doesn't include a timestamp on the transport layer (SSL). The include timestamp setting of the oracle/wss_username_token_over_ssl_client_policy policy was therefore disabled.

After the upgrade we noticed errors in the log files when the SOA composites invoked the Web Services exposed by the OSB: the SSL endpoints of the Web Services could not be reached. Although it seemed the Web Service was actually invoked, and the same SSL endpoint was reachable from a browser. To verify what went wrong we enforced the security/logging assertions of the policy. This causes the outbound SOAP call to be logged to the OWSM logfile: once before applying the policy (without WS-Security tokens) and once after applying the policy (message including the WS-Security tokens). The log showed that the SOAP request contained the correct WSS headers, and that a valid SOAP response was returned by the Web Service. So the Web Service call actually took place. However, the OWSM client policy returned an error that the SSL endpoint was unreachable. It seemed the timestamp setting was activated again by the patch causing a validation error on transport level. Disabling the include timestamp setting fixed the problem.

See Securing Heterogeneous Systems Using Oracle Web Services Manager on Slideshare for more info on this subject.

Using the default SOA Suite XSDs from MDS

SOA Suite supplies out-of-the-box XSDs for interacting with Human Workflow, using Sensors, catch BPEL-specific faults, and so on. These XSDs are preseeded in MDS: both in the MDS schema created by RCU and in JDeveloper after installing the SOA extension in $JDEVELOPER/jdeveloper/integration/seed/soa/shared. Applying the patch will upgrade some of these XSDs. It is therefore important to refer to those XSDs in your SOA composites using MDS. Don't copy and package these XSDs locally inside your composites. Doing so results in errors since SOA Suite detects discrepancies between the packaged SOA Suite XSDs (old patchset), and the same XSDs in MDS (that are updated by applying the patch).

Invoking the TaskService.acquireTask operation 

The Portal application retrieves Human Tasks by invoking the out-of-the-box SOA Suite APIs. When tasks are acquired by users, authentication information needs to be passed to the TaskService.acquireTask operation. A required token is obtained by invoking the TaskQueryService.authenticate operation. The SOAP request that is generated by the Portal includes both the user and onBehalfOfUser as part of the authentication information. While this worked in PS2, in PS5 this resulted in an authentication error indicating that the user had too few privileges to authenticate on behalf of another user. The issue is resolved by removing the onBehalfOfUser element in the SOAP request. Since the SOA Suite APIs are exposed through the OSB in this environment we could alter the request pipeline of the OSB routing flow to delete this element as a temporary solution. A better solution is to upgrade the Portal so the SOAP request doesn't include the onBehalfOfUser element altogether.

Workflow Notification settings

The SOA composites use the Workflow Notification capabilities to send (actionable) emails and email alerts for Human Worfklow. After applying the patch the following error occurs:


<Feb 7, 2013 12:04:38 PM CET> <Warning> <oracle.soa.services.notification> <BEA-000000> <<.> With the current setting, only Email notifications will be sent; Notifications via voice, SMS or IM will not be sent. If you would like to enable them, please configure corresponding sdpmessaging driver. Then modify the accounts and set NotificationMode attribute to ALL in workflow-notification-config.xml> 
<Feb 7, 2013 12:04:39 PM CET> <Error> <oracle.soa.services.notification> <BEA-000000> <<.> Incorrect incoming Notification Configuration.
Incorrect Incoming Configuration : AccountName : Default.
The AccountName in workflow-config.xml should be one of the names used in workflow-notification-config.xml.

ORABPEL-31031

Incorrect incoming Notification Configuration.
Incorrect Incoming Configuration : AccountName : Default.
The AccountName in workflow-config.xml should be one of the names used in workflow-notification-config.xml.

Reconfigure the Workflow Notification settings in Enterprise Manager to solve this issue.

Summary

All in all, the Oracle Fusion Middleware upgrade of all domains was done in a few days. We had some excellent help from Oracle Support for one issue (config.xml with an incorrect Coherence configuration after the patching) that was quickly resolved. Just as any project, good preparation is half the battle. Make sure to read and follow the Oracle patching guides and use the tips & tricks from the available blogs. Be sure to backup the environment, including the OFM database schema's such as MDS and SOAINFRA, and test the upgrade on a production look-a-like test environment. Even if all consoles such as the Service Bus Console and Enterprise Manager work correctly, also inspect the log files for any error. Finally, in case you have longer running SOA composites, make sure you have a testset of open and running composite instance before patching the environment for regression testing.

DOAG 2012

This week the German Oracle User Group, or DOAG as it is called in German, held their yearly conference. Like other years, the location was the conference center in Nuremberg, a beautiful city in the south.

Vennster was well represented in the SOA/BPM Space, we did the following sessions:

• SOA Made Simple: service design (Ronald van Luttikhuizen)
• SOA Made Simple: creating a roadmap for your SOA (Lonneke Dikmans) 
• Effective Fault Handling in Oracle SOA Suite 11g (Ronald van Luttikhuizen)
• Introduction in Eventing in SOA Suite 11g (Ronald van Luttikhuizen)
• Using the B2B Adapter in a Dutch government project (Ronald van Luttikhuizen)
• Securing heterogeneous systems using Oracle WebServices Manager (Ronald van Luttikhuizen and Jens Peters)
• Deployment in Oracle SOA Suite and Oracle BPM Suite (Lonneke Dikmans)
• Stop generating your User Interface! Start designing it (Lonneke Dikmans)

You can find the slides by Ronald and me on slideshare:

• http://www.slideshare.net/lonneked
• http://www.slideshare.net/rluttikhuizen

Of course there were also other presentations by other presenters ;) DOAG is a big conference, with over 400 presentations. Most of them cover cases, others explain the latest developments. There is a number of tracks that are of interest if you are working in the 'middleware space': BPM, Middleware & SOA, development, Java and Strategy and Business.  The English spoken sessions are not as popular as the German language sessions, but both are well visited. 

I visited three sessions, one case study titled "Dynamische Benutzer-Workflows mit SOA und BPM-Suite" by Arne Brüning, one about the new developments in EclipseLink called "The Evolution of Java Persistence" by Doug Clarke and the last one was a session titled "NoSQL and SQL: Blending the Best of Both Worlds" by Andrew Morgan. All three happened to be presented by Oracle. They were very different in nature. The workflow session discussed a customer case. It was interesting from that point of view. I would have preferred more technical depth, but the presenter was well prepared and had an interesting story to tell. The session by Doug about Eclipse gave a nice overview of the latest developments and put them into perspective of the history of TopLink and EclipseLink. I think that this is a good strategy: it shows that EclipseLink is both proven and modern: it has been around for years and part of the original team is still working there PLUS they have solutions for new developments like JSON, REST services, NoSQL and multi-tenancy. The final presentation was an example how not to do that. The presenter put NoSQL in the title in an attempt to attract a crowd. But the session was really about MySQL clusters. A lot of people left the session while he was talking, because it was completely off topic. The presentation itself was not bad, but the title was misleading.  

Unfortunately I did not have time to see more sessions, because of all the presentations we were doing ourselves. There certainly was a lot more I would have liked to listen to and I hope we will be back next year!

Security-per-environment using config plans

Software artifacts normally flow through several environments; for example development, test, integration, acceptance, and production. Some piece of software may be developed locally on your laptop, can be deployed to a central test environment by a developer, scheduled for deployment to the integration environment by the build manager, and then formally promoted to acceptance and production by system administrators after successful testing. These various environments rarely look the same. While a production environment might consist of a clustered and load-balanced configuration with multiple servers running on Linux, your development environment may consist of a bunch of laptops all running a single integrated server on Windows.

Not only sizing, server versions, hardware specs, and OS specifics can vary between these environments, also security configuration. It could be that the production environment enforces SSL/TLS for all internal and external Web Service calls, only uses official certificates issued by a trusted CA, applies WS-Security based message encryption for outbound Web Service calls, uses WS-Security SAML Tokens for authentication, and an appliance for SSL offloading instead of the application server itself. Maybe in the integration environment self-signed certificates and WS-Security UserName Tokens are used, while the development environment enforces no security measures whatsoever.

One way to deal with environment-specific settings is to manually configure these settings while promoting software items from one environment to the other. Maybe this is manageable for a small and simple application, this approach will soon be error-prone and high-maintenance when the size of the application increases. This is especially the case in SOA systems, where there is a high(er) number of different software components involved. A different approach is to use scripting and automated builds in which settings can be configured per environment.

Oracle SOA Suite 11g uses so-called config(uration) plans for this purpose. Environment-specifics like Web Service endpoint locations invoked by an SCA composite can be extracted from the SCA composite and stored in an XML config plan per environment. So we could have a MyComposite_cfgplan_dev.xml that indicates the endpoint of MyWebService is located at http://localhost:7001/MyWebService, while the endpoint is configured as https://some-server:8011/external/MyWebService in the MyComposite_cfgplan_prod.xml config plan for the production environment.

Oracle Web Service Manager (OWSM) is used by Oracle SOA Suite to secure services, references, and components of an SCA composite. You can for example apply the out-of-the-box oracle/wss_username_token_client_policy enforcing a WSS UserName Token to be included in the invocation of an external Web Service. While there are numerous examples online that explain the use of config plans and other examples explaining design-time addition of OWSM policies using JDeveloper, there is less information on how to include and configure OWSM-specific settings in config plans for SOA Suite 11g.

Adding OWSM policies to an SCA composite at designtime in JDeveloper

Adding OWSM policies to an SCA composite at designtime in JDeveloper

OWSM policies can be applied and configured per reference, service, or component. When applying security to a reference and configuring it for a specific environment, the OWSM settings need to be placed as wsp:PolicyReference element between the attribute and property elements of the reference in the config plan. The wsp prefix refers to the http://schemas.oracle.com/ws/2006/01/policy namespace.

For example (snippet from a configuration plan):


<reference name="MyExternalService">
  <binding type="ws">
    <attribute name="location">
      <replace>http://server:8011/SomeService-1.3?wsdl</replace>
    </attribute>
    <wsp:PolicyReference 
      orawsp:category="security" 
      orawsp:status="disabled"    
      URI="oracle/wss_username_token_client_policy"/>
    <property name="csf-key">
      <replace>BPMS_USER</replace>
    </property>
  </binding>
</reference>


In the above example, the oracle/wss_username_token_client_policy is applied on the MyExternalService reference but disabled during deployment. It furthermore indicates that the credentials stored in the BPMS_USER key that is located on the server should be used in constructing the WS-Security UserName Token for the outbound SOAP call. While this may be an appropriate configuration for the test environment, for production you might want the policy to be enforced during deployment, perhaps use another policy that enforces SSL, and possibly use another CSF key. In order to do this, you can add these OWSM-specifics to the reference element in the configur plan for production.

For example (snippet from a configuration plan):

<reference name="MyExternalService">

  <binding type="ws">

    <attribute name="location">

      <replace>http://prodserver:8012/Service-1.3?wsdl</replace>

    </attribute>

    <wsp:PolicyReference 

      orawsp:category="security" 

      orawsp:status="enabled"    

      URI="oracle/wss_username_token_over_ssl_client_policy"/>

    <property name="csf-key">

      <replace>PROD_PROXY_USER</replace>

    </property>

  </binding>

</reference>