To Pull or to Push

In a recent project various designs were discussed to integrate business processes implemented in Oracle BPM with a Java front-end application. More specifically, how status changes in these processes could be communicated to the end user via the front-end. The possible scenarios quickly boiled down to a poll versus push approach and various in-between scenarios. This blog lists the various approaches and design considerations.

Case

The custom-built Java application provides a thin and stateless User Interface that is used by local branch employees to sell products to customers. The process of selling these products is coordinated through a business process that is implemented in Oracle BPM 11g. The configuration of the products for a particular customer is done by the employee through the Java application. After the configuration of the product is completed, the Java application sends a signal to the process. The process will initiate various automated tasks such as credit checking. In the meantime the employee waits until these tasks are executed, then views the results of those checks and continues with the completion of the order.

The question we discussed is how to inform the employee, though the Java application, that the checks are executed by the process and the employee can move on. We know the milestone in the process is reached through a combination of the occurrence of a particular Human Task and the process state itself.

Solution based on polling

In a polling approach, the Java front-end application polls for the process status and workflow status every few seconds using the out-of-the-box Oracle BPM and Human Workflow APIs. The client applications poll to see if process instances have reached a particular milestone; if so the application moves forward. Since the Java application is stateless and has no server side, the custom application directly polls from every client session and not from a central web server. This means every new client will increase the load on the BPM platform. Stress tests showed that in this scenario the OBPM engine would be mostly busy handling polling requests and couldn’t accommodate for a high degree of concurrently running process instances.

The basic polling approach is simple to implement from the perspective of the Java front-end, but not very scalable.

We could consider optimizing the polling. For example to use a Coherence Grid or Service Bus Result Cache as caching layer between the Java clients and the Oracle BPM APIs and have the clients poll the cache instead of the APIs directly. Upside is that such a cache is easily created and that we limit the number of requests on the backend BPM engine by centralizing the polling requests and using a default caching mechanism. Downside is that the polling results can be outdated when the actual process state has changed and the process status data in the cache is not yet invalidated.

Solution based on pushing

The Java front-end clients are interested in some status change. For this purpose, a natural approach would be to push an event from the process to the Java clients when that particular milestone is reached. The preferred choice in this scenario would be to use JMS for this since this standard is both supported by Java as well as Oracle BPM 11g through the use of JCA Adapters. The pro of this approach is that it is much more scalable. We only perform an activity (publish an event) when needed, we don’t do useless polling. Downside however in this case is that the front-end application doesn’t support event subscription using JMS and this would require a significant change in that application.

Approach

We used a three step approach:

• Short-term. Expose a new service that encapsulates the polling logic. Implement this service on the Service Bus and use Result Caching to cache the polling results. This service is easily created and alleviates the back-end usage of OBPM APIs. The business needs to accept that results can be outdated in some scenarios as a trade-off for the increased load that can be supported.
• Middle-term. Push events from the BPM process using Mediator and a JCA JMS Adapter when a certain milestone is reached and store the event data in a custom table. Have the Java front-end poll on the custom table. In this approach the process uses events, which is the desired approach. We only need a small change to the Java front-end to poll on the custom table instead of the OBPM engine itself, thereby offloading the engine. The Java front-end however still incurs overhead by using a polling approach. 
• Long-term. Change the custom Java application to support event subscription, for example using JMS. This way the scalability of the front-end application will improve since the number of events is much lower than the number of polling requests. Also, on average the front-end application knows about reaching the milestone faster using events, than it does when periodically polling for that information. 

Tracking progress of your BPEL process using sensors

Often when you start using Oracle SOA Suite11g and BPEL you need a mechanism to help the end users keep track of the progress of the overall process instances. The EM shows this to administrators, but this is not suitable for end users.

What about the worklist application?

The SOA Suite offers the worklist application to handle tasks and to view progress using views and reports. It shows human tasks, crossing multiple process definitions. It does not show invocation of services or specific data changes in the process. The figure below shows three instances of one specific process definition. In this example relevant milestones are reached when the process starts, the first automated step is executed, the second human task is executed and when the process ends. In the worklist application you would be able to see what human tasks are open or executed by which user. With the worklist application you can't keep track of the first, second and last milestone, only of the third one because that is the only one associated with a human task. 

Sensors and sensor actions

The required functionality can be accomplished using sensors and sensor actions on relevant parts in the BPEL process. There are different types of sensors that can be used in SOA Suite 11g: activity sensors, variable sensors and fault sensors.

After you have configured the sensor in your BPEL process, you need to define one or more sensor actions. There are four types of actions: three sensor actions and one BAM sensor action:

• Database to publish it to the BPEL dehydration store
• JMS to publish it to a JMS topic or queue. If the JMS provider is not local, you can use the JMS adapter
• Custom Java class to handle it in a different way. 
• BAM Sensor actions to publish them to Oracle BAM.

You can chose to publish the sensor to AQ, so you can handle it both with Java clients and with PL/SQL because JMS queues in WebLogic can be implemented using AQ. 

Displaying the information with BAM or a custom application

There are two ways to show the progress to the end user: using BAM or building a custom application on top of the sensor data. You can easily decide to implement BAM later: you can either add BAM sensor actions to the BPEL sensor, or define a Enterprise Message Source in BAM. 

Option 2 (defining an Enterprise Message Source) has the advantage of not having to change the BPEL process (design time) when you decide to start using BAM rather than or on top of the custom application. 

Steps

Now that we have established how we want to implement the functionality, we can get started :) The following steps need to be taken:

1. Define the relevant milestones you want to show to the user;
2. Configure the sensors;  
3. Configure the sensor actions;
4. Create the database user and queues for AQ;
5. Configure WebLogic Server and add JMS objects;
6. Create PL/SQL code to read from the queue and store it in the table;
7. Deploy the BPEL process;

In parallel you can build the GUI or configure BAM, depending on what you decided upon. 

Resources

• Oracle® Fusion Middleware Developer's Guide for Oracle SOA Suite11g Release 1 (11.1.1.7), chapter 18;
• Oracle® Fusion Middleware Developer's Guide for Oracle SOA Suite11g Release 1 (11.1.1.7), part X;
• Oracle® Fusion Middleware Configuring and Managing JMS for Oracle WebLogic Server11g Release 1 (10.3.6)

Supporting multiple ebMS service versions in Oracle B2B

In recent years I've been involved in different Oracle B2B implementations and troubleshooting projects based on ebMS and AS2 protocols. Not so long ago, I came across a project that needed to support multiple versions of an ebMS service. Versioning is quite trivial for SOAP and RESTful Web Services using e.g. different namespaces, version indicators in endpoint locations, and so on. However, it is a bit different for ebMS-based services implemented in a B2B gateway.

Case

ebMS is one of the protocols supported by the Dutch government standard DigiKoppeling for exchanging information between governments such as municipalities, provinces, and departments. In this particular case, the existing integration flow that needed to support a new version of an ebMS service is as follows:

OLO (Omgevingsloket Online) is a government solution where companies and citizens can apply for permits. OLO routes those permit requests, using the ebMS protocol, to the applicable government agency based on the type of permit. In this case Oracle B2B is used by one of these government agencies to process these permit requests. B2B transforms and routes the inbound messages to Oracle Service Bus using JMS, while OSB forwards the requests to a backend application by invoking a SOAP Web Service. Vice versa, outbound messages are transformed by B2B to ebMS and send to OLO. You can read more about this case in a presentation I gave at OUGN.

Problem

OLO offers different versions of its service, among others the StUF LVO 3.05 and StUF LVO 3.11 version. The challenge was to add support for StUF LVO 3.11 while maintaining support for StUF LVO 3.05. In OSB we simply added a new service according to the StUF LVO 3.11 specification, alongside the existing service that processes 3.05 messages. The next section describes the solution for the B2B configuration.

Solution

First step is to define a new "Document Protocol Version" of type "Custom" in Oracle B2B and add the 3.11 "Document Types" to it. For StUF LVO 3.05 we already created a Document Protocol Version containing all of the 3.05 document types.

Next we add the 3.11 document types to the two existing trading partners: OLO and the government agency that processes the permit requests. We also specify if the document can be send or received by that particular trading partner.

To complete the B2B configuration we define the agreements. Here we can make sure that service versioning is supported:

Inbound messages from OLO to B2B to OSB

• ebMS requires a design-time contract between the service provider and the service consumer. This contract is called CPA (Collaboration Protocol Agreement) and CPP (Collaboration Protocol Profile). For the new version of the service that supports 3.11 a new CPA was created. This CPA has a different CPA identifier than the CPA we created for the 3.05 version. By assigning this new CPA identifier to the newly added agreements using the "Agreement Id" attribute, B2B can distinguish between 3.05 and 3.11 messages.
• Messages from B2B to OSB are delivered on a JMS queue using a partner channel that is defined for the government agency in B2B. Here we define a new JMS channel that points to a different queue, for example the StUF_LVO_311_Queue. The new OSB service that processes 3.11 messages is listening on this queue, while the existing service processes messages from the 3.05 queue.

Outbound messages from OSB to B2B to OLO

OSB can integrate with B2B by setting specific JMS headers that B2B uses to correlate JMS messages to specific agreements. One of these headers is the DOCTYPE_REVISION attribute. In the new OSB service, we can simply set the header value to the Document Protocol Version that was configured for the 3.11 messages. Now B2B receives both 3.05 and 3.11 messages but is able to differentiate between them using the header properties.

Summary

I'm not the biggest fan of ebMS, but if you need support for it then Oracle B2B is a good product. The ebMS protocol is a point-to-point protocol build upon SOAP. Due to its point-to-point nature and complexity, the ebMS protocol isn't as flexible and doesn't promote reuse the way that "plain" SOAP Web Services and RESTful Web Services do.

However, if ebMS is a constraint in your project than Oracle B2B offers a good out-of-the-box solution that helps you to implement ebMS-based integrations that also support versioning. 

Eventing Hello World

This week I presented the "Introduction in Eventing in Oracle SOA Suite 11g" session at the DOAG conference in Nürnberg. I used several demos in this session to show the eventing capabilities of SOA Suite. This blog contains the source code and accompanying explanation so you can replay the demo yourself.

Introduction
An event is the occurrence of something relevant, signals a change in state that might require an action. Examples of events are: an invoice that has been paid, a customer that moved to a new address, a new purchase order, and so on. Events are complimentary to processes and services: processes and services describe what should be done, events about when something important occurs. SOA is not only about (synchronous) services and processes (what); but also about events (when). Eventing improves decoupling in your SOA landscape.

Slides
The presentation slides can be viewed and downloaded from my SlideShare account. The presentation introduces the concept of eventing, discusses some basic eventing patterns, and talks about the implementation of eventing in SOA Suite using AQ, JMS, and EDN.

DOAG 2012 - Introduction in Eventing in SOA Suite 11g from Ronald van Luttikhuizen

Code
You can download a zipfile containing the JDeveloper workspaces for the SOA Composites (DOAG2012_Eventing_SOASuite) and Java projects (DOAG2012_Eventing_Java) used in the demo here.

Prerequisites
You'll need an Oracle SOA Suite 11g runtime to deploy and run the SOA Composites. Oracle provides a pre-built SOA and BPM virtual machine for testing purposes that is easy to install.

Download JDeveloper 11g (11.1.1.6) or later from Oracle Technology Network to inspect and modify the SOA Composite and Java projects.

Setup
In order for the demos to run some plumbing needs to be done for AQ and JMS:

• The configuration steps required for the AQ demos are listed in the aq_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• The configuration steps required for the JMS demos are listed in the jms_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• In JDeveloper modify the existing Database Connection to point to your DOAG12_JMSUSER database schema.
• In JDeveloper modify the GenerateInvoice File Adapter of the Billing SOA Composite to write the invoice file to an existing directory of the SOA Suite runtime.
• In JDeveloper modify the WriteSensorToFile File Adapter of the ProcessSensors SOA Composite to write sensor values to an existing directory of the SOA Suite runtime.
• Use Enterprise Manager to create a partition in SOA Suite called doag2012_eventing.
• Deploy the following SOA Composites to the doag2012_eventing partition of the SOA Suite runtime using JDeveloper, Enterprise Manager or scripts: Order2Cash, Dunning, Billing, and ProcessSensors.
• Deploy the Java application CRM to the WebLogic Server.

Advanced Queuing (AQ)
The first demo shows the queuing capabilities of AQ and shows how to integrate SOA Composites with AQ queues:

1. Insert a record in the ORDERS table in the DOAG12_WEBSHOP schema. 
2. The BIT_ENQUEUE_NEW_ORDER_EVENT trigger will execute the ENQUEUE_NEW_ORDER_EVENT procedure that enqueues a message to the NEW_ORDER_QUEUE. 
3. You should see a new event in the NEW_ORDER_QT queue table of the DOAG12_JMSUSER schema.
4. If the Order2Cash SOA Composite is deployed you should see a new instance in Enterprise Manager. This process is started by receiving events from the NEW_ORDER_QT queue using an AQ Resource Adapter.

Java Message Service (JMS)
The next demo shows the publish/subscribe capabilities of JMS and shows how to integrate SOA Composites with JMS:

1. Complete the Book Order Human Task of the Order2Cash process instance that was started in the previous step.
2. The Order2Cash instance will use a JMS Adapter to publish an event to the JMS topic DOAG12_BillingTopic. You can inspect the topic in WebLogic Console and see that an event has been published. There are two subscribers to the JMS topic: the Billing SOA Composite (using a JMS Adapter) and the CRM Java application (using a Message-Driven Bean or MDB).
3. In Enterprise Manager you should see a new instance of the Billing SOA Composite that is started through the JMS Adapter. The instance should have written an invoice file using the File Adapter.
4. In the SOA Suite log file (e.g. user_projects/domains/[domain]/servers/[server]/logs) you should see a log statement produced by the CRM application that indicates that the event is received.

Event Delivery Network (EDN)
This demo shows the publish/subscribe capabilities of EDN and shows how to use EDN from SOA Composites:

1. Wait for 5 minutes after the billing event has been published from the Order2Cash process instance using JMS. The OnAlarm branch will be activated and publish a DunningEvent using a Mediator component.
2. The Dunning SOA Composite is subscribed to the DunningEvent, a new instance of this SOA Composite will be started and can be inspected from Enterprise Manager.

A second usage of EDN is demonstrated using the following steps:

1. Start a new instance of the Order2Cash process by inserting a new order in the database. Complete the Human Task again. Only this time use Enterprise Manager to fire a PaymentEvent to EDN. This can be done by right-clicking the soa-infra node and selecting Business Events. Select the PaymentEvent and hit the Test button. You can use the PaymentExample.xml file from the Order2Cash JDeveloper project as example event payload. 
2. Using correlation the running Order2Cash process instance will receive an in-flight event and will continue without starting the Dunning process.

Sensors and Composite Sensors
The last demo shows the sensor and composite sensor capabilities of SOA Suite. Composite sensors are used to add metadata to running instances so they are easier to find in Enterprise Manager or via the Java APIs of SOA Suite:

1. Inspect the composite sensors of the Order2Cash SOA Composite in JDeveloper.
2. In Enterprise Manager navigate to the instances of the Order2Cash SOA Composite. Use the Add Fields button to search for instances based on functional data such as customer name and order id. 

Alternatively, composite sensors can also be published using JMS.

Sensors and monitoring objects can be used to feed runtime data of BPEL component instances into Oracle BAM, or publish this data to JMS, Database, or AQ in a non-intrusive way. Sensors are based on configuration instead of adding additional activities and components to your SOA Composites:

1. Inspect the sensor configuration of the Dunning BPEL component in JDeveloper. These sensors publish to the DOAG12_SensorQueue queue.
2. In Enterprise Manager navigate to the instances of the ProcessSensors SOA Composite. You should see new instances that are started based on the published sensor events.

Conclusion
The presentation and the demos in this blog give you an overview of the eventing capabilities of SOA Suite. The demo is stuffed with different eventing techniques (AQ, JMS, EDN), this should not be considered a best-practice ;-) Analyze what the best eventing implementation is for your specific scenario.

Some best-practices:

• Model events in your BPM and SOA projects;
• Use events to notify running processes;
• Expand the service registry to include events;
• Use events for additional decoupling between processes and services;
• Events is not just queuing, also consider publish/subscribe and event stream processing;
• There is not a single best technology for eventing implementation in SOA Suite.

Vennster @ UKOUG 2011

The UK Oracle User Group (UKOUG) Conference 2011 in Birmingham is a wrap! Well, at least it was when I started writing this blog :-) Now it has been over a week since I returned on a bumpy flight from Birmingham from my first UKOUG conference which I really enjoyed! Good atmosphere, friendly people, quality presentations, lots of familiar people, and surprisingly good food (at some places at least).

Here's a short summary of the sessions that were hosted by Vennster. Due to last minute cancellations we were asked to fill in for some of the speakers that couldn't attend and presented some more sessions than expected.

BPA Suite to BPEL case study
Lonneke presented on how to model business processes in Oracle BPA Suite and how to transform business process definitions from BPMN to BPEL along with the advantages and disadvantages of this approach. You can find the slides and more background information in this previous blog post.

Effective Fault Handling in Oracle SOA Suite 11g
I presented on fault prevention and handling in SOA and BPM environments and the mechanisms Oracle SOA Suite 11g offers for fault handling. A fault can be defined as something that happened outside the expected operational activity or “happy flow”. Faults can be categorized into:

• Technical faults (network errors, full tablespace, etc.)
• Programming faults (nullpointer exception, cursor not closed, etc.)
• Business faults (credit check failure, invoice amount higher than the ordered amount, etc.)
• Faulty user input (return date of a flight before the departure date, wrong credit card number, etc.).

To prevent faults you can apply various techniques: configure a robust infrastructure to avoid technical errors (clustering, redundancy, high availability), use pair-programming and peer-reviews to avoid programming errors, identify business requirements and implement these to avoid and correctly deal with business faults, and apply user experience techniques to lower the chance of faulty user input and its impact. This session mostly focused on fault prevention and handling for technical and business faults.

Effective Fault Handling in Oracle SOA Suite 11g

View more presentations from Ronald van Luttikhuizen

Event-Driven SOA: Events meet Services
Lonneke and I presented (unexpectedly since the would-be presenter was unable to attend) the use of events and how events, services, and processes can work together to provide flexibility, decoupling, and realize business requirements. Events are important from a business as well as technology perspective. Oracle SOA Suite supports eventing through several implemenations such as Advanced Queuing (AQ), Java Message Service (JMS), and the Event Delivery Network (EDN).

I included the slides from my earlier eventing presentation at Kaleidoscope 2011 which was quite similar to the UKOUG session.

Overview of Eventing in Oracle SOA Suite 11g

View more presentations from Ronald van Luttikhuizen

Some background information can be found in this 2008 blog post explaining the importance of events.

Approach to SOA: Making This a Successful Endeavor for the Whole Organization
Lonneke was asked last-minute to present on why and when to use SOA and what will makes it successful using the (according to some) infamous "breakfast" example. At the end of this session, you will know what SOA means, when it is best used, and how you can get there. 

Wednesday Wizardry
Lonneke, Lucas, and Ronald showed how to rapidly build an enterprise application based on SOA and BPM principles using Oracle Fusion Middleware in only a couple of hours. Read all about it in this blog post.

Other sessions
Due to the unexpected sessions we needed to host I wasn't able to attend that many other presentations. However, I did visit and enjoyed the "WebLogic Server and Oracle RAC" session by Simon Haslam and Frances Zhao on the integration between Oracle WebLogic Server and the Oracle (RAC) Database; especially Active GridLink and the enhancements it offers.

Christmas in Birmingham: Lonneke in front of a square with a giant polar bear and snooker on the big screen

As for me, until next year at UKOUG 2012!