To Pull or to Push

In a recent project various designs were discussed to integrate business processes implemented in Oracle BPM with a Java front-end application. More specifically, how status changes in these processes could be communicated to the end user via the front-end. The possible scenarios quickly boiled down to a poll versus push approach and various in-between scenarios. This blog lists the various approaches and design considerations.

Case

The custom-built Java application provides a thin and stateless User Interface that is used by local branch employees to sell products to customers. The process of selling these products is coordinated through a business process that is implemented in Oracle BPM 11g. The configuration of the products for a particular customer is done by the employee through the Java application. After the configuration of the product is completed, the Java application sends a signal to the process. The process will initiate various automated tasks such as credit checking. In the meantime the employee waits until these tasks are executed, then views the results of those checks and continues with the completion of the order.

The question we discussed is how to inform the employee, though the Java application, that the checks are executed by the process and the employee can move on. We know the milestone in the process is reached through a combination of the occurrence of a particular Human Task and the process state itself.

Solution based on polling

In a polling approach, the Java front-end application polls for the process status and workflow status every few seconds using the out-of-the-box Oracle BPM and Human Workflow APIs. The client applications poll to see if process instances have reached a particular milestone; if so the application moves forward. Since the Java application is stateless and has no server side, the custom application directly polls from every client session and not from a central web server. This means every new client will increase the load on the BPM platform. Stress tests showed that in this scenario the OBPM engine would be mostly busy handling polling requests and couldn’t accommodate for a high degree of concurrently running process instances.

The basic polling approach is simple to implement from the perspective of the Java front-end, but not very scalable.

We could consider optimizing the polling. For example to use a Coherence Grid or Service Bus Result Cache as caching layer between the Java clients and the Oracle BPM APIs and have the clients poll the cache instead of the APIs directly. Upside is that such a cache is easily created and that we limit the number of requests on the backend BPM engine by centralizing the polling requests and using a default caching mechanism. Downside is that the polling results can be outdated when the actual process state has changed and the process status data in the cache is not yet invalidated.

Solution based on pushing

The Java front-end clients are interested in some status change. For this purpose, a natural approach would be to push an event from the process to the Java clients when that particular milestone is reached. The preferred choice in this scenario would be to use JMS for this since this standard is both supported by Java as well as Oracle BPM 11g through the use of JCA Adapters. The pro of this approach is that it is much more scalable. We only perform an activity (publish an event) when needed, we don’t do useless polling. Downside however in this case is that the front-end application doesn’t support event subscription using JMS and this would require a significant change in that application.

Approach

We used a three step approach:

• Short-term. Expose a new service that encapsulates the polling logic. Implement this service on the Service Bus and use Result Caching to cache the polling results. This service is easily created and alleviates the back-end usage of OBPM APIs. The business needs to accept that results can be outdated in some scenarios as a trade-off for the increased load that can be supported.
• Middle-term. Push events from the BPM process using Mediator and a JCA JMS Adapter when a certain milestone is reached and store the event data in a custom table. Have the Java front-end poll on the custom table. In this approach the process uses events, which is the desired approach. We only need a small change to the Java front-end to poll on the custom table instead of the OBPM engine itself, thereby offloading the engine. The Java front-end however still incurs overhead by using a polling approach. 
• Long-term. Change the custom Java application to support event subscription, for example using JMS. This way the scalability of the front-end application will improve since the number of events is much lower than the number of polling requests. Also, on average the front-end application knows about reaching the milestone faster using events, than it does when periodically polling for that information. 

SOA Made Simple

The book SOA Made Simple is published! SOA Made Simple is written by Vennster's Lonneke Dikmans and Ronald van Luttikhuizen. You can download a sample chapter and order the book from the Packt Publishing website. 

SOA Made Simple is a concise and indispensable handbook for finally understanding exactly what Service Oriented Architecture is. Split into three clear sections, in this book you’ll learn from both theory as well as step-by-step implementation examples to aid in your understanding of this often poorly- articulated industry term.

A short abstract for SOA Made Simple:

SOA is an industry term which is often preached like a religion rather than taught like a technology, and over time, grasping the concept has become unnecessarily difficult. Many companies proclaim that they don’t know where to begin with SOA, while others have begun their SOA effort but haven’t reaped the benefits they were convinced it would bring. “SOA Made Simple” explains what SOA is in simple terminology and by using real-life examples. Service-orientation is already a very natural way of thinking for business stakeholders that want to realize and sell services to potential clients, and this book helps you to realize that concept both in theory and practice. After reading “SOA Made Simple” you will have a clear understanding ofwhat SOA is so you can implement and govern SOA in your own organization. If you are an architect who wants to be completely clear in your understanding of what SOA is, then this book is essential. In fact, anyone (designer, developer, administrator or team lead) who is implementing or about to implement an architecture in an IT environment should not miss out on “SOA Made Simple”. 

Eventing Hello World

This week I presented the "Introduction in Eventing in Oracle SOA Suite 11g" session at the DOAG conference in Nürnberg. I used several demos in this session to show the eventing capabilities of SOA Suite. This blog contains the source code and accompanying explanation so you can replay the demo yourself.

Introduction
An event is the occurrence of something relevant, signals a change in state that might require an action. Examples of events are: an invoice that has been paid, a customer that moved to a new address, a new purchase order, and so on. Events are complimentary to processes and services: processes and services describe what should be done, events about when something important occurs. SOA is not only about (synchronous) services and processes (what); but also about events (when). Eventing improves decoupling in your SOA landscape.

Slides
The presentation slides can be viewed and downloaded from my SlideShare account. The presentation introduces the concept of eventing, discusses some basic eventing patterns, and talks about the implementation of eventing in SOA Suite using AQ, JMS, and EDN.

DOAG 2012 - Introduction in Eventing in SOA Suite 11g from Ronald van Luttikhuizen

Code
You can download a zipfile containing the JDeveloper workspaces for the SOA Composites (DOAG2012_Eventing_SOASuite) and Java projects (DOAG2012_Eventing_Java) used in the demo here.

Prerequisites
You'll need an Oracle SOA Suite 11g runtime to deploy and run the SOA Composites. Oracle provides a pre-built SOA and BPM virtual machine for testing purposes that is easy to install.

Download JDeveloper 11g (11.1.1.6) or later from Oracle Technology Network to inspect and modify the SOA Composite and Java projects.

Setup
In order for the demos to run some plumbing needs to be done for AQ and JMS:

• The configuration steps required for the AQ demos are listed in the aq_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• The configuration steps required for the JMS demos are listed in the jms_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• In JDeveloper modify the existing Database Connection to point to your DOAG12_JMSUSER database schema.
• In JDeveloper modify the GenerateInvoice File Adapter of the Billing SOA Composite to write the invoice file to an existing directory of the SOA Suite runtime.
• In JDeveloper modify the WriteSensorToFile File Adapter of the ProcessSensors SOA Composite to write sensor values to an existing directory of the SOA Suite runtime.
• Use Enterprise Manager to create a partition in SOA Suite called doag2012_eventing.
• Deploy the following SOA Composites to the doag2012_eventing partition of the SOA Suite runtime using JDeveloper, Enterprise Manager or scripts: Order2Cash, Dunning, Billing, and ProcessSensors.
• Deploy the Java application CRM to the WebLogic Server.

Advanced Queuing (AQ)
The first demo shows the queuing capabilities of AQ and shows how to integrate SOA Composites with AQ queues:

1. Insert a record in the ORDERS table in the DOAG12_WEBSHOP schema. 
2. The BIT_ENQUEUE_NEW_ORDER_EVENT trigger will execute the ENQUEUE_NEW_ORDER_EVENT procedure that enqueues a message to the NEW_ORDER_QUEUE. 
3. You should see a new event in the NEW_ORDER_QT queue table of the DOAG12_JMSUSER schema.
4. If the Order2Cash SOA Composite is deployed you should see a new instance in Enterprise Manager. This process is started by receiving events from the NEW_ORDER_QT queue using an AQ Resource Adapter.

Java Message Service (JMS)
The next demo shows the publish/subscribe capabilities of JMS and shows how to integrate SOA Composites with JMS:

1. Complete the Book Order Human Task of the Order2Cash process instance that was started in the previous step.
2. The Order2Cash instance will use a JMS Adapter to publish an event to the JMS topic DOAG12_BillingTopic. You can inspect the topic in WebLogic Console and see that an event has been published. There are two subscribers to the JMS topic: the Billing SOA Composite (using a JMS Adapter) and the CRM Java application (using a Message-Driven Bean or MDB).
3. In Enterprise Manager you should see a new instance of the Billing SOA Composite that is started through the JMS Adapter. The instance should have written an invoice file using the File Adapter.
4. In the SOA Suite log file (e.g. user_projects/domains/[domain]/servers/[server]/logs) you should see a log statement produced by the CRM application that indicates that the event is received.

Event Delivery Network (EDN)
This demo shows the publish/subscribe capabilities of EDN and shows how to use EDN from SOA Composites:

1. Wait for 5 minutes after the billing event has been published from the Order2Cash process instance using JMS. The OnAlarm branch will be activated and publish a DunningEvent using a Mediator component.
2. The Dunning SOA Composite is subscribed to the DunningEvent, a new instance of this SOA Composite will be started and can be inspected from Enterprise Manager.

A second usage of EDN is demonstrated using the following steps:

1. Start a new instance of the Order2Cash process by inserting a new order in the database. Complete the Human Task again. Only this time use Enterprise Manager to fire a PaymentEvent to EDN. This can be done by right-clicking the soa-infra node and selecting Business Events. Select the PaymentEvent and hit the Test button. You can use the PaymentExample.xml file from the Order2Cash JDeveloper project as example event payload. 
2. Using correlation the running Order2Cash process instance will receive an in-flight event and will continue without starting the Dunning process.

Sensors and Composite Sensors
The last demo shows the sensor and composite sensor capabilities of SOA Suite. Composite sensors are used to add metadata to running instances so they are easier to find in Enterprise Manager or via the Java APIs of SOA Suite:

1. Inspect the composite sensors of the Order2Cash SOA Composite in JDeveloper.
2. In Enterprise Manager navigate to the instances of the Order2Cash SOA Composite. Use the Add Fields button to search for instances based on functional data such as customer name and order id. 

Alternatively, composite sensors can also be published using JMS.

Sensors and monitoring objects can be used to feed runtime data of BPEL component instances into Oracle BAM, or publish this data to JMS, Database, or AQ in a non-intrusive way. Sensors are based on configuration instead of adding additional activities and components to your SOA Composites:

1. Inspect the sensor configuration of the Dunning BPEL component in JDeveloper. These sensors publish to the DOAG12_SensorQueue queue.
2. In Enterprise Manager navigate to the instances of the ProcessSensors SOA Composite. You should see new instances that are started based on the published sensor events.

Conclusion
The presentation and the demos in this blog give you an overview of the eventing capabilities of SOA Suite. The demo is stuffed with different eventing techniques (AQ, JMS, EDN), this should not be considered a best-practice ;-) Analyze what the best eventing implementation is for your specific scenario.

Some best-practices:

• Model events in your BPM and SOA projects;
• Use events to notify running processes;
• Expand the service registry to include events;
• Use events for additional decoupling between processes and services;
• Events is not just queuing, also consider publish/subscribe and event stream processing;
• There is not a single best technology for eventing implementation in SOA Suite.

Vennster @ UKOUG 2011

The UK Oracle User Group (UKOUG) Conference 2011 in Birmingham is a wrap! Well, at least it was when I started writing this blog :-) Now it has been over a week since I returned on a bumpy flight from Birmingham from my first UKOUG conference which I really enjoyed! Good atmosphere, friendly people, quality presentations, lots of familiar people, and surprisingly good food (at some places at least).

Here's a short summary of the sessions that were hosted by Vennster. Due to last minute cancellations we were asked to fill in for some of the speakers that couldn't attend and presented some more sessions than expected.

BPA Suite to BPEL case study
Lonneke presented on how to model business processes in Oracle BPA Suite and how to transform business process definitions from BPMN to BPEL along with the advantages and disadvantages of this approach. You can find the slides and more background information in this previous blog post.

Effective Fault Handling in Oracle SOA Suite 11g
I presented on fault prevention and handling in SOA and BPM environments and the mechanisms Oracle SOA Suite 11g offers for fault handling. A fault can be defined as something that happened outside the expected operational activity or “happy flow”. Faults can be categorized into:

• Technical faults (network errors, full tablespace, etc.)
• Programming faults (nullpointer exception, cursor not closed, etc.)
• Business faults (credit check failure, invoice amount higher than the ordered amount, etc.)
• Faulty user input (return date of a flight before the departure date, wrong credit card number, etc.).

To prevent faults you can apply various techniques: configure a robust infrastructure to avoid technical errors (clustering, redundancy, high availability), use pair-programming and peer-reviews to avoid programming errors, identify business requirements and implement these to avoid and correctly deal with business faults, and apply user experience techniques to lower the chance of faulty user input and its impact. This session mostly focused on fault prevention and handling for technical and business faults.

Effective Fault Handling in Oracle SOA Suite 11g

View more presentations from Ronald van Luttikhuizen

Event-Driven SOA: Events meet Services
Lonneke and I presented (unexpectedly since the would-be presenter was unable to attend) the use of events and how events, services, and processes can work together to provide flexibility, decoupling, and realize business requirements. Events are important from a business as well as technology perspective. Oracle SOA Suite supports eventing through several implemenations such as Advanced Queuing (AQ), Java Message Service (JMS), and the Event Delivery Network (EDN).

I included the slides from my earlier eventing presentation at Kaleidoscope 2011 which was quite similar to the UKOUG session.

Overview of Eventing in Oracle SOA Suite 11g

View more presentations from Ronald van Luttikhuizen

Some background information can be found in this 2008 blog post explaining the importance of events.

Approach to SOA: Making This a Successful Endeavor for the Whole Organization
Lonneke was asked last-minute to present on why and when to use SOA and what will makes it successful using the (according to some) infamous "breakfast" example. At the end of this session, you will know what SOA means, when it is best used, and how you can get there. 

Wednesday Wizardry
Lonneke, Lucas, and Ronald showed how to rapidly build an enterprise application based on SOA and BPM principles using Oracle Fusion Middleware in only a couple of hours. Read all about it in this blog post.

Other sessions
Due to the unexpected sessions we needed to host I wasn't able to attend that many other presentations. However, I did visit and enjoyed the "WebLogic Server and Oracle RAC" session by Simon Haslam and Frances Zhao on the integration between Oracle WebLogic Server and the Oracle (RAC) Database; especially Active GridLink and the enhancements it offers.

Christmas in Birmingham: Lonneke in front of a square with a giant polar bear and snooker on the big screen

As for me, until next year at UKOUG 2012!