Tracking progress of your BPEL process using sensors

Often when you start using Oracle SOA Suite11g and BPEL you need a mechanism to help the end users keep track of the progress of the overall process instances. The EM shows this to administrators, but this is not suitable for end users.

What about the worklist application?

The SOA Suite offers the worklist application to handle tasks and to view progress using views and reports. It shows human tasks, crossing multiple process definitions. It does not show invocation of services or specific data changes in the process. The figure below shows three instances of one specific process definition. In this example relevant milestones are reached when the process starts, the first automated step is executed, the second human task is executed and when the process ends. In the worklist application you would be able to see what human tasks are open or executed by which user. With the worklist application you can't keep track of the first, second and last milestone, only of the third one because that is the only one associated with a human task. 

Sensors and sensor actions

The required functionality can be accomplished using sensors and sensor actions on relevant parts in the BPEL process. There are different types of sensors that can be used in SOA Suite 11g: activity sensors, variable sensors and fault sensors.

After you have configured the sensor in your BPEL process, you need to define one or more sensor actions. There are four types of actions: three sensor actions and one BAM sensor action:

• Database to publish it to the BPEL dehydration store
• JMS to publish it to a JMS topic or queue. If the JMS provider is not local, you can use the JMS adapter
• Custom Java class to handle it in a different way. 
• BAM Sensor actions to publish them to Oracle BAM.

You can chose to publish the sensor to AQ, so you can handle it both with Java clients and with PL/SQL because JMS queues in WebLogic can be implemented using AQ. 

Displaying the information with BAM or a custom application

There are two ways to show the progress to the end user: using BAM or building a custom application on top of the sensor data. You can easily decide to implement BAM later: you can either add BAM sensor actions to the BPEL sensor, or define a Enterprise Message Source in BAM. 

Option 2 (defining an Enterprise Message Source) has the advantage of not having to change the BPEL process (design time) when you decide to start using BAM rather than or on top of the custom application. 

Steps

Now that we have established how we want to implement the functionality, we can get started :) The following steps need to be taken:

1. Define the relevant milestones you want to show to the user;
2. Configure the sensors;  
3. Configure the sensor actions;
4. Create the database user and queues for AQ;
5. Configure WebLogic Server and add JMS objects;
6. Create PL/SQL code to read from the queue and store it in the table;
7. Deploy the BPEL process;

In parallel you can build the GUI or configure BAM, depending on what you decided upon. 

Resources

• Oracle® Fusion Middleware Developer's Guide for Oracle SOA Suite11g Release 1 (11.1.1.7), chapter 18;
• Oracle® Fusion Middleware Developer's Guide for Oracle SOA Suite11g Release 1 (11.1.1.7), part X;
• Oracle® Fusion Middleware Configuring and Managing JMS for Oracle WebLogic Server11g Release 1 (10.3.6)

Eventing Hello World

This week I presented the "Introduction in Eventing in Oracle SOA Suite 11g" session at the DOAG conference in Nürnberg. I used several demos in this session to show the eventing capabilities of SOA Suite. This blog contains the source code and accompanying explanation so you can replay the demo yourself.

Introduction
An event is the occurrence of something relevant, signals a change in state that might require an action. Examples of events are: an invoice that has been paid, a customer that moved to a new address, a new purchase order, and so on. Events are complimentary to processes and services: processes and services describe what should be done, events about when something important occurs. SOA is not only about (synchronous) services and processes (what); but also about events (when). Eventing improves decoupling in your SOA landscape.

Slides
The presentation slides can be viewed and downloaded from my SlideShare account. The presentation introduces the concept of eventing, discusses some basic eventing patterns, and talks about the implementation of eventing in SOA Suite using AQ, JMS, and EDN.

DOAG 2012 - Introduction in Eventing in SOA Suite 11g from Ronald van Luttikhuizen

Code
You can download a zipfile containing the JDeveloper workspaces for the SOA Composites (DOAG2012_Eventing_SOASuite) and Java projects (DOAG2012_Eventing_Java) used in the demo here.

Prerequisites
You'll need an Oracle SOA Suite 11g runtime to deploy and run the SOA Composites. Oracle provides a pre-built SOA and BPM virtual machine for testing purposes that is easy to install.

Download JDeveloper 11g (11.1.1.6) or later from Oracle Technology Network to inspect and modify the SOA Composite and Java projects.

Setup
In order for the demos to run some plumbing needs to be done for AQ and JMS:

• The configuration steps required for the AQ demos are listed in the aq_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• The configuration steps required for the JMS demos are listed in the jms_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• In JDeveloper modify the existing Database Connection to point to your DOAG12_JMSUSER database schema.
• In JDeveloper modify the GenerateInvoice File Adapter of the Billing SOA Composite to write the invoice file to an existing directory of the SOA Suite runtime.
• In JDeveloper modify the WriteSensorToFile File Adapter of the ProcessSensors SOA Composite to write sensor values to an existing directory of the SOA Suite runtime.
• Use Enterprise Manager to create a partition in SOA Suite called doag2012_eventing.
• Deploy the following SOA Composites to the doag2012_eventing partition of the SOA Suite runtime using JDeveloper, Enterprise Manager or scripts: Order2Cash, Dunning, Billing, and ProcessSensors.
• Deploy the Java application CRM to the WebLogic Server.

Advanced Queuing (AQ)
The first demo shows the queuing capabilities of AQ and shows how to integrate SOA Composites with AQ queues:

1. Insert a record in the ORDERS table in the DOAG12_WEBSHOP schema. 
2. The BIT_ENQUEUE_NEW_ORDER_EVENT trigger will execute the ENQUEUE_NEW_ORDER_EVENT procedure that enqueues a message to the NEW_ORDER_QUEUE. 
3. You should see a new event in the NEW_ORDER_QT queue table of the DOAG12_JMSUSER schema.
4. If the Order2Cash SOA Composite is deployed you should see a new instance in Enterprise Manager. This process is started by receiving events from the NEW_ORDER_QT queue using an AQ Resource Adapter.

Java Message Service (JMS)
The next demo shows the publish/subscribe capabilities of JMS and shows how to integrate SOA Composites with JMS:

1. Complete the Book Order Human Task of the Order2Cash process instance that was started in the previous step.
2. The Order2Cash instance will use a JMS Adapter to publish an event to the JMS topic DOAG12_BillingTopic. You can inspect the topic in WebLogic Console and see that an event has been published. There are two subscribers to the JMS topic: the Billing SOA Composite (using a JMS Adapter) and the CRM Java application (using a Message-Driven Bean or MDB).
3. In Enterprise Manager you should see a new instance of the Billing SOA Composite that is started through the JMS Adapter. The instance should have written an invoice file using the File Adapter.
4. In the SOA Suite log file (e.g. user_projects/domains/[domain]/servers/[server]/logs) you should see a log statement produced by the CRM application that indicates that the event is received.

Event Delivery Network (EDN)
This demo shows the publish/subscribe capabilities of EDN and shows how to use EDN from SOA Composites:

1. Wait for 5 minutes after the billing event has been published from the Order2Cash process instance using JMS. The OnAlarm branch will be activated and publish a DunningEvent using a Mediator component.
2. The Dunning SOA Composite is subscribed to the DunningEvent, a new instance of this SOA Composite will be started and can be inspected from Enterprise Manager.

A second usage of EDN is demonstrated using the following steps:

1. Start a new instance of the Order2Cash process by inserting a new order in the database. Complete the Human Task again. Only this time use Enterprise Manager to fire a PaymentEvent to EDN. This can be done by right-clicking the soa-infra node and selecting Business Events. Select the PaymentEvent and hit the Test button. You can use the PaymentExample.xml file from the Order2Cash JDeveloper project as example event payload. 
2. Using correlation the running Order2Cash process instance will receive an in-flight event and will continue without starting the Dunning process.

Sensors and Composite Sensors
The last demo shows the sensor and composite sensor capabilities of SOA Suite. Composite sensors are used to add metadata to running instances so they are easier to find in Enterprise Manager or via the Java APIs of SOA Suite:

1. Inspect the composite sensors of the Order2Cash SOA Composite in JDeveloper.
2. In Enterprise Manager navigate to the instances of the Order2Cash SOA Composite. Use the Add Fields button to search for instances based on functional data such as customer name and order id. 

Alternatively, composite sensors can also be published using JMS.

Sensors and monitoring objects can be used to feed runtime data of BPEL component instances into Oracle BAM, or publish this data to JMS, Database, or AQ in a non-intrusive way. Sensors are based on configuration instead of adding additional activities and components to your SOA Composites:

1. Inspect the sensor configuration of the Dunning BPEL component in JDeveloper. These sensors publish to the DOAG12_SensorQueue queue.
2. In Enterprise Manager navigate to the instances of the ProcessSensors SOA Composite. You should see new instances that are started based on the published sensor events.

Conclusion
The presentation and the demos in this blog give you an overview of the eventing capabilities of SOA Suite. The demo is stuffed with different eventing techniques (AQ, JMS, EDN), this should not be considered a best-practice ;-) Analyze what the best eventing implementation is for your specific scenario.

Some best-practices:

• Model events in your BPM and SOA projects;
• Use events to notify running processes;
• Expand the service registry to include events;
• Use events for additional decoupling between processes and services;
• Events is not just queuing, also consider publish/subscribe and event stream processing;
• There is not a single best technology for eventing implementation in SOA Suite.

Vennster @ UKOUG 2011

The UK Oracle User Group (UKOUG) Conference 2011 in Birmingham is a wrap! Well, at least it was when I started writing this blog :-) Now it has been over a week since I returned on a bumpy flight from Birmingham from my first UKOUG conference which I really enjoyed! Good atmosphere, friendly people, quality presentations, lots of familiar people, and surprisingly good food (at some places at least).

Here's a short summary of the sessions that were hosted by Vennster. Due to last minute cancellations we were asked to fill in for some of the speakers that couldn't attend and presented some more sessions than expected.

BPA Suite to BPEL case study
Lonneke presented on how to model business processes in Oracle BPA Suite and how to transform business process definitions from BPMN to BPEL along with the advantages and disadvantages of this approach. You can find the slides and more background information in this previous blog post.

Effective Fault Handling in Oracle SOA Suite 11g
I presented on fault prevention and handling in SOA and BPM environments and the mechanisms Oracle SOA Suite 11g offers for fault handling. A fault can be defined as something that happened outside the expected operational activity or “happy flow”. Faults can be categorized into:

• Technical faults (network errors, full tablespace, etc.)
• Programming faults (nullpointer exception, cursor not closed, etc.)
• Business faults (credit check failure, invoice amount higher than the ordered amount, etc.)
• Faulty user input (return date of a flight before the departure date, wrong credit card number, etc.).

To prevent faults you can apply various techniques: configure a robust infrastructure to avoid technical errors (clustering, redundancy, high availability), use pair-programming and peer-reviews to avoid programming errors, identify business requirements and implement these to avoid and correctly deal with business faults, and apply user experience techniques to lower the chance of faulty user input and its impact. This session mostly focused on fault prevention and handling for technical and business faults.

Effective Fault Handling in Oracle SOA Suite 11g

View more presentations from Ronald van Luttikhuizen

Event-Driven SOA: Events meet Services
Lonneke and I presented (unexpectedly since the would-be presenter was unable to attend) the use of events and how events, services, and processes can work together to provide flexibility, decoupling, and realize business requirements. Events are important from a business as well as technology perspective. Oracle SOA Suite supports eventing through several implemenations such as Advanced Queuing (AQ), Java Message Service (JMS), and the Event Delivery Network (EDN).

I included the slides from my earlier eventing presentation at Kaleidoscope 2011 which was quite similar to the UKOUG session.

Overview of Eventing in Oracle SOA Suite 11g

View more presentations from Ronald van Luttikhuizen

Some background information can be found in this 2008 blog post explaining the importance of events.

Approach to SOA: Making This a Successful Endeavor for the Whole Organization
Lonneke was asked last-minute to present on why and when to use SOA and what will makes it successful using the (according to some) infamous "breakfast" example. At the end of this session, you will know what SOA means, when it is best used, and how you can get there. 

Wednesday Wizardry
Lonneke, Lucas, and Ronald showed how to rapidly build an enterprise application based on SOA and BPM principles using Oracle Fusion Middleware in only a couple of hours. Read all about it in this blog post.

Other sessions
Due to the unexpected sessions we needed to host I wasn't able to attend that many other presentations. However, I did visit and enjoyed the "WebLogic Server and Oracle RAC" session by Simon Haslam and Frances Zhao on the integration between Oracle WebLogic Server and the Oracle (RAC) Database; especially Active GridLink and the enhancements it offers.

Christmas in Birmingham: Lonneke in front of a square with a giant polar bear and snooker on the big screen

As for me, until next year at UKOUG 2012!

Fault handling in Oracle SOA Suite 11g - Part I

You generally want to differentiate between technical errors and functional faults within your processes and services. Functional faults are those that have meaning to the business and might be expected. Functional faults and handling these faults can be part of a process. Consider the example of electronic invoice handling in which an invoice is processed that has a total amount of $2000 while an organization only approved an amount of $1500. In this scenario we can use a human task to halt this particular process instance and assign it to the finance department. An employee of the finance department acquires the task and investigates the issue. He or she may conclude that the client sending the invoice was mistaken, that the invoice approval was not entered correctly in our backend IT-systems or that someone put a coffee mug on the invoice and hence the amount was wrongly interpreted by our scanning and OCR software. In any case, after this human intervention the process may continue again and follow the “happy flow” in our BPEL or BPM processes.

When it comes to technical faults you probably do not want to design error handling in the process itself. If you do, your processes and services will end up being cluttered with all kinds of additional process logic such as while loops, gotos, catches, event handling, and so on to try to recover from technical errors. Technical errors might not be recoverable at all; think of an invoice file that is incorrectly formatted, an invoice file that contains negative numbers while your service or process only accepts positive values, or an invoice file that is mangled during transport. Besides, trying to handle these errors makes your SCA composites look like a mix of spaghetti and circuit boards. Not exactly flexible, agile and manageable: the things we wanted to achieve with service- and process-orientation in the first place.

This blog series contains a possible mechanism to generically handle technical errors in your processes and services -that are wrapped as SCA composites- in Oracle SOA Suite 11g.

In one of our projects we came across a scenario in which administrators need to be notified in case of technical errors in any of the SCA composites. Next to the notification they want the corresponding composite to be terminated. Administrators then investigate the cause of the problem and possibly restart the process instances that are involved. Since every employee uses a task-driven portal, administrators want the error to be presented as a human task in this portal instead of receiving a bunch of e-mails. This needed to be implemented with a minimum of additional (business or process) logic.

To achieve this the following mechanism is used:

• Use Oracle SOA Suite’s Fault Management Framework to redirect (technical) errors to a custom Java class;
• Have the Java class fire an event containing the unique id of the instance using the Event Delivery Network (EDN) or Advanced Queuing (AQ);
• Terminate the composite instance by using the Fault Management Framework and the outcome of the custom Java class;
• Create a single SCA composite to handle all technical errors. This composite subscribes to the event, gathers information on the faulted composite instance, and presents this information as a human task that is assigned to administrators.

Read more on fault handling in part II, part III and part IV of this blog series.