SOA Black Belt Workshop, Day 2: Engine Internals

The second day of the workshop took a deep dive into EDN and two service engines: the Mediator and BPEL.

Mediator Service Engine: parallel routing and the resequencer 

This was very interesting because the threading and transactions within the parallel routing rules were explained. I like to use mediator in composites, but had never encountered the need for parallel routing rules. The explanation helps especially when you look at performance issues in your composite.

The storyline about the fabric was continued nicely in the discussion about the Mediator component: Simone explained what operations were implemented by the service engine. On top of that she told us what type of threads are active when handling parallel routing rules. 

Next up was the resequencer. It is a very powerful mechanism to put messages that are out of order back in sequence. The only caveat is that it is sometimes really difficult to determine the right order. Unfortunately this can't be easily solved by a tool ;) 

EDN: making governance easier

Business events is a concept that is well known to E-Business Suite users. This concept has been used in SOA Suite too. It abstracts away the implementation of a the events (JMS or AQ) from the developer and (the most important feature) makes governance of events easier. Because you can keep track of the events you can subscribe to, and of the subscribers to events. This features is applied in Fusion apps and OER. During this presentation I learned a number of new things: The AQ implementation has a Java API that allows you to publish business events. Secondly, there is an extra queue for Once and Only once implementations (and it makes sense ;) ).

BPEL Service Engine: the Cube engine and idempotent invokes at Starbucks 

I know the BPEL engine fairly well after doing work both in BPEL10g and BPEL11g. So I was afraid that this part would be a bit boring for me. Fortunately, it started with a section about the Cube Engine and the components of this engine. It validated the mental model in my mind and added a lot of information to it. An interesting detail about BPEL 2.0 is it generates the runtime model in memory during load(). Some things about the BPEL engine were new to me: the possibility to set an invoke to non-idempotent (causing it to dehydrate the process), and the methods from the Fabric that are implemented by the engine. Last but not least, the function of the two threads, invoke threads and engine threads were explained. This helps a great deal in tuning the performance of your BPEL engine! 

In the session about correlation Simone told us about a mnemonic from Gregory Hohpe I will never forget: only the customer cares about the correlation id (your name and your drink type). The same is true for your BPEL process: Whenever you implement correlation, remember where to define it by realizing that only the 'waiting' process needs to define correlation sets, like the customer at Starbucks. The barista couldn't care less. He or she only cares about preparing the drink!

We did not have enough time for all the labs; I am going to do the resequencer lab as soon as I publish this post. However, the correlation lab and the performance lab were a lot of fun and especially the performance lab was challenging!

One thing is for sure, after today I know a lot more about the database structure and the threading models that are used by the service engines. This will make tuning a lot easier. Again, I can't wait what tomorrow will bring....

Eventing Hello World

This week I presented the "Introduction in Eventing in Oracle SOA Suite 11g" session at the DOAG conference in Nürnberg. I used several demos in this session to show the eventing capabilities of SOA Suite. This blog contains the source code and accompanying explanation so you can replay the demo yourself.

Introduction
An event is the occurrence of something relevant, signals a change in state that might require an action. Examples of events are: an invoice that has been paid, a customer that moved to a new address, a new purchase order, and so on. Events are complimentary to processes and services: processes and services describe what should be done, events about when something important occurs. SOA is not only about (synchronous) services and processes (what); but also about events (when). Eventing improves decoupling in your SOA landscape.

Slides
The presentation slides can be viewed and downloaded from my SlideShare account. The presentation introduces the concept of eventing, discusses some basic eventing patterns, and talks about the implementation of eventing in SOA Suite using AQ, JMS, and EDN.

DOAG 2012 - Introduction in Eventing in SOA Suite 11g from Ronald van Luttikhuizen

Code
You can download a zipfile containing the JDeveloper workspaces for the SOA Composites (DOAG2012_Eventing_SOASuite) and Java projects (DOAG2012_Eventing_Java) used in the demo here.

Prerequisites
You'll need an Oracle SOA Suite 11g runtime to deploy and run the SOA Composites. Oracle provides a pre-built SOA and BPM virtual machine for testing purposes that is easy to install.

Download JDeveloper 11g (11.1.1.6) or later from Oracle Technology Network to inspect and modify the SOA Composite and Java projects.

Setup
In order for the demos to run some plumbing needs to be done for AQ and JMS:

• The configuration steps required for the AQ demos are listed in the aq_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• The configuration steps required for the JMS demos are listed in the jms_configuration.txt file that is part of the Queuing_Utilities JDeveloper project.
• In JDeveloper modify the existing Database Connection to point to your DOAG12_JMSUSER database schema.
• In JDeveloper modify the GenerateInvoice File Adapter of the Billing SOA Composite to write the invoice file to an existing directory of the SOA Suite runtime.
• In JDeveloper modify the WriteSensorToFile File Adapter of the ProcessSensors SOA Composite to write sensor values to an existing directory of the SOA Suite runtime.
• Use Enterprise Manager to create a partition in SOA Suite called doag2012_eventing.
• Deploy the following SOA Composites to the doag2012_eventing partition of the SOA Suite runtime using JDeveloper, Enterprise Manager or scripts: Order2Cash, Dunning, Billing, and ProcessSensors.
• Deploy the Java application CRM to the WebLogic Server.

Advanced Queuing (AQ)
The first demo shows the queuing capabilities of AQ and shows how to integrate SOA Composites with AQ queues:

1. Insert a record in the ORDERS table in the DOAG12_WEBSHOP schema. 
2. The BIT_ENQUEUE_NEW_ORDER_EVENT trigger will execute the ENQUEUE_NEW_ORDER_EVENT procedure that enqueues a message to the NEW_ORDER_QUEUE. 
3. You should see a new event in the NEW_ORDER_QT queue table of the DOAG12_JMSUSER schema.
4. If the Order2Cash SOA Composite is deployed you should see a new instance in Enterprise Manager. This process is started by receiving events from the NEW_ORDER_QT queue using an AQ Resource Adapter.

Java Message Service (JMS)
The next demo shows the publish/subscribe capabilities of JMS and shows how to integrate SOA Composites with JMS:

1. Complete the Book Order Human Task of the Order2Cash process instance that was started in the previous step.
2. The Order2Cash instance will use a JMS Adapter to publish an event to the JMS topic DOAG12_BillingTopic. You can inspect the topic in WebLogic Console and see that an event has been published. There are two subscribers to the JMS topic: the Billing SOA Composite (using a JMS Adapter) and the CRM Java application (using a Message-Driven Bean or MDB).
3. In Enterprise Manager you should see a new instance of the Billing SOA Composite that is started through the JMS Adapter. The instance should have written an invoice file using the File Adapter.
4. In the SOA Suite log file (e.g. user_projects/domains/[domain]/servers/[server]/logs) you should see a log statement produced by the CRM application that indicates that the event is received.

Event Delivery Network (EDN)
This demo shows the publish/subscribe capabilities of EDN and shows how to use EDN from SOA Composites:

1. Wait for 5 minutes after the billing event has been published from the Order2Cash process instance using JMS. The OnAlarm branch will be activated and publish a DunningEvent using a Mediator component.
2. The Dunning SOA Composite is subscribed to the DunningEvent, a new instance of this SOA Composite will be started and can be inspected from Enterprise Manager.

A second usage of EDN is demonstrated using the following steps:

1. Start a new instance of the Order2Cash process by inserting a new order in the database. Complete the Human Task again. Only this time use Enterprise Manager to fire a PaymentEvent to EDN. This can be done by right-clicking the soa-infra node and selecting Business Events. Select the PaymentEvent and hit the Test button. You can use the PaymentExample.xml file from the Order2Cash JDeveloper project as example event payload. 
2. Using correlation the running Order2Cash process instance will receive an in-flight event and will continue without starting the Dunning process.

Sensors and Composite Sensors
The last demo shows the sensor and composite sensor capabilities of SOA Suite. Composite sensors are used to add metadata to running instances so they are easier to find in Enterprise Manager or via the Java APIs of SOA Suite:

1. Inspect the composite sensors of the Order2Cash SOA Composite in JDeveloper.
2. In Enterprise Manager navigate to the instances of the Order2Cash SOA Composite. Use the Add Fields button to search for instances based on functional data such as customer name and order id. 

Alternatively, composite sensors can also be published using JMS.

Sensors and monitoring objects can be used to feed runtime data of BPEL component instances into Oracle BAM, or publish this data to JMS, Database, or AQ in a non-intrusive way. Sensors are based on configuration instead of adding additional activities and components to your SOA Composites:

1. Inspect the sensor configuration of the Dunning BPEL component in JDeveloper. These sensors publish to the DOAG12_SensorQueue queue.
2. In Enterprise Manager navigate to the instances of the ProcessSensors SOA Composite. You should see new instances that are started based on the published sensor events.

Conclusion
The presentation and the demos in this blog give you an overview of the eventing capabilities of SOA Suite. The demo is stuffed with different eventing techniques (AQ, JMS, EDN), this should not be considered a best-practice ;-) Analyze what the best eventing implementation is for your specific scenario.

Some best-practices:

• Model events in your BPM and SOA projects;
• Use events to notify running processes;
• Expand the service registry to include events;
• Use events for additional decoupling between processes and services;
• Events is not just queuing, also consider publish/subscribe and event stream processing;
• There is not a single best technology for eventing implementation in SOA Suite.

Fault handling in Oracle SOA Suite 11g - Part I

You generally want to differentiate between technical errors and functional faults within your processes and services. Functional faults are those that have meaning to the business and might be expected. Functional faults and handling these faults can be part of a process. Consider the example of electronic invoice handling in which an invoice is processed that has a total amount of $2000 while an organization only approved an amount of $1500. In this scenario we can use a human task to halt this particular process instance and assign it to the finance department. An employee of the finance department acquires the task and investigates the issue. He or she may conclude that the client sending the invoice was mistaken, that the invoice approval was not entered correctly in our backend IT-systems or that someone put a coffee mug on the invoice and hence the amount was wrongly interpreted by our scanning and OCR software. In any case, after this human intervention the process may continue again and follow the “happy flow” in our BPEL or BPM processes.

When it comes to technical faults you probably do not want to design error handling in the process itself. If you do, your processes and services will end up being cluttered with all kinds of additional process logic such as while loops, gotos, catches, event handling, and so on to try to recover from technical errors. Technical errors might not be recoverable at all; think of an invoice file that is incorrectly formatted, an invoice file that contains negative numbers while your service or process only accepts positive values, or an invoice file that is mangled during transport. Besides, trying to handle these errors makes your SCA composites look like a mix of spaghetti and circuit boards. Not exactly flexible, agile and manageable: the things we wanted to achieve with service- and process-orientation in the first place.

This blog series contains a possible mechanism to generically handle technical errors in your processes and services -that are wrapped as SCA composites- in Oracle SOA Suite 11g.

In one of our projects we came across a scenario in which administrators need to be notified in case of technical errors in any of the SCA composites. Next to the notification they want the corresponding composite to be terminated. Administrators then investigate the cause of the problem and possibly restart the process instances that are involved. Since every employee uses a task-driven portal, administrators want the error to be presented as a human task in this portal instead of receiving a bunch of e-mails. This needed to be implemented with a minimum of additional (business or process) logic.

To achieve this the following mechanism is used:

• Use Oracle SOA Suite’s Fault Management Framework to redirect (technical) errors to a custom Java class;
• Have the Java class fire an event containing the unique id of the instance using the Event Delivery Network (EDN) or Advanced Queuing (AQ);
• Terminate the composite instance by using the Fault Management Framework and the outcome of the custom Java class;
• Create a single SCA composite to handle all technical errors. This composite subscribes to the event, gathers information on the faulted composite instance, and presents this information as a human task that is assigned to administrators.

Read more on fault handling in part II, part III and part IV of this blog series.