Publishing process information using sensors and AQ

Sensors in Oracle BPEL PM provide a nice mechanism to publish in-flight process data in a fire-and-forget fashion. The information that is published (sensor data) is separated from the communication mechanism (sensor action) which is a good thing: it promotes separation of concerns. JMS and BAM are supported communication mechanisms to publish sensor data, native Oracle AQ is not. It is however possible to use JMS over AQ instead of in-memory or file-based JMS for transactional and persisted message delivery and integration with database components. I’ve described the involved steps to publish sensor data from BPEL PM to AQ-backed JMS and subscribe to these events from ESB since it is bit of a configuration-nightmare. These steps apply to SOA Suite 10.1.3.4:

• Create a new AQ queue or topic (multi-consumer queue) in the database; e.g. MyQueue. These AQ destinations are typically created in the schema “JMSUSER”. Make sure its payload is of type JMS text (SYS.AQ$_JMS_TEXT_MESSAGE) and not XMLType. If not already there, configure a connection pool and data source in OC4J that refers to the database schema that owns the AQ destinations.
• Create a Database Persistence provider in OC4J. In Enterprise Manager go to oc4j_soa –> Administration –> Database Persistence and click “Deploy”. You could e.g. use JMSUserRP as Resource Provider name and JMSUserRA as Resource Adapter name. Point to the data source as described in the previous step.
• Create a Resource Adapter connection factory for the newly created Resource Adapter by navigating to oc4j_soa –> Applications –> Default. Select the module from the Resource Adapter list that you’ve just created. Create a new Connection Factory and new Administered Object. Use the queue interfaces -e.g. javax.jms.XAQueueConnectionFactory- in case you publish to an AQ queue, and the topic interfaces if you use a multi-consumer AQ queue -e.g. oracle.j2ee.ra.jms.generic.AdminObjectQueueImpl. Examples of JNDI names are JMSUserRA/XAQCF for the connection factory and JMSUserRA/Queue for the administered object.
• Define a sensor and sensor action in BPEL PM. Either choose “JMS Queue” or “JMS Topic” as “Publish Type” in the sensor action. Enter the connection factory JNDI name from the previous step as JMS Connection Factory, e.g. JMSUserRA/XAQCF and the JNDI location of the AQ queue or topic. The last value should be like: [JNDI name of the "Administered Object"]/[either "Queues" or "Topics" depending on the AQ type]/[Name of the AQ queue or topic]; for example JMSUserRA/Queue/Queues/MyQueue.

Now, you can create an ESB project that subscribes to the published sensor events and actually does something useful with it.

• Create an inbound JMS adapter in a new ESB project. Enter the Resource Adapter connection factory -e.g. JMSUserRA/XAQCF- as JNDI name in the first step and select the correct AQ destination from the browse list in the next step.

And voila you’re done.

There are two sides to this. First of all -not the most important one however- is the technical implementation and configuration details as described above.

Secondly it provides a nice pattern. You can use sensors to publish relevant in-flight process data from running instances. This promotes decoupling since BPEL PM does not (need to) know who is interested in the sensor data. It might be that nobody is interested in it. Secondly, you can use ESB’s fan-out and content-based routing patterns and connectivity features to route process data to all interested components, possibly filtering and transforming its data.

Behind the scenes of OBUG, literally!

This week I attended the OBUG Benelux Connect 2009 user conference. OBUG stands for the Oracle BeNeLux (Belgium, Netherlands, and Luxembourg) User Group. The conference was held at the Metropolis Cinema Complex in Antwerp. One of the sessions I planned to attend was the “Report from the R&D Lab – Analyzing the upcoming 11g Release of Fusion Middleware” session by Lonneke Dikmans and Lucas Jellema. The presentation was held on behalf of WAAI, a collaboration of four Dutch companies in which Lucas, Lonneke, and I participate. Goal of the WAAI collaboration is to share, bundle, and expand knowledge on the upcoming Fusion Middleware 11g release. In this presentation, the initial findings and research of WAAI would be made available to the audience, including real world examples and demonstrations. At least, that was the plan...

From Kuassi Mensah from Oracle -who had an earlier presentation on Database Web Services and SOA- we heard that you couldn’t hook up your laptop. That meant no demo’s! And nobody had informed the speakers about this! Even stranger since Javapolis is held at the same venue and tons of demo’s are given there each year. While Lonneke, Lucas, and I chatted about this we got the idea that I could do the demo backstage while they did the presentation. We asked the Metropolis crew, who was more than willing to help out. Thanks a lot for that guys!

Time was running out while Lonneke and Lucas showed me what to demo and virtual machines, SOA composites, schema’s, and other stuff quickly moved from one laptop to another. Lonneke and Lucas stayed remarkably calm given the situation. Just a few minutes before the presentation we crawled through a secret hatch in the back of the cinema. While I “installed” myself right next to the digital projectors, professional sound systems, buttons, wires, and tons of popcorn Lonneke and Lucas went back to do the presentation. Guided by their instructions during the presentation -while we couldn’t see each other- we still managed to show Fusion Middleware 11g stuff to the audience :-) It turned out to be a great presentation after all, despite the organization or lack of it. Also quite educational when it comes to the inner workings of cinemas and good for the friday afternoon talks with colleagues :-)

Custom adapter article on OTN

A while ago someone asked me if you could create your own adapter and use it from Oracle SOA Suite. More specifically if you could create an inbound e-mail adapter (not available out-of-the-box in SOA Suite) that polls for new mail messages and use it as activation agent for starting a BPEL process or ESB flow.


I knew this was possible -at least the possibility was documented- and searched for a how-to. I couldn’t find one explaining all steps involved. However, I did find lots of questions on the OTN forums asking how to achieve this. I thought it would be nice to write an article about it after I got it working. It’s published on OTN.


The article includes a step-by-step tutorial for building the adapter and plugging it into Oracle SOA Suite components such as BPEL PM and ESB. The article also briefly discusses adapter support, offerings, and convergence in future Oracle Fusion Middleware releases that incorporate former BEA products.


Drop me a line if you’re interested in an example for using a outbound adapter instead of an inbound one. I’ll see if I can add such an example in the future.

Deep-dive into our OOW 2008 demo’s

Sunday Lonneke and I did the Oracle vs BEA shootout presentation. We compared products from both the Oracle Fusion Middleware stack (BPA Suite, BPEL PM and OESB) with their BEA counterparts (ALBPM, WLI and ALSB). If you attended the presentation and want some more info on the demo’s or couldn’t make it to the presentation, we’ll be at the Oracle ACE Office Hours in the OTN Lounge this wednesday from 4.00 to 5.30 pm. The demo’s include closed-loop integration between different components, creating custom adapters in Oracle ESB, performing data enrichment in ALSB, etc. etc. The OTN Lounge is a really cool place to hang out. It’s located on the 3rd floor of Moscone West. See you there!

P.S. Visit the Oracle Wiki for a complete listing of the Oracle ACE Office Hours.

SCA to eliminate “over-servicing”

When Oracle acquired Collaxa and its BPEL product, there was no Oracle ESB yet. All adapters -such as file, FTP, JMS and database adapters- and complementary routing and transformation functionality were directly used from BPEL processes. Later on, Oracle introduced its ESB product and advocated to place adapters in the ESB instead of BPEL PM. That made sense since it promotes a cleaner separation of concerns. After all, adapters are more a technical, infrastructural concern than an orchestration concern. Mostly due to backwards compatibility, adapter functionality remained a part of the BPEL PM product. This caused much confusion since it was not always clear from where to use these adapters.

When you follow the above design guideline of placing adapter functionality in the ESB-layer, you could end up with masses of ESB projects in a real-life SOA implementation. Think of it, there can be tens -or maybe hundreds of BPEL processes and composite services- of which several will invoke partnerlinks that are in fact ESB projects wrapping adapter functionality. Examples are retrieving data from a database, publishing an event, FTP-ing an order file to a supplier, etc. This leads to “over-servicing” since some of these ESB projects are not reusable (low-level) services, but local to a single composite service or process only.

The amount of ‘none-reusable’ services depends on the approach taken. Losing overview is one of the risks of implementing SOA in bottom-up fashion only. This can result in too many granular, and possibly none-reusable services. It can be prevented by applying a more top-down or meet-in-the-middle approach. This will lead to less none-reusable services. However, you’ll always end-up with some services that are ‘private’ or ‘local’, meaning that they are only used by one other process or composite service. In such cases you would like the ESB service to be local -or encapsulated- to the composite service or process without moving the adapter functionality to BPEL.

The great thing about the Service Component Architecture (SCA) standard -on which Oracle SOA Suite 11g is based- is that you do not need to expose low-level mediator services and adapter functionality as a separate external service. It can be a local component to a composite and therefore not visible to other service composites. This promotes much better integration and encapsulation of infrastructure and low-level services. You only need to expose those artifacts that are -or will be- reusable. This resembles some of the OO-principles such as encapsulation. However, if a project ends up with lots and lots of none-reusable or ‘private’ services, it might be a good idea to analyze the services and process in a more top-down fashion to be able to create better reusable services.

BPEL, Beehive and Service Repository at OOW

This recap of some interesting OOW2008 sessions is posted a bit later than expected since my baggage -including notes- was stuck on the airport for a few days. Coincidentally my baggage was stranded at the same airport for which I codesigned the new baggage handling system. Maybe software can have a grudge against its creator after all? Luckily it was another terminal than the one I transferred through.

BPEL PM
There were several interesting sessions on BPEL PM by Clemens Utschig and Robin Zimmermann on the new features in 10.1.3.4, upcoming features in 10.1.3.5, and some useful tips and tricks for problem solving BPEL projects. A summary of the new and improved features in the Oracle BPEL PM 10.1.3.4 patch can be found here. The main objective of this patch is to make the BPEL Console a one-stop-shop. It therefore mainly introduces administrative improvements. The most interesting of these are:

Lost BPEL instances
Actually these instances are not lost, they just don’t show up in the BPEL Console. This is due to rollbacks in asynchronous process instances that are not yet dehydrated. This can e.g. be the case when a time-out occurs and the global BPEL transaction is rolled-back. The problem is solved by using a separate transaction for dehydration and not doing the actual instance’s work in the same transaction as the dehydration.

Deployment plans
This looks very much like the deployment plans already available in Oracle ESB. These plans are used to extract most of the configurable process information that differs per environment. Such information includes URL’s and ports of invoked services, adapter-specific information like inbound file names, JNDI locations of JMS queues, database adapter names, etc. etc. Ant tasks can be used to deploy BPEL processes to a target environment with the configuration of that specific environment. This information is wrapped in a BPEL suitcase. Part of the environment-specific information -not all, especially adapter-related information- could already be externalized using customized Ant builds. When an ESB is used to wrap adapter functionality, the need for deployment plans is not as urgent.

Other improvements in the 10.1.3.4 patch include improved visibility of engine threading model, improved statistics collection, minimization of XML coding errors through compliance testing and enhanced debugging of XML payloads, improved automated recovery agent (this feature was disabled in previous releases), and collection of support information when creating service requests.

Note that some of the latest 10.1.3.3 MLR’s are not included in the 10.1.3.4 patch. You’ll need to apply patch 10.1.3.4 followed by some additional MLR’s to update to the newest version. A preview of the new 10.1.3.5 features are also available in the PDF.

Some other cool stuff presented at OOW2008:

Beehive
Oracle Beehive is launched. Beehive is an integrated, open, and secure collaborative platform. Sort of a new and improved OCS, but then build from scratch. It provides seamless integration with -and abstraction of- all kinds of collaborative tools and technologies such as mail, file system, content management, feeds, calendar, mobile devices, chat, protocols, etc., etc. This is done through the notion of team and personal workspaces. Beehive also includes a Web based interface. Integration with existing user-interfaces or building more advanced user-interfaces can be achieved through its Java API and/or WebCenter Suite. That way you would have a WebCenter frontend communicating with a Beehive backend. See the Beehive website and the Beehive forum.

Enterprise Repository
Some products that were lacking from the Oracle stack prior to the BEA acquisition were related to governance. In the beginning of smaller, integration-aimed, and not enterprise-wide SOA projects technology usually poses a bigger risk than governance. However, in the course of SOA-projects lack of governance quickly becomes the main risk. Next to the runtime Service Registry product from Systinet, Oracle Web Service Manager, and the Enterprise Manager SOA Management Pack that Oracle offers, governance support now also includes the former BEA product Enterprise Repository. This product supports and enables governance at design-time. With this product you can -among others- “harvest” BPEL projects to retrieve artifacts such as processes, WSDL’s, XSD’s, and so on. Enterprise Repository creates a taxonomy out of this and graphically presents this. This way one can see for example what XSD is used by what processes, what policies are attached to what processes, and if these policies are met. Later versions will automate the retrieval of runtime information to automatically determine whether policies such as service response times are met. Publishing repository information to the development environment instead of the other way around should also be possible in future releases.

And this was just a small portion of all the OOW2008 news! See OTN for more information!

The feared “demo-effect” – bluescreen just before our OOW session

Yesterday I arrived in San Francisco to attend and present on Oracle Open World 2008. After a really nice dinner with most of the ODTUG presenters, I quickly went to sleep. The day after -today as I’m writing this blog-, Lonneke and I were going to present the Oracle versus BEA shootout session in Moscone West. Exciting, moreover since the session was fully booked, with more than fifty people on the waiting list. So naturally we wanted to prepare, test, and fine-tune our demo’s this morning.

The day starts good. I had slept for more than 10 hours, my jetlag -that I really felt during the ODTUG dinner- was reduced to a minor disorientation. So far so good. I met with Lonneke to go over the demo’s. First thing that happens when I start my laptop is that I see the feared bluescreen of death telling me that a fatal core dump has occurred (sounds just like Star trek). Meanwhile the presentation was only a couple of hours away. Aarrrgghhh! To keep in Star trek terms, I only needed to realign the dilithium matrix to stabilize the warp field to fix this “coredump”. My laptop caught the demo-effect in its worst form: half our presentation is demo and my laptop is dead! That’s when my heart rate doubled and my blood pressure went to a new all-time high. Lonneke -normally making jokes to cheer you up when something goes wrong- kept quiet this time and looked at me alarmingly. Luckily, restarting computers when they’re broken or don’t do what you want can result in miracles: no bluescreen this time. Guess my computer had a jetlag too. After fixing our BEA demo’s and a quick rehearsal we left for Moscone. Luckily, the demo-virus stayed in the hotel and all demo’s worked perfectly during our presentation!

Later on this week we’ll post more on our Oracle Open World. We’ll also post the demo’s from our presentation later on.

Lonneke at OOW 2008.

Collect all the ribbons and become a fout-star general!