Database adapters, TopLink and Transformations

In one of our current customer SOA projects we’re using Oracle Enterprise Service Bus (ESB) to implement and expose services. One of these services involves transformation of inbound data and persisting this data in a database. We’re using XSL to transform the inbound data and Oracle Application Server’s database adapter to persist data into a relational database. JDeveloper, Oracle’s IDE, provides wizards to configure database adapters. TopLink (an object-relational mapping framework) mappings are generated as a result of this configuration. These mappings are XML files containing metadata such as the structure and format of the database tables. In our ESB flow data is transformed into nested XML format, which is persisted in multiple master-detail tables.

This week -after a modification to the database adapter configuration- data was still persisted, but lots of database records that were previously populated, were suddenly empty. The ESB console indicated all instances were valid and debugging the ESB flows showed that the input XML consumed by the database adapter was unchanged. After some more investigation it turned out that the order of tables in some of the generated TopLink mapping files (OurService_table.xsd, OurService_toplink_mappings.xml and OurService.RootTable.ClassDescriptor.xml) had altered; e.g. instead of table 1, table 2, table 3, the order of tables was table 1, table 3, table 2. The transformation activity still generated input XML data as table 1, table 2, table 3. Synching the order of tables in the transformation and mappings files resolved this issue. This means that runtime, the database adapter strictly follows the order of tables indicated in the TopLink mappings. You can argue whether this is too strict or not, especially since the same tools are very “tolerant” of other faults, omissions, validation, etc. Well, at least you would like to see some kind of error when the XML input format does not conform to the TopLink mapping definition.

Anyway, this was a tricky issue since the input XML seemed valid, the ESB console indicated all instances were valid, no error was logged and some data (but not all!) was persisted in the database.

Events and SOA

In my opinion events are just as important in a SOA as services. I think it’s not possible to achieve a “real” SOA without addressing events separately, just as we do for services. It can be somewhat confusing that events have there own acronym emphasizing their importance, namely Event Driven Architecture (EDA). Some coin this as the upgraded version of SOA, or SOA 2. How we love 3 (or 4 or 5) letter acronyms :-) Most of these different acronyms only add to the confusion, so I prefer sticking to SOA and emphasize events as integral part of it.

So why are events important?
A service-oriented approach should result in business-value (ROI), otherwise SOA only introduces technical complexity through additional middleware. One of the ways to achieve this is through business-IT alignment. Events play a major role in both of these worlds (business and IT).

Business
Businesses deal with (and are causing) events all the time: a customer moving to a different address, a new purchase order, receiving an invoice from a supplier, sending a bill to a partner, and so on. Entities in a business’ ecosystem such as employees, partners, suppliers and customers all react to these events. They initiate new processes, perform activities, propagate events, and so on. It is therefore logical to incorporate events when modeling business processes, which are at the heart of SOA. Just as in all modellng practices, one wants to model and focus on important aspects. In case of SOA this includes processes, services and events.

IT
From a technical point of view events can achieve asynchronicity and de-coupling. By using publish/subscribe and queueing mechanisms, software components are not required to know of each others existence. They simply subscribe to a topic and act based on received (and subscribed to) events. The other way around, if components have some (intermediate) result or state, they can share this with the rest of the world by publishing an event and then forget about it. These components don’t need to know what other components are interested in this information. Of course you’ll need some glue (i.e. middleware) to implement this.

So what does this mean concretely in a SOA-project? For one thing, besides trying to identify what types of services you have (business services, composite technical services, etc.), also investigate what types of events can be identified. There should be an event-registry besides a service-registry. This does not need to be a full-blown tool at first but can simply be documented using Excel in the beginning. Also define the relationship between processes, services and events. What services publish what type of events? What processes are initiated by what events? What running processes are influenced by what events?

Experiences with Vista and Oracle software

This is a quick overview of my attempts to install and run (Oracle) software on Microsoft Vista. A few months ago I got this new Dell laptop pre-installed with Vista Business. Last couple of years I mostly worked with 2000, XP and Unix/Linux so Vista was getting used to in the beginning; especially the enormous amounts of “Are you sure”, “Do you really want this?”, “You’re not allowed to do this”, etc. pop-ups (even more than XP had). Although I must admit that some of the new UI-stuff is real eye-candy.

Some general (and sometimes really frustrating) issues I ran into when trying to get Vista up and running was the new IPv6 network protocol and updated WPA stuff (my existing router and wifi configuration really had a hard time connecting to my laptop or vice versa – I’m not a network guru, maybe that’s the problem after all :-), the new user account control (UAC) policy in Vista (I couldn’t stop some Windows services in the beginning even when logged on as admin) and the pre-installed firewall, virus scanner, etc. which made my (finally up-and-running) Internet connection and network adapter crash. So after some reboots and resetting my laptop to factory settings I thought I had the new Vista “under control”.

Time to install some real business software.

Up to now I have the following software up and running (didn’t really test all software extensively yet):

• Sparx Systems Enterprise Architect
• JDK 5 and 6
• Eclipse SDK
• Eclipse WTP
• Oracle BPA Suite
• JDeveloper 10g
• JDeveloper 11g – Technology Preview including SOA Suite and WebCenter
• Oracle XE 10g
• Oracle SQL Developer
• Oracle OC4J 10g

Some tips:

• Don’t use the standard Vista (un)zipper when unzipping software with long directory- and filenames. For example, I couldn’t run Eclipse WTP since the Vista unzipper didn’t extract all the extension files. This can be tricky since some installers won’t even indicate that all files are not properly unzipped. You can start Eclipse WTP, develop some code, but when trying to add a server, there’s nothing there to add?! Use another tool such as 7-Zip.
• Run all installations and configurations as administrator; right-click and choose “Run as administrator”.
• I ran into an error when installing a standalone OC4J. This was caused by compatibility issues between Sun’s JDK and the new IPv6 protocol. Trying to shut down OC4J returns an error indicating “Error: Missing ormi[s]://:”. See this forum post for a solution. I simply added the -Djava.net.preferIPv4Stack=true
  parameter to the OC4J startup command in oc4j.cmd.
• Read the installation manual. Ok, I know this sounds a bit daft, but there are numerous forum questions in which it turns out people didn’t configure a loopback adapter, didn’t have the right JDK installed, etc. So after all, it’s really worth reading the manual.
• Make sure the directory structure for your Java and Oracle installations don’t contain whitespaces or symbols such as # or @. I only use letters, digits, dots and underscores. Not all software can handle other characters.

Ok, in the next couple of days/weeks I’m going to (or at least try to :-) install SOA Suite 10g (lot of forum posts on installing this on Vista), Oracle Database 10g Enterprise Edition and playing around with Coherence.

Eliminating paperwork by combining digital imaging tools, content management systems and SOA

It was once said that the advance of ICT technologies such as e-mail would reduce the cost of handling documents and be beneficial for the environment. It would result in less printed paper, right? Everything would be done digitally, right? Well, it doesn’t seem to work that way. Even while technology exists to store most information digitally, lots of organizations still have an archive filled with all kinds of printed documents (invoices, orders, etc.). This is costly (you have to build, rent and/or buy archive space) and not really ‘CO2-neutral’. Especially in case documents are delivered electronically.

One of the related problems is that the process of electronically storing documents has to be compliant with a growing number of standards and laws. ‘Just’ storing documents manually on a file system won’t do.
In a customer case the following technologies are/will be combined to accomplish certified electronic storage of documents:

• Scanners with OCR (Optical Character Recognition) software
• Process orchestration using services
• CMS (Content Management System)

How are these technologies combined?
Once documents arrive they are scanned. The OCR software outputs an image of the document and extracts metadata into an XML file. An automated process picks up the XML file, transforms its contents to a canonical data model and invokes a custom SOAP service, also passing the file location of the associated image. The invoked service then uses a Web Service API to store the image into a CMS database and uses the XML data to create and store metadata/attributes for the document.

Why are these technologies complementary?
While a CMS provides reliable and robust storage of electronic documents and provides multiple interfaces (both programmatic and UI-based), process orchestration and SOA implement traceable, auditable and robust processes governing the retrieval and storage of electronic documents. This in turn enables official certification in which stored documents are also ‘legal’ documents.

In this particular case Oracle products are used: the SOA Suite and Content Services (part of Oracle Collaboration Suite). But the same principal can of course be implemented using other technologies available on the market.

Using TopLink native sequencing in an ESB/BPEL project

In ESB and BPEL projects you can use and configure a database adapter to, among others, insert data into a database. TopLink is used to implement this. In the generated TopLink mappings you can specify if you want to use sequencing, either native sequencing or a sequence table. If you want to use Oracle database sequences you need to:

• Create the database sequence(s) in the Oracle database.
• Select “Use Native Sequencing” in the “Database Info” tab of the TopLink map.
• For each entity for which you want a field to be generated using sequences: configure the database sequence name and the entity fields which need to be populated. This can be done in the “Use Sequencing” tab of the Toplink mapping for that entity.

Finally, make sure that:

• The preallocation size in the TopLink map is equal to the increment by value of the Oracle database sequence.
• The TopLink sequencing settings match those of the Database Adapter Connection Factory settings configured in the OC4J container. This one can be tricky, since for “normal” Java/JEE projects you probably do not need to configure this.

How to enable the Enterprise Manager for SOA Suite

I found out that for some reason, on Linux the EM is not configured to start after installation of SOA Suite on Oracle Application Server 10.1.3.1 For those of you who want to use the Enterprise Manager, follow the following steps to get the EM started:

Be careful when editing files, make sure you have made a backup of the files.

1. Edit the file: default-web-site.xml in the $OC4J home/config folder
   Search for the following line and change ohs-routing from false to true
   application="ascontrol" name="ascontrol" load-on-startup="true" root="/em" ohs-routing="true"
   
2. Edit the file: server.xml in the $OC4J/home config folder
   Search for the following line and change start van false in true
   application name="ascontrol" path="../../home/applications/ascontrol.ear" parent="system" start="true"
3. That’s all. Just restart your Oracle Application Server:
   $ORACLE_HOME/opmn/bin/opmnctl stopall
   $ORACLE_HOME/opmn/bin/opmnctl startall
4. Now you should be able to browse to the Enterprise Manager.
   http://servername:[http_port]/em
   

If you for some reason forgot the http port, just look at the file:
$ORACLE_HOME/install/readme.txt

How to get the password of ORASSO

If you need a password of some exotic user you never heard of before, like ORASSO, chances are you are configuring Oracle Single Sign On of some Application Server release.

About five years ago I found out this trick the hard way. I say the hard way because there are similar users like ORASSO. The first time I needed the user orasso. I got on the internet and found out where to look, as you can read here. Then I needed the user orasso_[xyz] again. So I looked again. But the password didn’t work. Ok, there are more users here and some times you need a similar one like I did that day. The user orasso_[xyz] looked to me the same but it isn’t.

But now here’s the way to find out the user passwords which Oracle uses to log in to the Repository.

1. Start the tool OIDADMIN.
   On Linux:
   - be sure you have a X-Server application (eg. Kea!X, Exceed etc.) running on you’re pc.
   - log on to the Linux box (I mostly use the program putty.exe for this)
   - export DISPLAY=[i .p. address of XP machine]:0 (syntax varies for other shells)
   - $ORACLE_HOME/bin/oidadmin
2. Log on with cn=orcladmin
3. In the left frame you need to click trough the following following entries:
   Entry Management
   cn=OracleContext
   cn=Products
   cn=IAS
   cn=IAS Infrastructure Databases
   orclReferenceName=[infrastruct_db_servicename]
   OrclResourceName=ORASSO

The orclpasswordattribute text box on the OrclResourceName=ORASSO tab contains the schema password.

You can also use ldapsearch to get the password. For more info, you can check the Oracle Application Server Single Sign-On Administrator’s Guideracle site.