In the real business scenario of integrating SOA/AIA applications with Oracle Business Activity Monitoring (BAM), one common requirement is that such integration should not impact the operations of business systems running on SOA/AIA in terms of high availability and performance. To achieve this goal, one option is to use BAM Adapter or BAM Sensor Actions with Batching enabled. This article explains how batching works in Oracle BAM Adapter.

Why You Need Batching

When talking about batching, the first impression you may have is that you put a set of data into a package or a batch, and send it across for processing as a whole. In the BAM Adapter context, batching not only allows a group of XML payloads to be sent to the BAM Server side, but also changes the behavior of a synchronous invocation to BAM Server API to Async.

In mission critical applications, the preferred approach to integrate Oracle BAM with live data feed is using the Fire and Forget Message Exchange Pattern (MEP). Fire and Forget MEP has two major benefits compared to synchronous integration pattern:

• High Performance – Threads invoking BAM Adapter APIs are not blocked by down stream operations. Thus the slow execution of down stream code should not impact the performance of the core business system.
• High Availability – The failures including application failure such as errors due to invalid payload,  and system failures such as system outage due to maintenance, will not impact the normal operations of the main system. For example, in the Order to Cash (O2C) flow of AIA Communication PIPs, data integration between AIA and BAM should be in an asynchronous manner, thus, failures and outage of BAM will not impact AIA operations.

When Oracle BAM Adapter or BAM Sensor Actions is used, you should always enable batching to leverage the Fire and Forget MEP. In the next section, you will see how BAM Adapter works with batching enabled.

How Batching Works

Oracle BAM Adapter is a JCA compliant adapter that primarily converts Oracle BAM operations (Insert, Upsert, Update, and Delete) into the standard Web Services Interface described by WSDL.

By default, BAM Adapter operates in a synchronous manner as shown in the following diagram.

The caller thread T1 initiated in Oracle BPEL Process Manager invokes BAM Adapter operations in a synchronous way, and the same thread T1 continues executing BAM Server (Active Data Cache) API.

The behavior changes if batching is enabled shown in the following diagram.

Oracle BAM Adapter works in the following way when batching is enabled:

1. The caller thread T1 sends the request in the XML format to Oracle BAM Adatper.
2. Oracle BAM Adapter processes the request and delegates the operation to BatchProcessor, which is a module in BAM to handle batch processing.
3. BatchProcessor inserts the request message into a batch and release the caller thread T1.
4. BatchProcessor determines when to send the batch to an internal queue based on the following rules: The number of messages in the batch reaches the threshold specified by the Batch_Lower_Limit attribute or the timer specified by the Batch_Timeout attribute expires (5 secs by default).
5. The batch in the queue is scheduled to be flushed to BAM by an external thread T2. The queue size is specified by the Number_Batches atribute.

So you can see that the caller thread T1 is not block by BAM server operations.

Summary

In summary, batching should be always enabled if integrating BAM with mission critical applications (for example, O2C PIP) using Oracle BAM Adapter.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

The configuration plan for composite deployment is composite-specific.  Therefore, when you deploy the SOA composite application into different environments (e.g. development, test, production), you need to substitute the binding variables in each configuration plan for each environment, this is a tedious and time consuming task, and prone to mistakes. In PS6, there is a new feature called global token, this feature allows you to define global token variables for specific URLs in SOA composite applications.  The documentation at http://docs.oracle.com/cd/E28280_01/admin.1111/e10226/soainfra_config.htm#CEGCJICF shows you how to configure the global token using Oracle Enterprise Manager Fusion Middleware Control, and the documentation at http://docs.oracle.com/cd/E28280_01/dev.1111/e10224/sca_bindingcomps.htm#CIHFJFJC shows you how to create token in Oracle JDeveloper using the Binding URLs of External References feature.

Global Token

The global token is based on key value pair. As described in Oracle documentation, you cannot create a token name that contains special characters such as dashes.  This limitation is due to the fact that the token name is being processed using regular expression with pattern = \$\{\w+\}, “\w” which means  a word character, short for letters, digits, and underscores, hence the token key cannot accept any special characters. If you use any non-words characters, the invocation will fail with null pointer exception. However, you can use “_” (underscores) instead of dashes, for example you can have a token name with domain/department, application name and host/port separated by underscores (e.g. HR_HRMS_HOST, HR_HRMS_PORT).

Other than the user defined global token, there are 2 system predefined global token variables: serverURL and applicationURL, which can be used instead of defining your own token. You cannot use serverURL and applicationURL as the token name when you define a global token using Oracle Enterprise Manager Fusion Middleware Control.

If you use serverURL (SOA Infrastructure>Common Properties), you need to provide the protocol (http or https), for example:

When you define the token in your composite.xml file, the web service binding will look like this:

<binding.ws  port=”http://xmlns.oracle.com/bpel_101_HelloWorld_jws/bpel_101_HelloWorld/HelloWorldProcess#wsdl.endpoint(helloworldprocess_client_ep/HelloWorldProcess_pt)” 
location="${serverURL}/soa-infra/services/default/bpel-101-HelloWorld/helloworldprocess_client_ep?WSDL"  supports="" soapVersion="1.1">

You can also use applicationURL when you want to invoke another composite application on the same SOA platform, however, the target composite application must be deployed to the same SOA partition as the calling composite application. This is because the applicationURL token will be substituted at runtime with the soa-infra application URL with partition name of the calling composite application, for example: http://localhost:7001/soa-infra/services/default for the default SOA partition. Below is a snippet of web service binding in the composite.xml file:

<binding.ws port=”http://xmlns.oracle.com/bpel_101_HelloWorld_jws/bpel_101_HelloWorld/HelloWorldProcess#wsdl.endpoint(helloworldprocess_client_ep/HelloWorldProcess_pt)”  
location="${applicationURL}/bpel-101-HelloWorld/helloworldprocess_client_ep?WSDL" supports="" soapVersion="1.1">

When the target URL is resolved, the final endpoint URL will look like “http://localhost:7001/soa-infra/services/default/bpel-101-HelloWorld/helloworldprocess_client_ep?WSDL”

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

Careful consideration needs to be given to the manner in which files are created and written for (read) processing by the File and FTP Adapters

Main Article

Note

Hereinafter the term “the Adapter” means either the File or FTP Adapter. The abbreviation EA means External Application.

General usage of the Adapter

The objective of the Adapter (when reading files) is simple – viz. identify file(s) to be read, process the file(s), (optionally) delete the file and (optionally) archive the file. [ I recommend always archiving the files to aid recovery in case of disaster. ]
And so, conceptually, the requirements are extremely simple. But, as is often the case, the simplest of scenarios may present problems.

What’s the problem?

When either of the Adapters is configured, the file(s) to be processed will be identified according to some pattern. Furthermore, the Adapter will be configured to poll the source location at a predefined interval.
At each polling interval, the source location is examined for any file(s) that fit the pattern-matching criterion and (optionally) are of at least a certain age. More on this last point later.

When considering what problem(s) might arise, one needs to think about what the Adapter does internally when polling. In the simplest case, a source directory listing is acquired and all files observed are examined to see if they fit all of the criteria specified – i.e. name and (perhaps) age. If all criteria are matched then the file(s) will be processed.

The Adapter has no way of knowing if writing of the source file has been completed. This is the problem. A file that is still being written by the EA may be selected by the Adapter as a candidate for further processing.
However, if the file has not been completely written, then it is possibly going to be malformed or at the very least missing data.

Now we have to think about how the EA writes the file(s) into the source location. How does that application work? How big are the files? And, most importantly, how long does it take the EA to write the files? For example, the EA may create the file but then during its own acquisition of data to populate that file, there may be delays of indeterminate time.
If you know how long (at a maximum) the length of time taken from creation to full population is, then the minimum file age should be configured. And this is really the predicament.
Can you know without any doubt whatsoever that a file will have been completely written once it’s 5 seconds old or 2 minutes old or however old it is.
I suggest that the only reasonable answer to this is “probably” and that’s just not good enough for a robust implementation.
Therefore, configuring the Adapter for a file age is almost certainly a compromise.

The minimum age dilemma

This is a double-edged sword. Firstly there is the question of uncertainty around how long the EA will take to write a file. Secondly, the FMW developer will usually want to be able to process fully formed files as soon as they are ready. [ I recently worked with a client who had the unfortunate brief of providing near real-time processing with the FTP Adapter which was, quite frankly, an unrealistic proposition ]

The EA

It is not uncommon for the EA (i.e. the process by which files are written into the source location) to be some piece of legacy software that everyone’s afraid to touch or maybe not allowed to modify. If that is the case, then great care needs to be taken when considering the polling rate and the file age specification. Always allow generous margins beyond what you would hope to be reasonable. That’s probably the best you can do. Unfortunately, this is likely to mean that files will not be processed as quickly as they otherwise might. High rate polling (e.g. once per second) probably makes little sense if the file age is set to, say, 10 seconds!

However, if you do have control over the EA, then there is a better option.

A solution

I contend that the optimum solution is for the EA to create and write to a file in the source location using a name that will not match what any instance of the Adapter is expecting. In other words, it will be ignored.
Once the EA has finished writing, the file should be renamed to something that will be recognized by the Adapter.
If it is not practical for the EA to write what is effectively a temporary file into the source location, then there is another option but with limitations. The EA can write to some location that the Adapter knows nothing about. In this case, fairly obviously, the file can be created with its ultimate name. Once writing is complete, this file can be moved to its final location. But beware! The way the file is “moved” needs to be an atomic action. The way the “move” is performed may depend on the underlying operating system and / or the physical locations of the temporary and ultimate source locations. Think about the traditional Unix mv command and you’ll get the point.

If either of these strategies can be employed then the polling rate of the Adapter can be set to whatever best represents business needs. The minimum age should be set to zero. This works because we now know with absolute certainty that any file that matches the configured pattern is guaranteed to be ready for processing.

Addendum

The Adapter has a facility to work with a so-called “trigger” file. This is where a file with a well-known name (not pattern matched) is created by the EA (or something working in associated with it) in some well-known location. The mere existence of this file indicates to the Adapter that all / any files in the source location are ready for processing. This is most useful when, for example, source files may be constructed during a business working day for processing at some later time. At the appropriate time, the trigger file is created and files matching the criteria specified in the Adapter will be processed. It is essential when employing this strategy to ensure that there is no possibility that files that might otherwise match the selection criteria are created in the source location after the trigger file has been created.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

In Part 1 of the B2B Agreement Life-Cycle Management Series, we looked at the best practices to import high volume of agreement metadata into the repository [1]. In this post, we will take a look at the best practices to deploy the agreement metadata for run-time processing.

Background

B2B 11g supports the notion of agreements, which serve as the binding contract between different partners and documents defined within the repository. In order to facilitate run-time processing of messages, these agreements must be deployed so that the corresponding metadata can be queried for runtime validation of message content.

In production systems with large B2B repository containing several partners, documents and agreements, the system performance could be severely affected, if the deployment of the agreements is not managed properly.

This note highlights the best practices for deployment of agreements, when large numbers of partners and documents in the excess of several hundreds are required to be maintained within the B2B repository.

Symptoms

The run-time processing of inbound messages, typically go through a validation phase, where the message contents are tested against the metadata pre-defined for the particular document.

Usually these operations complete fairly quickly. However, as the number of deployed agreements within the repository goes up to several hundreds and beyond, the time to complete internal processing by the engine could go up by a significant amount. If that happens, it may be worth looking into the state of the MDS labels active within the repository.

Remedy

Bulk Deploy

All the agreements created for all trading partners and all document definitions within the B2B repository, can be deployed by a single invoke of the ant utility for B2B deploy. This would ensure that only 1 active label is created for all the agreements created. Thus, the number of labels within MDS will be at a minimum and not add aditional processing overhead on the system perfromance when labels need to be referred to at runtime for metedata retrieval.

Multi-Agreement Deploy

In certain situations, the operating constraints on a production system, may limit the possibility of carrying out a bulk deploy, as mentioned previously. In that case, it might still be helpful to deploy multiple agreements in batches. The greater the batch sizes are, the less will be the performance overhead, since they will reduce the number of active MDS labels in turn. The key objective here is to minimize the number of active MDS labels as much as possible.

Results

Let us take a look at the numbers of active MDS labels created as a result of our 2 types of agreement deployment operations.The examples cited here use the command-line utilities that are available as part of the B2B 11g install. These operations can also be performed via the B2B console, after selecting multiple agreements for deployment. However, for production systems, it is always recommended to develop custom scripts based on the command-line utilities available.

In the first case, we are deploying the 2 agreements were individually deployed one at a time.

• ant -f ant-b2b-util.xml b2bdeploy -Dtpanames=”MarketInc_OracleServices_X12_4010_850_Agr_In” 
  • ant -f ant-b2b-util.xml b2bdeploy -Dtpanames=”MarketInc_OracleServices_X12_4010_997_Agr_Out”

When we investigate the B2B_LIFECYCLE table, we see that there were 2 MDS labels created for each deployment of 2 agreements. It should also be noticed that both of these agreements and labels are in active state.

SQL> select state, count(*), count(distinct label)

from b2b_lifecycle
group by state
/

STATE             COUNT(*) COUNT(DISTINCTLABEL)
—————   —————- ——————————————
Active                           2                                         2

SQL>

Alternatively, when both of these agreements were deployed together, only 1 active MDS label was created for both the active agreements.

• ant -f ant-b2b-util.xml b2bdeploy -Dtpanames=”MarketInc_OracleServices_X12_4010_850_Agr_In,MarketInc_OracleServices_X12_4010_997_Agr_Out”

SQL> select state, count(*), count(distinct label)

from b2b_lifecycle
group by state
/

STATE             COUNT(*) COUNT(DISTINCTLABEL)
—————    —————- ——————————————
Active                           2                                         1

SQL>

The basic syntax for bulk deploy is the same as shown earlier, without any agreement names in the command-line. e.g.

• ant -f ant-b2b-util.xml b2bdeploy

For few agreements, the bulk deployment can also be achieved via the B2B console UI, by selecting all the agreements before clicking for deployment. However, in general, UI is not recommended for deployment. Scripts using the command-line utilities should be developed for deployment of agreements as a best practice.

After the deployment is completed, all the existing active agreements and labels will go into an inactive state. A standard maintenance procedure should be implemented to purge all the inactive agreements and labels for proper housekeeping.

Summary

The key objective is to minimize the number of active MDS labels generated as a result of deployment of agreements for run-time processing. Ideally, a bulk deploy would result in 1 active MDS label for the entire repository.

In situations where bulk deploy is not possible, minimizing the number of active MDS labels should be a top priority. So, techniques like deployment of several agreements in a comma-separated list via one command-line invoke should be used in non-ideal situations, whenever possible.

Acknowledgements

The material posted here has been compiled with the help from B2B Engineering and Product Management teams.

References

1. B2B Agreement Life-Cycle Mgmt Part 1 – Best Practices for High Volume Import of CPA Metadata into B2B Repository. https://blogs.oracle.com/ateamsoab2b/entry/best_practices_for_high_volume

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Background

SOA Suite for Healthcare Integration pack comes with a pre-installed simulator that can be used as an external endpoint to generate inbound and outbound HL7 traffic on specified MLLP ports. This is a command-line utility that can be very handy when trying to build a complete end-to-end demo within a standalone, closed environment.

The ant-based utility accepts the name of a configuration file as the command-line input argument. The format of this configuration file has changed between PS4 and PS5. In PS4, the configuration file was XML based and in PS5, it is name-value property based.

The rest of this note highlights these differences and provides samples that can be used to run the first scenario from the product samples set.

PS4 – Configuration File

The sample configuration file for PS4 is shown below.

The configuration file contains information about the following items:

• Directory for incoming and outgoing files for the host running SOA Suite Healthcare
• Polling Interval for the directory
• External Endpoint Logical Names
• External Endpoint Server Host Name and Ports
• Message throughput to be simulated for generating outbound messages
• Documents to be handled by different endpoints

A copy of this file can be downloaded from here.

PS5 – Configuration File

The corresponding sample configuration file for PS5 is shown below.

The configuration file contains similar information about the sample scenario but is not in XML format. It has name-value pairs specified in the form of a properties file. This sample file can be downloaded from here.

Simulator Configuration

Before running the simulator, the environment has to be set by defining the proper ANT_HOME and JAVA_HOME. The following extract is taken from a working sample shell script to set the environment:

Also, as a part of setting the environment, template jndi.properties and logging.properties can be generated by using the following ant command:

• ant -f ant-b2bsimulator-util.xml b2bsimulator-prop

Sample jndi.properties and logging.properties are shown below and can be modified, as needed.

The jndi.properties contains information about connectivity to the local Weblogic Managed Server instance and the logging.properties file controls the amount of logging that can be generated from the running simulator process.

Simulator Usage – Start and Stop

The command syntax to launch the simulator via ant is the same in PS4 and PS5. Only the appropriate configuration file has to be supplied as the command-line argument, for example:

• ant -f ant-b2bsimulator-util.xml b2bsimulatorstart -Dargs=”simulator1.hl7-config.xml”

This will start the simulator and will keep running to provide an active external endpoint for SOA Healthcare Integration engine. To stop the simulator, a similar ant command can be used, for example:

• ant -f ant-b2bsimulator-util.xml b2bsimulatorstop

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

Integration flows can fail at run-time with a variety of errors. The cause of these failures could be either Business errors or System errors.  When Synchronous Integration Flows fail, they are restarted from the beginning. On the other hand, Asynchronous Integration flows when they error can potentially be resubmitted/recovered from designated/pre-configured milestones within the flow. These milestones could be persistence points like queues topics or database tables, where the state of the flow was last persisted. Recovery is a mechanism whereby a faulted Asynchronous Flow can be rerun from such a persistence milestone.

The SOA Suite 11g and AIA products provides various Automated and Manual recovery mechanisms to recover from asynchronous fault scenarios. They differ based on the SOA component that encounters the error. For instance, recovering from a BPEL fault may be quite different than recovering from a Resequencer fault. In this blog, we look at the various Manual Recovery mechanisms and options available to an end user. Manual recovery mechanisms require an Admin user to take appropriate action on the faulted instance from the Oracle Enterprise Manager Fusion Middleware Control [EM FMWC Console]

The intention of this blog is to provide a quick reference for Manual Recovery of Faults within the SOA and AIA contexts. It aims to present some of the valuable information regarding Manual recovery in one place. These are currently available across many sources such as SOA Developers Guide, SOA Admin Guide, AIAFP Developers Guide and AIAFP Infrastructure and Utilities Guide.

Next we look at the various Manual recovery mechanisms available in SOA Suite 11g and AIA, starting with the BPEL Message Recovery.

BPEL Message Recovery

To understand the BPEL Message Recovery, let us briefly look into how BPEL Service engine performs asynchronous processing. Asynchronous BPEL processes use an intermediate Delivery Store in the SOA Infrastructure Database to store the incoming request. The message is then picked up and further BPEL processing happens in an Invoke Thread. The Invoke Thread is one among the free threads from the ‘Invoke Thread Pool’ configured for BPEL Service Engine. The processing of the message from the delivery Store onwards until the next dehydration in the BPEL process or the next commit point in the flow constitutes a transaction. Figure below shows at a high level the Asynchronous request handling by BPEL Invoke Thread.

Any unhandled errors during this processing will cause the message to roll back to the delivery Store. The Delivery Store acts as a safe milestone for any errors that cause the asynchronous BPEL processing to rollback. In such scenarios these messages sitting in the delivery store can be resubmitted for processing using the BPEL Message Recovery.

It is quite similar in case of Callback messages that arrive for in-flight BPEL process instances. The Callback messages are persisted in the Delivery Store and a free thread from the Engine Thread Pool will perform correlation and asynchronously process the callback activities. Callback messages from Faulted activities are available at the Delivery Store for Recovery.

Refer to the section from FMW Administrator’s Guide here – http://docs.oracle.com/cd/E28280_01/admin.1111/e10226/bp_config.htm#CEGFJJIF for details on configuring the BPEL Service Engine Threadpools.

The Recovery of these Invoke/Callback messages can be performed from the Oracle Enterprise Manager Fusion Middleware Control [EM FMWC Console] [SOA->Service Engine->BPEL->Recovery]. The admin user can search for recoverable messages by filtering based on available criteria on this page. The figure below shows the BPEL Engine Recovery page where the messages eligible for recovery are searched based on the message type and state.

During Recovery of these messages, the end user cannot make any modifications to the original payload. The messages marked recoverable can either be recovered or aborted. In the former case, the original message is simply redelivered for processing again.

The BPEL Configuration property ‘MaxRecoverAttempt’ determines the number of times a message can be recovered manually or automatically. Messages go to the exhausted state after reaching the MaxRecoverAttempt. They can be selected and ‘Reset’ back to make them available for manual/automatic recovery again.

The BPEL Service Engine can be configured to automatically recover failed messages, either on Server startup or during scheduled time periods. Refer to the section from FMW Administrator’s Guide here – http://docs.oracle.com/cd/E28280_01/admin.1111/e10226/bp_config.htm#CDEHIIFG for details on setting up auto recovery.

SOA Fault Recovery

The Fault Handling for invocations from SOA Components can be enhanced, customized and externalized by using the Fault Management Framework (FMF). We will not go into the details of Fault Management Framework here. Refer to this a-team blog post here – http://www.ateam-oracle.com/fault-management-framework-by-example for insights into the FMF.

In short FMF, allows a Fault Policy with configurable Actions to be bound to SOA Component. These can be attached at the Composite, Component or Reference levels. The configured Actions will be executed when the invocation fails. The available Actions could be retry, abort, human intervention, custom java callout, etc. When the Action applied is human intervention the faults become available for Manual Recovery from the Oracle Enterprise Manager Fusion Middleware Control [EM FMWC Console]. They show up as recoverable instances in the faults tab of ‘SOA->faults and rejected messages’ as shown in the figure below

During the recovery, the Admin user can opt for one among Retry, Replay, Abort, Rethrow or Continue as the Recovery option. For Retry options, the EM User has access to the payload. The payload can be changed and resubmitted during recovery. While this is a useful feature, it could pose audit/security issue from an administrative perspective if it is not properly controlled using Users/Roles.

The retry action can also chain the execution into a custom java callout to do additional processing after a successful Retry. This is selected from the ‘After Successful Retry’ option during Retry. The custom java callout should be configured in the Fault Policy file attached to the composite.

Resequencer Recovery

Mediator Resequencer groups which end up in Errored or Timed Out states can be recovered from the EM Console by an Admin user. In fact Resequencer faults do not have other automated recovery mechanisms and rely on only Manual recovery by the admin for their remediation. Mediator Resequencer faults can be searched and filtered from the faults page of the Mediator component. Figure below shows a search of faults by the Resequencing Group.

An Errored group can be recovered by either choosing to Retry or Abort. Retry will reprocess the current failed message belonging to the faulted Group. In case of abort, the current failed message is marked as failed and processing will resume from the next in sequence available message for the faulted Group. In both cases the group itself is unlocked and set to ready so that it can process further messages. As can be seen in the figure below, the Admin user can modify the request payload during this recovery.

In case of Standard Resequencer, groups can end up as Timed Out when the next in sequence message does not arrive until the timeout period. Such groups can be recovered by skipping the missing message. Figure below shows such a recovery. In this case the processing of the group will continue from the next message rather than wait for the missing sequence id.

AIA Message Resubmission

This section deals with Integrations built using the AIA Foundation Pack. Refer to the AIA Concepts and Technologies Guide at http://docs.oracle.com/cd/E28280_01/doc.1111/e17363/toc.htm to familiarize with the AIA concepts. Let us see a common design pattern employed in AIA Integrations. The Figure below is from the AIA Foundation Pack Developers Guide Document available and shows an architecture used for Guaranteed Message Delivery between Source and Target applications with no intermediate persistence points. The blocks shown are SOA Suite Composites. The Source and Target milestones are persistence points such as Queue, Topics or Database tables. The same design can also be enhanced to have multiple intermediate milestones in case of more complex flows.

Such Flows are commonly seen in AIA Pre Built Integrations which use Asynchronous flows to integrate systems. E.g. Communications O2C Integration Pack for Siebel, BRM and OSM. Refer to the Pre Built Integrations Documentation here-  http://www.oracle.com/technetwork/apps-tech/aia/documentation/index.html

The salient points of this design are

• A Single transaction performs the message consumption from source milestone, the processing and the message delivery to target milestone
• All blocks are implemented using SOA Suite composites
• Any errors/faults in processing, rollback the message all the way to the source milestone
• Milestone Queues and Topics are configured with Error Destinations to hold the rolled back messages for resubmission.
• An enhanced Fault message (AIAFault) is raised and stored in the AIA Error Topic. This Fault has sufficient information to resubmit the message from the Source milestone.

The Faults can be recovered using the AIA Error Resubmission Utility. In AIA Foundation Pack 11.1.1.7.0 the AIA Error Resubmission Utility is a GUI utility and can be used for single or bulk fault recoveries. The Resubmission Utility can be accessed from AIA home-> Resubmission Utility of the AIA application, as shown in figure below. Earlier versions of AIA foundation Pack 11g only have a command line utility for Error Resubmission. This is available at <AIA_HOME>/util/AIAMessageResubmissionUtil.

Any fault within the flow will roll back to the previous milestone or recovery point and enable resubmission from that point. The milestones could be Queues, Topics or AQ destinations. The Queues and Topics designed to be milestones are associated with corresponding Error Destinations. This is where the faulted messages reside. The AIA Resubmission Utility simply redelivers the messages from the fault destination back to the milestone destination for reprocessing in case of Queue or Topic.

In the case of Resequencer errors, the Resequencer is the Recovery point and holds the message for resubmission. Note that Resequencer is not typically designed as a milestone in the flow but acts as a recovery point for Resequencer errors. For such errors, the AIA Resubmission utility recovers the failed Resequencer message and also unlocks the faulted Resequencer Group for further processing.

It is important to note here that the AIA Error Handling and Resubmission Mechanism is a designed solution. It relies on the fact that the Integration implements the principles and guidelines of AIA Foundation Pack and AIA Guaranteed Message Delivery pattern for its accurate functioning.

Refer to the AIA Foundation Pack Infrastructure and Utilities Guide at http://docs.oracle.com/cd/E28280_01/doc.1111/e17366/toc.htm for details of the AIA Error Handling Framework and AIA Resubmission utility. Refer to the AIA Foundation Pack Developers Guide at http://docs.oracle.com/cd/E28280_01/doc.1111/e17364/toc.htm for implementing AIA Error Handling and Recovery for the Guaranteed Message Delivery Pattern.

Use Case: Message Resubmission with Topic Consumers

Let us next look at a use case from one of our customer engagements. It is a Custom Integration developed using AIA Guaranteed Message Delivery pattern and employing the AIA Resubmission utility for recovery. We can see how the above recovery mechanisms offer different options when designing a typical integration flow

Without going deep into the details, the figure below shows at a high level the design used for delivering messages to 3 End Systems using a Topic and 3 Topic Consumers. The BPEL ABCS components consume the canonical message, convert it to the respective Application specific formats and deliver to the End Systems. The requirement was the guarantee delivery of the message to each of the 3 systems within the flow, which the design achieves under normal circumstances.

However issues were observed at run-time for failure cases. When the message delivery fails for one of the Systems e.g. System B, the design caused a rollback of the message to the previous milestone which in this case is the Topic. The rolled back message residing in the Error destination is then redelivered to the Topic. The message is picked for processing again by all 3 Topic Consumers causing duplicate message delivered to Systems A and C.

This issue can be addressed in a few ways;

1) Introducing an intermediate milestone after the message is consumed off the topic. For instance we could introduce a queue to hold the converted messages. (Indicated by point 1 in the figure)

2) Use separate queues instead of a topic to hold the canonical messages.

In case of failures, only the message in the failed branch would have to be recovered using AIA Message Resubmission as seen in section above.

However, both these options introduce additional queues which need to be maintained by the operations team. Also if in future an additional end systems were to be introduced, it would necessitate adding new queues in addition to new JMS consumers and ABCS components.

3) Introduce a transaction boundary: This can be done by changing the BPEL ABCS component use an Asynchronous One Way Delivery Policy. In this case, any failures cause the message to rollback not to the topic but to the internal BPEL Delivery store. (Indicated by point 2 in the figure) These messages can then be recovered manually using Bpel Message Recovery as we saw in the first section above. The recovery is limited only to the faulted branch of the integration.

4) Another option is to employ Fault Policies. We can attach a Fault Policy to the BPEL ABCS component. The policy invokes the human intervention action for faults encountered during end system invoke. The message can then be manually recovered from the EM FMWC Console as seen in the SOA Fault Recovery Section above. This would apply only to the faulted branches and hence avoid the duplicate message delivery to the other End Systems.

Also another issue seen was that the end systems would lose messages that arrived when the consumers are offline. This problem can be addressed by configuring durable subscription for the Topic consumers. In the absence of Durable subscriptions, a Topic discards a message once it has been delivered to all the active subscribers. With Durable Subscribers, the message is retained until delivered to all the registered durable subscribers hence guaranteeing message delivery. Refer to the Adapters Users Guide here – http://docs.oracle.com/cd/E28280_01/integration.1111/e10231/adptr_jms.htm for details on configuring Durable Subscriptions for Topic Consumers.

Summary

Table below summarizes the different Manual recoveries that we have seen and their main characteristics

In this blog, we have seen the various Manual Fault Recovery mechanisms provided by SOA Suite and AIA 11g versions. We have also seen the security requirements for the Admin user to perform recovery and the variety of options available to handle the faults. This knowledge should enable us to design and administer robust integrations which have definite points of recovery.

-Shreeni

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

This blog is intended to share some knowledge about the effects of Queue Level Redelivery Settings and Adapter level Retry Settings on message processing by JMS Adapter.  It is also intended to provide some useful insights that help in designing retry mechanisms into an integration system. Specifically, this blog illustrates the Retry behavior of JMS Adapter and how it is impacted by these settings.

Detail

Consider an Integration system that uses JMS Adapter to consume message from a queue and deliver to an End System after performing BPEL processing. The Figure below depicts such a simple system. Note that the source queue is configured with an error queue to hold the failed messages.

Adapter level Retry Settings

First, let’s look at the Adapter level Retry Settings that are available to configure on the JMS Adapter Consumer service. The settings below are typically used to configure the retry behavior of inbound JMS Adapter consumers:

2. Jca.retry.count –> example value 3
   Jca.retry.interval –> example value 2
   Jca.retry.backoff –> example value 2

Assume that the end system is down. During such an error condition when the message cannot be successfully processed and delivered to the End System, the JMS Adapter retries the processing of the failed message, using the above retry settings. For the above example values, the adapter retries a failed message after 2 , 6  and 14 seconds from the time of first failure.

Now, assume the end system is down even after the 3 retries.  The expectation in most integration flows is that the message rolls back to the source queue and can be found in the error destination. This will help in manually recovering the failed message after the error condition is resolved.

All messaging providers support some form of queue redelivery settings. For instance, a Weblogic JMS has the Redelivery Limit setting, and AQ JMS provides the same using the max_retries setting of a queue.

Note that Weblogic JMS can be configured to discard or log the failed messages instead of redirecting to an error destination.

Failure to Rollback

The table below lists some sample values of the settings and the behavior observed after repeated failures:

Summary

Incorrect settings of the queues and adapters could lead to undesired behavior in recovery of messages during failure conditions. We have seen a few such situations in this blog. With proper settings, we can design integration systems to exhibit consistent error handling and recovery behaviour.

References

-Shreeni

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

In 10g, there was a mechanism for suppressing consumption of messages at the Adapter level. That mechanism can not be used in 11g. But there is a way…

Main Article

The way to do this is to set the InboundThreadCount in the appropriate MBean to zero. This will effectively suppress consumption of messages – e.g. from MQ or JMS or whatever. Setting this value to something greater than zero will cause consumption to resume.  Making such a change is dynamic – i.e. no restart required.

This situation is easily handled through Enterprise Manager and can be done in either of two ways.

1. Navigate through the System MBean Browser and make the change there
2. Identify the composite, click through the appropriate item in the Services & References section, select the Properties tab, make the property change and Apply

Either of these techniques are reasonable when there are very few composites to manage.

However, now consider the situation when your installation has many Adapters across many Composites. And, for whatever reason, you need to (effectively) stop or restart the Adapters.  In this case, a programmatic approach will be less cumbersome and more efficient.

What we need is a program that uses a control file that defines the Services in the context of the Composite that uses it / them

Here’s the definition of the control file:-

<?xml version="1.0"?>
<xs:schema version="1.0"
           xmlns:xs="http://www.w3.org/2001/XMLSchema"
           elementFormDefault="qualified">
    <xs:element name="AdapterControl">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="compositeDetail" minOccurs="1" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element name="composite" type="xs:string"/>
                            <xs:element name="revision" type="xs:string"/>
                            <xs:element name="partition" type="xs:string"/>
                            <xs:element name="service" type="xs:string"/>
                            <xs:element name="location" type="xs:string"/>
                            <xs:element name="application" type="xs:string" default="soa-infra" minOccurs="0"/>
                            <xs:element name="threads" type="xs:positiveInteger" default="1"  minOccurs="0"/>
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>

So that’s the schema definition. It’s self-explanatory. Now let’s look at an actual control file that manages just one Adapter (defined by its Service name) within a specific Composite.

<?xml version="1.0" encoding="UTF-8"?>
<AdapterControl
    xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
    <compositeDetail>
        <composite>MQResponder</composite>
        <revision>1.1</revision>
        <partition>default</partition>
        <service>MQInbound</service>
        <location>AdminServer</location>
        <application>soa-infra</application>
        <threads>1</threads>
    </compositeDetail>
</AdapterControl>

All of the elements apart from <threads> are needed to identify the MBean that needs to be modified. The MBean object name is complex. Here’s an example:-

oracle.soa.config:SCAComposite.SCAService=%SERVICE%,name=AdapterBinding,revision=%REVISION%,partition=%PARTITION%,SCAComposite=\"%COMPOSITE%\",Location=%LOCATION%,label=%LABEL%,j2eeType=SCAComposite.SCAService.SCABinding,Application=%APPLICATION%

Actually, that’s not a real example. What you see here is an MBean object name with embedded tokens. The application that I’ll present here replaces these tokens prior to performing the lookup. Note that the control XML does not contain the “label”. This is derived from the other data provided in the control file.

If you review the XSD at this point, you will note that the <compositeDetail> element may repeat. Furthermore, note that the <application> and <threads> elements are optional.

The <threads> element is needed where users run with multiple adapter threads. If you only ever have one thread, then you don’t need it. When stopping a service, this element is not used (because we’re going to set the InboundThreadCount to zero).

How it works

The application usage is as follows:-

StopStart stop|start host port username password control_file

More than likely, you’ll run it something like this:-

java -jar AdapterControl.jar start localhost 7001 weblogic welcome1 “c:\users\aknight\my documents\AdapterController.xml”

The program does not validate the XML control file against the XSD at runtime. Bad things will happen if the XML structure is non-conformant. But don’t worry too much as it can’t damage your runtime system.

Internally, the application is using the Fabric API (Locator) to work out what the %LABEL% needs to be. Other than that, it’s using javax.management.MBeanServerConnection to find and invoke the appropriate method on the MBean.

The application distribution is available here. The Java code is included in the JAR. I have also included all dependent JARs in the distribution (so it’s rather large).

The application is offered as is. It is not a tool offered formally by Oracle.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

This is a very quick observation on a simple performance tuning fix for SOA Suite on Exalogic.

The problem

SOA Suite appears to grind to a halt when a load is imposed upon it, when running on Exalogic. CPU may or may not be spiked on SOA at this point in time. SOA may become completely unresponsive, or just be very slow. You may see 504 gateway timeout errors, servers in doubt in the admin server screen, or other symptoms of a “barely responding” SOA Suite system.

The resolution

Turn ON “Always use keep-alive” for the origin server in Oracle Traffic Director (this defaults to OFF). Information on how to enable it at http://docs.oracle.com/cd/E23389_01/doc.11116/e21036/perf014.htm.

The details

The recommended architecture for SOA Suite on Exalogic is to use Oracle Traffic Director to route webservice callouts between SOA Composites. This allows for broad load distribution as well as some resiliency against failure. It also provides the capability to use Infiniband class connection speeds and latency between SOA Composites, which is not a bad thing.

Oracle Traffic Director, as detailed in the documentation above, defaults to NOT using HTTP Keep Alive for PUT and POST requests to it’s origin servers. For SOA on Exalogic, the SOA Suite servers are considered origin servers.

SOA Suite requests are 99% POST requests. This means that every request between composites (if you are using the recommended setup) will generate a NEW HTTP connection at the remote SOA Server. This causes a build up of “stale” connections, which are slow to garbage collect for technical reasons. Eventually, with sufficient load, the pile up will be so great that SOA ends up stuck in a Garbage collection loop and will either throw OutOfMemoryError or slow to a crawl. By changing the Oracle Traffic Director setting to true, Oracle Traffic Director will cache and reuse the same HTTP connection for multiple requests (it actually has a small pool of cached connections), the build up won’t happen, and performance will dramatically improve as a result.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Dealing with high volume of transactions in Oracle Business Activity Monitoring (BAM) 11g can be challenging. Making the right design decision is critical for the stability and performance of your system. This post covers the common issues and general design recommendations for Oracle BAM 11g in the context of high volume of transactions.

The common issue in the context of  high volume of transactions, is that Active Data processing at the BAM Server Side may not catch up the speed of data received in BAM Active Data Cache (ADC), which in turn may cause slight or serious delays in report refreshing with real-time data. When there are a huge backlog of unprocessed data in BAM, the performance and functionality of the whole system will be significantly impacted. Now, let’s take a look at the following report design aspects which are the key factors impacting the scalability and performance, and our recommendations

Star Schema with High Volume of Data

It is common to use Star Schema as the data model in BAM to store business data. Whenever a message is received by BAM, it will be stored in the schema, and triggers Active Data processing within BAM Server, Under small or median load, say less than 10 – 20 messages per second (the rate of incoming messages received by BAM ADC), Active Data processing can keep up with the incoming data rate, and report should be working as expected. However, if the load goes higher, say over 100 messages per second, Active Data processing will consume higher resources (e.g. CPU, Memory, IO, etc), and the chance of performance degradation will be higher as well. Thus, our main design consideration here is to find out a way to shrink the size and data volumes of the star schema.

Recommendations

1. If you have one single Star Schema that contains a lot of look-up and calculated fields, consider breaking the one big Star Schema into multiple smaller schemas. The benefit of having multiple schemas is that the size and data volumes for each schema is reduced, therefore reducing server side contention and Active Data processing time.

2. If the transaction volume is high, say more than 60 transactions per second, consider adding additional pre-processing for the business data before sending them to BAM. For example, you can use Oracle Event Processing (OEP) to apply filters or aggregation for the business data before sending them to BAM. Using this approach, the data volume in BAM will be dramatically reduced.

3. Another approach for dealing with the high volume of data is to use Poll Mode versus Push Mode for BAM dashboards. BAM Dashboards in Poll Mode bypass Active Data Processing and reload itself at a time interval. Under heavy load, this approach consumes less system resources (CPU, Memory, Threads, etc.), and therefore performs better in terms of throughput and response time. The charts below shows the difference between Push and Poll Mode in terms of CPU usage and thread count.

CPU and Thread Count – Push Mode

CPU and Thread Count – Poll Mode

As you can see, when a report is running in Push Mode, CPU usage is consistently around 20%, and the system creates more threads for handling active data. From the performance perspective, a report in Poll Mode uses less resources and perform better under heavy load.

4. Consider normalizing your current data model to reduce the data volume of a single Data Object. If the main fact table (main Data Object of Star Schema) contains redundant fields that cause the data volume growing dramatically, consider normalizing the data model by moving these fields to separate Data Objects or External Data Objects. If you need to drill down the current view to show the values of these redundant fields, create a new report displaying these fields in Poll Mode and use the drill across feature to link the original view to the new view. 

Manipulating Data in Views

BAM reports can include data manipulation functions, such as filters, calculations, drill down, drill across, driving, etc. Data manipulation is expensive in BAM, and can seriously impact performance if overused in BAM reports under heavy load. Applying filters, drill down, or drill across in a BAM report will get all report views reloaded. Reloading a report is expensive as it requires querying database, re-compiling internal classes built for evaluating calculated fields, and establishing persistent HTTP connections between client and server. Frequent reloading will increase the changes of getting contentions for accessing server resources.

Recommendations

1. Minimize the usage of data manipulating functions if possible. If these functions have to be used, ensure the underling Data Object does not contain high volume of data. In the context of high transaction rate, consider normalizing Data Objects instead of using single Star Schema or using Poll Mode for dashboards.

2. If drill down or drill across function is used in the report design, we recommend that you use separate Data Objects for the main and target view. Using separate Data Objects can help to reduce the data volume, which is the key factor for improving performance under heavy load.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

In Fusion Applications, the status of SOA composite instances are either running, completed, faulted or staled. The composite instances become staled immediately (irrespective of current status) when the respective composite is redeployed with the same version. The messages (payload) are stored in SOA audit tables until they are purged. The users can go through Enterprise Manager and view audit trails and respective messages of each composite. This is good for debugging composite instances. However there are situations where you want to re-submit initiation of SOA composite instances in bulk for the following reasons:

• The composite was redeployed with the same version number that resulted in all respective instances (completed successfully, faulted or in-flight) becoming stale (“Staled” status)
• Instances failed because down-stream applications failed and the respective composite did not have an ability to capture the initial message in persistence storage to retry later

In these cases, it may be necessary to capture the initial message (payload) of many instances in bulk to resubmit them. This can be managed programmatically through SOA Facade API. The Facade API is part of Oracle SOA Suite’s Infrastructure Management Java API that exposes operations and attributes of composites, components, services, references and so on. As long as instances are not purged, the developer can leverage SOA Facade API to retrieve initial messages of either Mediator or BPEL components programmatically. The captured messages can be either resubmitted immediately or stored in persistence storage, such as file, jms or database, for later submission. There are several samples, but this post takes the approach of creating a SOA composite that provides the ability to retrieve initial message of Mediator or BPEL components. The sample provides the frame work and you can tailor it to your requirements.

Main Article

SOA Facade API

Please refer to this for complete SOA Facade API documentation. The SOA audit trails and messages work internally as follows:

• The “Audit Level” should be either Production or Development to capture the initial payload
• The “Audit Trail Threshold” determines the location of the initial payload.  If the threshold is exceeded, the View XML link is shown in the audit trail instead of the payload. The default value is 50,000 bytes. These large payloads are stored in a separate database table: audit_details.

Please refer to the following document for more details on these properties.

Since the SOA composite we are developing will be deployed in the same respective SOA Server, you do not require user credentials to create the locator object. This is all you need:

Locator locator = LocatorFactory.createLocator();

Please see the SOA Facade API document for more information the Locator class.

Once the Locator object is created, you can lookup composites and apply various filters to narrow down the search to respective components. This is all explained in detail with examples in the SOA Facade document. Here, we focus on how to retrieve the initial messages of the Mediator and BPEL components to resubmit them.

How to retrieve initial payload from BPEL?

In BPEL, the initial payload is either embedded in the audit trail or has a link to it. This is controlled by the audit trail threshold value. If the payload size exceeds the audit threshold value then the audit trail has a link. This is the main method to get audit trail:

auditTrailXml = (String)compInst.getAuditTrail
/* The “compInst” is an instance Component that is derived from: */
Component lookupComponent = (Component)locator.lookupComponent(componentName);
ComponentInstanceFilter compInstFilter = new ComponentInstanceFilter(); compInstFilter.setId(componentId);

If the payload size exceeds the audit threshold value, then the actual payload is an XML link that is stored in the “audit_details” table. The following is the API facade to get it:

auditDetailXml = (String)locator.executeComponentInstanceMethod(componentType +”:”+ componentId, auditMethod, new String[]{auditId});

The “auditId” for SOA is always “0″.

How to retrieve initial payload from Mediator

The initial payload in Mediator is never embedded in the Audit Trail. It is always linked and the syntax is similar to BPEL (where payload size exceeds the audit threshold value). However, the “auditID” is in the Mediator audit trail and it must be parsed to get that value for the initial payload. This is the code snippet to get the “auditId” from Mediator audit trail:

if (componentType.equals("mediator")) {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document document = db.parse(new InputSource(new StringReader(auditTrailXml)));
NodeList nodeList = document.getElementsByTagName("event");
String attribute = nodeList.item(0).getAttributes().getNamedItem("auditId").getNodeValue();
addAuditTrailEntry("The Audit is: " + attribute); 
auditId = attribute;auditMethod="getAuditMessage";} 

/* Once you have the "auditID" from above code, the syntax to get the initial payload is the same as in BPEL.*/
auditDetailXml = (String)locator.executeComponentInstanceMethod(componentType +":"+ componentId, auditMethod, new String[]{auditId});

Complete Java embedded code in BPEL

try { 
String componentInstanceID = new Long(getInstanceId()).toString();    
addAuditTrailEntry("This Run time Component Instance ID is "+componentInstanceID);  

XMLElement compositeNameVar = (XMLElement) getVariableData("inputVariable", "payload", "/client:process/client:compositeName");
String compositeName = compositeNameVar.getTextContent();  

XMLElement compositeIdVar = (XMLElement) getVariableData("inputVariable", "payload", "/client:process/client:compositeId");
String compositeId = compositeIdVar.getTextContent();  

XMLElement componentTypeVar = (XMLElement) getVariableData("inputVariable", "payload", "/client:process/client:componentType");
String componentType = componentTypeVar.getTextContent();  

XMLElement componentNameVar = (XMLElement) getVariableData("inputVariable", "payload", "/client:process/client:componentName");
String componentName = componentNameVar.getTextContent();  

XMLElement componentIdVar = (XMLElement) getVariableData("inputVariable", "payload", "/client:process/client:componentId");
String componentId = componentIdVar.getTextContent();  

String auditDetailXml = "null";
String auditTrailXml = "null";
String auditMethod = "getAuditDetails";
String auditId = "0";

addAuditTrailEntry("The lookup Composite Instance Name is "+compositeName);  
addAuditTrailEntry("The lookup Composite Instance ID is "+compositeId);  
addAuditTrailEntry("The lookup Component Instance Name is "+componentName);
addAuditTrailEntry("The lookup Component Instance Type is " + componentType);
addAuditTrailEntry("The lookup Component Instance ID is "+componentId);  

Locator locator = LocatorFactory.createLocator();  
Composite composite = (Composite)locator.lookupComposite(compositeName);  
Component lookupComponent = (Component)locator.lookupComponent(componentName);  

ComponentInstanceFilter compInstFilter = new ComponentInstanceFilter();  

compInstFilter.setId(componentId);

List<ComponentInstance> compInstances = lookupComponent.getInstances(compInstFilter);  
if (compInstances != null) {  
    addAuditTrailEntry("====Audit Trail of Instance===");  
    for (ComponentInstance compInst : compInstances) {  
        String compositeInstanceId = compInst.getCompositeInstanceId(); 
        String componentStatus = compInst.getStatus(); 
        addAuditTrailEntry("Composite Instance ID is "+compositeInstanceId);  
        addAuditTrailEntry("Component Status is "+componentStatus);  

        addAuditTrailEntry("Get Audit Trail");
        auditTrailXml = (String)compInst.getAuditTrail();

        if (componentType.equals("mediator")) {
            DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
            DocumentBuilder db = dbf.newDocumentBuilder();
            Document document = db.parse(new InputSource(new StringReader(auditTrailXml)));
            NodeList nodeList = document.getElementsByTagName("event");
            String attribute = nodeList.item(0).getAttributes().getNamedItem("auditId").getNodeValue();
            addAuditTrailEntry("The Audit is: " + attribute);

            auditId = attribute;
            auditMethod="getAuditMessage";
            }

        addAuditTrailEntry("Received Audit Trail");

        addAuditTrailEntry("Get Audit Details of: "+ componentType +":"+ componentId + "for auditId: " + auditId);

        try {
            auditDetailXml = (String)locator.executeComponentInstanceMethod(componentType +":"+ componentId, auditMethod, new String[]{auditId});
        } catch (Exception e) { 
        addAuditTrailEntry("Exception in getting audit details:" + e);
        }

        addAuditTrailEntry("Received Audit Details");

        setVariableData("auditTrailString", "payload", "/client:AuditTrailString/client:auditTrail", auditTrailXml);
        setVariableData("auditDetailString", "payload", "/client:AuditDetailString/client:auditDetail", auditDetailXml);

        addAuditTrailEntry("BPEL Variables set");
    }  
} 

} catch (Exception e) { 
    addAuditTrailEntry("Exception in getting Audit Trails and Details"); 
}

The sample payload to run above composite is:

    <element name="process">
        <complexType>
            <sequence>
                <element name="compositeName" type="string"/>
                                <element name="compositeId" type="string"/>
                                <element name="componentType" type="string"/>
                                <element name="componentName" type="string"/>
                                <element name="componentId" type="string"/>
            </sequence>
        </complexType>
    </element>

Sample Code

Please get the complete Jdeveloper Project as follows:

1. DummySOAApplication to retrieve initial payload of Mediator and BPEL components

2. The SOA Audit Trail Composite “SOAAuditTrails” that contains the logic to get initial payload of “Dummy Composite”

3. Sample Payload “SOA_audit_payload”

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

In PS3 (11.1.1.4) and PS4 (11.1.1.5), the SOA-INFRA application will not be able start up when you set both immediate and deferred audit policy MBean attributes to active. This is a known bug (13384305), and there is a patch for PS5 (11.1.1.6) to resolve this issue, and there is also a cumulative patch (18254378) for PS4. If you need a quick workaround to start up the SOA-INFRA application before the patch is fully tested, this blog describes how to find the MBean configuration in the MDS schema and change the value in order to start up the SOA-INFRA application.

When you encountered this issue, the Weblogic console would display the server status as “RUNNING” but SOA-INFRA wouldn’t show up in EM Console. In the SOA server log file, you would see the following exception:

[/WEB-INF/fabric-config-core.xml]: Cannot resolve reference to bean
  'DOStoreFactory' while setting bean property 'DOStoreFactory'; nested
  exception is org.springframework.beans.factory.BeanCreationException: Error
  creating bean with name 'DOStoreFactory' defined in ServletContext resource
  [/WEB-INF/fabric-config-core.xml]: Invocation of init method failed; nested
  exception is java.lang.NullPointerException
  at
  org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolve
  Reference(BeanDefinitionValueResolver.java:275)
  at
  org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolve
  ValueIfNecessary(BeanDefinitionValueResolver.java:104)
  at
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.
  applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1245)
  at
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.
  populateBean(AbstractAutowireCapableBeanFactory.java:1010)
  at
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.
  doCreateBean(AbstractAutowireCapableBeanFactory.java:472)

The audit policy attribute settings are stored in the MDS schema. There are 3 tables we can use to fix this problem: MDS_ATTRIBUTES, MDS_COMPONENTS and MDS_PARTITIONS. The MDS_COMPONENTS table stores the version information; the latest version has the highest value in the COMP_CONTENTID column.

select * from mds_components

The MDS_ATTRIBUTES stores the attribute values of the MBean configuration properties, in this case “audit-config”. As we are only interested in audit-config settings related to the SOA-INFRA application for this issue, we can find the correct partition id for the SOA-INFRA application in MDS_PARTITIONS table (see below):

select * from mds_partitions

To find the latest audit-config attribute values, run the following SQL statement to retrieve the latest audit policy configuration:

SELECT *
FROM MDS_ATTRIBUTES
WHERE ATT_CONTENTID=
(SELECT MAX(COMP_CONTENTID)
FROM MDS_COMPONENTS
WHERE COMP_LOCALNAME = 'audit-config'
)
AND MDS_ATTRIBUTES.ATT_PARTITION_ID=
(SELECT PARTITION_ID FROM MDS_PARTITIONS WHERE PARTITION_NAME='soa-infra'
);

The default configuration is shown below:

To resolve this issue as the workaround, you will need to ensure that one of the active attribute value (either immediate or deferred) is set to “true”, and the other attribute value is set to “false”, then you will be able to proceed to start the SOA-INFRA application.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

11g Resequencer Health Check

In this Blog we will see a few useful queries to monitor and diagnose the health of resequencer components running in a typical SOA/AIA Environment.

The first query is a snapshot of the current count of Resequencer messages in their various states and group_statuses.

Query1: Check current health of resequencers

select to_char(sysdate,'YYYY-MM-DD HH24:MI:SS') time, gs.status group_status, m.status msg_status, count(1), gs.component_dn 
from mediator_group_status gs, mediator_resequencer_message m
where m.group_id = gs.group_id
and gs.status < = 3
and gs.component_status!=1
group by gs.status, m.status, gs.component_dn
order by gs.component_dn, gs.status;

Table below lists a representative sample output of the above query from a running SOA Environment containing Resequencers collected at 12:04:50

For our analysis, let us collect the same data again after a few seconds

Refer to the appendix  for a quick glossary of Resequencer group_status and message_status state values

Let us dive a bit deeper into each of the above state combinations, their counts and what they imply.

1. GRP_STATUS/MSG_STATUS = 0/0 – READY

These show the messages which are ready for processing and eligible to be Locked and processed by the resequencer.  For a healthy system, this number would be quite low as the messages will be locked and processed continuously by the resequencer.  When the messages arriving into the system have stopped, this count should drop to zero.

A high count for this combination would suggest that not enough groups are being locked by the resequencer for the rate at which messages are arriving for processing.  The Mediator property – “Resequencer Maximum Groups Locked” should be adequately increased to lock groups at a higher rate.

Refer here to see how this property can be changed from EM Console

2. GRP_STATUS=0/MSG_STATUS=2 – PROCESSED

This count indicates the number of processed messages. This number will be seen to be growing over time. A Very high count (like > 1 million in the above example) indicates that the Resequencer purge is due and should be run soon to delete the processed messages.

2. 3. GRP_STATUS=0/MSG_STATUS=5 – ABORTED

   This count shows the number of message that are currently manually aborted by the administrator.  Refer here for how Resequencer messages can be aborted using the SOA EM Console.

2. 4. GRP_STATUS=1/MSG_STATUS=0 – LOCKED

   This combination of states shows the messages within groups which are being currently processed. For a healthy system, this number would be quite low as the messages belonging to locked groups are processed continuously by the Resequencer Worker threads.  When the messages arriving into the system have stopped, this count should drop to zero.

A high count for this combination would suggest that not enough worker threads are available to process the messages for the rate at which groups are locked for processing.  The Mediator property – “Resequencer Worker Threads” should be adequately increased to boost the message processing rate.

Refer here to see how this property can be changed from EM Console

5. GRP_STATUS=1/MSG_STATUS=2 – LOCKED

The count against this combination shows the number of messages which are processed for locked groups. This is a transient state and once all messages for the locked groups are processed, these counts change status to GRP_STATUS=0/MSG_STATUS=2

6. GRP_STATUS=3 – ERRORED

These show the messages against error’ed groups. These will need to be manually recovered from EM Console or the AIA Resubmission tool. They indicate messages which have failed processing due to various errors. If these messages can be recovered and processed successfully, in which case they transition to state GRP_STATUS=0/MSG_STATUS=2. If the errors are non recoverable, then they can be aborted from the EM Console and they move to GRP_STATUS=0/MSG_STATUS=5.

Refer to my earlier blog here for details on recovery of resequencer errors.

Query2: Check ContainerID’s  health

select * from MEDIATOR_CONTAINERID_LEASE ;

Table below shows a sample output for the above query from a 2 node clustered SOA installation.

It shows that time when both the nodes last renewed their mediator containerids. These containerid renewals serve as heartbeats for the mediator/Resequencer. It is vital in maintaining the load balance of messages among the nodes and failover of groups/messages that were allocated to expired nodes.

Query3: Load Balance between cluster nodes

select to_char(sysdate,'YYYY-MM-DD HH24:MI:SS') time, gs.container_id container, gs.status group_status, m.status msg_status, count(1)
from mediator_group_status gs, mediator_resequencer_message m
where m.group_id = gs.group_id
and   gs.status  in (0,1)
and component_status!=1 
group by  gs.container_id, gs.status, m.status
order by gs.container_id, gs.status;

The above query can be used to monitor the load balance of messages between nodes of a cluster. Sample output below shows an output for a 2 node clustered SOA environment.

This sample output shows the messages of ready and locked messages are roughly evenly distributed across the cluster. If a major skewness is observed for a specific container, then further analysis may be required. Thread dumps and Diagnostic logs of the slower node may indicate towards the cause of the skewness.

Appendix:

Below table lists the important status values of MEDIATOR_GROUP_STATUS and MEDIATOR_GROUP_STATUS tables and how the values can be interpreted.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

One of the consequences of Asynchronous SOA-based integration patterns is that it does not guarantee that messages will reach their destination in the same sequence as initiated at the source.

Ever faced an integration scenario where

- an update order is processed in the integration layer before the create order?

- the target system cannot process two orders for the same customer?

Common fixes used in the field include

- Singleton BPEL implementations, singleton JCA adapters, custom sequencing logic using tables etc.

These common ‘fixes’ often result in performance bottlenecks since all messages are usually funneled through a single threaded component. These approaches also become unreliable and counter-productive when used in clustered deployments. Error scenarios can also cause unexpected behavior.

To address the sequencing requirement without these shortcomings, Oracle SOA Suite provides the Mediator Resequencer component that allows you to build/rebuild a sequence from an out-of-sequence set of input messages. The Resequencer enforces sequential processing of related messages and performs parallel processing of unrelated messages, thereby keeping up the performance levels.

The white paper below aims to provide a common set of use cases for using a Resequencer, Resequencer modes, best practices, configurations, handling error scenarios, HA, failover, etc.

Oracle Mediator Resequencer.pdf

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

In 11g, mediator executes routing rules either sequentially or in parallel.  If you are planning to use parallel routing rules, you would need to understand how the mediator queues and evaluates routings in parallel in different threads. This article describes 2 different threads used in the parallel routing rules and the design consideration if you are planning to use parallel routing rules in your implementation.

Main Article

By using parallel routing rules, services can be designed to be truly asynchronous. However, the service engine executes these requests in a rather unique fashion. Let’s say that the average time taken to complete execution of each service is one second. If you were to receive 10 requests on the first Mediator service and the Mediator Service Engine is configured with 10 threads, the expectation is that all requests would be completed within one second. However, that is not the case. Let me elaborate further.

Let’s say you have 3 mediator services deployed to your SOA server and each of these has a single parallel routing rule. When the mediator service received a message, the message would be inserted in Mediator store by the dispatcher, its message metadata would be written to the MEDIATOR_DEFERRED_MESSAGE table, and the payload goes into MEDIATOR_PAYLOAD table. All these would occur on the original calling thread.

The mediator service engine has one thread called Locker thread. The locker thread would surface the message metadata from the MEDIATOR_DEFERRED_MESSAGE table into an internal memory queue. The locker thread does this in its own transaction. The MEDIATOR_DEFERRED_MESSAGE table also has a state listed below:
0 – READY
1 – LOCKED
2 – COMPLETED SUCCESSFULLY
3 – FAULTED

Hence, it is important to understand how the locker thread works behind the scene, as this would affect your design decision:

The locker thread has an algorithm to process the message stored by the dispatcher, and there is only 1 locker thread per managed server. After the dispatcher has stored the message data in the MEDIATOR_DEFERRED_MESSAGE table, the locker thread will lock message(s) that has a state=“0”, the number of message to be locked by the locker thread will be dependent on the Parallel Maximum Rows Retrieved parameter that you have set in the EM->SOA-INFRA->SOA Administration->Mediator Properties. The locker thread will cycle through 1 mediator component at a time and check if there are any requests to process from the internal queue. It will process the message(s) by changing its state to “1” for that component for processing, and then sleep for the configured interval defined in the “parallel locker thread sleep” setting before moving on to the next mediator component. If it finds no message to process, it would move on to the next, and the next, until it loops back to the first composite, where it would then process its next request.

For example: If there are 3 mediator components m1, m2, m3 with priority 1, 2, 3 then the algorithm goes as lock m1 -> sleep -> m2 -> sleep -> m3 -> sleep -> m2 -> sleep -> m3 -> sleep -> m3, in 6 iterations of locker thread, m1 messages are locked once, m2 messages are locked twice and m3 messages are locked 3 times as per the priority. All this happen in a single Locker cycle. Only after the locker thread locks the retrieved messages will they be queued for the worker threads to process the messages. So if you have many mediator components (e.g. 50 mediator components) with parallel routing rule, it will take a considerable amount of time for the locker thread to lock the message to complete one locker cycle. If you have mediator components with lower priority, it will take a longer time for the locker thread to lock the message for the low priority mediator component. The locker cycle will be reset if you undeploy or deploy a new mediator component with parallel routing rules, this is to ensure the mediator component with higher priority will be processed in the next cycle.  You will be able to observe these behaviors when you set logging level to the Trace:32 FINEST level in Oracle Enterprise Manager Fusion Middleware Control.

• oracle.soa.mediator.common
• oracle.soa.mediator.service
• oracle.soa.mediator.dispatch
• oracle.soa.mediator.serviceEngine

Example of Trace:32 diagnostic log:

[2013-12-02T16:06:21.696-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: run] Locker running
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: lockMessages] Trying to obtain locks
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: beginTransaction] Transaction  begins
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] TransactionManager status
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] Getting Transaction status
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: beginTransaction] TransactionManager begin
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] TransactionManager status
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] Getting Transaction status
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.dispatch.db] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.dispatch.db.DeferredDBLocker] [APP: soa-infra] [SRC_METHOD: lock] Obtaining locks for max rows 200
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.dispatch.db] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.dispatch.db.DeferredDBLocker] [APP: soa-infra] [SRC_METHOD: lock] Removing ABCS/MediatorTest!1.0*soa_e302c3ae-9b29-4bd5-802a-6052389f7be3/Mediator1 from counter 
[2013-12-02T16:06:21.697-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.dispatch.db] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.dispatch.db.DeferredDBLocker] [APP: soa-infra] [SRC_METHOD: lock] Obtaining locks for ABCS/MediatorTest!1.0/Mediator1 and counter 1 at index 0
[2013-12-02T16:06:21.703-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: lockMessages] Obtained locks
[2013-12-02T16:06:21.703-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: commitTransaction] Commiting Transaction
[2013-12-02T16:06:21.703-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] TransactionManager status
[2013-12-02T16:06:21.704-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] Getting Transaction status
[2013-12-02T16:06:21.704-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: commitTransaction] TransactionManager commit
[2013-12-02T16:06:21.713-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: enqueueLockedMessages] Spining ...........Sleeping for 1000 milliseconds oracle.tip.mediator.dispatch.db.DeferredDBLocker
[2013-12-02T16:06:22.716-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: run] Locker running
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: lockMessages] Trying to obtain locks
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: beginTransaction] Transaction  begins
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] TransactionManager status
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] Getting Transaction status
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: beginTransaction] TransactionManager begin
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] TransactionManager status
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] Getting Transaction status
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.dispatch.db] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.dispatch.db.DeferredDBLocker] [APP: soa-infra] [SRC_METHOD: lock] Obtaining locks for max rows 200
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.dispatch.db] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.dispatch.db.DeferredDBLocker] [APP: soa-infra] [SRC_METHOD: lock] Removing ABCS/MediatorTest!1.0*soa_b0b789b9-0114-4138-ac50-9331e044af38/Mediator2 from counter 
[2013-12-02T16:06:22.717-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.dispatch.db] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.dispatch.db.DeferredDBLocker] [APP: soa-infra] [SRC_METHOD: lock] Obtaining locks for ABCS/MediatorTest!1.0/Mediator2 and counter 1 at index 0
[2013-12-02T16:06:22.724-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: lockMessages] Obtained locks
[2013-12-02T16:06:22.724-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: commitTransaction] Commiting Transaction
[2013-12-02T16:06:22.724-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] TransactionManager status
[2013-12-02T16:06:22.724-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: getTransactionStatus] Getting Transaction status
[2013-12-02T16:06:22.724-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.JTAHelper] [APP: soa-infra] [SRC_METHOD: commitTransaction] TransactionManager commit
[2013-12-02T16:06:22.735-07:00] [WLS_SOA1] [TRACE:32] [] [oracle.soa.mediator.common.listener] [tid: Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms\n] [userId: <anonymous>] [ecid: a88161838353ad87:-406b0e26:142b4bfd15c:-8000-0000000000000004,0] [SRC_CLASS: oracle.tip.mediator.common.listener.DBLocker] [APP: soa-infra] [SRC_METHOD: enqueueLockedMessages] Spining ...........Sleeping for 1000 milliseconds oracle.tip.mediator.dispatch.db.DeferredDBLocker

After the locker thread locked the message, the worker thread will retrieve the message from the in memory queue and process the message. The number of worker thread can be tuned by changing the Parallel Worker Threads property in EM->SOA-INFRA->SOA Administration->Mediator Properties. Once the message is processed, the worker thread will change the state of the message to either “2” –Completed successfully or “3” – Faulted.

The engine was designed in order to prevent starving of threads caused by load on a single composite. What the engine wants to avoid is that, if you have a Mediator service that has received hundreds of thousands of requests and another one having received two requests, each service is given a fair amount of time to be serviced, otherwise the two requests may have to wait for hours to execute. Thus, the three settings to consider in asynchronous Mediator services are the following:

• The Parallel Locker Thread Sleep setting: This is defined at the Mediator Service Engine level
• The number of threads allocated to the Mediator Service Engine: This is defined by the Parallel Worker Threads parameter
• The Priority property: Which is set at design time and applicable only to parallel routing rules

Another important point to note is when a mediator service engine is started, it registers itself in the database table called MEDIATOR_CONTAINERID_LEASE and gets a container ID. This is important because when the row is inserted into the MEDIATOR_DEFERRED_MESSAGE table, it round-robins’s the deferred message to one of its containers, the engine will then assigns the ID of the container that should process the message.

Hence, I would strongly suggest and recommend that you would take into account the list of design considerations that I have listed below when you are designing your composite using mediator parallel routing rules:

• The priority property is only applicable to the parallel routing rules, so you need to consider the mediator component priority base on your business requirement.
• The locker thread will cycle through all mediator components with parallel routing rules deployed in your environment regardless of whether it has been retired or shutdown.
• Use sequential routing rules if latency is important and that you are expecting the message to be processed without delay.
• If you have well over 100 parallel mediator components deployed in your environment, the time to complete the locker cycle grew exponentially and could not be further tuned because there is only 1 locker thread and the lowest parallel locker thread sleep time that you can set is 1 second.
• If you have a mediator component that contain both sequential and parallel routing rules, sequential routing rules will be executed before parallel routing rules.
• Fault policy is only applicable to parallel routing rules only. For sequential routing rules, the fault goes back to the caller and it is the responsibility of the caller to handle the fault. If the caller is an adapter, then you can define rejection handlers on the inbound adapter to take care of the errored out messages, that is, the rejected messages.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

This article explains how to encrypt sensitive data (such as ssn, credit card number, etc ) in the incoming payload and decrypt the data back to clear text (or original form) in the outgoing message. The purpose is to hide the sensitive data in the payload, in the audit trail, console and logs.

Main Article

Oracle provides Oracle Web Services Manager (OWSM) message protection, but it encrypts the entire payload. However, Oracle OWSM gives us the capability to create our own custom policies and custom assertions. The framework is implemented in Java and allows us to write our own custom assertions which can be attached to a policy to encrypt and decrypt message data. These policies must be attached to the SOA composites in order to execute the policy assertion.

Step by step guide:

1. 1. Create a custom Java encryptor class

This is the Java implementation class for encrypting the data in incoming messages. It must extend oracle.wsm.policyengine.impl.AssertionExecutor and must have the method execute

 public IResult execute(IContext iContext)

This method is invoked by the policy framework. The execute method  gets the xml nodes in the SOAP message that require encryption from the SOAP message and encrypts the value. It then sets the node value to the encrypted value.

1. 2. Create a custom Java decryptor class

This is the Java implementation class for decrypt the data in outgoing message. It must extend oracle.wsm.policyengine.impl.AssertionExecutor and must have method execute

 public IResult execute(IContext iContext)

This method is invoked by the policy framework. The execute method  gets the xml nodes in the SOAP message that require decryption from the SOAP message and decrypts the value. It then sets the node value to the decrypted value.

3. Compile and build Java encryptor and decryptor in a jar file

Required libraries are:

$ORACLE_COMMON_HOME\modules\oracle.wsm.common_11.1.1\wsm-policy-core.jar

$ORACLE_COMMON_HOME\modules\oracle.wsm.agent.common_11.1.1\wsm-agent-core.jar

$ORACLE_COMMON_HOME\modules\oracle.osdt_11.1.1\osdt_wss.jar

$ORACLE_COMMON_HOME\modules\oracle.osdt_11.1.1\osdt_core.jar

4. Copy the jar file to $SOA_HOME\soa\modules\oracle.soa.ext_11.1.1

5. Run ant in $SOA_HOME\soa\modules\oracle.soa.ext_11.1.1

6. Restart SOA server

7. Create a custom encryption assertion template

This custom assertion template calls the custom Java encryptor class which encrypts the message data.

When this assertion is attached to a policy that is attached to the SOA composite web service then whenever a request is made to a SOA composite service, OWSM applies the policy enforcement and the execute method of the custom encryptor Java class is invoked.

<orawsp:AssertionTemplate xmlns:orawsp="http://schemas.oracle.com/ws/2006/01/policy"
                          orawsp:Id="soa_encryption_template"
                          orawsp:attachTo="generic" orawsp:category="security"
                          orawsp:description="Custom Encryption of payload"
                          orawsp:displayName="Custom Encryption"
                          orawsp:name="custom/soa_encryption"
                          xmlns:custom="http://schemas.oracle.com/ws/soa/custom">
  <custom:custom-executor orawsp:Enforced="true" orawsp:Silent="false"
                   orawsp:category="security/custom"
                   orawsp:name="WSSecurity_Custom_Assertion">
    <orawsp:bindings>
      <orawsp:Implementation>fully qualified Java class name that will be called by this assertion </orawsp:Implementation>
      <orawsp:Config orawsp:configType="declarative" orawsp:name="encrypt_soa">
        <orawsp:PropertySet orawsp:name="encrypt">
          <orawsp:Property orawsp:contentType="constant"
                           orawsp:name="encryption_key" orawsp:type="string">
            <orawsp:Value>MySecretKey</orawsp:Value>
          </orawsp:Property>
        </orawsp:PropertySet>
      </orawsp:Config>
    </orawsp:bindings>
  </custom:custom-executor>
</orawsp:AssertionTemplate>

8. Use Enterprise Manager (EM) to import the custom encryption assertion template into the Weblogic domain Web Services Policies

9. Create an assertion using the encryption assertion template that was imported

10. Create custom decryption assertion template

This custom assertion template calls the custom Java decryptor class which decrypts the message data.

When this assertion is attached to a policy that is attached to the SOA composite web service then whenever a request is made to that SOA composite web service then OWSM applies the policy enforcement  and the execute method of the custom outbound decryptor is invoked.

<orawsp:AssertionTemplate xmlns:orawsp="http://schemas.oracle.com/ws/2006/01/policy"
                          orawsp:Id="soa_decryption_template"
                          orawsp:attachTo="binding.client" orawsp:category="security"
                          orawsp:description="Custom Decryption of payload"
                          orawsp:displayName="Custom Decryption"
                          orawsp:name="custom/soa_decryption"
                          xmlns:custom="http://schemas.oracle.com/ws/soa/custom">
  <custom:custom-executor orawsp:Enforced="true" orawsp:Silent="false"
                   orawsp:category="security/custom"
                   orawsp:name="WSSecurity Custom Assertion">
    <orawsp:bindings>
      <orawsp:Implementation>fully qualified Java class name that will be called by this assertion</orawsp:Implementation>
      <orawsp:Config orawsp:configType="declarative" orawsp:name="encrypt_soa">
        <orawsp:PropertySet orawsp:name="decrypt">
          <orawsp:Property orawsp:contentType="constant"
                           orawsp:name="decryption_key" orawsp:type="string">
            <orawsp:Value>MySecretKey</orawsp:Value>
          </orawsp:Property>
        </orawsp:PropertySet>
      </orawsp:Config>
    </orawsp:bindings>
  </custom:custom-executor>
</orawsp:AssertionTemplate>

11. Create an assertion using the decryption assertion template that was imported

1. 12. In Enterprise Manager (EM), export custom encryption policy to a file and save it to $JDEV_USER_DIR/system11.1.1.x.x.x.x/DefaultDomain/oracle/store/gmds/owsm/policies/oracle

13. In Enterprise Manager (EM), export custom decryption policy to a file and save it to $JDEV_USER_DIR/system11.1.1.x.x.x.x/DefaultDomain/oracle/store/gmds/owsm/policies/oracle

14. In JDeveloper, attach the custom encryption policy to the SOA composite inbound services that require message data encryption

15. In JDeveloper, attach custom decryption policy to the SOA composite outbound services that have message data are in encryption format but need to be decrypted for outbound message

16. Compile and deploy the SOA composite

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

As SOA Suite for HealthCare (HC) gains popularity among providers, I have seen the need to separate the SOA Suite (SOA) and SOA Suite for HealthCare into separate Weblogic domains. Generally this is done for performance reasons; more specifically it is done when the customer has a high transaction throughput rate coupled with relatively short and stringent service level agreements. In a multi-domain architecture such as this, you will need to use a method other than the default, in-memory binding to pass messages between your SOA Composite and HealthCare.

SOA Suite for HealthCare can access JMS queues from other domains in one of three ways: you can setup and use Weblogic’s Store and Forward capability for JMS, you can create a foreign JMS server in the domain and use the local jndi references when creating the queues for HC, or you can specify the details of the remote JNDI location in the Destination Provider attribute of an Internal Delivery Channel (IDC) and include the IDC in your endpoint.

For this article, we will be using the later method where the remote JNDI location is defined in the transport details of the Internal Delivery Channel. I think this is the simplest method to use and since HC persists all of its messages in its own repository, you don’t have to worry about losing them if the remote JMS provider is down. Consider the message flow in the following diagram:

Figure 1 – Message flow across separate HC and SOA Domains

Here messages flow from a single endpoint via MLLP to the HealthCare adapter in the HC domain. The HealthCare adapter processes the message and writes it remotely to the queue RMT_ADM_GENERIC_ADT in the SOA domain. The SOA composite reads the message locally from RMT_ADM_GENERIC_ADT, processes the message, and then writes it to another local queue RMT_LAB_GENERIC_ADT. SOA Suite for HealthCare then reads the message remotely from RMT_LAB_GENERIC_ADT, processes it, and sends it to the target endpoint via MLLP. Here is how to configure a remote JMS queue in SOA Suite for HealthCare.

Transport Details

Either create or open an Internal Delivery Channel (IDC) to be used to access the remote JMS queue. Add the remote JNDI details to the transport protocol parameters of the IDC. Both sending and receiving IDCs can access remote JMS queues. In the IDC, click the Transport Details button to open a pop-up window where the details are entered.

Figure 2 – Adding the Destination Name and Connection Factory to the Internal Delivery Channel

In the Basic tab, add the destination name for the remote JMS queue in the Destination Name field. Next enter the name of the remote Connection Factory used to connect to the queue. Now switch to the Advanced tab.

Figure 3 – Adding the Destination Provider Location, Username, and Password to the Internal Delivery Channel

In the Destination Provider field under the Advanced tab, add the following JNDI destination location details for the remote JMS provider. Here is a sample of what the location information will look like:

java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory;java.naming.provider.url=t3://soahost:8001

Provide the hostname and port number for the remote JMS provider. Add a valid username for the remote location along with the password, then save and enable the channel. After enabling the channel, you should see an additional listener on the remote queue when you look at it in the monitoring tab of the WLS console.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

During the design of SOA applications it is inevitable that from time to time you will need to interface with Microsoft-based applications. While technologies like SOAP and REST do a great job when request-reply communication is needed, most people struggle when a messaging-based communication is required. This blog will present two approaches to get messaging working between Microsoft and SOA Suite 12c.

Which Choices Do I have?

SOA Suite 12c offers a complete set of tools to integrate with Microsoft applications using messaging. Which one to use is a simple question of asking where the messaging system resides. If the messaging system to be accessed sits on SOA Suite side (WebLogic JMS) then you should use the WebLogic JMS .NET Client. If the messaging system to be accessed sits on Microsoft side (Microsoft Message Queuing) then you should use the JCA adapter for MSMQ. Using the WebLogic JMS .NET Client allows code written in .NET to access the WebLogic JMS server using the T3 protocol, just like any other Java application. Using the JCA adapter for MSMQ allows SOA composites and OSB applications to send/receive messages to/from MSMQ queues.

Using the WebLogic JMS .NET Client

The implementation of the WebLogic JMS .NET Client is very straightforward. All you have to do is deploy your .NET application with the WebLogic.Messaging.dll assembly file. You still need to code how your application will send/receive messages to/from the WebLogic JMS destinations. You can easily find the WebLogic.Messaging.dll assembly file in the following location: $FMW_HOME/wlserver/modules/com.bea.weblogic.jms.dotnetclient_x.x.x.x.

In the same location you can find the WebLogic JMS .NET Client API documentation. For those of you that are familiar with the JMS API, it will be easy to understand since the API design is almost the same. For beginners, I have provided the following C# sample code that shows how to publish messages to an WebLogic JMS queue.

using System;
using System.Collections.Generic;
using System.Text;
using WebLogic.Messaging;

namespace com.oracle.fmw.ateam.soa
{

    public partial class SampleDotNetApplication
    {

        private void sendMessage()
        {

            IConnectionFactory wlsConnectionFactory;
            IQueue ordersQueue;

            IDictionary<string, Object> environment;
            IContext jndiContext;

            IConnection connection = null;
            ISession session = null;
            IMessageProducer messageProducer = null;
            ITextMessage message = null;

            try
            {

                environment = new Dictionary<string, Object>();
                environment[Constants.Context.PROVIDER_URL] = "t3://soa.suite.machine:8001";
                environment[Constants.Context.SECURITY_PRINCIPAL] = "weblogic";
                environment[Constants.Context.SECURITY_CREDENTIALS] = "welcome1";
                jndiContext = ContextFactory.CreateContext(environment);

                wlsConnectionFactory = jndiContext.LookupConnectionFactory("jms/wlsConnectionFactory");
                ordersQueue = (IQueue) jndiContext.LookupDestination("jms/ordersQueue");

                connection = wlsConnectionFactory.CreateConnection();
                connection.Start();

                session = connection.CreateSession(Constants.SessionMode.AUTO_ACKNOWLEDGE);
                messageProducer = session.CreateProducer(ordersQueue);
                message = session.CreateTextMessage();
                message.SetStringProperty("customProperty", "123456789");
                message.Text = "<message>Oracle SOA Suite 12c Rocks</message>";
                messageProducer.Send(message);

            }
            finally
            {

                messageProducer.Close();
                session.Close();
                connection.Stop();
                connection.Close();

            }

        }

    }

}

Note that in the sample code above, an initial context object is instantiated during every method call. The code was written this way for clarity purposes but in the real world you should avoid this practice. That could lead to potential performance issues as explained in the next section. As a best practice instantiate only one initial context object per CLR process.

// environment = new Dictionary<string, Object>();
// environment[Constants.Context.PROVIDER_URL] = "t3://localhost:7101";
// environment[Constants.Context.SECURITY_PRINCIPAL] = "weblogic";
// environment[Constants.Context.SECURITY_CREDENTIALS] = "welcome1";
// jndiContext = ContextFactory.CreateContext(environment);

/* Get the JNDI context from an local cache manager
 * instead of creating it multiple times. The first
 * call create the instance and the subsequent calls
 * just retrieve it from a private static variable. */

jndiContext = JndiContextManager.getInstance();

wlsConnectionFactory = jndiContext.LookupConnectionFactory("jms/wlsConnectionFactory");
ordersQueue = (IQueue)jndiContext.LookupDestination("jms/ordersQueue");

The above code was slightly modified to handle the JNDI context management, using the Singleton design pattern to restrict the number of instances into just one.

Special considerations when using the WebLogic JMS .NET Client

Before moving applications from development/staging to production, maybe you should step back and take a look in some of the following guidelines:

* Make sure that the T3 protocol is available in the managed server accessed by the .NET application. When pointing to the admin server you don’t have to worry about this, but perhaps other managed servers must have networks channels configured to get this protocol enabled.

* Use the -Dweblogic.protocol.t3.login.replyWithRel10Content=true JVM property to allow WebLogic JMS .NET client applications written prior of 12.1.3 version to interoperate with the 12.1.3 version.

* Be aware that the following features are not currently supported: Queue browsers, XA transactions, SSL, HTTP tuneling, SAF clients, multicast subscribers and automatic reconnect.

* How you implement your messaging code has a significant impact on the overall performance of your application and will affect the rate of message production and/or consumption. According to the Microsoft CLR specification, each process uses a fixed thread pool with 25 threads per available processor. Each time you create an initial context that uses the T3 protocol you burn a thread from that pool and also create a socket connection to the WebLogic server. On the WebLogic server side you also create a thread to handle requests coming from that socket, using the traditional socket-reader muxer thread pool. That means if you create a large number of concurrent initial context objects, there will be a correspondingly large number of socket connections to manage in the client thread pool. You can also run out threads in the client application if the number of initial contexts created exceeds the thread pool size. Consider using one shared initial context per process to optimize the client thread pool and minimize the performance hit on the server incurred when there are too much searches in the JNDI tree.

Using the JCA Adapter for MSMQ

This adapter leverages the jCOM technology available in WebLogic to provide connectivity to the MSMQ server. The first thing to do is enable jCOM in all servers where the adapter will be deployed. You can easily do this using the WebLogic administration console. In the managed server settings page, go to the “Protocols” > “jCOM” tab. Select the “Enable COM” check box as shown in the screen shot below:

Due to the nature of the JCA adapter, you will also need to create an outbound connection pool. In the deployments page, search for the “MSMQAdapter” and then go to the “Configuration” > “Outbound Connection Pools” tab. Create a new javax.resource.cci.ConnectionFactory and provide it with a proper JNDI name. After that click in the newly created outbound connection pool and go to the “Properties” tab. Here is the summary of the main properties:

After setting these properties, you can optionally go to the “Connection Pool” tab and fine tune the connection pool, specifically the “Initial Capacity”, “Max Capacity” and “Capacity Increment” parameters. That’s it, this is the minimal configuration needed to start using the JCA adapter for MSMQ. The following section discusses some special considerations for this adapter.

Special considerations when using the JCA adapter for MSMQ

Using the JCA adapter for MSMQ in JDeveloper is equal to any other SOA Suite technology adapter. All you need to do is drag it from the components palette to the composite designer and inform the wizard about the JNDI name of the adapter and details about the queue.

But before start deploying applications into the SOA server, review the follow recommendations to help ensure connectivity, high availability and performance:

* If you intend to access public queues and/or distribution lists, an Active Directory Domain Services (AD_DS) must be configured on a Windows 2008 Server system. This requirement does not apply for private queues.

* When the SOA server is not co-located with the MSMQ server or is installed in an operating system different than Windows (e.g.: Linux, Solaris) you need to use DCOM as access mode. in that case you need to set the value of the property “AccessMode” to “DCOM”. In addition, you need to install the MSMQ DCOM Proxy in the machine where the MSMQ server is running.

* When the MSMQ Adapter needs to make an outbound connection to the MSMQ server, it must sign on with valid security credentials. In accordance with the J2CA 1.5 specification, the WebLogic server supports both container-managed and application-managed sign-on for outbound connections. The MSMQ adapter can leverage either of these methods to sign on to the EIS. The credentials must include a user that has proper permissions to interact with the MSMQ server otherwise you will get exceptions during deployment.

* The MSMQ adapter supports high availability through the active-active topology. It has a poller thread to poll the queue for the next available message. Each poller thread uses the MSMQ receive API and only removes the message from the queue after successful read. This ensures there is no message duplication when the MSMQ Adapter is deployed in an active-active topology.

* Use the adapter.msmq.dequeue.threads binding property to increase the number of poller threads during endpoint activation. The default value of this property is “1″ which is good for simple tests but with a higher value you can achieve a better degree of parallelism. You can set this property at runtime using the Enterprise Manager FMW Control Console.

* Enabling streaming during MSMQ message consumption can significantly reduce the SOA server memory footprint when large payloads are used, specially if there is a mediator applying content-based routing rules before delivering the message for processing.

Conclusion

Enterprise application developers and architects today rarely ask the “Why SOA?” question. They are more often asking about how to implement SOA using best practices in order to build robust and scalable applications that maximize their SOA infrastructure investment. SOA Suite 12c has been designed to be the “Industrial SOA” solution that organizations need to deliver these solutions. Hopefully this blog has provided some useful information and best practices on integrating Microsoft-based applications with SOA Suite 12c using messaging.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

In a previous blog post, we saw a few useful tips to help us quickly monitor the health of resequencer components in a soa system at runtime. In this blog post, let us explore some tips to diagnose mediator resequencer issues. During the diagnosis we will also learn some key points to consider for Integration systems that run Mediator Resequencer composites.

Please refer to the Resequencer White paper for a review of the basic concepts of resequencing and the interplay of various subsystems involved in the execution of Resequencer Mediator composites.

Context

In this blog post we will refer to the AIA Communications O2C Pre-Built Integration pack (aka O2C PIPs) as an example for understanding some issues that can arise at runtime with resequencer systems and how we can diagnose the cause of such issues. The O2C PIP uses resequencing-enabled flows. One such is the UpdateSalesOrder flow between OSM and Siebel. It is used to process the OSM status of Sales Orders in proper time sequence within the Siebel system.

Isolate the server within the soa cluster

Many a times the resequencer health check queries point us to an issue occurring only in one server within the soa cluster. While the Database queries mentioned here give us the containerId of the specific server, it does not specify the server name. This is because mediator uses a GUID to track a runtime server.

Trace Log messages generated by the Mediator can help us correlate this GUID to an individual server running in the cluster at runtime. The oracle.soa.mediator.dispatch runtime logger can be enabled from the FMW EM console to TRACE:32 level. Figure below shows the screenshot.

Enabling this logger just for a few minutes will suffice and one can see messages such as below in soa servers’ diagnostic logs, once every lease refresh cycle. The default refresh cycle is 60s apart.

[APP: soa-infra] [SRC_METHOD: renewContainerIdLease] Renew container id [34DB0F60899911E39F24117FE503A156] at database time :2014-01-31 06:11:18.913

It implies, the server which logged the above message is running with a containerId of 34DB0F60899911E39F24117FE503A156 !

Locker Thread Analysis

When one observes excessive messages piling up with a status of GRP_STATUS=READY and MSG_STATUS=READY, it usually indicates that the locker thread is not locking the groups fast enough for processing the incoming messages. This could be due to Resequencer Locker thread stuck or performing poorly. For instance the locker thread could be stuck executing updates against the MEDIATOR_GROUP_STATUS table.

It is generally useful to isolate the server which is creating the backlog using health check queries and then isolate the server name by using the logger trace statements as described in previous section. Then a few thread dumps of this server, could throw more light on the actual issue affecting the locker thread.

Usually thread dumps show a stack such as below for a resequencer Locker thread.

"Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms
" id=330 idx=0x1b0 tid=28794 prio=10 alive, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at oracle/tip/mediator/common/listener/<strong>DBLocker.enqueueLockedMessages</strong>(DBLocker.java:213)
    at oracle/tip/mediator/common/listener/DBLocker.run(DBLocker.java:84)
    at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:120)
    at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
    at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

In the above thread the Locker is enqueing messages from locked groups into the in memory queue for processing by the worker threads.

During times of any issue, the Locker thread could be seen stuck doing database updates. If this is seen across thread dumps with no progress made by the thread, then it could point to a database issue which needs to be attended.

A poor performance of the locker query on the database side will adversely impact the Resequencer performance and hence decrease the throughput of the integration flow that uses Resequencers.

Recollect that the Locker thread runs an update query continuously attempting to lock eligible groups. Below shown is a sample FIFO Resequencer Locker query as seen database AWR reports.

update mediator_group_status a set a.status=7 where id in ( select id from (select distinct b.id, b.lock_time from 
mediator_group_status b, mediator_resequencer_message c where b.id=c.owner_id and b.RESEQUENCER_TYPE='FIFO' and 
b.status=0 and b.CONTAINER_ID=:1 and c.status=0 and b.component_status!=:2 ORDER BY b.lock_time) d where rownum<=:3 )

The Database AWR reports can also very useful to check the average Elapsed Time and other performance indicators for the locker query.

Huge data volume due to no proper purging strategy for Mediator tables is a common reason for deteriorated Locker query performance. Regular data purging, partitioning, statistics gathering and creation of required indexes on MEDIATOR_GROUP_STATUS will usually ensure good performance of locker query.

Note that there is only one Resequencer Locker thread running per server at runtime. Any database issue that impacts the locker thread will impair all the Mediator Composites that use the same resequencing strategy. The mediator resequencer uses database for storage, retrieval of messages to implement the reordering and sequencing logic. Hence, the proper and timely maintenance of SOA database goes a long way in ensuring a good performance.

Worker Thread Analysis

Recollect that Worker threads are responsible for processing messages in order. There are multiple worker threads per server to parallel-process multiple groups, while ensuring that each group is exclusively processed by only one worker thread to preserve the desired sequence. Hence, the number of worker threads configured in Mediator properties (from FMW EM console) is a key parameter for optimum performance.

Below sample snippets from server thread dumps show Resequencer Worker threads. The first stack shows a worker thread which is waiting for messages to arrive on the internal queue. As and when Locker thread, locks new eligible groups, such available worker threads will process the messages belonging to the locked groups.

Idle Worker Thread:

"Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms
" id=208 idx=0x32c tid=26068 prio=10 alive, parked, native_blocked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2230)
    at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:616)[inlined]
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:196)[inlined]
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)[optimized]
    at java/util/concurrent/<strong>LinkedBlockingQueue.poll</strong>(LinkedBlockingQueue.java:424)[optimized]
    at oracle/tip/mediator/common/listener/<strong>AbstractWorker.run</strong>(AbstractWorker.java:63)
    at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:120)
    at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
    at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

The next partial stack shows a worker thread which is processing a message from a group that has been locked by the Locker.

Busy Worker Thread:

….
    at oracle/tip/mediator/service/BaseActionHandler.requestProcess(BaseActionHandler.java:75)[inlined]
    at oracle/tip/mediator/service/OneWayActionHandler.process(OneWayActionHandler.java:47)[optimized]
    at oracle/tip/mediator/service/ActionProcessor.onMessage(ActionProcessor.java:64)[optimized]
    at oracle/tip/mediator/dispatch/MessageDispatcher.executeCase(MessageDispatcher.java:137)[optimized]
    at oracle/tip/mediator/dispatch/InitialMessageDispatcher.processCase(InitialMessageDispatcher.java:500)[optimized]
    at oracle/tip/mediator/dispatch/InitialMessageDispatcher.processCases(InitialMessageDispatcher.java:398)[optimized]
    at oracle/tip/mediator/dispatch/InitialMessageDispatcher.processNormalCases(InitialMessageDispatcher.java:279)[inlined]
    at oracle/tip/mediator/dispatch/resequencer/ResequencerMessageDispatcher.processCases(ResequencerMessageDispatcher.java:27)[inlined]
    at oracle/tip/mediator/dispatch/InitialMessageDispatcher.dispatch(InitialMessageDispatcher.java:151)[inlined]
    at oracle/tip/mediator/dispatch/resequencer/ResequencerMessageHandler.handleMessage(ResequencerMessageHandler.java:22)[optimized]
    at oracle/tip/mediator/resequencer/<strong>ResequencerDBWorker.handleMessag</strong>e(ResequencerDBWorker.java:178)[inlined]
    at oracle/tip/mediator/resequencer/ResequencerDBWorker.process(ResequencerDBWorker.java:343)[optimized]
    at oracle/tip/mediator/common/listener/AbstractWorker.run(AbstractWorker.java:81)
    at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:120)
    at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
    at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
    -- end of trace

It should be noted that all further processing of the message until the next transaction boundary happens in the context of this worker thread. For example, the diagram below shows the O2C UpdateSalesOrder Integration flow, from a threads perspective. Here, the BPEL ABCS processing, the calls to AIA SessionPoolManager, as well as the Synchronous invoke to the Siebel Webservice, all happen in the resequencer worker thread.

Now consider an example thread stack as shown below seen in the server thread dump. It shows a worker thread seen to be engaged in http communication with an external system.

Stuck Worker Thread:

"Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time: 0 ms
 " id=299 idx=0x174 tid=72518 prio=10 alive, in native, daemon
  at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
  at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
  at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
  at java/net/<strong>SocketInputStream.read</strong>(SocketInputStream.java:129)[optimized]
  at HTTPClient/BufferedInputStream.fillBuff(BufferedInputStream.java:206)
  at HTTPClient/BufferedInputStream.read(BufferedInputStream.java:126)[optimized]
  at HTTPClient/StreamDemultiplexor.read(StreamDemultiplexor.java:356)[optimized]
  ^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1758a7ae0[recursive]
  at HTTPClient/RespInputStream.read(RespInputStream.java:151)[optimized]
….
….
  at oraclele/tip/mediator/dispatch/resequencer/ResequencerMessageDispatcher.processCases(ResequencerMessageDispatcher.java:27)
  at oracle/tip/mediator/dispatch/InitialMessageDispatcher.dispatch(InitialMessageDispatcher.java:151)[optimized]
  at oracle/tip/mediator/dispatch/resequencer/ResequencerMessageHandler.handleMessage(ResequencerMessageHandler.java:22)
  at oracle/tip/mediator/resequencer/<strong>ResequencerDBWorker.handleMessage</strong>(ResequencerDBWorker.java:178)[inlined]
  at oracle/tip/mediator/resequencer/ResequencerDBWorker.process(ResequencerDBWorker.java:343)[optimized]
  at oracle/tip/mediator/common/listener/AbstractWorker.run(AbstractWorker.java:81)
  at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerExecutor.java:120)
  at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
  at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:30)
  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
  -- end of trace

If this thread remains at the same position across thread dumps spanning few minutes, it would indicate that the worker thread is blocked on external Webservice application. If such external system issues block a significant number of worker threads from the pool of available worker threads, it will impact the overall throughput of the system. There will be fewer workers available to process all the groups that are being locked by the Locker thread, across all composites that use resequencers. When the rate of incoming messages during such time is high, this issue will show up as a huge backlog of messages with status GRP_STATUS=LOCKED and MSG_STATUS=READY in the resequencer health check query.

Note that JTA timeout will not abort these ‘busy’ threads. Such threads may eventually return after the JTA transaction has rolled back, or in some cases depending on how sockets are handled by the external system, may not return at all.

For such integration flows, it is advisable to configure HTTP Connect and Read timeouts for Webservice calls at the composite’s Reference properties. Figure below shows a screenshot of the properties. This will ensure that worker threads are not held up due to external issues and affect processing of other components that rely on worker threads.

Few more Loggers

The below loggers can be enabled for trace logging to gather diagnostic information on specific parts of the Mediator/resequencer.

- Logger oracle.soa.mediator.dispatch  for Initial message storage, Group Creation, Lease Renew, Node failover

- Loggers oracle.soa.mediator.resequencer  and oracle.soa.mediator.common.listener for Resequencer Locker, Resequencer Worker, Load Balancer

Conclusion

We have explored into how problems at various different layers can manifest at the resequencer in an Integration system and how the cause of these issues can be diagnosed.

We have seen

- Useful pointers in diagnosing resequencer issues and where to look for relevant information

- How a good SOA database maintenance strategy is important for resequencer health

- How timeout considerations play a role in resequencer performance

-Shreeni

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

If there is one universal truth about implementing parallelism using Java threads, it is that you will need to pay special attention when writing your code. Writing custom OEP adapters that spawn multiple threads is no different. If you have a strong requirement for handling high-throughput events, you have two choices:

1. Scaling out horizontally using multiple JVMs in cluster.

2. Scaling up a single JVM that spreads the adapter logic into multiple threads.

While scaling out is a good choice, sometimes you just can’t use this approach; or maybe you can but are constrained by lack of resources. When that happens, you need to leverage the scale up approach, especially if the machine where the OEP application will run has enough CPU cores and memory.

The idea behind scaling up using a single JVM is simple: you just need to create two or more threads and equally partition the amount of work between them, so they can process work in parallel. Let’s understand how this is possible considering the built-in JMS adapter available in OEP. The OEP JMS adapter has a property called concurrent-consumers. When you set this property to any positive integer greater than one, you are instructing the adapter to create parallel consumers, each one running on its own thread, which will increase message consumption and also application throughput.

Now consider the scenario where you need your own custom adapter, and you desire to implement parallelism. Implementing multi-threaded code in Java is fairly straightforward once you understand what is happening in the JVM. However, the truth is that even the most well written code in the world, has a good chance of blowing up and doing unexpected things when executing in the OEP runtime due to poor life cycle management of threads. This article will explain some of the possible issues that can occur when you implement threads on your own and, most importantly, how you can leverage the work manager support available in OEP to help ensure that you achieve the scalability and resiliency that you require.

This article will assume that you have some basic understanding about how to create a custom adapter in OEP. If you need information about how to create custom adapters before continue reading this article, I strongly recommend reviewing the product documentation section that covers this topic. Another great source of information is the book Getting Started with Oracle Event Processing 11g, written by some of the folks behind the OEP product at Oracle.

Testing a Simple OEP Application

Consider the following scenario: an event-driven application written in OEP that uses a custom inbound adapter, generates a random amount of events every five seconds. Those events will flow through simple pass-through channels and be queried out by a processor with the following CQL statement: SELECT * FROM inboundChannel [NOW]. Finally, those events will be printed out in the console by a custom outbound adapter. The picture below show the EPN of this application.

Aiming to generate events in parallel, the first version of the custom inbound adapter was written to perform its work using regular Java threads. The listing below shows the custom adapter implementation:

package com.oracle.fmw.ateam.soa;

import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import com.bea.wlevs.ede.api.RunnableBean;
import com.bea.wlevs.ede.api.StreamSender;
import com.bea.wlevs.ede.api.StreamSource;

public class CustomInboundAdapter implements RunnableBean, StreamSource {

	private static final int NUMBER_OF_THREADS = 4;

	private StreamSender streamSender;
	private ExecutorService executorService;
	private boolean suspended;

	@Override
	public void run() {

		executorService = Executors.newFixedThreadPool(NUMBER_OF_THREADS);

		for (int i = 0; i < NUMBER_OF_THREADS; i++) {

			RegularJavaThread thread = new RegularJavaThread();
			thread.setName(RegularJavaThread.class.getSimpleName() + "-" + i);

			executorService.execute(thread);

		}

	}

	@Override
	public synchronized void suspend() throws Exception {

		executorService.shutdown();
		this.suspended = true;

	}

	@Override
	public void setEventSender(StreamSender streamSender) {
		this.streamSender = streamSender;
	}

	private class RegularJavaThread extends Thread {

		@Override
		public void run() {

			final Random random = new Random(System.currentTimeMillis());
			int count = 0;

			try {

				while (!suspended) {

					count = random.nextInt(10);

					for (int i = 0; i < count; i++) {

						streamSender.sendInsertEvent(new Tick(getName()));

					}

					Thread.sleep(5000);

				}

			} catch (Exception ex) {

				ex.printStackTrace();

			}

		}

	}

}

As you can see in the listing above, a regular Java thread is created to perform the main logic of the adapter; which is to generate a random number of events every five seconds. Note that the thread is supposed to finish its work when the OEP application changes its status to suspended. This can happen in two ways: when the application is uninstalled from the server or when the user intentionally set its status as suspended through the OEP visualizer.

According to the run() method of the custom inbound adapter, four instances of the thread are scheduled to work using the java.util.concurrent.Executor service. This code runs perfectly well and produces the desired behavior, as you can see in the console output listed below:

<Aug 6, 2014 9:10:01 PM EDT> <Notice> <Deployment> <BEA-2045000> <The application bundle "wm-driven-threads-in-oep" was deployed successfully to file:/oracle/user_projects/domains/oep-development/defaultserver/applications/wm-driven-threads-in-oep/wm-driven-threads-in-oep.jar with version 1407373801915> 
<Aug 6, 2014 9:10:02 PM EDT> <Notice> <Spring> <BEA-2047000> <The application context for "wm-driven-threads-in-oep" was started successfully> 
Tick [uuid=13074bae-554d-49d1-934f-c2cadfadcaec, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=377a896e-d26b-4a8c-ad6a-585495ce78c2, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=488ea3c1-0515-471c-ba46-dd07f446437f, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=b75c5a24-a82d-49d2-87df-49f3900059e1, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=69df06ca-a57a-494a-b20b-976047537fce, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=a5855897-eeef-44d1-85d8-480b4745ac20, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=f71308ef-797a-4457-a5a6-7f91fe10d773, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=2164390b-8c74-4f4a-942e-7c60a9656c57, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=f82a2376-ce56-41e9-90a3-119e4e827ff5, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-3]
Tick [uuid=660e074e-991f-4082-8210-8960582aa129, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-0]
Tick [uuid=32b2a37e-235d-460d-82b2-8537dddc7703, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=4bea1680-0d1e-4f7a-9ae4-e78a23bb1229, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=215ef396-2210-40b5-88f1-4e1c86c4b93f, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=27cfebbd-649c-4f00-ac8d-33948754298b, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=cffdab2e-daca-4145-8e37-b808a5f4b412, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-2]
Tick [uuid=35a8ea83-130e-48ac-bd47-20bed9c20d82, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=5c61c10a-e61e-42d6-8931-fbf3ee092b6f, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=e5d070e2-bdb9-4fb9-bc12-7edf0165d258, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=a3488aba-00ee-4dd0-93fa-29517f69ce7b, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]
Tick [uuid=1669f735-35a4-4b2f-bad3-02117ef94991, dateTime=Wed Aug 06 21:10:07 EDT 2014, threadName=RegularJavaThread-1]

If the code runs perfectly well, you are probably wondering what’s wrong with the current implementation. The next section will illustrate the problem.

Problem: Server Cannot Control the Allocated Threads

With the OEP application running on the server, watch the threads running on top of the JVM. There are many ways to accomplish this, from taking thread dumps to using more sophisticated tools. I chose to use the JRockit Mission Control which provides a nice view of the active threads:

As expected, there are four threads allocated in the JVM executing the work defined in the code. Now let’s consider for a moment the idea of the developer being able to define the number of threads used as an instance property that could be changed through a configuration change. Just like the concurrent-consumers property of the JMS adapter, it is scenario perfectly amenable to happen. Let’s not forget that the developer can also hard code the number of threads, making it impossible to change the value even at the configuration level.

The development practices explained above can lead to some potential problems. What if the developer defines a number of threads so high that it affects the performance of other OEP applications running in the same server? Or worse, what if the developer defines a number of threads so high that the JVM itself can’t handle comply due to a lack of resources?

To protect the health of the server and prevent those situations from happening, administrators can use work managers. A work manager is a OEP server feature that controls the threading behavior using constraints. Once created in the server, the administrator can associate the work manager with one or multiple OEP applications, but this is something that should be done manually. When you create a domain for the first time, a default work manager called JettyWorkManager is created, and it is primarily used by the Jetty engine: a Java web server used to deploy HTTP servlets and static resources. You can create and/or change work managers in the domain configuration file found in the config folder of your server.

Back to the scenario, let’s assume that a work manager named wm-driven-threads-in-oep is created in the domain configuration file and limits the number of threads to a minimum of one and a maximum of two. The listing below shows how this work manager should be defined.

<?xml version="1.0" encoding="UTF-8"?>
<n1:config xsi:schemaLocation="http://www.bea.com/ns/wlevs/config/server wlevs_server_config.xsd"
xmlns:n1="http://www.bea.com/ns/wlevs/config/server"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

   <netio>
      <name>NetIO</name>
      <port>9002</port>
   </netio>

   <netio>
      <name>sslNetIo</name>
      <ssl-config-bean-name>sslConfig</ssl-config-bean-name>
      <port>9003</port>
   </netio>

   <work-manager>
      <name>wm-driven-threads-in-oep</name>
      <min-threads-constraint>1</min-threads-constraint>
      <max-threads-constraint>2</max-threads-constraint>
   </work-manager>

   <work-manager>
      <name>JettyWorkManager</name>
      <min-threads-constraint>5</min-threads-constraint>
      <max-threads-constraint>10</max-threads-constraint>
   </work-manager>

   <jetty>
      <name>JettyServer</name>
      <network-io-name>NetIO</network-io-name>
      <work-manager-name>JettyWorkManager</work-manager-name>
      <scratch-directory>Jetty</scratch-directory>
      <secure-network-io-name>sslNetIo</secure-network-io-name>
   </jetty>

   <!-- the rest of the file was dropped for better clarity -->

</n1:config>

Work managers can be associated with one or multiple applications, and also can be associated with one particular component of the application. Having said that, keep in mind that sharing the same work manager across multiple applications can limit throughput since the max-threads-constraint value will be the same for all of them, forcing some applications to queue requests while they keep waiting for available threads in the pool. And that could mean keep waiting for ever. As a best practice, if you run more than one application per server, consider creating one work manager for each application to manage its threads boundaries individually.

Once created in the domain configuration file, the work manager can be further configured using the OEP Visualizer:

Now we are all set. We can easily restart the server and test the OEP application again. With a work manager in place limiting the maximum number of threads to only two, even if the code tries to allocate four threads there will be only two threads running. That’s the theory anyway. Unfortunately, the reality is slightly different. Back to the JRockit Mission Control we still see four threads allocated in the JVM:

So, what is really happening here? Since those threads were created using standard JDK techniques, the server has no control over the number of threads. That’s why even creating a work manager and associating it with the OEP application, the constraints were ignored. Also, the main adapter code runs on its own separate thread managed by the server, as a result of implementing the com.bea.wlevs.ede.api.RunnableBean interface. That means that the spawned threads have no relationship with the adapter thread, forcing them to behave like deamon threads: instead of having its life cycle associated with the application, they will have its life cycle associated with the JVM. The problem can become even worse if the spawned threads hold references to objects belonging to the adapter thread. If for some reason the adapter thread dies, its garbage will be retained in memory since the spawned threads will still have references of it’s objects, causing memory leak problems that could lead to out of memory errors.

With this problem in mind, there is a clearly a need for some technique that would allow threads created from an adapter implementation to: have the chance to clean up its garbage independent of the run() method logic; respect the constraints imposed by an associated work manager and have the flexibility to be marked as deamon or non-deamon depending of the need. The solution for this problem will be explored in the next section.

Solution: Implementing the WorkManagerAware Interface

When designing custom adapters, if you need to spawn threads to perform some work and need server control over those threads, you can use the com.bea.wlevs.ede.spi.WorkManagerAware interface. It is available through the OEP API and all you need is to make sure that your adapter class is implementing this interface. This allows your adapter class to receive a reference of the work manager associated with the application. From this work manager reference, you can request work scheduling for threads in a safe manner. The listing below shows the updated custom adapter implementation.

package com.oracle.fmw.ateam.soa;

import java.util.Random;

import com.bea.wlevs.ede.api.RunnableBean;
import com.bea.wlevs.ede.api.StreamSender;
import com.bea.wlevs.ede.api.StreamSource;
import com.bea.wlevs.ede.spi.WorkManagerAware;
import commonj.work.Work;
import commonj.work.WorkManager;

public class CustomInboundAdapter implements WorkManagerAware, RunnableBean,
		StreamSource {

	private static int NUMBER_OF_THREADS = 4;

	private StreamSender streamSender;
	private WorkManager workManager;
	private boolean suspended;

	@Override
	public void run() {

		String threadName = null;

		try {

			for (int i = 0; i < NUMBER_OF_THREADS; i++) {

				threadName = WorkManagerBasedThread.class.getSimpleName() + "-" + i;

				workManager.schedule(new WorkManagerBasedThread(threadName));

			}

		} catch (Exception ex) {

			ex.printStackTrace();

		}

	}

	@Override
	public synchronized void suspend() throws Exception {
		this.suspended = true;
	}

	@Override
	public void setEventSender(StreamSender streamSender) {
		this.streamSender = streamSender;
	}

	@Override
	public void setWorkManager(WorkManager workManager) {
		this.workManager = workManager;
	}

	private class WorkManagerBasedThread implements Work {

		private String threadName;

		public WorkManagerBasedThread(String threadName) {
			this.threadName = threadName;
		}

		@Override
		public void run() {

			final Random random = new Random(System.currentTimeMillis());
			int count = 0;

			try {

				while (!suspended) {

					count = random.nextInt(10);

					for (int i = 0; i < count; i++) {

						streamSender.sendInsertEvent(new Tick(threadName));

					}

					Thread.sleep(5000);

				}

			} catch (Exception ex) {

				ex.printStackTrace();

			}

		}

		@Override
		public boolean isDaemon() {
			
			// This way you can inform to the
			// OEP runtime that this thread is
			// deamon but without losing the
			// association with the WM...
			
			return false;
			
		}

		@Override
		public void release() {
			
			// Here you can put the logic to clean up
			// any resources allocated by the thread,
			// in a safe and guaranteed manner...
			
		}

	}

}

As you can see in the listing above, no changes were made in the thread logic. The main difference is that the threads are scheduled through the work manager reference. Also, the threads have the isDeamon() and release() callback methods, which can be used to change the way the thread behaves regarding its life cycle and how resources are cleaned up. After installing the new version of the OEP application, it is possible to see that the constraints imposed by the work manager are now being respected:

The console output listed below also shows that there are only two threads now performing the work:

<Aug 7, 2014 1:47:20 PM EDT> <Notice> <Deployment> <BEA-2045000> <The application bundle "wm-driven-threads-in-oep" was deployed successfully to file:/oracle/user_projects/domains/oep-development/defaultserver/applications/wm-driven-threads-in-oep/wm-driven-threads-in-oep.jar with version 1407433640951> 
<Aug 7, 2014 1:47:23 PM EDT> <Notice> <Spring> <BEA-2047000> <The application context for "wm-driven-threads-in-oep" was started successfully> 
Tick [uuid=a33b7f84-2489-4737-9e33-18cabd6cca9f, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=c84e2f0c-37fa-4dca-a897-4f75fa55a7b9, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=74023209-4964-4181-8f18-40eef8ef2476, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=f9ccce52-fee6-49ad-91cc-6d9023d2c967, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=a065ebf9-1320-4cc0-a9c3-b2e004398931, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-0]
Tick [uuid=43e4db5b-72a1-4760-8205-b92a8a784cbb, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]
Tick [uuid=3bf7cbc3-09b9-4f75-99e4-d223b1a400c3, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]
Tick [uuid=48e2ab1f-4806-4d3b-8474-10fdb51d346b, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]
Tick [uuid=413803e2-c5c2-4f49-879e-06c356bca0c0, dateTime=Thu Aug 07 13:47:23 EDT 2014, threadName=WorkManagerBasedThread-1]

You can download the final implementation of the project used in this article here.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission