Recent Blog Entries

Wednesday, 19 August 2009

Oracle Application Server 10.1.3.5 Patchset Released

10.1.3.5 patchset has been released for Linux x86 and Windows 32 bit (64-bit Linux coming soon).
Patch is downloadable from Metalink:
Patchset 8626084
Description PLACEHOLDER BUG TO UPLOAD AS 10.1.3.5 PATCH-SET ON ARU

One of the major new features is for example as BPEL instance migration feature, where you can “move” one running instance from one revision to another revision. More on this feature coming here soon!

Wednesday, 12 August 2009

Load Balancing Strategies for SOA Infrastructures

When designing SOA infrastructures or Enterprise Architectures, most architects miss one component which may impact the overall system performance dramatically – the hardware load balancer.

Typically a hardware load balancer like F5 BigIP exists as a shared component for many applications and services to distribute load between multiple servers or applications and to ensure failover in case of hardware or software failure.

However, the typical setup of a load balancer is rarely suited for typical SOA services as the following example shows:

A sample real life SOA service consists of a large order system which does the backend processing of customer orders. It was implemented using Oracle SOA Suite 10g and is based mainly on BPEL processes which contain the business logic and orchestrate  the backend systems like Product Catalog, Fulfillment and Billing. All the requests (orders) come from a CRM portal.

This SOA implementation is running on an 8-node Linux-based BPEL cluster running Oracle Fusion Middleware 10g. In front of these 8-nodes, a F5 BigIP is distributing the load.

Initially the load balancer setup was the following:

- static round-robin to all 8 nodes
- session affinity (persistence) for a duration of one hour

The corresponding rule for the BigIP configuration looks like

persist source_addr_1h

Let’s look at the impact of these settings:

Session Affinity (persistence) means that all requests coming from the same originating IP address will be routed by the load balancer to the same cluster node. This rule is mostly suited for typical web applications where it is desired to keep one HTTP session on the same cluster node.

However this setup has a severe negative impact for backend SOA services as we can see shortly:

The SOA order service consists of one invocation endpoint and some partnerlinks for calling the backend systems. Most of this communication is asynchronous, that means the backend systems are called via one-way requests and use callbacks to return messages back to the order system.

What now happens is that the first request to the order system comes from the CRM system, resulting in a routing to one node (say “Node A”) of the cluster. If these requests come over the load balancer, then all subsequent request coming from the CRM system will also be routed to the same cluster node, because the originating IP is equal!  This results in a very poor overall load distribution. In fact only one of the 8 nodes will get most of the requests. When this node is at 100% resource utilization, the system will dramatically slow down even though the other 7 nodes are almost idle.

But it is even getting worse.

First of all, the same effect is in place for every callback from a backend system. So, if a callback (return message) is received from the Billing System, also the same node gets these answer repeatedly within one hour (though this need not be Node A, of course).

Secondly, the static round-robin algorithm does not take in effect, which state each cluster node has. So, for example if one cluster node is heavily under load, because it processes some complex orders, and this results in 100% cpu load, then the load balancer will not recognize this but route lots of other requests to this node causing overload and saturation.

In summary, a small misconfiguration of the load balancer will lead to a system which does not use the hardware of 8 nodes effectively, which will not be able to handle lots of requests and which will not scale well.

So what are the recommendations?

1. There should be no session affinity (persistence) at all for a runtime SOA system. There may be some exceptions, for example at deployment time. When deploying SOA services with multiple artifacts (for example multiple BPEL processes, WSDLs and XSDs), this should happen in general on the same cluster node first to prevent lots of replication and inconsistencies in the cluster. But this can be configured, for example by using a dedicated deployment server or setting up a virtual host as a deployment target.

2. BigIP offers more sophisticated load balancing algorithms than dumb static round-robin. For example you can use the dynamic ratio load balancing (described in Chapter "Configuring servers for SNMP Dynamic Ratio load balancing" of the BigIP Reference Guide). This algorithms uses metrics which are calculated dynamically by SNMP agents running on each node. These SNMP agents are typically included in the LINUX distribution and just need to be started on each node. The load balancer then regularly queries these agents about the values, dynamically calculates the metrics and routes the requests accordingly. This means that the distribution of requests will be proportionally to the metrics of each node for each time frame.

An example might look like

monitor type real_server {
interval 5
timeout 16
dest *.12345
method "GET"
cmd "GetServerStats"
metrics "ServerBandwidth:1.5,CPUPercentUsage, MemoryUsage,
TotalClientCount"
agent "Mozilla/4.0 (compatible: MSIE 5.0; Windows NT)
}

The configuration of the Dynamic Ratio Load Balancing needs the following settings on BigIP side (example):

Interval:  1 minute
Timeout:  10 seconds
Community Strings:  "public"
SNMP Version: V2

The overall effect is that only nodes which are not under load will get requests, and that the overall distribution of load will be much more effective.  All cluster nodes will be utilized and the overall system will scale well up to the limit of the all available nodes. Then, of course, when all 8 cluster nodes are at 100%, this algorithm cannot help anymore of course….

References:

- BigIP Reference Guide

- Oracle SOA Suite 11g Enterprise Deployment Guide

- Oracle SOA Suite 10g Best Practises

- Oracle Fusion Middleware 11g High Availability Guide

Wednesday, 22 July 2009

Enterprise Architecture Roundtable in Berlin and Frankfurt – Bookmark Your Calendars

On September 2nd, 2009 in Frankfurt and September 3rd, 2009 in Berlin, Oracle will organize an Enterprise Architecture Roundtable to discuss trends in infrastructures and methods for modern IT platforms. Main focus is to facilitate the exchange of Enterprise Architecture knowledge and experience between lead architects of our main customers. You are welcome to join. Register directly here:

2. September 2009 – Frankfurt

3. September 2009 - Berlin

Friday, 17 July 2009

Running Oracle Soa Suite 11g on OpenSuse 11.1 – Part 2

PART 2 – Starting SOA Suite and Creating a First SOA Composite in JDeveloper 11g Version 11.1

This continues Part 1 which can be found here.

After installation we need to take one steps before starting the servers:

  • Configure a boot.properties file in the soa_domain root directory:
    This is simply a text file with 2 lines:

    username=weblogic
    password=welcome1

    After first system start the password entry will be encrypted.

Starting SOA Suite contains of 2 servers:

  • an AdminServer (running WebLogic Console and Grid Control Console)
  • a ManagedServer soa_server1

At runtime, only the ManagedServer is required.

Now we can start the servers with

<wl_home>/user_projects/domains/soa_domain/startWebLogic.sh

Test if you can access the WebLogic console at http://<host>:7001/console

clip_image002

After completion, start the Managed Server:

<wl_home>/user_projects/domains/soa_domain/startManagedWebLogic.cmd soa_server1

Check if you can access the soa infra test page at :8001/soa-infra">http://<host>:8001/soa-infra

clip_image002[5]

Now you can also login to Grid Control at :7001/em">http://<host>:7001/em

clip_image002[9]

clip_image002[11]

Notice the Linux System Resources: Altogether we use only 2,5 GB of RAM (including Linux…!)

clip_image002[13]

Next install and run JDeveloper 11.1 (I used the generic JAR installer) and create an Application Server Connection:

clip_image002[15]

clip_image002[17]

clip_image002[19]

clip_image002[21]

From the About Menu, Check for Updates and install the SOA Composite Editor (this is NOT included in the default installation!):

clip_image002[23]

Now you are ready to create your first application:

  1. New Application
  2. New Project, for example a SOA Composite with a BPEL process:

clip_image002[25]

With Project –> right click you can Deploy the composite to the SOA server:

clip_image002[27]

Wednesday, 15 July 2009

Running Oracle Soa Suite 11g on OpenSuse 11.1 – Part 1

PART 1 – Installation of Oracle Database, Weblogic Application Server and SOA Suite 11gR1.

Not only the version numbers match ;-)  Also the products – OpenSuse 11.1 and Oracle Fusion Middleware 11gR1 form an excellent couple even though not officially certified. While Oracle Enterprise Linux is the officially certified version, it lacks more recent kernel support which means also that WLAN drivers might not work for example. So  if you are looking for a lightweight Open Source Linux based installation (as development environment) of SOA Suite 11gR1 here is the way how I did it (Installation on Toshiba Tecra M9 Notebook with 4 GB RAM with Gnome desktop).

1. Be sure to install OpenSuse 11.1 with the development tools and libraries and with 32-bit support for gcc, glibc and glibc-devel.

2.  Create user oracle and group oracle. Login as user oracle.

3. Install Oracle Database 11.1.0.6 Standard Edition and create a default OLTP database with the creation assistant. Be sure to use character set AL32UTF8. I used an SGA size of 512 MB with autoextend=on.

4. Install Weblogic Server 11g (10.3.1) – I used the generic jar installer.

5. Install the 11gR1 Database Repository Creation Utility and start it from <install>/bin/rcu

clip_image002

 

clip_image002[6]

 

clip_image002[8]

clip_image004

clip_image002[10]

clip_image002[12]

To save space, I created single tablespaces “DEV_SOA” and “TEMP” for all repository schemas:

clip_image002[18]

clip_image002[20]

clip_image002[22]

6. Install the Oracle SOA Suite 11gR1 into the recently created WebLogic (BEA) Home.

7. Next, execute <WLS_HOME>/common/config.sh to create a new WLS domain for SOA.

First check if the Sun JDK has been set in commEnv.sh as JAVA_HOME – otherwise set the <WLS_HOME>/jdk160_11 as JAVA_HOME.

clip_image002[24]

clip_image002[26]

clip_image002[28]

clip_image002[30]

clip_image002[32]

Set the database connections to the ones created with RCU:

clip_image002[34]

clip_image002[36]

clip_image002[38]

clip_image002[40]

clip_image002[42]

Finally you are done!

Troubleshooting:

  • When you receive an error at Database linking in ins_client.mk at database installation time: Skip this step, complete the DB installation and execute <ORACLE_HOME>/bin/relink client after installation.
  • If executing runInstaller fails, then include the option –jreLoc <path to JDK160_11> on the command line

Thursday, 7 May 2009

JRockit JVM now certified for SOA Suite 10.1.3.4

Oracle JRockit (formerly BEA JRockit) is a well recognized JVM which has many advantages over Sun JVM. For example it overcomes the usual difficulties with PermGen space limitations of the Sun JVM, just to name one. In benchmarks in a real complex SOA project we have experienced performance advantages of JRockit over Sun on Linux of more than 50%!

Since today, JRockit 1.6 (R27.6.3) is now certified for SOA Suite 10.1.3.4 on Linux and Windows!

Please see the updated Oracle Fusion Middleware 10gR3 Certification Matrix.

More on monitoring and performance tuning with JRockit and its superior Mission Control tool soon…

Thursday, 30 April 2009

Using Oracle JMS Adapter – Part 1: Publish from BPEL to AQ via Partnerlink Invoke

Using the Oracle JMS Adapter is the standard way to inititate asynchronous communication to partner systems, for example JMS-based services, to databases using Oracle AQ or to IBM Websphere MQ.

A complete sample (for SOA Suite 10.1.3.4) demonstrates the steps required for implementing a fire-and-forget (one-way) invoke to an AQ queue on an Oracle database:

1. Create the following datasource and connection pool (in oc4j_soa/config/data-sources.xml)

<managed-data-source name="SensorTestDS" connection-pool-name="SENSOR_CONNECTION_POOL" jndi-name="jdbc/SensorTestDS"/>

<connection-pool name="SENSOR_CONNECTION_POOL">
<connection-factory factory-class="oracle.jdbc.pool.OracleDataSource" user="jmsuser" password="jmsuser" url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=tcp)(HOST=skoser-de.de.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=DB10G.SKOSER)))"/>
</connection-pool>

(Note: The data source is named “Sensor..” because part 2 shows how to use this from a bpel activity sensor)

2. Enter resource provider in oc4j_soa/config/application.xml:

<resource-provider class="oracle.jms.OjmsContext" name="SendorTestRP">
<description>Sensor JMS/AQ</description>
<property name="datasource" value="jdbc/SensorTestDS"/>
</resource-provider>

The value of the datasource property must match the JNDI lookup (“jndi-name”) of the datasource in data-sources.xml.

3. Create database objects: create a queue named “JMS_OUT” and a queue table “JMS_OUT_TABLE”.

/* Execute as SYS */
CONNECT SYS AS SYSDBA
DROP USER jmsuser CASCADE ;

GRANT connect,resource,AQ_ADMINISTRATOR_ROLE TO jmsuser IDENTIFIED BY jmsuser ;
GRANT execute ON sys.dbms_aqadm TO jmsuser;
GRANT execute ON sys.dbms_aq TO jmsuser;
GRANT execute ON sys.dbms_aqin TO jmsuser;
GRANT execute ON sys.dbms_aqjms TO jmsuser;

connect jmsuser/jmsuser;

begin
sys.dbms_aqadm.create_queue_table(
queue_table => 'JMS_OUT_TABLE',
queue_payload_type => 'SYS.AQ$_JMS_MESSAGE',
sort_list => 'PRIORITY, ENQ_TIME',
compatible => '10.0.0',
primary_instance => 0,
secondary_instance => 0,
storage_clause => 'tablespace users pctfree 10 initrans 1 maxtrans 255 storage ( initial 64K minextents 1 maxextents unlimited )');
end;
/

begin
sys.dbms_aqadm.create_queue(
queue_name => 'JMS_OUT',
queue_table => 'JMS_OUT_TABLE',
queue_type => sys.dbms_aqadm.normal_queue);
end;
/

begin
dbms_aqadm.start_queue( queue_name =>'JMS_OUT' ,enqueue => true ,dequeue => true );
end;

5. Create a new connection factory entry in JMSAdapter oc4j-ra.xml
(oc4j-soa/application-deployments/default/JMSAdapter/oc4j-ra.xml)

<connector-factory location="eis/Jms/11gAQ" connector-name="Jms Adapter">
<config-property name="connectionFactoryLocation" value="java:comp/resource/SensorTestRP/QueueConnectionFactories/QCF"/>
<config-property name="factoryProperties" value=""/>
<config-property name="acknowledgeMode" value="AUTO_ACKNOWLEDGE"/>
<config-property name="isTopic" value="false"/>
<config-property name="isTransacted" value="false"/>
<config-property name="username" value="jmsuser"/>
<config-property name="password" value="jmsuser"/>
<connection-pooling use="private">
<property name="waitTimeout" value="180" />
<property name="scheme" value="fixed_wait" />
<property name="property-check-interval" value="5" />
<property name="inactivity-timeout-check" value="all" />
<property name="abandoned-connection-timeout" value="20" />
<property name="autoCloseSession" value="" />
<property name="inactivity-timeout" value="120" />
</connection-pooling>
<security-config use="none">
</security-config>
</connector-factory>

The string after “java:comp/resource/” must match the resource provider name in application.xml.

IsTransacted must be set to false for outbound JMSAdapter (see Metalink Note 418884.1 ).

6. Restart oc4j

7. Design a bpel process using the JMS Adapter:

Create a default async bpel process and insert one assign and one invoke activity.

Create a new partnerlink using the “Define Adapter Service” icon:

image

image

Define a db connection to the jmsuser schema:

image

image

image

image

Continue with the Adpater service creation

image

image

image

The Destination Name above must contain

java:comp/resource/SensorTestRP/Queues/JMSUSER.JMS_OUT

We want to publish employees to the queue so copy the addemployees.xsd form here to the BPEL project (in directory bpel) and reference it:

image

The resulting bpel process should look like this:

image

You can download the BPEL Process project for JDeveloper 10.1.3.4 here.

8. Deploy the BPEL process to the default domain.

9. Test the process from the BPEL Console:

image

image

Verify in the database that the message has been written to the JMS_OUT_TABLE:

SQL> select count(*) from JMS_OUT_TABLE;

COUNT(*)
----------
1

Next time, we will cover using sensors with JMS Adapter and then Native MQ Adapter…..

Troubleshooting:

If you receive an Orabpel-12101/ORABPEL-12102 ERRJMS_TRX_COMMIT/ERRJMS_TRX_ROLLBACK then you have not set isTransacted to false for outbound JMS Adapter invocation….

Thursday, 19 March 2009

Governance: Installation Tip for Oracle Service Registry on Weblogic 10.3

If you follow the Installation Guide for installation of Oracle Service Registry on Weblogic Server 10.3. you might run into a problem, that you cannot login into the Registry Console.

You might get an error like

ERROR: uddi_security_v3.com.systinet.uddi.v3.InterceptorProxy - No Configuration was registered that can handle the configuration named NamePasswordAN

EXCEPTION: No Configuration was registered that can handle the configuration named NamePasswordAN

image

The solution is described in Metalink Note 785097.1: Installing Oracle Service Registry 10.3 on Oracle Weblogic Server 10.3

You have to include the login module for weblogic in the JAVA_OPTIONS:

set JAVA_OPTIONS=%SAVE_JAVA_OPTIONS% -Djava.security.auth.login.config=%OSR_HOME%\conf\jaas.config

Be sure to replace %OSR_HOME% with your OSR installation directory…..