WebLogic Portal Propagation Tool
Field Guide
Peter Laird, WebLogic Portal Engineering
March 12, 2007
Revision 2.9
This is a supplemental guide for the Propagation Tool released with WebLogic Portal 9.2. Most of the information also applies to the version of the tool released with WebLogic Portal 8.1 SP4 and later service packs, however that release is not the focus of this document.
This guide includes information that supplements the existing documentation on the official BEA documentation site, and is a result of customer experiences with the tool. This document presents the following sections of information:
The Propagation Tool is an active project within the WebLogic Portal product team. Future releases of the tool will benefit from feedback from users of the tool. The information presented in this document will be used to guide enhancements and fixes for future versions of the tool. As always, BEA encourages users to file Support Cases and post to the BEA newsgroups for issues and ideas on how the tool can be improved.
This is a living document. Since the content of this document is derived largely from the feedback of actual users, this document will change over time as new information is obtained from new reports and interactions with users.
This section lists a number of best practices to employ when using the Propagation Tool. Most of this information can also be found in the full set of official documentation. This document presents the information in the form of a list and highlights the most common recommendations given to users.
The Propagation Tool offers powerful control over a propagation using the Scoping and Policy features. However, with this power can come some complexity. When first starting out with the Propagation Tool it is recommended to retain the default scope (the full application) and the default Policy (accept all Adds, Updates, and Deletes). Once you are comfortable with the concepts and process of propagation, it is appropriate to start using the advanced features.
The Eclipse User Interface (UI) provides many features for defining how the assets to be propagated, and a user interface for visualizing the combined inventory. The UI is the best choice when promoting an application for the first few times from one environment to another. However, over time, the promotion process generally becomes routine with the same options chosen every time. The Ant tasks are more appropriate for supporting a repeatable process when the propagation scope and policy choices are well understood and held constant.
The Eclipse User Interface allows the user to Import and Export inventories from running WLP applications. However, the UI does not provide access to all of the options available for these operations.
It is important to understand that the Eclipse UI is actually using the Ant tasks when invoking Import and Export. It invokes an ant script behind the scenes when these features are used from the UI. The dialog allows the user to save the Ant script it uses, which will help you understand what those operations are doing. The onlineDownload, onlineUpload, and onlineCommit tasks have numerous options that can be used when using the Ant tasks directly.
The Import and Export operations in the Eclipse UI, and the onlineDownload, onlineUpload, and onlineCommit Ant tasks open an HTTP connection to the propagation servlet running in the WLP application running on WebLogic Server. The URL specified for these commands indicate how the servlet is addressed.
For WLP applications deployed in a cluster, a proxy server or load balancer sits in front of the cluster to marshal traffic to the cluster nodes. While it is possible to target a propagation to the proxy, a better approach is to identify a single cluster managed node and to address it directly. This approach has three advantages:
The Propagation Tool primarily commits configuration changes into the WLP application’s database schema. Portal desktops, portlet preferences, content, and most other artifacts are persisted into the database. However, some security information such as entitlements and delegated administration policies are persisted into the embedded LDAP server intrinsic to every WebLogic Server instance.
In a cluster, the Admin Server is responsible for distributing updates to the embedded LDAP server to all nodes in the cluster. If the Admin Server is down, the nodes in the cluster can get out of sync if updates are made to entitlements or delegated administration. Therefore, it is strongly recommended to have the Admin Server running when the Propagation Tool is updating an application. This applies specifically to the duration of an Export operation in the Eclipse UI, or an onlineCommit Ant task invocation.
Unlike any other activity in a WebLogic Portal application, the Propagation Tool iterates through the configuration of the entire WLP application. Therefore, if there is a latent problem in the application configuration, the Propagation Tool is apt to encounter it. While these problems adversely affect the propagation, the source of the error is at a deeper level.
When troubleshooting such a problem, it is important to attempt to exercise the area having trouble using another means such as the Portal Administration Tool (PAT). Doing this will expedite the resolution of the issue with BEA Support because it more correctly identifies the actual problem, and may provide a simpler reproducible test case.
Some cases have arisen where an unsupported database driver or corrupt security repository have negatively impacted the Propagation Tool. While these problems affect the Propagation Tool, the solution lies in addressing the root cause.
The Propagation Tool contains a rich verbose logging facility. Verbose logging can be enabled in multiple places.
Client Side
When working with offline features (most Eclipse UI features, offline ant tasks), verbose logging is enabled on the client side. This can be done using the verboseLogFile attribute available on some tasks, as seen in the official documentation:
http://edocs.bea.com/wlp/docs92/prodOps/scripts.html
For verbose logging in the Eclipse UI, follow these instructions:
http://edocs.bea.com/wlp/docs92/prodOps/propTool.html#wp1056339
Server Side
For verbose logs on operations that are invoked against a running application (like download, commit), verbose logging is enabled on the server side. Configuring verbose logging requires an update to the Propagation web.xml, or the use of a deployment plan (covered later in this document). See:
http://edocs.bea.com/wlp/docs92/prodOps/propToolAdvanced.html#wp1071690
In the deployment plan, you must provide a valid directory to output the verbose logs. This folder must already exist for the log files to be written.
During an Export (Eclipse) or onlineCommit (Ant) operation, the Propagation Tool will process a list of changes to make against the destination application. At times, this list can be enormous, with thousands of necessary changes. This operation can take a significant amount of time.
The Propagation Tool does not create an encompassing transaction for all changes for several reasons:
Each change is executed in its own transaction. If the propagation is interrupted in the middle of the list of changes, the configuration will be left in an incomplete state. This can happen if the managed server hardware fails during the operation, for example.
Fortunately, the Propagation Tool operation can simply be rerun in such a case. The tool will detect the new, shorter, list of remaining changes that need to be applied and will start again where the previous run stopped.
The Import and Export operations in the Eclipse UI, and the onlineDownload, onlineUpload, and onlineCommit Ant tasks are considered the online operations because they communicate with the propagation servlet running in the WLP application. These operations read and write large amounts of database records while working with the WLP application configuration. These operations are notable in that they tend to take the longest amounts of time to execute.
The solution to improving the speed of these operations is to improve the performance of the database. Allocating more powerful hardware is an effective but expensive solution. Often, having a DBA tune the database is effective.
The Microsoft Windows file system is more restrictive than those of other operating systems. Specifically, there are file path length limitations that are problematic when working with deeply nested folder structures.
The Propagation Tool makes use of the file system extensively when doing certain operations. It uses a designated working folder to write large numbers of files for the artifacts that it is exporting, importing, differencing and committing. This ordinarily is not a problem because the Propagation Tool uses the working folder judiciously and does not create deeply nested folders. However, if the working folder itself is set to be a deeply nested folder, the Propagation Tool may fail while trying to create, read, delete or update some of its files because the file path may exceed the limit.
By default, the propagation servlet will use the temporary folder provided by the WebLogic Server servlet container as the working folder. This path is a nested folder under the domain directory, like:
C:\bea92GA\user_projects\domains\wlpHome_domain\servers\AdminServer\tmp\_WL_user\wlpBEA\c0b5ju\public
This path can grow very long, especially if the domain folder path is lengthy. In some cases, this will cause the Propagation Tool to fail because the file path exceeds the limit.
A best practice for Windows computers is to not allow the Propagation Tool servlet to use the servlet container temporary space for its working directory. The working folder should have a small folder path length, like C:\bea92GA\propagation. The working folder may be set in web.xml for the Propagation Tool web application, as described in the official documentation at:
http://edocs.bea.com/wlp/docs92/prodOps/propToolAdvanced.html#wp1071690
The Import/Export or onlineDownload/onlineUpload operations allow for inventory files to be retrieved/pushed to a remote server that is hosting a WLP application. Because these operations use a servlet, this transfer uses HTTP. When firewalls are an issue, this can be a convenient approach to move the inventory files. It is also the only approach supported by the Import and Export operations in the Eclipse UI.
To ensure that the Export/onlineUpload operation succeeds, it is sometimes necessary to increase the allowed upload size of the servlet. Because upload capabilities have historically been used in Denial of Service attacks on web applications, the default limit for any uploaded inventory is one megabyte. This can be increased in a deployment plan by following the instructions here:
http://edocs.bea.com/wlp/docs92/prodOps/propToolAdvanced.html#wp1070764
Additionally, the user must be granted upload rights in the WebLogic Server console. Only users in the Administrator and Deployer roles are granted this right by default. Other users can be given this right in the WebLogic Server Console. For more information see:
http://e-docs.bea.com/wls/docs92/secwlres/types.html
While using HTTP to download and upload inventory files works well in most cases, some users would prefer to use an out of band approach to moving the files such as FTP, a source code control client, or a physical medium such as a DVD.
For these users, the onlineDownload and onlineUpload tasks support an option to avoid using HTTP for the file transfer. They allow the administrator to position the inventory file on the server’s local file system, and reference the file location in the ant task.
For example:
<target
name="downloadProductionInventory">
<onlineDownload
servletURL="http://myhost/propagation/inventorymanagement"
username="weblogic” password="weblogic"
allowHttp="true"
outputInventoryFile="/opt/inventories/prod.zip"
outputToServerFileSystem=”true”
/>
</target>
This invocation would not download the inventory file to the machine running the ant task. This invocation would cause the inventory file to be written to the file system of the server machine, at location /opt/inventories/prod.zip.
The onlineUpload task has a similar flag, called readFromServerFileSystem.
The Propagation Tool can commit nearly all of the changes to configuration artifacts that it detects. However, there are a few types of changes that the Propagation Tool can detect, but cannot automatically commit. These types of changes are called Manual Changes. The administrator must manually make these changes using the Portal Administration Tool or the WebLogic console.
The following is a list of changes that are detected, but always manual:
To view the required Manual Changes in the Eclipse UI, the best approach is to right click on the Merged Inventory view, and filter for Manual Changes only. When using the Ant tasks, the offlineExtractTask can produce a file that lists the Manual Changes.
Any user invoking the Propagation Tool online tasks must be a member of the PortalSystemAdministrators role. However, unless the user is also a member of the Administrators role, that user will not be able to view or update any configuration artifacts in the WLP application by default. That user must be granted DA rights to the artifacts he wishes to propagate.
In most cases, it is recommended that the propagation user be a member of the Administrators role. This allows the user to propagate any artifact in the application. This is the safest approach to propagation because the Propagation Tool has full visibility into the application configuration.
However, there are use cases in which the propagation is targeted to a specific area of the application and this advice does not apply. For example, a content administrator may want to propagate a single folder of content items. This user will likely not have DA rights to any Portal Framework assets, but that is not a problem in this use case.
In these cases it is acceptable for a user without full DA rights to the application to do the propagation. It is important to understand that this may cause errors in cases where the user is attempting to add items that exist in destination, but are not visible to the user because of the absence of DA rights. The user will see errors on the console about the conflicted artifacts failing to add because they already exist.
There are two mechanisms to improve the experience when propagating Content Repositories. In general, repositories are more susceptible to performance issues due to the number of items stored, and the size of the binary properties (images, documents) that are stored.
Two approaches can help mitigate performance issues seen while propgating content. First, putting large numbers of content items under a single parent (thousands of documents under a single folder) is known to cause slower response times from the Content Repository (see CR306895 below). Second, use scoping to exclude parts of the Content Repository that are not related to the particular propagation use case.
It is a best practice to periodically download and retain a series of historical inventories for the environment. This is a good idea for so many reasons that we recommend that inventories be retained in source code control (if size allows) or a backed up disk drive (for huge inventories). The recommendation is to automate a script to periodically take an inventory snapshot of the environment and store that inventory. Think of your inventory as a peer to your code assets - the same care you put into maintaining historical versions of your code should be applied to inventories.
Note: inventory snapshots do not replace the need to do database and file system backups.
Before embarking on authoring your own Ant script, be sure to review and consult the sample that has been provided. The sample contains a comprehensive set of steps for a propagation. You will likely need a reduced set to perform your propagation. Using the sample Ant script as a start is a good practice.
You may find the sample in the install at this location:
BEA_HOME/weblogic92/portal/bin/propagation/
propagation_ant.xml
The section following this one will list a number of known issues that have been found by users of the Propagation Tool. While all issues are important to note, those in the following section generally have workarounds and are usually isolated to specific use cases.
This section discusses a more serious problem that can occur in any WLP application with or without the Propagation Tool. This problem is being discussed in this document because it manifests itself most clearly during the use of the Propagation Tool and it can cause the operations to fail.
For customers using WebLogic Portal 9.2, an important patch has been made available to prevent problems with the LDAP repository. See:
http://dev2dev.bea.com/pub/advisory/223
What is the Embedded
LDAP Repository?
Every WebLogic Server instance has an internal LDAP repository that stores certain security information for the server. Every WebLogic Portal application additionally stores security information such as visitor roles, visitor entitlements, delegated administration (DA) roles, and DA policies in the repository.
What are the ER
tables in the WebLogic Portal schema?
To aid in management of the entitlement and DA policies for the application, the Portal Administration Tool needs to perform queries over the set of policies in the LDAP repository. To make such queries perform adequately, additional information about the policies in LDAP are written into the ER tables in the WLP database schema. The tables which contain data that relate to embedded LDAP can be found in:
BEA_HOME/weblogic92/common/p13n/db/[vendor]/er_create_tables.sql.
The Synchronization Issue
In some cases, the embedded LDAP and the ER tables in the database may become out of sync, and they will disagree on the policy data. For runtime operations involved in servicing WLP requests, the application functions as per the embedded LDAP repository.
The runtime services do not consult the database tables, and so will operate without any problems.
On the other hand, the policy management tools will be affected by the issue. For the Portal Administration Tool, the policies or roles that are in dispute will not appear. The Propagation Tool however uses different APIs and is more negatively affected by this problem.
Avoiding the
Synchronization Issue
The possible reasons for these two stores to get out of sync are as follows:
http://dev2dev.bea.com/pub/advisory/223
The following is a list of tips to avoid this issue:
WebLogic Portal 9.2
GA Propagation Tool Symptoms
Users of the Propagation Tool on WLP 9.2 GA have noted serious problems with the Propagation Tool when the embedded LDAP and database have become out of sync. In general, the problem will cause any propagation operation on the affected application to fail. The following are partial stack traces seen while performing propagations on affected applications:
Stack Trace 1:
java.lang.NullPointerException
at com.bea.p13n.entitlements.management.internal.
RDBMSRolePolicyManager.getRolePolicy(RDBMSRolePolicyManager.java:394)
at com.bea.p13n.entitlements.management.internal.
RDBMSRolePolicyManager.getRolePolicy(RDBMSRolePolicyManager.java:297)
Stack Trace 2:
java.lang.NullPointerException
at
com.bea.p13n.management.inventory.hierarchy.nodes.appresident.security.roles.nodes.
DARoleMainServiceNode.getDARoleXML(DARoleMainServiceNode.java:143)
at
com.bea.p13n.management.inventory.hierarchy.nodes.appresident.security.roles.nodes.
DARoleMainServiceNode.><init>(DARoleMainServiceNode.java:128)
Stack Trace 3:
<Error> <Entitlements> <BEA-402726>
<Attempt to access Entitlement Policy Mgmt API by user in invalid role. Entitlement Policy operation attempted by
disallowed user ["principals=[portaladmin,
Administrators, PortalSystemAdministrators]"].>
Recovery Procedure
A support case should be opened with BEA Customer Support and reference CR299272. The support organization will provide the official recovery procedure.
The following is an unofficial recovery procedure that can be followed for non-critical environments. All entitlement and delegated administration data will be lost using this procedure.
Quick Fix:
New Domain:
This section contains a list of issues that users may encounter while using the Propagation Tool. Not all of these are product issues – some are common user mistakes that get reported as product issues. In all cases, future versions of the Propagation Tool should improve upon these issues.
If you encounter any issue listed as a product bug and the provided workaround is not sufficient, you may open a case with BEA Support and request a patch.
|
Propagation fails to propagate Security Roles and
Entitlements |
|
Reference: CR305936 |
|
Status: Product Bug, or Works by
Design |
|
Symptom: Security roles and entitlements that were set to propagate will not appear on the destination server. Errors may or may not appear on the console indicating that there are problems adding an artifact such as a Role or Entitlement. The Roles and Entitlements will not propagate. Exceptions will appear such as:
|
|
Causes: There are two causes for this problem. One is “Works by Design”, and the other is a Product Bug. Cause 1: The enterprise application name must match between the two environments. If they don’t, the Propagation Tool will add the Roles and Entitlements using an incorrect application scope to add those artifacts. Changing the application name is not supported in WLP 9.2 Propagation Tool, and so the solution is to make the application names the same. Cause 2: Due to a Product Bug, this can also be caused by the source and destination using different form factors for deployment. If the source is an Exploded Directory, while the destination application is deployed as a compressed EAR, this problem will occur. This causes a subtle change in the application name. |
Workaround: Cause 1: use matching application names between source and destination environments. Cause 2: use matching application deployment formats in both environments as a temporary solution. Request a patch for CR305936 for a fix to the application deployment problem. |
|
Propagation fails
trying to add an artifact because the artifact already exists |
|
Reference: CR296677 |
|
Status: Works as designed |
|
Symptom: Errors will appear on the console indicating that there are problems adding an artifact in any area of the inventory (Content, Portal, Security, etc) |
|
Cause: The user invoking propagation has more DA rights to the source application than the destination application. Because of this, the source inventory contains more artifacts than the destination inventory. The Propagation Tool therefore believes that those artifacts need to be added to the destination. Upon doing so, the Add operation fails because that artifact already exists in the destination, but is obscured by DA. |
Solution: Works by design. In general, it is best to propagate with a user that has full DA rights to an application. See the Best Practice above. |
|
Propagation onlineUpload
task will not work if user is not given Upload permission in WLS Console |
|
Reference: CR296752 |
|
Status: Works as designed |
|
Symptom: Errors will appear on the console indicating that there are problems reading the uploaded inventory. <Error>
<InventoryServices> <000000>
<FAILURE: There was no uploaded inventory in the request.> <Error> <InventoryServices> <000000> <The files in the upload directory could not be read. |
|
Cause: WebLogic Server restricts the right to uploading files to a server to the Administrator and Deployer roles. This is a security mechanism to defend against Denial of Service attacks. |
|
Solution: Works by design. The user invoking propagation must be in the Administrator or Deployer roles, or explicitly granted this right in the WebLogic Console. See http://e-docs.bea.com/wls/docs92/secwlres/types.html |
|
Cannot propagate
between Applications that are not identical in the J2EE sense |
|
Reference: N/A |
|
Status: Works as designed |
|
Symptom: Errors will appear on the console indicating nodes cannot be added because its parent does not exist. The parent is usually a web application name, or a resource that comes from files within the web application, like .portlet, .shell, .laf files for example. |
|
Cause: The Propagation Tool assumes that the .EAR file that was used in source will be moved to destination and the core J2EE structure will not be modified by deployment plans. This includes maintaining the same name for the application, and context paths for all web applications. |
Solution: Works by design. However, BEA is currently evaluating a feature that would allow for certain aspects of the EAR to change during promotion, like web application context paths. |
|
FileNotFoundException due to “Too many open files” on Unix
platforms |
|
Reference: CR266997 |
|
Status: Product bug. |
|
Symptom: Invoking the onlineCommit task (or Eclipse Export) on a WLP application will cause errors to appear on the console like: java.io.FileNotFoundException and
(Too many open files) |
|
Cause: a bug causes a certain number of file descriptors to be left open in certain cases. |
|
Workaround: Increase the number of file descriptors (limits.conf). |