Enterprise Administration > Windchill ESI > Administering Windchill ESI in an ORACLE Applications Environment > Guidelines for Monitoring, Diagnosing, and Resolving Problems
  
Guidelines for Monitoring, Diagnosing, and Resolving Problems
This section provides you with:
General guidelines on how to monitor and diagnose problems and lists some specific techniques you can use to resolve them.
A listing of specific problems, their causes, and resolutions.
Monitoring Problems
As a Windchill ESI Administrator you are alerted to problems through reporting channels such as user reports which includes trouble tickets logged via a help desk or through e-mails and through automated system alerts.
User reports are a passive, reactive means of detecting problems. To improve Windchill ESI system stability and lower the overall cost-of-ownership, it is recommended that you supplement user reports with automated system alerts which is a proactive approach that can prevent problems before they become critical.
The following describe how to monitor problems using the following:
TIBCO BusinessWorks monitoring services
Error handling processes and logging services
Event rules and problem-detection approaches
Monitoring with BusinessWorks
TIBCO BusinessWorks is used by the Windchill ESI Error Handling process to monitor the Windchill EAI software components and to handle system problems. TIBCO Administrator provides you with a Graphical User Interface and microagents to monitor and maintain hardware and software components within TIBCO administrative domains. These tools can be configured to monitor BusinessWorks process engines, adapters, log file entries, available disk space, operating system parameters, and more. When defining a deployment configuration, TIBCO Administrator provides facilities to create alert and escalation rules triggered by predefined events, such as a component failure, suspended process, or log event.
You can use BusinessWorks to monitor the following:
BusinessWorks engine problems
Adapter problems
JMS problems
BusinessWorks Engine Problems
TIBCO BusinessWorks engines need to be running for every process used in Windchill ESI. Therefore the BusinessWorks monitoring component must be able to do the following:
Handle events raised when a failure of a BusinessWorks engine causes messages to be lost or for processing to stop
Send alerts to TIBCO Administrator monitoring when failures occur
Escalate actions as failure frequency increases
Failure Categories
In a deployment, failures can be separated into the following categories:
Any Failure: Catches any failure and performs an action
First Failure: Catches the first failure of an engine and performs an action
Second Failure: Catches the second failure of an engine and performs an action
Subsequent Failure: Catches any failure that occurs subsequent to the second failure and performs an action
Suggested Deployment Configuration
The following lists suggested actions that you can configure during deployment, for the various types of failures:
Any Failure: Raise an alert to the administrator
First Failure: Restart the engine
Second Failure: Restart the engine and raise a second failure alert
Subsequent Failure: Restart the engine and send e-mail to the administrator
A configurable counter and timer exists which determines when to reset the failure count to first. Use the timer setting on this counter to set to a particular time frame in which more than two failures in that time frame raises considerable concern about the system’s overall integrity.
1. Navigate down through the TIBCO Administrator GUI:
Application Management > <Application Name> > Configuration > Process Archive.par
2. Click the Monitoring tab:
3. Click the Add button under Events. The Add Event screen will appear:
4. Select the type of event you would like to monitor. For example, when a JMS queue fails, a process becomes suspended. You may configure an alert to monitor for a process suspension:
A systems administrator has the ability to restart the engine using TIBCO BusinessWorks Administrator. For more information, see topic JMS Problems below.
Adapter Problems
The adapter’s distribution target system needs to be running for many processes used in Windchill ESI. Therefore, the TIBCO BusinessWorks monitoring component must be able to do the following:
Restart adapter instances when they fail
Send alerts to TIBCO Administrator when failures occur
Escalate actions as failure frequency increases
Failure Categories
In a deployment, failures can be separated into the following categories
Any Failure: Catches any failure and performs an action
First Failure: Catches the first failure of an adapter and performs an action
Second Failure: Catches the second failure of an adapter and performs an action
Subsequent Failure: Catches any failure that occurs subsequent to the second failure and performs an action
Suggested Deployment Configuration
The following lists suggested actions that you can configure during deployment, for the various types of failure:
Any Failure: Raise an alert to the administrator
First Failure: Restart the adapter
Second Failure: Restart the adapter and raise a Second Failure alert
Subsequent Failure: Restart the adapter and send an e-mail to the administrator
A configurable counter and timer exists which determines when to reset the failure count to first. Use the timer setting on this counter to set to a particular time frame in which more than two failures in that time frame raises considerable concern about the system’s overall integrity.
JMS Problems
An EMS Server needs to be running for the master process flow and error handling processes to work properly. The TIBCO Administrator must be able to do the following:
Handle events raised when an EMS server failure occurs
Send errors to the error handling process when a failure occurs
Unfortunately, TIBCO BusinessWorks does not create an event when EMS servers fail. To combat this, you can do one of the following:
No suspend: When a JMS queue does not respond or cannot be connected to at runtime after a repeat-on-error-until-true group with the suspend checkbox option not selected, BusinessWorks creates a process engine log entry indicating a timeout of the JMS activity. You could configure a log event in TIBCO Administrator to monitor the log for this and raise an alert.
Suspend: When a JMS queue does not respond or cannot be connected to at runtime after a repeat-on-error-until-true group with the suspend checkbox option selected, BusinessWorks suspends the process and you can configure a TIBCO Administrator rule base to raise an alert.
By default, Windchill ESI is configured for the suspend approach, as this allows for the process that caught the JMS queue failure to continue after restarting the EMS server.
Special Guidelines for EMS Server Problems
Windchill PDMLink allows using a single JMS provider at a time. Therefore, the TIBCO EMS server becomes an integral part of the Windchill ESI architecture, and Windchill PDMLink users may use it for functions other than Windchill ESI applications. Therefore, it may not be prudent to configure Windchill ESI to automatically restart the EMS server when there is a problem with Windchill ESI that is related to a JMS problem. Rather, the default and recommended configuration is to have the error-handling process place the current BusinessWorks job in suspend mode when it detects a serious JMS problem.
Suspend mode allows you to manually intervene and does not pose a risk of data loss. Only the affected BusinessWorks job - that is, the product data transaction that is being published - is suspended while other jobs in the same process engine may continue. If other jobs encounter the same JMS problem, they get individually suspended as well. You can individually restart jobs via the TIBCO BusinessWorks Administrator. Jobs get resumed from the point of suspension, not from the last checkpoint.
TIBCO Administrator does not provide built-in administration domain monitoring of TIBCO EMS. You may, however, configure TIBCO BusinessWorks to issue alerts in the event that the EMS server is suspended or requires restarting. End-users can use the basic administration console in TIBCO EMS to configure TIBCO BusinessWorks to issue alerts to in the event that the EMS server is suspended or requires to be restarted. For more information, see topic Configuring E-Mail Alerts below.
Oracle Applications Problems
The TIBCO Adapter for Oracle Applications log files can report problems such as an invalid Oracle ApplicationsWindchill ESI user account. These situations generally occur because of problems such as typographical errors made during installation or passwords expiring after a predetermined period. These can be unexpected and difficult to diagnose. Therefore, you may wish to configure TIBCO BusinessWorks to monitor these log files for the relevant message text and to issue alerts should the event occur.
Using Error Handling Processes and Logging Services
In addition to configuring alerts for problems with the underlying TIBCO products, you can also use the error handling and logging shared services to detect and pinpoint Windchill ESI problems.
Developing Event Rules and Problem-Detection Approaches
Since event rules are closely tied to hardware deployment, it is not feasible for Windchill ESI to provide a predefined, ready-to-use configuration. However, you can develop and adjust event rules as needed, over time, in a non-invasive manner to the underlying TIBCO products and Windchill ESI business logic component. You can prioritize developing rules based on the most common or troublesome issues you have encountered, and thus gradually, move from a reactive to a proactive problem-detection approach.
Diagnosing Problems
After detecting a problem that cannot be corrected automatically or by the user, you need to begin diagnosing the problem. This involves categorizing and localizing the problem to determine its root cause.
Localizing Problems
To localize the source of the problem, you need to ask questions such as:
Is the problem associated with a business process issue such as a system-of-record violation, a functional issue such as invalid data, or a technical issue such as a server being down?
Is the problem associated with Windchill PDMLink, TIBCO, or the distribution target ERP system?
If it is a problem related to TIBCO, is it associated with TIBCO EMS, BusinessWorks, or Adapter for Oracle Applications?
Is the problem associated with the BusinessWorks deployment and/or run-time agents?
Is the problem associated with the underlying physical network and computing, rather than Windchill ESI?
Can the problem scenario be duplicated in a test system with the same configuration as the production environment?
Categorizing Problems: Key Areas of Focus for Troubleshooting
To categorize problems, you need to focus on key problem areas and get familiar with error handling reports such as error logs and error handling codes.
Most system-related, technical issues can be categorized according to the location of the root cause. Refer to the Administering Windchill ESI in an ORACLE Applications Environment and applicable documentation provided by your systems integrator, as necessary, to verify and restore correct configuration settings.
It is also important for you to become familiar with the business process and functional troubleshooting information. Users who are not familiar with this information may escalate such issues to you.
The following categories of problems and their descriptions are not intended to be exhaustive with detailed step-by-step procedures. Rather, they are provided to help you to focus on some of the key or potential root causes of technical issues:
Problems originating from Windchill ESI
Problems originating from TIBCO components such as:
EMS
BusinessWorks
Adapter for Oracle Applications
Problems originating from the distribution target.
Problems indicated in Windchill ESI logs.
Windchill ESI Problems
The following lists how to deal with problems that may originate from Windchill ESI services:
Verify that the Windchill server and application are running
Verify that the Windchill JMS client is connecting to the TIBCO EMS server
Verify that the Windchill ESI user account for the EMS server is correctly configured
Verify all other JMS-related configurations
Scan for error messages in the Windchill administrative logs
Validate the distribution targets and their attributes for the given release
Verify correct operation of the Windchill Enterprise Integration Access Remote Procedure Calls (RPCs)
Verify the correctness of the values set for the various Windchill ESI preferences
TIBCO Problems
The following list how to deal with problems that may originate from TIBCO components:
EMS Problems
Verify that the EMS server is running
Verify the EMS server is correctly configured
Ensure that the required Windchill ESI JMS queues are defined
Verify that the BusinessWorks Windchill ESI user account for the EMS server is correctly configured
Verify the secure queue authentication configuration
Isolate any Java Virtual Machine (JVM) issues, such as unsupported versions or memory overflow errors
BusinessWorks Problems
Verify that all required TIBCO services are running. Process engines do not start if the following required services are not running. Verify that all required TIBCO services are running:
TIBCO Administrator <version> (Domain name)
TIBCO HAWK Agent (Domain name)
Verify the existence and contents of required Windchill ESI BusinessWorks application properties and configuration files:
ESIORAEMailMessageLookups.properties
ESIORADefaults.properties
ESIORAErrorHandlingCodes.properties
ESIORALookups.properties
ESIORAMessageLookups.properties
FilesToRead_ORA.properties
Verify that the BusinessWorks JMS client is connecting to the TIBCO EMS server
Verify all other EMS-related configuration
Isolate any Java Virtual Machine (JVM) issues, such as unsupported versions or memory overflow errors
Verify that the Windchill ESI components are deployed to the correct domain
Verify the Windchill ESI business logic configuration and deployment settings, including the global variable values
* 
You must restart the process engine for new global variable values to take effect.
Verify the Java classpath
Validate the configuration settings in the TRA file
Validate the BusinessWorks Administrator server configuration by:
Authorizing user accounts
Making sure that multiple repositories with the same Rendezvous configuration (network, service, and daemon) on the same subnet have unique names, regardless of the BusinessWorks administration domain
Ensuring that the server-based repository (.dat) files are not deleted directly. To properly delete a server-based project from an administration domain, follow these steps:
Undeploy the project (this removes all of the .tra and .cmd files)
Stop the administration server service
Delete the contents of <Tibco_Home>/tra/domain/<Domain_Name>/application/<Application_Name>/working folder
Restart the administration server service
* 
The meaning of <Application_Name> in the above referenced path may be understood from the fact that the deployment name of an application is of the form <Domain_Name>-<Application_Name>.
Isolate any issues with internationalization or locale configuration settings. Check out the following:
com_infoengine_locale attribute in the JMS header of Windchill ESI messages
ESIOMAdapter/Locale global variable
Defaults and cross-referencing lookup file entries used in data mapping
Adapter for Oracle Applications Problems
The following lists how to deal with problems that may originate in an Oracle Applications distribution target. Refer to the Administering Windchill ESI in an ORACLE Applications Environment for more details.
Verify that all Oracle Applications servers are running.
Verify that the Oracle Applications system is running a version that is supported
Verify the existence of a valid ESISYS Oracle Applications user account used by the TIBCO Adapter for Oracle Applications in all ESITarget instances.
Verify that the ESISYS user account has the required security authorization profile.
Isolate any issues with Internationalization or locale configuration.
Oracle Applications Problems
The following lists how to deal with problems that may originate in an Oracle Applications distribution target. Refer to the Administering Windchill ESI in an ORACLE Applications Environment for more details.
Verify that all Oracle Applications servers are running.
Verify that the Oracle Applications system is running a version that is supported
Verify the existence of a valid ESISYS Oracle Applications user account used by the TIBCO Adapter for Oracle Applications in all ESITarget instances.
Verify that the ESISYS user account has the required security authorization profile.
Isolate any issues with Internationalization or locale configuration.
Logs and Error Handling Codes
To help diagnose problems, you need to be thoroughly familiar with the key Windchill ESI logs, described in the Administering Windchill ESI in an ORACLE Applications Environment. You also need to be familiar with the EAI error-handling architecture, and the error codes. The following lists the available logs and how to access them:
Logs
How to Access
Windchill Enterprise Systems Transaction Log
Windchill PDMLink user interface
TIBCO BusinessWorks Process engine logs
* 
Windchill ESI application messages appear with a role designation of ESI to distinguish them from the standard TIBCO product messages.
TIBCO BusinessWorks Administrator. You can find the log files at <TIBCO_HOME>/tra/domain/<domain_name>/application/logs/<Application_Name>-Process_Archive.log[.sequence#]
TIBCO EMS logs
File name and path are determined by log file related configuration parameters in the tibemsd.conf file.
TIBCO Adapter for Oracle Applications logs
<TIBCO_HOME>/tra/domain/<domain_name>/application/logs/<Application_Name>-ESIOMAdapter.log.[.sequence#]
Oracle Applications logs
Consult your Oracle Applications administrator.
You should also be familiar with the Oracle Applications distribution targets and the data objects and attributes impacted by Windchill ESI. You may have direct responsibility for viewing and manipulating data in Oracle Applications, or may simply have to coordinate these activities with Oracle Applications administrators. To view the data that was transferred into Oracle Applications by Windchill ESI, the following windows are particularly important:
Master Item
Bills of Materials
Engineering Change Orders
Routings
Resolving Problems
After you have monitored, detected, and diagnosed problems, you need to resolve them. The following lists some general techniques you can use when troubleshooting as well as a list of specific problems with their solutions.
Techniques for Resolving Problems
Following lists some of the troubleshooting techniques you can use to resolve problems.
Coordinating Troubleshooting Teams and Escalating Problems
As a Windchill ESI administrator, you may have to involve and coordinate a wide variety of specialists to fully resolve production issues. These parties may include end-users, functional experts, Windchill PDMLink administrators, Oracle Applications administrators, database specialists, operating system specialists, network administrators, and others. If, after diagnosing and localizing an issue, you determine that the problem cannot be solved in-house, you may need to escalate it to systems integration partners or to PTC Technical Support, or both, for assistance. Systems integration partners may be particularly critical to help you resolve problems involving special configuration or customizations made to the standard Windchill ESI product.
Manually Overriding Checkpoints
The Windchill ESI application leverages checkpoint activities at key points in the TIBCO BusinessWorks processing flow. A checkpoint saves the process data and state of the current running process instance so that it can be recovered at a later time in the event of a failure. If a process engine fails, all process instances can be recovered and can resume execution at the location of their last checkpoint in the process definition. Only the most recent state is saved by a checkpoint. If you have multiple checkpoints in a process, only the state from the last checkpoint is available for recovering the process.
The location of Windchill ESI checkpoints has been strategically designed to ensure robust transaction integrity while minimizing adverse performance effects. Occasionally, you may need to manually override an activated checkpoint to restart transaction processing from scratch when the process engine is restarted.
* 
Overriding checkpoints should be done with extreme caution, as it can lead to duplicate or incomplete data, incorrect export history records, and data corruption.
However, if circumstances warrant it, you can override checkpoints by deleting the <Tibco_Home>/tra/domain/<Domain_Name>/application/<Application_Name>/working directory and its contents. Depending on the process engine mode, there may be many other /working subdirectories under the /tibco path that would need to be deleted. Do a search for /working to locate and remove all of these subdirectories.
Resolving Solaris Platforms Issues
On Solaris platforms you may have to resolve problems that arise from:
Environment variable issues
Designer deployment configuration issues
The following describe techniques to resolve these.
Environment Variable Issues
On various Unix platforms certain TIBCO components fail to launch because of library dependencies. If there are any *.sh or *.csh files present in the bin directory of that particular component, they should be executed and then the ‘ldd’ command can be used to identify library dependencies. All the required library paths should be added to the tibco.env.CUSTOM_EXT_PREPEND variable present in the *.tra file.
Designer Deployment Configuration Issues
Occasionally, the TIBCO Administrator and the TIBCO Runtime Agent (TRA) become corrupted and without warning, shut down on the Solaris platforms. If this occurs while TIBCO Designer is running, deployment configurations and TIBCO Designer may behave strangely, for example:
Server-based repositories do not appear in the project pull-down list for the current domain when opening projects
Deployment components are displayed as new, even though the deployment history show they had been deployed
The deploy and undeploy options are unavailable for deployed, changed, or even new deployment components
If any of these symptoms occur, reboot the Solaris machine, restart the TRA and TIBCO Administrator, and then restart Designer.
Finding and Killing TIBCO Processes on UNIX platforms
Occasionally, as part of resolving problems you may need to kill all running TIBCO processes.
On UNIX platforms, you can use the UNIX command ‘ps’ to list all the processes that are running. However, the standard ‘all’ flag, ‘-a’, does not necessarily work to list low-level TIBCO processes such as TIBCO Rendezvous. To view such processes, you can execute the following command:
# ps -ef | grep <filter>
For example:
# ps -ef | grep tibco
The above results in several lines that look like this:
root 5559 1 0 09:10:02 pts/1 0:18
/opt/tibco/administrator/<version>/bin/tibcoadmin --propfile /opt/
You can force-kill the processes listed by using the ‘kill’ command with the flag -9 and the process ID (the second column from your ‘ps -ef’ command)
# kill -9 5559
Using the -9 flag ensures the process is really dead. Generally speaking, to ensure all TIBCO processes are dead, you should run
ps -ef | grep tibco
ps -ef | grep hawk
and
ps -ef | grep rvd
and verify that no running processes are found.
Configuring E-Mail Alerts
As mentioned earlier, Windchill ESI can be configured to send you e-mail alert messages when the Windchill ESI business logic experiences an error:
Sending an ESIPostResult message to Windchill PDMLink
Waiting for an ESIResultResponse message from Windchill PDMLink
Parsing the ESIResultResponse message
This configuration is performed through the following global variables:
ESIMail/ToAddress
ESIMail/FromAddress
ESIMail/CCAddress
ESIMail/BCCAddress
The server is configured using the global variable: ESIMail/SMTPHostServer.
Refer to theAdministering Windchill ESI in an ORACLE Applications Environment for details on how to configure the global variables.
The Windchill ESI business logic attempts to retry any EMS-related actions the number of times specified by the TIBCO BusinessWorks global variable, ESIJMS/RetryCount. Therefore, you may receive multiple e-mail messages for the same issue. After a final unsuccessful attempt, the Windchill ESI business logic suspends the associated BusinessWorks process, allowing you to examine the errors described in the e-mail alert messages, fix the underlying problems, and resume or kill the BusinessWorks process using TIBCO Administrator. Refer to the TIBCO Administrator User's Guide for details on how to use this interface.
Resolving Specific Problems
The following section lists specific problems, provides possible causes, and suggests solutions to resolve them.
TIBCO Administrator shows status as unknown for BWengine or adapter processes
Occasionally, as part of resolving problems you may need to kill all running TIBCO processes.
On UNIX platforms, you can use the UNIX command ‘ps’ to list all the processes that are running. However, the standard ‘all’ flag, ‘-a’, does not necessarily work to list low-level TIBCO processes such as TIBCO Rendezvous. To view such processes, you can execute the following command:
# ps -ef | grep <filter>
For example:
# ps -ef | grep tibco
The above results in several lines that look like this:
root 5559 1 0 09:10:02 pts/1 0:18
/opt/tibco/administrator/<version>/bin/tibcoadmin --propfile /opt/
You can force-kill the processes listed by using the ‘kill’ command with the flag -9 and the process ID (the second column from your ‘ps -ef’ command)
# kill -9 5559
Using the -9 flag ensures the process is really dead. Generally speaking, to ensure all TIBCO processes are dead, you should run
ps -ef | grep tibco
ps -ef | grep hawk
and
ps -ef | grep rvd
and verify that no running processes are found.
Configuring E-Mail Alerts
As mentioned earlier, Windchill ESI can be configured to send you e-mail alert messages when the Windchill ESI business logic experiences an error:
Sending an ESIPostResult message to Windchill PDMLink
Waiting for an ESIResultResponse message from Windchill PDMLink
Parsing the ESIResultResponse message
This configuration is performed through the following global variables:
ESIMail/ToAddress
ESIMail/FromAddress
ESIMail/CCAddress
ESIMail/BCCAddress
The server is configured using the global variable: ESIMail/SMTPHostServer.
Refer to theAdministering Windchill ESI in an ORACLE Applications Environment for details on how to configure the global variables.
The Windchill ESI business logic attempts to retry any EMS-related actions the number of times specified by the TIBCO BusinessWorks global variable, ESIJMS/RetryCount. Therefore, you may receive multiple e-mail messages for the same issue. After a final unsuccessful attempt, the Windchill ESI business logic suspends the associated BusinessWorks process, allowing you to examine the errors described in the e-mail alert messages, fix the underlying problems, and resume or kill the BusinessWorks process using TIBCO Administrator. Refer to the TIBCO Administrator User's Guide for details on how to use this interface.
Resolving Specific Problems
The following section lists specific problems, provides possible causes, and suggests solutions to resolve them.
TIBCO Administrator shows status as unknown for BWengine or adapter processes
There are two possible causes for this issue:
A particular service’s microhawk agent is taking time to send its status to the Administrator; it should be resolved in few minutes.
A configuration change has not been picked up by hawk service; restarting the hawk service will resolve this issue
Adapter fails to start with JMS transport
After installing TIBCO Runtime Agent <version> and TIBCO Runtime Agent <version>.1, TIBCO Adapter projects using Enterprise Message Service as transport will not start up from TIBCO Designer and give the following error:
Message "Code = AESDKC-0156,Category = JmsComm, Severity = errorRole, Description = could not open JMS shared library jms." displays
This could be due to incompatible Adapter SDK libraries. To resolve this issue:
On Windows: Remove the libeay32.dll and ssleay32.dll files from <TIBCO_HOME>/adapters/sdk/version/lib
On UNIX: Remove the libssl and libcrypto openssl libraries from the <TIBCO_HOME>/adapters/sdk/version/lib directory.
Unresolved library dependencies
After installing TIBCO Runtime Agent <version> and TIBCO Runtime Agent <version>.1, TIBCO Adapter projects using Enterprise Message Service as transport will not start up from TIBCO Designer and give the following error:
Message "Code = AESDKC-0156,Category = JmsComm, Severity = errorRole, Description = could not open JMS shared library jms." displays
This could be due to incompatible Adapter SDK libraries. To resolve this issue:
On Windows: Remove the libeay32.dll and ssleay32.dll files from <TIBCO_HOME>/adapters/sdk/version/lib
On UNIX: Remove the libssl and libcrypto openssl libraries from the <TIBCO_HOME>/adapters/sdk/version/lib directory.
“Coyote connector has not been started” error appears in TIBCO BusinessWorks Engine Logs
This could be due to incorrect values for ESIOthers/WSHost and ESIOthers/WSPort variables. Providing the proper values should resolve this issue.
All EMS Server configurations disappear after the EMS server is started manually
This could be because the command to start the EMS server has been changed in the 5.1.4 version. In EMS version 4.x the command to start EMS was "./tibemsd". In EMS v5.1.4 the command is "./tibemsd64 -config ../tibco/cfgmgmt/ems/data/tibemsd.conf". The command uses relative path and it should be executed from "<TIBCO_HOME>\ems\<version>\bin".
To resolve this problem stop the process started by the command:
"./tibemsd"
and start the ems server with correct command:
"./tibemsd64 -config ../tibco/cfgmgmt/ems/data/tibemsd.conf"
Deployment of Applications using the TIBCO Administrator GUI displays a Failure message
This could happen if any previous deployment or un-deployment accidentally acquires a lock and has not released it.
To resolve this issue run the AppManage command from command prompt to diagnose and resolve the problem accordingly:
Browse to <Tibco_Home>\tra\<version>\bin
Run the following command:
AppManage -deploy -app <Application name> -user <username>
-pw <password> -domain <Domain_Name>
For Example:
AppManage -deploy -app ORA_ESI_Solution -user admin -pw admin
-domain I4590
For more information refer to the <Tibco_Home>\tra\domain\<Domain_Name>\logs\ApplicationManagement.log file.
TIBCO BusinessWorks returns error messages
TIBCO BusinessWorks returns error messages such as the following:
-2003 Feb 19 13:27:12:294 GMT -5 Engine Error [] PE-Error
process initialization failed for
ProcessDefinitions/Services/WCResult_Service
--Initialization error in
[ProcessDefinitions/Services/WCResult_Service/JMSRepeatUntilTru
e_ESIPostResult_Result/JMSSender_ESIResult_PostResult]
--javax.naming.ServiceUnavailableException: Failed to query
JNDI: Failed to connect to the server at tcp://localhost:7222.
Root exception is javax.jms.JMSException: Failed to connect to
the server at tcp://localhost:7222
This could happen if the EMS server name is invalid.
To resolve this problem confirm that the ESIJMS/JNDIContextURL global variable in the deployment configuration of the process engine matches the value in the QueueConnectionFactory URL, in the factories.conf file located in <Tibco_Home>/ems/<version>/tibco/cfgmgmt/ems/data
The ESIJMS/JNDIContextURL global variable should be set to:
tibjmsnaming://<machinename>:<port>
and the QueueConnectionFactory URL should be set to:
tcp://<machine name>:<port>.
The machine name and port values should match.
For example, if ESIJMS/JNDIContextURL is set to tibjmsnaming://mymachine.mycompany.com:7222, then the QueueConnectionFactory URL should be set to tcp://mymachine.mycompany.com:7222.
Refer to “Global Variables for Process Engines” theAdministering Windchill ESI in an ORACLE Applications Environment for details on this variable.
BusinessWorks returns error messages
BusinessWorks returns error messages such as the following:
-2003 Feb 19 18:57:29:051 GMT -5 Engine Error [] PE-Error process
initialization failed for
ProcessDefinitions/DataProcessing/JMS_ESIEvent_TransactionRelease_
End_PD
--Initialization error in
[ProcessDefinitions/DataProcessing/JMS_ESIEvent_TransactionRelease
_End_PD/JMSReceiver_Event_ESIEvent]
--Could not establish connection or session with EMS provider.
--javax.naming.AuthenticationException: Not permitted: invalid name
or password. Root exception is javax.jms.JMSSecurityException:
invalid name or password
This could happen if the user name and password for the JMS server is invalid.
To resolve this issue confirm that the ESIJMS/Username and ESIJMS/Password global variables in the deployment configuration of the process engine match the values of the username and password specified on the EMS server for the queues.
The username and password values set for the EMS server is in the users.conf file which is located in the <Tibco_Home>/ems/<version>/tibco/cfgmgmt/ems/data directory. Refer to "Global Variables for Process Engines" in theAdministering Windchill ESI in an ORACLE Applications Environment for details on this variable.
The passwords are obfuscated if they were set using the EMS Administration tool. If you do not remember the passwords you set, you will have to reset them.
Upon starting a TIBCO component on UNIX, you receive errors:
Upon starting a TIBCO component on UNIX, you receive the following errors:
/usr/lib/dld.sl: Call to mmap() failed - TEXT (UNIX path)
/usr/lib/dld.sl: Permission denied
For example,
/usr/lib/dld.sl: Call to mmap() failed - TEXT
/opt/tibco/tra/5.6/lib/libmaverick50.sl
/usr/lib/dld.sl: Permission denied
This could happen if permissions for the file have not been set properly.
To resolve this issue use the command chmod 555 on the file listed after TEXT. For example,
# chmod 555 /opt/tibco/tra/<version>/lib/libmaverick50.sl
BusinessWorks, Windchill ESI, or both are failing to connect to EMS
This could happen if the EMS server is not configured properly. When you specify the name of the EMS server as localhost, that server is only recognized on the box that it is running on. No other machine can connect to it. An application that is set to localhost will attempt to find the EMS server running on the same machine. If it is not found, it will error. When you specify a machine name as your server name, other machines can connect to your EMS server.
To resolve this issue check that the QueueConnectionFactory value located in the <Tibco_Home>/ems/<version>/tibco/cfgmgmt/ems/data/factories.conf file and the global variable ESIJMS/JNDIContextURL in the deployment of the Process Engine are set accordingly.
QueueConnectionFactory in factories.conf file = tcp://<machinename>:7222 Where machine name is the machine where the EMS server is running.
Set the global variable ESIJMS/JNDIContextURL in BW Engine = tibjmsnaming://<machinename of where EMS server is running>:7222
It does not matter where this EMS server resides. It can reside on the same machine as Windchill PDMLink, same machine as the TIBCO process engine, or a different machine altogether. As long as the above values described are set appropriately (and the machines are on the same network), Windchill ESI and EAI components are able to connect to the correct EMS server.
To determine which machine and username is connected to an EMS server, type the following command in the TIBCO EMS administration tool:
>show connections
This gives you a list of which users are connected and from which machine.
Transaction remains in pending state
This could happen if the JMS server queues are not subscribed to as described in Configuring and Administering JMS Queues.
Common root causes are: the JMS server is not running when the Windchill method server starts, misconfigurations in the JMS properties of the Windchill adapter, or misconfigurations in the JMS server.
When Windchill ESI services are successful in subscribing to the JMS queue meant for receiving result messages, the following information is included in the Windchill PDMLink method server log:
Thread-10: subscription to queue "com.ptc.windchill.esi.Result" successful
If this information does not appear in the log, it means Windchill was unable to subscribe to the said queue successfully. In this case, all Windchill ESI transactions are left in a pending state. The transactions are processed once Windchill is able to subscribe to the queue successfully.
Normally, when Windchill ESI services are unable to subscribe to the queue because the JMS server is unavailable, Info*Engine receives an exception. Windchill ESI services log this exception in the Windchill Adapter transaction log. If TIBCO EMS is the JMS provider, the message contains the following text:
javax.ems.JMSException: Failed to connect to the server at tcp
As a detection measure, you may consider configuring your monitoring software to look for this or similar text. To resolve the issue, make sure the EMS server is up before the method server, and resolve any issues with the Windchill adapter JMS configuration and the JMS server configuration.
To avoid this issue, ensure the following:
Only one WCESI user is connected to EMS server (can be viewed from EMS Administration Tool> Show connections)
The number of ESISYS connections with ClientID (BW-ESIMaster_JMSConnection-queue-<Application Name>-Process_Archive) should be equal to number of ERP instances configured (can be viewed from EMS Administration Tool> Show connections)
All the connections are from either TIBCO or Windchill server in the current testing suite. There are no connections from a previous suite or from an alien machine (can be viewed from EMS Administration Tool> Show connections)
Windchill and Process Archive connect to same JMS Queue (can be viewed from EMS Administration Tool> Show queues)
The queue com.ptc.windchill.esi.Result has only one receiver (can be viewed from EMS Administration Tool> Show queues)
There should be no messages in any of the queues (can be viewed from EMS Administration Tool> Show queues)
Windchill has subscribed to the DataResponse queue successfully. Connect to JMS server and check if the DataResponse queue is created and the file WCESI user is granted send rights on the DataResponse queue. If show queues displays * in front of DataResponse queue name, it indicates that the queue is temporary and needs to be created. This issue is observed if the EAR is deployed manually. Run the following commands in JMS Administration window to resolve the issue:
create queue <DataResponse>
setprop queue <DataResponse> secure
grant queue <DataResponse> <EAI User> receive
grant queue <DataResponse> <WC ESI User> send
setprop factory QueueConnectionFactory url=tcp://<JMSServer>:7222
commit
Process Archive is connected to the same DataResponse queue that Windchill has subscribed to. Check the JMS Administration window, to confirm that the DataResponse queue is subscribed to by the Process Archive. In case of manual deployment this step is missed sometimes. If the DataResponse queue is subscribed to, check TIBCO Administrator--> Application Management --> Application Name --> Configuration-->Deployment Name--> Advanced -->ESIJMS/DataResponseQueue
TIBCO BusinessWorks Process engine logs error messages
TIBCO BusinessWorks Process engine log contains the following message text:
Job terminated: Please check following 1) FilesToRead_ORA.properties exists at %TIBCO_HOME%\\esi\\bin 2) %TIBCO_HOME%\esi\bin exists in tibco.env.STD_EXT_CP path in %TIBCO_HOME%\
bw\<version>\bin\bwengine.tra
These symptoms usually indicate that, when starting up, the TIBCO BusinessWorks process engine could not load the Windchill ESI business logic properties files. These properties files contain the cross-reference and default values used in data mapping, texts used in logging, etc. There are several possible causes for this issue:
1. The properties files are not properly installed in the <TIBCO HOME>\esi\bin directory.
2. The FilesToRead_ORA.properties file specifies an invalid value for the Path variable. The Path variable must point to the place where the files are located. This value must contain a slash at the end of the path name. For example: C:/tibco/esi/bin (Windows) or /opt/tibco/esi/bin (UNIX).
3. The properties files' security authorizations do not allow read access to the TIBCO BusinessWorks process engine.
To resolve this issue:
1. If the FilesToRead_ORA.properties file specifies an invalid value for the Path variable, enter in the correct value, save the file, and then restart the TIBCO BusinessWorks process engine.
2. If the properties files have restricted read access, grant this access to the TIBCO BusinessWorks process engine. See theAdministering Windchill ESI in an ORACLE Applications Environment for further details on configuring the Windchill ESI business logic properties files and specifying class paths during deployment.
The TIBCO BusinessWorks process engine does not connect to the TIBCO EMS server
This could happen if the BusinessWorks engine was started before the EMS server.
To avoid this problem always start the EMS server before the BusinessWorks engine. For further information on detecting this problem, see topic JMS Problems.
On a Windows server, Adbagent fails to start with the following error: The ordinal 3823 could not be located in dynamic link library LIBEAY32.dll
To resolve this issue run the following commands:
1. MOVE /Y <Tibco_Home>/adapter/sdk/<version>/bin/libeay32.dll <Tibco_Home>/adapter/sdk/5.5/bin/libeay32_bk.dll
2. MOVE /Y <Tibco_Home>/adapter/sdk/<version>/bin/ssleay32.dll <Tibco_Home>/adapter/sdk/5.5/bin/ssleay32_bk.dll
3. COPY /Y <Tibco_Home>/tibrv/<version>/bin/libeay32.dll <Tibco_Home>/adapter/sdk/<version>/bin/libeay32.dll
4. COPY /Y <Tibco_Home>/tibrv/<version>/bin/ssleay32.dll <Tibco_Home>/adapter/sdk/<version>/bin/ssleay32.dll
Windchill ESI fails to publish any business objects to the distribution target system
Windchill ESI fails to publish any business objects to the distribution target system. Errors include overall business object failures, "missing a required field" errors, etc. In addition, the TIBCO BusinessWorks Process engine log does not contain any Windchill ESI message text, as shown in the following example:
2003 Aug 11 18:55:31:722 GMT -4 Engine ESI [] PE-ESI Job-15000
[ProcessDefinitions/Services/Logging_Service/WriteLog_ESIMaster
]: ,98,,3,1,0,40012,,,, 2003 Aug 11 18:55:32:563 GMT -4 Engine
ESI [] PE-ESI Job-15000
[ProcessDefinitions/Services/Logging_Service/WriteLog_ESIMaster
]: ,98,,3,1,0,40013,,,, 2003 Aug 11 18:55:32:864 GMT -4 Engine
ESI [] PE-ESI Job-15000
[ProcessDefinitions/Services/Logging_Service/WriteLog_ESIMaster
]: ,98,,3,1,0,40016,,,,
These symptoms usually indicate that, when starting up, the TIBCO BusinessWorks process engine could not load the Windchill ESI business logic properties files. These properties files contain the cross-reference and default values used in data mapping, texts used in logging, etc. There are several possible causes for this issue:
1. The properties files are not properly installed in the <TIBCO HOME>\esi\bin directory.
2. The FilesToRead_ORA.properties file specifies an invalid value for the Path variable. The Path variable must point to the place where the files are located. This value must contain a slash at the end of the path name. For example: C:/tibco/esi/ (Windows) or /opt/tibco/esi/ (UNIX).
3. The properties files' security authorizations do not allow read access to the TIBCO BusinessWorks process engine.
To resolve this problem verify the following:
1. If the FilesToRead_ORA.properties file specifies an invalid value for the Path variable, enter in the correct value, save the file, and then restart the TIBCO BusinessWorks process engine.
2. If the properties files have restricted read access, grant this access to the TIBCO BusinessWorks process engine. See theAdministering Windchill ESI in an ORACLE Applications Environment for further details on configuring the Windchill ESI business logic properties files and specifying class paths during deployment.
TIBCO BusinessWorks indicates an error similar to one of the following
TIBCO BusinessWorks indicates an error similar to one of the following:
ORA-01461 can bind a LONG value only for insert into a LONG column
[DataDirect][ODBC Oracle driver]String data, right truncated. Error in parameter
This could happen if you have attempted to map a field which has a longer field length in Windchill PDMLink than is allowed in Oracle Applications. The most likely fields having this issue are CN number and Reference Designator.
Windchill PDMLink should be modified or business processes should be in place to prevent CN Numbers or Reference Designators over 10 characters in length. Refer to the Administering Windchill ESI in an ORACLE Applications Environment for further details on managing field length differences between systems.
TIBCO Administrator fails with an "Apache/Tomcat 404" Web browser error
This could happen if you are using an unsupported web browser version.
To resolve this problem upgrade your web browser to Microsoft Internet Explorer version 6.0 SP1, or later.
The TIBCO Adapter for Oracle Applications log contains error messages
The TIBCO Adapter for Oracle Applications log contains error messages similar to the following:
2004 Mar 11 14:43:20:226 GMT -5
BOMConfiguration.BOMConfiguration Error [Adapter] AEADB-700071
Duplicate agent found on DEV01MACHINE.mydomain.mycompany.com
(132.253.82.60)2004 Mar 11 14:43:20:226 GMT -5
MasterConfiguration.MasterConfiguration Info [Adapter] AEADB-
700095 Terminating agent
This could happen if multiple developers are using the same adapter deployment configuration names on the same subnet. Differentiating adapter deployment configurations by their domain is insufficient.
To avoid this problem ensure that all adapter deployment configuration names are unique across the subnet.
When starting an adapter instance or process engine in TIBCO Administrator, the status remains stuck on "Starting Up"
When starting an adapter instance or process engine in TIBCO Administrator, the status remains stuck on "Starting Up" and never changes to "Started".
This is a known issue with the TIBCO Adapter for Oracle Applications and the TIBCO Runtime Agent (TRA). It is possible that starting the TIBCO services automatically (as is the default on Windows systems) may cause them to sometimes start in the wrong sequence and prevent proper communication between the Administrator Server and TRA.
Check the component's log file to determine if it has, in fact, started successfully, despite the "Starting Up" status display. In most cases, it probably has. If not, try the following procedure:
1. Shut down the TIBCO Administrator application and all running TIBCO components and services.
2. Start up the TIBCO services manually, in the following order:
a. TIBCO Administrator <version> (Domain Name)
b. TIBCO HAWK Agent (Domain Name)
3. Restart the TIBCO Administrator application and try to start up the components again.
Multiple "Activity timed out" errors appear in the TIBCO BusinessWorks process engine log
Multiple "Activity timed out" errors appear in the TIBCO BusinessWorks process engine log; however, Windchill ESI reports a successful result in the Enterprise Systems Transaction Log Graphical User Interface.
This is an expected behavior. Windchill ESI automatically compensates for certain timeout events with subsequent Oracle Applications queries, and processing proceeds normally.
No action is required.
The Distribution Targets table is not displayed in the properties page of custom parts
The Distribution Targets table is not displayed in the properties page of custom parts like 'wt.wadm.FADProduct' after the targets are created in the database.
This could happen if the default version of the file <Windchill>\codebase\netmarkets\jsp\tgt\distributionList.jsp is not designed to display the distribution targets table for custom parts.
To enable the Distribution Targets table for custom parts such as wt.wadm.FADProduct:
1. Open the file: <Windchill>\codebase\netmarkets\jsp\tgt\distributionList.jsp
2. Modify the if statement as follows, by adding the custom part type.
if (oid.indexOf("wt.doc") != -1 ||
oid.indexOf("wt.epm") != -1 ||
oid.indexOf("wt.part") != -1 ||
oid.indexOf("com.ptc.windchill.mpml.processplan.MPMProcessPlan") != -1 ||
oid.indexOf("com.ptc.windchill.mpml.resource.MPMProcessMaterial") != -1 ||
oid.indexOf("com.ptc.windchill.mpml.resource.MPMTooling") != -1 ||
oid.indexOf("com.ptc.windchill.mpml.resource.MPMSkill") != -1 ||
oid.indexOf("wt.wadm.FADProducts") != -1)
3. Save the file and restart servlet engine.
Other Configuration Problems
Verify the following settings while manually deploying the EAR:
DataResponse Queue name, it should have the value of ESIOMAdapter/Datasource (like ORAPTCPROD) appended.
Java Max Heap Size and Java Thread Stack Size settings: These should be more than 512 MB and 512 KB respectively.
Now deploy the application, but do not start the services.
EMS: Run the following commands in the JMS Administration window to configure JMS. Replace <DataResponse> and <EAI User> with the applicable values. For example, com.ptc.windchill.esi.DataResponse.ORAPTCPROD and ESISYS and so on.
create queue <DataResponse>
setprop queue <DataResponse> secure
grant queue <DataResponse> <EAI User> receive
grant queue <DataResponse> <WC ESI User> send
setprop factory QueueConnectionFactory url=tcp://<JMSServer>:7222
commit
Now start the services.
Service Startup Errors
Verify the following:
<Tibco_Home>/tra/domain/<Domain_Name>/application/logs
If no logs are generated for service startup and process id shows "-1", try running the corresponding *.sh or *.cmd file from <Tibco_Home>/tra/domain/<Domain_Name>/application/<Application_Name> folder. This would indicate any dependency failures, which should be addressed before starting the service.
Multiple Release To Manufacturing workflows are created upon releasing a promotion request from Windchill PDMLink
This happens if the Windchill ESI preference Publish Promotion Requests has a value No. Set the preference to Yes in order that a single RTM workflow is created upon releasing a promotion request.
* 
With the Windchill ESIPublish Promotion Requests set to No, releasing a promotion request results in as many RTM workflows as there are promotables in the promotion request.