Saturday, June 16, 2012

Verbose garbage collection

When you turn the verbose garbage collection on, WAS will start printing garbage collection related information in your native_stderr.log file. You can turn the vebose garbage collection on using WAS Admin Console by going to Application servers < servername servers < Process Definition servers < Java Virtual Machine. Once you enable the verbose garbage collection on the WAS server will start writing messages into native_stderr.log file every time it executes garbage collection.

These are couple of entries from my native_stderr.log file.


<af type="tenured" id="49" timestamp="Jul 05 13:13:17 2009" intervalms="32.872">
<minimum requested_bytes="16776" />
<time exclusiveaccessms="0.044" />
<tenured freebytes="3624584" totalbytes="116873216" percent="3" >
<soa freebytes="3624584" totalbytes="116873216" percent="3" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="49" totalid="49" intervalms="34.359">
<refs_cleared soft="0" threshold="32" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="107.399" sweep="1.583" compact="0.000" total="109.117" />
<tenured freebytes="43894704" totalbytes="116873216" percent="37" >
<soa freebytes="43894704" totalbytes="116873216" percent="37" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
</gc>
<tenured freebytes="43877928" totalbytes="116873216" percent="37" >
<soa freebytes="43877928" totalbytes="116873216" percent="37" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<time totalms="110.648" />
</af>

<af type="tenured" id="50" timestamp="Jul 05 13:13:18 2009" intervalms="232.708">
<minimum requested_bytes="32" />
<time exclusiveaccessms="0.031" />
<tenured freebytes="0" totalbytes="116873216" percent="0" >
<soa freebytes="0" totalbytes="116873216" percent="0" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="50" totalid="50" intervalms="234.140">
<classloadersunloaded count="13" timetakenms="48.694" />
<expansion type="tenured" amount="19868672" newsize="136741888" timetaken="0.152" reason="excessive time being spent in gc" gctimepercent="49" />
<refs_cleared soft="0" threshold="32" weak="3" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="118.982" sweep="2.639" compact="0.000" total="170.899" />
<tenured freebytes="55392816" totalbytes="136741888" percent="40" >
<soa freebytes="55392816" totalbytes="136741888" percent="40" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
</gc>
<tenured freebytes="55392176" totalbytes="136741888" percent="40" >
<soa freebytes="55392176" totalbytes="136741888" percent="40" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<time totalms="172.525" />
</af>


The way IBM JDK works is if it is not able to allocate a memory then it will execute garbage collection to free up the memory. The J9 VM used in WAS 6.1 generates one <af> element every time a garbage collection works.

The af element has following elements

  • type:

  • id: The id represents how many times the gc was executed

  • intervalms: The time in ms since last time gc was executed

  • timestamp: time of gc


The minimum represents the number of bytes that were requested and JVM couldnot allocate them so it had to trigger garbage collection cycle.

The af element has 3 main child elements first tenured element has data about the tenured memory position before gc then gc element represents data about what happened during gc, such as time spent in mark, sweep and compact phases, The second tenured element represents the position of tenured memory after gc.


The IBM Support assistance has IBM Pattern modeling and Analysis tool for Java Garbage collection tool that can be used to analyze the garbage collection.

WebSphere Log Files /Logging performance data

Plug-In Logs
WebServer http Plugin will create log, by default named as http-plugin.log, placed under PLUGIN_HOME/logs/
Plugin writes Error messages into this log. The attribute which deals with this is
< Log > in the plugin-cfg.xml
For Example
< Log LogLevel=”Error” Name=”/opt/IBM/WebSphere/Plugins/logs/http_plugin.log” / >
To Enable Tracing set Log LogLevel to “Trace”.
< Log LogLevel=”Trace” Name=”/opt/IBM/WebSphere/Plugins/logs/http_plugin.log” / >
JVM logs
$ find /opt/IBM/WebSphere/ -name SystemOut.log -print
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Node/logs/member1/SystemOut.log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Node/logs/member2/SystemOut.log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Node/logs/nodeagent/SystemOut.log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Dmgr/logs/Dmgr/SystemOut.log
NodeAgent Process Log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Node/logs/nodeagent/native_stdout.log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Node/logs/nodeagent/native_stderr.log
IBM service logs – activity.log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Node/logs/activity.log
/opt/IBM/WebSphere/AppServer/profiles/%Profile%/Dmgr/logs/activity.log
——————————————————————————–
Enabling automated heap dump generation, DONT DO THIS IN PRODUCTION
  1. Click Servers > Application servers in the administrative console navigation tree.
  2. Click server_name >Performance and Diagnostic Advisor Configuration.
  3. Click the Runtime tab.
  4. Select the Enable automatic heap dump collection check box.
  5. Click OK
Locating and analyzing heap dumps
Goto profile_root\myProfile. IBM heap dump files are usually named as heapdump*.phd
Download and use tools like heapAnalyzer, dumpanalyzer
——————————————————————————–
Logging performance data with TPV(Tivoli Performance Viewer)
    1. Click Monitoring and Tuning > Performance Viewer > Current Activity > server_name > Settings > Log in the console navigation tree. To see the Log link on the Tivoli Performance Viewer page, expand the Settings node of the TPV navigation tree on the left side of the page. After clicking Log, the TPV log settings are displayed on the right side of the page.
    2. Click on Start Logging when viewing summary reports or performance modules.
    3. When finished, click Stop Logging . Once started, logging stops when the logging duration expires, Stop Logging is clicked, or the file size and number limits are reached. To adjust the settings, see step 1.
    By default, the log files are stored in the profile_root/logs/tpv directory on the node on which the server is running. TPV automatically compresses the log file when it finishes writing to it to conserve space. At this point, there must only be a single log file in each .zip file and it must have the same name as the .zip file.
  • View logs.
    1. Click Monitoring and Tuning > Performance Viewer > View Logs in the console navigation tree.
    2. Select a log file to view using either of the following options:
      Explicit Path to Log File
      Choose a log file from the machine on which the browser is currently running. Use this option if you have created a log file and transferred it to your system. Click Browse to open a file browser on the local machine and select the log file to upload.
      Server File
      Specify the path of a log file on the server.In a stand-alone application server environment, type in the path to the log file. The profile_root\logs\tpv directory is the default on a Windows system.
    3. Click View Log. The log is displayed with log control buttons at the top of the view.
    4. Adjust the log view as needed. Buttons available for log view adjustment are described below. By default, the data replays at the Refresh Rate specified in the user settings. You can choose one of the Fast Forward modes to play data at rate faster than the refresh rate.
      Rewind Returns to the beginning of the log file.
      Stop Stops the log at its current location.
      Play Begins playing the log from its current location.
      Fast Forward Loads the next data point every three (3) seconds.
      Fast Forward 2 Loads ten data points every three (3) seconds.
    You can view multiple logs at a time. After a log has been loaded, return to the View Logs panel to see a list of available logs. At this point, you can load another log.
    TPV automatically compresses the log file when finishes writing it. The log does not need to be decompressed before viewing it, though TPV can view logs that have been decompressed.

JVM Logs

The JVM Logs are the SystemOut.log and SystemErr.log files stored in each App Server, Node agent, and Deployment Manager profile. They contain messages sent to the Java System.out and System.err streams.
Location:
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/SystemOut.log
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/nodeagent/SystemOut.log
Where:
  • AppSrv01 is the App Server profile name.
  • server1 is the actual server name.
  • nodeagent is the actual node agent name for that profile.
Example event:
[2/17/12 13:52:00:440 PST] 00000057 CoordinatorCo W   HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 10 seconds.

Time stamp, ThreadId, Message shortname, EventType, MessageId: Message

Native (Process) Logs

The native_stdout.log or native_stderr.log files are stored in each App Server, Node agent, and Deployment Manager profile. They contain messages sent to stdout and stderr from native code including the JVM.
File location: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/native_stdout.log
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/nodeagent/native_stderr.log
where:
  • AppSrv01 is the App Server profile name.
  • server1 is the actual server name.
  • nodeagent is the actual node agent name for that profile.

Other server specific log files

All log files stored as part of a profile in individual server directories have a similar structure to the JVM logs. An example of this is startServer.log.These files contain information about specific activities such as starting and stopping servers, adding nodes and so on.

wsadmin.traceout

These files contain data for each wsadmin session and the content of the files is refreshed each time a new wsadmin session is created. They are stored in the Profile's log directory (Structure is similar to the JVM logs)
Dmgr01 is the Deployment Manager's profile name.
/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/wsadmin.traceout 

FFDC logs

First Fail Data Capture logs contain information generated from a processing failure. The files are stored in the ffdc directory under the Profile's log directory.
Dmgr01 is the Deployment Manager's profile name.
/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_cd30cd3_12.02.09_23.17.18.1876809.txt

Optional files

If you are a sophisticated Splunk user you can customize the Splunk App for WAS and create views and dashboards to look at the data in the files listed below. You can search and index these files using Splunk and do some basic field extractions on them. There are no out-of-the-box views that display this data.
Filename Location
javacore*.txt /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/javacore*.txt
activity.log /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/activity.log
Server Exception log /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log
App logs stored in the Profile's Log folders

Latency Vs Bandwidth

One of the most commonly misunderstood concepts in networking is speed and capacity. Most people believe that capacity and speed are the same thing. For example, it's common to hear "How fast is your connection?" Invariably, the answer will be "640K", "1.5M" or something similar. These answers are actually referring to the bandwidth or capacity of the service, not speed.

Speed and bandwidth are interdependent. The combination of latency and bandwidth gives users the perception of how quickly a webpage loads or a file is transferred. It doesn't help that broadband providers keep saying "get high speed access" when they probably should be saying "get high capacity access". Notice the term "Broadband" - it refers to how wide the pipe is, not how fast.

Latency:

Latency is delay.

For our purposes, it is the amount of time it takes a packet to travel from source to destination. Together, latency and bandwidth define the speed and capacity of a network.

Latency is normally expressed in milliseconds. One of the most common methods to measure latency is the utility ping. A small packet of data, typically 32 bytes, is sent to a host and the RTT (round-trip time, time it takes for the packet to leave the source host, travel to the destination host and return back to the source host) is measured.

The following are typical latencies as reported by others of popular circuits type to the first hop. Please remember however that latency on the Internet is also affected by routing that an ISP may perform (ie, if your data packet has to travel further, latencies increase).

Ethernet                  .3ms
Analog Modem              100-200ms
ISDN                      15-30ms
DSL/Cable                 10-20ms
Stationary Satellite      >500ms, mostly due to high orbital elevation
DS1/T1                    2-5ms


Bandwidth:

Bandwidth is normally expressed in bits per second. It's the amount of data that can be transferred during a second.

Solving bandwidth is easier than solving latency. To solve bandwidth, more pipes are added. For example, in early analog modems it was possible to increase bandwidth by bonding two or more modems. In fact, ISDN achieves 128K of bandwidth by bonding two 64K channels using a datalink protocol called multilink-ppp.

Bandwidth and latency are connected. If the bandwidth is saturated then congestion occurs and latency is increased. However, if the bandwidth of a circuit is not at peak, the latency will not decrease. Bandwidth can always be increased but latency cannot be decreased. Latency is the function of the electrical characteristics of the circuit.