RPC Averaged Latency is high during the online maintenance window (Full Version)

All Forums >> [Microsoft Exchange 2003] >> Information Stores



Message

rajesh.gowda -> RPC Averaged Latency is high during the online maintenance window (10.Oct.2010 3:36:35 AM)

Hi,

I have an exchange 2003 SP2 infrastructure.
The issue is that RPC Averaged latency is going high (>50) during the online maintenance window (default 1:00AM to 6:00AM server local time)

Action taken:
========
I have disabled the online maintenance for three days and found RPC Averaged latency was normal & we have not received any alert from SCOM.

Checked and found disk latency is normal during the online maitenance window.

Can sombody help me as to what could be causing this issue? and what action need to be taken?

Thanks & Regards,
Rajesh Gowda



uemurad -> RE: RPC Averaged Latency is high during the online maintenance window (10.Oct.2010 3:04:05 PM)

The online maintenance window performs several health-related tasks.  You should not disable it completely, but you can modify its schedule.

Tasks line online defragmentation, deleted item and deleted mailbox cleanup, etc., are done during that window.  The more important question is, what else is happening with your server during that time (inbound/outbound mail related)?



rajesh.gowda -> RE: RPC Averaged Latency is high during the online maintenance window (11.Oct.2010 1:46:51 AM)

Hello Dean T. Uemura

Thanks a ton for your quick response.

1. there is no acitivity (not even a single mail transaction) during this RPC Latency.
2. Have around 20 exchange servers out of which 12 servers have this issue.

as I already mentioned, it is happening during the online maitenance window (servers local time (2:00AM to 6:00AM) )

My mailbox is full of SCOM alerts only.

Please let me know if you need more information.

Thank you once again
Rajesh Gowda



uemurad -> RE: RPC Averaged Latency is high during the online maintenance window (11.Oct.2010 11:20:23 AM)

Are you getting System and/or Application log events? What are their IDs?



rajesh.gowda -> RE: RPC Averaged Latency is high during the online maintenance window (12.Oct.2010 1:33:50 AM)

Hello Dean T. Uemura,

What I have noticed is when the RPC Avergaed latency goes high I have noticed following Event IDs in the
Application
Source - ESE
Event - 700 & 701

System
========
Source - Foundation Agent
Event - 1180
Description:
OS Information Agent: The Logical Disk Performance Instance, 'D:' is failed with Disk Busy Time of 100 percent.
[SNMP TRAP: 19008 in CPQWINOS.MIB]

 

NOTE: Event 1180 is NOT found on one of the server but Latency is still high during the online maitnenance window ( so i suspect event 1180 is not an issue )

Thanks
Rajesh



uemurad -> RE: RPC Averaged Latency is high during the online maintenance window (12.Oct.2010 1:57:49 AM)

You are making this harder by not including all the information.  Are the 700 and 701 events related to the online defragmentation?

I'd be more concerned about the 1180 events.  What is stored on your drive D:?  Describe your storage configuration - controller(s), drive(s), RAID configuration.

When you have the high latency, what symptoms are your users experiencing?



rajesh.gowda -> RE: RPC Averaged Latency is high during the online maintenance window (12.Oct.2010 9:24:00 AM)

I am really sorry for that....
1. yes 700 & 701 Events are related to Public folder defragments
2. D: drive - we have used for database
E: drive - is used for Log files
3. Users are not connected to exchange during this issue (as i said it is happening during the maintenance schedule 2:00 AM to 6:00 AM server Local time)

Hope this helps!

Regards,
Rajesh Gowda



uemurad -> RE: RPC Averaged Latency is high during the online maintenance window (12.Oct.2010 9:50:36 AM)

The 700 and 701 events are normal occurences during the maintenance window.

As noted before, I'd be concerned about the 1180 events since it's telling you that something about the D: drive is intermittently failing.

Do you have any AntiVirus software running on this OS? If so, have you excluded the folders where the database files and transactional log files reside?

Do you have any disk-level defragmentation software running? You don't want to perform that type of defrag on a drive with database files.

Why do you have users connecting to Exchange from 2:00 am to 6:00 am? You said you have 20 Exchange servers in your environment. Can you segregate the users so that there is some quiet time for each server?

How much RAM do you have installed in the servers giving you problems?



rajesh.gowda -> RE: RPC Averaged Latency is high during the online maintenance window (13.Oct.2010 7:27:31 AM)

Hello,

we have excluded the folder path of database files and Transactional Log files

We have no software for disk-level defragmentation

There are no users connected to Exchange between 2:00AM to 6:00AM.
No backup jobs are running at this time.

4GB RAM is installed on the exchange server

Thanks & kind regards,
Rajesh Gowda



uemurad -> RE: RPC Averaged Latency is high during the online maintenance window (13.Oct.2010 10:17:34 AM)

Let's take a step backwards. What issue is being caused by RPC latency, and how are you measuring it? In other words, what problem are you trying to solve? Please include details of any Event IDs you are concerned about.



rajesh.gowda -> RE: RPC Averaged Latency is high during the online maintenance window (13.Oct.2010 10:48:58 PM)

Okay.
I will take an example of 1 exchange server.
The configuration & boot.ini settings of this server is as follows:-

------------------
System Information
------------------
Machine name: xxxxxxx
Operating System: Windows Server 2003, Standard Edition (5.2, Build 3790) Service Pack 2 (3790.srv03_sp2_gdr.070304-2240)
System Model: ProLiant BL20p G2
Processor: Intel(R) Xeon(TM) CPU 3.20GHz (4 CPUs), ~3.2GHz
Memory: 3904MB RAM
Page File: 3817MB used, 8107MB available
_______________
Disks
----------------------
OS & Page file on the same physical disk (Mirrored)
DB - is on one physical disk (RAID 5)
LOG - is on one physical disk
------------------------------------------
Boot.ini parameters
---------------------------------------------
[boot loader]
redirect=UseBiosSettings
timeout=4
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Standard" /noexecute=optout /fastdetect /3GB /USERVA=3030 /redirect /PAE


Issue - RPC Averaged Latency is high
When - Between 2:00AM to 6:00AM (during the online defrag mainteance)
Users - No users are connected to exchange at this time
Monitoring - as and when I receive an alert from SCOM, I logon to this server and open perfmon --> Add counter --> MsExchange IS--> RPC Averaged Latency
The value will be always above 60

Question: Why RPC Averaged latency is going high at this particular time when there are NO users connected to exchange?

NOTE: this similar issue is noticed on other serer as well



uemurad -> RE: RPC Averaged Latency is high during the online maintenance window (14.Oct.2010 12:12:14 AM)

So you're not experiencing a problem - you're reacting to what SCOM is telling you, right?

RPC latency will affect user experience, but since no users are connecting at that time it shouldn't be an issue.  You do not want to stop the Maintenance routines from running.

Still, run the ExBPA if you haven't already, and here's some Microsoft Documentation that may be of use.



rajesh.gowda -> RE: RPC Averaged Latency is high during the online maintenance window (8.Nov.2010 8:05:12 PM)

Hello,

Whenever I receive RPC Latency alert by email,  found online maintenance was running against PUBLIC FOLDER store. so I had to suspect this store.

I have now fixed the issue by running ISINTEG against the public folder store.

You believe it or not now the tickets are reduced by 89.36% compared to my last month report.
Received 133 alerts in each month of Sept 2010 & Oct 2010
Received only 2 alerts until today for the Nov 2010 (may be these two alerts are genuine)


Thanks & Regards,




Page: [1]