Exchange Server Forums
Forums |
Register |
Login |
My Profile |
Inbox |
RSS 
|
My Subscription |
My Forums |
Address Book |
Member List |
Search |
FAQ |
Ticket List |
Log Out
RPC Averaged Latency is high during the online maintenance window
|
Users viewing this topic:
none
|
Logged in as: Guest
|
Login  | |
|
RPC Averaged Latency is high during the online maintena... - 10.Oct.2010 3:36:35 AM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
Hi,
I have an exchange 2003 SP2 infrastructure.
The issue is that RPC Averaged latency is going high (>50) during the online maintenance window (default 1:00AM to 6:00AM server local time)
Action taken:
========
I have disabled the online maintenance for three days and found RPC Averaged latency was normal & we have not received any alert from SCOM.
Checked and found disk latency is normal during the online maitenance window.
Can sombody help me as to what could be causing this issue? and what action need to be taken?
Thanks & Regards,
Rajesh Gowda
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 10.Oct.2010 3:04:05 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
The online maintenance window performs several health-related tasks. You should not disable it completely, but you can modify its schedule.
Tasks line online defragmentation, deleted item and deleted mailbox cleanup, etc., are done during that window. The more important question is, what else is happening with your server during that time (inbound/outbound mail related)?
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 11.Oct.2010 1:46:51 AM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
Hello Dean T. Uemura
Thanks a ton for your quick response.
1. there is no acitivity (not even a single mail transaction) during this RPC Latency.
2. Have around 20 exchange servers out of which 12 servers have this issue.
as I already mentioned, it is happening during the online maitenance window (servers local time (2:00AM to 6:00AM) )
My mailbox is full of SCOM alerts only.
Please let me know if you need more information.
Thank you once again
Rajesh Gowda
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 11.Oct.2010 11:20:23 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
Are you getting System and/or Application log events? What are their IDs?
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 1:33:50 AM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
Hello Dean T. Uemura,
What I have noticed is when the RPC Avergaed latency goes high I have noticed following Event IDs in the
Application
Source - ESE
Event - 700 & 701
System
========
Source - Foundation Agent
Event - 1180
Description:
OS Information Agent: The Logical Disk Performance Instance, 'D:' is failed with Disk Busy Time of 100 percent.
[SNMP TRAP: 19008 in CPQWINOS.MIB]
NOTE: Event 1180 is NOT found on one of the server but Latency is still high during the online maitnenance window ( so i suspect event 1180 is not an issue )
Thanks
Rajesh
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 1:57:49 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
You are making this harder by not including all the information. Are the 700 and 701 events related to the online defragmentation?
I'd be more concerned about the 1180 events. What is stored on your drive D:? Describe your storage configuration - controller(s), drive(s), RAID configuration.
When you have the high latency, what symptoms are your users experiencing?
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 9:24:00 AM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
I am really sorry for that....
1. yes 700 & 701 Events are related to Public folder defragments
2. D: drive - we have used for database
E: drive - is used for Log files
3. Users are not connected to exchange during this issue (as i said it is happening during the maintenance schedule 2:00 AM to 6:00 AM server Local time)
Hope this helps!
Regards,
Rajesh Gowda
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 9:50:36 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
The 700 and 701 events are normal occurences during the maintenance window.
As noted before, I'd be concerned about the 1180 events since it's telling you that something about the D: drive is intermittently failing.
Do you have any AntiVirus software running on this OS? If so, have you excluded the folders where the database files and transactional log files reside?
Do you have any disk-level defragmentation software running? You don't want to perform that type of defrag on a drive with database files.
Why do you have users connecting to Exchange from 2:00 am to 6:00 am? You said you have 20 Exchange servers in your environment. Can you segregate the users so that there is some quiet time for each server?
How much RAM do you have installed in the servers giving you problems?
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 13.Oct.2010 7:27:31 AM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
Hello,
we have excluded the folder path of database files and Transactional Log files
We have no software for disk-level defragmentation
There are no users connected to Exchange between 2:00AM to 6:00AM.
No backup jobs are running at this time.
4GB RAM is installed on the exchange server
Thanks & kind regards,
Rajesh Gowda
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 13.Oct.2010 10:17:34 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
Let's take a step backwards. What issue is being caused by RPC latency, and how are you measuring it? In other words, what problem are you trying to solve? Please include details of any Event IDs you are concerned about.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 13.Oct.2010 10:48:58 PM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
Okay.
I will take an example of 1 exchange server.
The configuration & boot.ini settings of this server is as follows:-
------------------
System Information
------------------
Machine name: xxxxxxx
Operating System: Windows Server 2003, Standard Edition (5.2, Build 3790) Service Pack 2 (3790.srv03_sp2_gdr.070304-2240)
System Model: ProLiant BL20p G2
Processor: Intel(R) Xeon(TM) CPU 3.20GHz (4 CPUs), ~3.2GHz
Memory: 3904MB RAM
Page File: 3817MB used, 8107MB available
_______________
Disks
----------------------
OS & Page file on the same physical disk (Mirrored)
DB - is on one physical disk (RAID 5)
LOG - is on one physical disk
------------------------------------------
Boot.ini parameters
---------------------------------------------
[boot loader]
redirect=UseBiosSettings
timeout=4
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Standard" /noexecute=optout /fastdetect /3GB /USERVA=3030 /redirect /PAE
Issue - RPC Averaged Latency is high
When - Between 2:00AM to 6:00AM (during the online defrag mainteance)
Users - No users are connected to exchange at this time
Monitoring - as and when I receive an alert from SCOM, I logon to this server and open perfmon --> Add counter --> MsExchange IS--> RPC Averaged Latency
The value will be always above 60
Question: Why RPC Averaged latency is going high at this particular time when there are NO users connected to exchange?
NOTE: this similar issue is noticed on other serer as well
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 14.Oct.2010 12:12:14 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
So you're not experiencing a problem - you're reacting to what SCOM is telling you, right?
RPC latency will affect user experience, but since no users are connecting at that time it shouldn't be an issue. You do not want to stop the Maintenance routines from running.
Still, run the ExBPA if you haven't already, and here's some Microsoft Documentation that may be of use.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: RPC Averaged Latency is high during the online main... - 8.Nov.2010 8:05:12 PM
|
|
|
rajesh.gowda
Posts: 9
Joined: 18.Apr.2009
Status: offline
|
Hello,
Whenever I receive RPC Latency alert by email, found online maintenance was running against PUBLIC FOLDER store. so I had to suspect this store.
I have now fixed the issue by running ISINTEG against the public folder store.
You believe it or not now the tickets are reduced by 89.36% compared to my last month report.
Received 133 alerts in each month of Sept 2010 & Oct 2010
Received only 2 alerts until today for the Nov 2010 (may be these two alerts are genuine)
Thanks & Regards,
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
Printable Version
