• RSS
  • Twitter
  • FaceBook

Exchange Server Forums

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

RPC Averaged Latency is high during the online maintenance window

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [Microsoft Exchange 2003] >> Information Stores >> RPC Averaged Latency is high during the online maintenance window Page: [1]
Login
Message << Older Topic   Newer Topic >>
RPC Averaged Latency is high during the online maintena... - 10.Oct.2010 3:36:35 AM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
Hi,

I have an exchange 2003 SP2 infrastructure.
The issue is that RPC Averaged latency is going high (>50) during the online maintenance window (default 1:00AM to 6:00AM server local time)

Action taken:
========
I have disabled the online maintenance for three days and found RPC Averaged latency was normal & we have not received any alert from SCOM.

Checked and found disk latency is normal during the online maitenance window.

Can sombody help me as to what could be causing this issue? and what action need to be taken?

Thanks & Regards,
Rajesh Gowda
Post #: 1
RE: RPC Averaged Latency is high during the online main... - 10.Oct.2010 3:04:05 PM   
uemurad

 

Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
The online maintenance window performs several health-related tasks.  You should not disable it completely, but you can modify its schedule.

Tasks line online defragmentation, deleted item and deleted mailbox cleanup, etc., are done during that window.  The more important question is, what else is happening with your server during that time (inbound/outbound mail related)?

_____________________________

Regards,

Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com

(in reply to rajesh.gowda)
Post #: 2
RE: RPC Averaged Latency is high during the online main... - 11.Oct.2010 1:46:51 AM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
Hello Dean T. Uemura

Thanks a ton for your quick response.

1. there is no acitivity (not even a single mail transaction) during this RPC Latency.
2. Have around 20 exchange servers out of which 12 servers have this issue.

as I already mentioned, it is happening during the online maitenance window (servers local time (2:00AM to 6:00AM) )

My mailbox is full of SCOM alerts only.

Please let me know if you need more information.

Thank you once again
Rajesh Gowda

(in reply to uemurad)
Post #: 3
RE: RPC Averaged Latency is high during the online main... - 11.Oct.2010 11:20:23 AM   
uemurad

 

Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
Are you getting System and/or Application log events? What are their IDs?

_____________________________

Regards,

Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com

(in reply to rajesh.gowda)
Post #: 4
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 1:33:50 AM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
Hello Dean T. Uemura,

What I have noticed is when the RPC Avergaed latency goes high I have noticed following Event IDs in the
Application
Source - ESE
Event - 700 & 701

System
========
Source - Foundation Agent
Event - 1180
Description:
OS Information Agent: The Logical Disk Performance Instance, 'D:' is failed with Disk Busy Time of 100 percent.
[SNMP TRAP: 19008 in CPQWINOS.MIB]

 

NOTE: Event 1180 is NOT found on one of the server but Latency is still high during the online maitnenance window ( so i suspect event 1180 is not an issue )

Thanks
Rajesh

(in reply to uemurad)
Post #: 5
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 1:57:49 AM   
uemurad

 

Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
You are making this harder by not including all the information.  Are the 700 and 701 events related to the online defragmentation?

I'd be more concerned about the 1180 events.  What is stored on your drive D:?  Describe your storage configuration - controller(s), drive(s), RAID configuration.

When you have the high latency, what symptoms are your users experiencing?

_____________________________

Regards,

Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com

(in reply to rajesh.gowda)
Post #: 6
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 9:24:00 AM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
I am really sorry for that....
1. yes 700 & 701 Events are related to Public folder defragments
2. D: drive - we have used for database
E: drive - is used for Log files
3. Users are not connected to exchange during this issue (as i said it is happening during the maintenance schedule 2:00 AM to 6:00 AM server Local time)

Hope this helps!

Regards,
Rajesh Gowda

(in reply to uemurad)
Post #: 7
RE: RPC Averaged Latency is high during the online main... - 12.Oct.2010 9:50:36 AM   
uemurad

 

Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
The 700 and 701 events are normal occurences during the maintenance window.

As noted before, I'd be concerned about the 1180 events since it's telling you that something about the D: drive is intermittently failing.

Do you have any AntiVirus software running on this OS? If so, have you excluded the folders where the database files and transactional log files reside?

Do you have any disk-level defragmentation software running? You don't want to perform that type of defrag on a drive with database files.

Why do you have users connecting to Exchange from 2:00 am to 6:00 am? You said you have 20 Exchange servers in your environment. Can you segregate the users so that there is some quiet time for each server?

How much RAM do you have installed in the servers giving you problems?

_____________________________

Regards,

Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com

(in reply to rajesh.gowda)
Post #: 8
RE: RPC Averaged Latency is high during the online main... - 13.Oct.2010 7:27:31 AM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
Hello,

we have excluded the folder path of database files and Transactional Log files

We have no software for disk-level defragmentation

There are no users connected to Exchange between 2:00AM to 6:00AM.
No backup jobs are running at this time.

4GB RAM is installed on the exchange server

Thanks & kind regards,
Rajesh Gowda

(in reply to uemurad)
Post #: 9
RE: RPC Averaged Latency is high during the online main... - 13.Oct.2010 10:17:34 AM   
uemurad

 

Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
Let's take a step backwards. What issue is being caused by RPC latency, and how are you measuring it? In other words, what problem are you trying to solve? Please include details of any Event IDs you are concerned about.

_____________________________

Regards,

Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com

(in reply to rajesh.gowda)
Post #: 10
RE: RPC Averaged Latency is high during the online main... - 13.Oct.2010 10:48:58 PM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
Okay.
I will take an example of 1 exchange server.
The configuration & boot.ini settings of this server is as follows:-

------------------
System Information
------------------
Machine name: xxxxxxx
Operating System: Windows Server 2003, Standard Edition (5.2, Build 3790) Service Pack 2 (3790.srv03_sp2_gdr.070304-2240)
System Model: ProLiant BL20p G2
Processor: Intel(R) Xeon(TM) CPU 3.20GHz (4 CPUs), ~3.2GHz
Memory: 3904MB RAM
Page File: 3817MB used, 8107MB available
_______________
Disks
----------------------
OS & Page file on the same physical disk (Mirrored)
DB - is on one physical disk (RAID 5)
LOG - is on one physical disk
------------------------------------------
Boot.ini parameters
---------------------------------------------
[boot loader]
redirect=UseBiosSettings
timeout=4
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Standard" /noexecute=optout /fastdetect /3GB /USERVA=3030 /redirect /PAE


Issue - RPC Averaged Latency is high
When - Between 2:00AM to 6:00AM (during the online defrag mainteance)
Users - No users are connected to exchange at this time
Monitoring - as and when I receive an alert from SCOM, I logon to this server and open perfmon --> Add counter --> MsExchange IS--> RPC Averaged Latency
The value will be always above 60

Question: Why RPC Averaged latency is going high at this particular time when there are NO users connected to exchange?

NOTE: this similar issue is noticed on other serer as well

(in reply to uemurad)
Post #: 11
RE: RPC Averaged Latency is high during the online main... - 14.Oct.2010 12:12:14 AM   
uemurad

 

Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
So you're not experiencing a problem - you're reacting to what SCOM is telling you, right?

RPC latency will affect user experience, but since no users are connecting at that time it shouldn't be an issue.  You do not want to stop the Maintenance routines from running.

Still, run the ExBPA if you haven't already, and here's some Microsoft Documentation that may be of use.

_____________________________

Regards,

Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com

(in reply to rajesh.gowda)
Post #: 12
RE: RPC Averaged Latency is high during the online main... - 8.Nov.2010 8:05:12 PM   
rajesh.gowda

 

Posts: 9
Joined: 18.Apr.2009
Status: offline
Hello,

Whenever I receive RPC Latency alert by email,  found online maintenance was running against PUBLIC FOLDER store. so I had to suspect this store.

I have now fixed the issue by running ISINTEG against the public folder store.

You believe it or not now the tickets are reduced by 89.36% compared to my last month report.
Received 133 alerts in each month of Sept 2010 & Oct 2010
Received only 2 alerts until today for the Nov 2010 (may be these two alerts are genuine)


Thanks & Regards,


(in reply to uemurad)
Post #: 13

Page:   [1] << Older Topic    Newer Topic >>
All Forums >> [Microsoft Exchange 2003] >> Information Stores >> RPC Averaged Latency is high during the online maintenance window Page: [1]
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Follow TechGenix on Twitter