• RSS
  • Twitter
  • FaceBook

Exchange Server Forums

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

Exchange RPC requests and latency are extremely high

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [Microsoft Exchange 2007] >> Management >> Exchange RPC requests and latency are extremely high Page: [1]
Login
Message << Older Topic   Newer Topic >>
Exchange RPC requests and latency are extremely high - 4.Feb.2010 7:05:49 PM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
For some reason starting only a couple days ago, Exchange is unable to handle our load, which is very light compared to the hardware in the server. We have 2 xeon quad core 3GHz CPUs with 32GB of RAM in this server and RPC requests per user per second is reaching 1.3 with an RPC latency of over 400ms!!! There are over 22,000 emails in queue and send delays are reaching the 2 hour mark. We only have 61 mailboxes and and a handful of public folders on this server, there is no reason why this should be happening. The domain controller which is also global catalog is in the same rack as the Exchange server so there should be minimal delay on DC traffic. I checked the event viewer on both DC and Exchange server and do not see anything in there that should cause this...

Does anyone have any ideas as to why a 61 mailbox powerhouse of a server would have issues like this? Resource use is very low, stays around 15-20% CPU with no red flags on Spotlight (system monitoring program). Please help, we cannot keep going with 2 hour delays in email. It's a bit difficult to have email conversations like that...
Post #: 1
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 6:52:33 AM   
mark@mvps.org

 

Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
Give us your precise disk configurations.

_____________________________

Mark Arnold (Exchange MVP)
List Moderator

(in reply to sserwe)
Post #: 2
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 9:19:39 AM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
3x320GB HDDs no RAID yet, kinda threw it together in a hurry. I plan on rebuilding once I get a new server in and virtualizing. There are no red flags on disk activity though, they are all in the green. Exchange and stores are all on the C: drive. Exchange is the only thing on this server, it shouldn't be having problems like this with only 61 mailboxes... Is this possibly a domain issue?

(in reply to mark@mvps.org)
Post #: 3
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 9:26:52 AM   
mark@mvps.org

 

Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
Not enough. SATA or SAS?

_____________________________

Mark Arnold (Exchange MVP)
List Moderator

(in reply to sserwe)
Post #: 4
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 9:53:54 AM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
Just SATA, 7200 RPM 16MB cache.

(in reply to mark@mvps.org)
Post #: 5
I feel your pain man - 5.Feb.2010 10:15:58 AM   
catzodellamarina

 

Posts: 143
Joined: 26.Feb.2008
Status: offline
I have client latency baaaad. Management has accepted the risk of no limits on my CCR. It was brand new in 2008, now we get repeated calls that "Outlook is slow" "Can't connect to server". Acording to the Performance Troubleshooter, my problem is high RPC too. I have excessive devices & 3rd party crap. The piling on keeps coming.

My problem is management has my hands tied and changes the rules when they see fit. Impossible to stay within specs. Your problem may be more simple. So I'll share with you what I know, maybe your management allows you freedom to fix.

Blackberry, iPhones, encryption sw, multiple shared calendars & mailboxes, and up to 8 GB mailboxes in Online mode are killing me.

(in reply to sserwe)
Post #: 6
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 10:27:59 AM   
mark@mvps.org

 

Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
Those disks have about 65 IOPS of performance. Assuming you have your logs on one disk that's not a lot of IOPS for 11,000 messages an hour (22k messages, 2 hour queue (flakey assumption but work with me :) )

I know you said that disk performance wasn't your issue but what are the disk queues and disk layout?

I'd be pointing fingers there before looking elsewhere.

_____________________________

Mark Arnold (Exchange MVP)
List Moderator

(in reply to sserwe)
Post #: 7
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 11:43:48 AM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
Disk queue length rarely goes above 1, in fact it's sitting below 0.2 right now. I think this is a domain issue. The DC that I said is in the same rack as the mail server has been causing problems since I promoted it to DC in the first place so I demoted it to member. Now the only DC is the primary which at an offsite hosting location. We had it set up to use that one before I even promoted the new one and things were working just fine then, even with 20k+ email an hour (we have reporting services on our web servers that constantly send alerts via email to a public folder). I have disabled all the reporting for now until I get this resolved.

After demoting the local DC, the problem did get better but we still have RPC latency sitting at around 300ms...

(in reply to mark@mvps.org)
Post #: 8
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 11:45:10 AM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
I also tried rebooting both the primary domain controller this morning and the exchange server, which did nothing to solve the issue.

(in reply to mark@mvps.org)
Post #: 9
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 12:00:33 PM   
mark@mvps.org

 

Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
Yeah, that's looking like it's front ending the problem. Fix that and you should see the disk queues ramping up. I'm not saying it's going to be a problem but unless you increase the available IOPS available you're probably going to see something.

_____________________________

Mark Arnold (Exchange MVP)
List Moderator

(in reply to sserwe)
Post #: 10
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 12:26:51 PM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
After running the troubleshooting assistant in exchange, It is telling me "Paged pool memory on server <server> is over the error threshold of 220 MB. Current value: 236 MB." I'm not sure if this is related but seems like a problem none the less. It is also giving me warnings about the first and second storage groups not having a dedicated drive. Is this something I should make happen with the low load that we have?

(in reply to mark@mvps.org)
Post #: 11
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 12:29:36 PM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
quote:

ORIGINAL: mark@mvps.org

Yeah, that's looking like it's front ending the problem. Fix that and you should see the disk queues ramping up. I'm not saying it's going to be a problem but unless you increase the available IOPS available you're probably going to see something.


Do you know how I could fix this? What settings should I check? The server is caught up now but that is because we've basically stopped using it while it was backed up so I'm worried we will have the same problem once I start error reporting on the web servers and get back to our normal load.

(in reply to mark@mvps.org)
Post #: 12
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 1:05:09 PM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
I just ran the performance monitor again while there is next to no load on the server and the RPC operations per second per user rate is still at almost 0.7, way higher than the default error level of 0.25... How is this possible if the server isn't even being used?

(in reply to mark@mvps.org)
Post #: 13
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 1:58:22 PM   
mark@mvps.org

 

Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
Didn't you say you had a lot of "devices"?
Those BB's, they're gonna always be talkin' it's what they do.

_____________________________

Mark Arnold (Exchange MVP)
List Moderator

(in reply to sserwe)
Post #: 14
RE: Exchange RPC requests and latency are extremely high - 5.Feb.2010 3:50:10 PM   
sserwe

 

Posts: 14
Joined: 14.Oct.2009
Status: offline
It looks like I may have cleared up the problem. By demoting the problem DC and making some minor settings changes in Exchange including disabling more intense authentication for incoming emails, the server was eventually able to catch up. Once it caught up, RPC latency returned to normal levels and the amount of RPC requests dropped considerably. I am now working on seperating data stores and transaction logging to seperate physical drives to further increase performance.

Thanks for your help. We're not quite back to our usual load yet but I'm sure I'll find out over the weekend if we're back up to optimal.

(in reply to mark@mvps.org)
Post #: 15

Page:   [1] << Older Topic    Newer Topic >>
All Forums >> [Microsoft Exchange 2007] >> Management >> Exchange RPC requests and latency are extremely high Page: [1]
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Follow TechGenix on Twitter