Exchange Server Forums

Exchange 2003 performance issues

Users viewing this topic: none

Logged in as: Guest

Printable Version

All Forums >> [Microsoft Exchange 2003] >> Information Stores >> Exchange 2003 performance issues

Page: [1]

Message

<< Older Topic Newer Topic >>

Exchange 2003 performance issues - 8.Sep.2005 7:38:48 AM

howardphillis

Posts: 18
Joined: 30.Oct.2001
From: london
Status: offline

I am getting persistently slow performance from my Exchange 2003 server and I can't see where the bottleneck could be. Users are seeing the "Retrieving data from Exchange server" message all the time. When you click on a folder or open an email it can take up to 10 seconds to refresh. We have a lot of Mac users who connect via IMAP and they are getting disconnected all the time. At first I thought it was a network issue but I got a company in to monitor the traffic our network for a couple of weeks but they found no real problems with it. I was also seeing the time it took to do a full backup go from about 4 hours to over 24 hours. So then I though it must be a fragmentation problem. I ran an offline defrag and that seemed to cure it. My priv1.edb file shrank from 11Gb down to about 5Gb. Outlook started responding quickly again and even the IMAP users were stayign online. My backup times went back down to a couple of hours so all was looking good...

But that was about a month ago and now it seems the problem is back again. My priv1.edb file is still only reporting to be about 5Gb but when I run analysis in disk defragmenter it shows 49% total fragmentation and 99% file fragmentation. I haven't yet run another offline defrag, but surely I should need to be doing this once a month? If so I'll have to kiss goodbye to my weekends forever.

Any thoughts on this greatly appreciated.

Post #: 1

Featured Links*

RE: Exchange 2003 performance issues - 8.Sep.2005 3:36:36 PM

mark@mvps.org

Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline

Don't worry about looking at statistics from a file defragmenter on the exchange databases. If the edb and stm are 5GB combined then that's all you need to worry about.

You'll need to run some performance monitoring on the server itself, it does sound like that's a good candidate rather than the network. Look for an undue number of messages. If the store managed to go from 11GB to 5GB there was possibly some mail loop or virus scanning gone mad. Check the virus scanner. Are you excluding the directories you need to (see vendors instructions)

_____________________________

Mark Arnold (Exchange MVP)
List Moderator

(in reply to howardphillis)

Post #: 2

RE: Exchange 2003 performance issues - 9.Sep.2005 4:14:06 AM

howardphillis

Posts: 18
Joined: 30.Oct.2001
From: london
Status: offline

Mark, thanks for the feedback. Just to clarify a couple of isues. It was only my priv1.edb file that went from 11Gb down to 5Gb. My priv1.stm stayed at just under 4.5Gb. My total database shrank from just under 16Gb to just under 10Gb (I'm not running Enterprise so I was pretty near the limit).

I was running perfmon for about a month but that was mainly on network counters. Are there any other counters that you would recommend I watch? CPU usage stays around 5%-10% with the occasional spike up to 50%. The server is a dual xeon 2.8Ghz with 1Gb of RAM of which store.exe is using about 500Mb.

I am running Symantec Mail Security for Exchange and there doesn't seem to be anything odd happening with it. It's only picking up about 30 infections a day and it's up to date with definitions.

You say to look for an undue number of messages. Where should I be lookinmg for these?

Any help gratefully appreciated.

(in reply to mark@mvps.org)

Post #: 3

RE: Exchange 2003 performance issues - 26.Sep.2005 11:55:46 AM

zmalmgren

Posts: 3
Joined: 25.Sep.2005
Status: offline

How many IMAP users do you have and what are their mailbox sizes?

We are having the exact same problems you are, although we are still using Exchange 5.5. After much troubleshooting and chasing around, we found disk I/O to be the culprit. We have now setup a second server in the same site, removed the IMPA protocol from the first server, and have moved our IMAP users to the new server. (the new server is on a SAN)

So far we are seeing a performance increase, but I would hate to start celebrating as we are still in the watching and waiting mode.

But IMAP is heavy on I/O utilization, so I would look for that if I were you... Also, I would love to know if you are still having the problem or if it is fixed.

-Zach

(in reply to howardphillis)

Post #: 4

RE: Exchange 2003 performance issues - 26.Sep.2005 12:01:41 PM

howardphillis

Posts: 18
Joined: 30.Oct.2001
From: london
Status: offline

Zach

that sounds pretty interesting. I will definitely check out disk I/O and try moving the IMAP mailboxes. We have about 15 IMAP users with mailboxes up to 400MB each. I'll let you know how I get on and if you have any more developments or discoveries I'd love to hear them.

Howard

(in reply to zmalmgren)

Post #: 5

RE: Exchange 2003 performance issues - 27.Sep.2005 3:23:22 PM

zmalmgren

Posts: 3
Joined: 25.Sep.2005
Status: offline

I know it is now IMAP and the way Exchange handles it. Here is an overview of what happened in our environment.

We have an Exchange 5.5 SP4 server running on a Dual 600 MHz box with 1 gig RAM, 15000k RPM disks in RAID5 with Windows NT 4.0 SP6a.

4 Months ago our clients connecting to the server were all using MAPI (either through the outlook client on Windows or entourage on Macintosh [OS9]). We have 300 mailboxes varying in size from almost nothing to 1.5 gigs. There were no complaints about mail performance.

2 months ago, we started upgrading to Mac OSX. Apparently entourage will not connect to the old exchange 5.5 servers anymore, so we had to put them on IMAP. For whatever reason, most of these users have 400+ MB in their inboxes.
1 month ago, performance lags on the server. Users start to complain. Some IMAP users don�t get disconnected, but effectively can�t read their email as it Is so slow as to be unusable.

Troubleshooting starts. Server CPU is about 45-55% on both processors. The page file isn�t being used, so memory is fine. Perfmon is showing �0� for all I/O related counters (but this turns out to be in error), and networking (from perfmon) looks good. We see some runts and alignment errors reported on the switch, and start thinking its networking. We end up replacing the cable, switching to a new port, and replacing the NIC as well. At this point we realize that non-exchange related tasks on the server are suffering as well. It takes 20+ minutes to copy a 200MB directory to the server (this is on a Gb network). We do exchange related performance tasks, ask users to delete unneeded email, defrag and compress the priv.edb as well as the directory services database. Nothing works. (The NIC replacement finally ended the runts and alignment errors we were seeing, but performance was as awful as ever.) Finally someone actually goes and physically looks at the server and notices the hard drive lights are solid on. It turns out that the server is getting killed on I/O, but for whatever reason, the perfmon counters for �physical disk� we not loaded or not working. We decided that the best thing to do is offload the IMAP users to a separate server connected to the SAN. That should be able to take the I/O no problem, and it�s only for about 30 users that we have on IMAP, so we assume there won�t be any problems with this.

New Server stats: Exchange 5.5 w SP4, Windows 2000 SP4, 1 GB RAM, dual 700 Mhz CPU, Mirrored 18GB 10000 rpm drives for the O/S, SAN attached storage for the �D� drive and exchange data.
We disable IMAP on our first exchange server because now it won�t respond to ANY email requests. We complete the email mailbox move of our 30 users. This is now the current state of both boxes:

First Email Server, running only Outlook Clients:
270 users, CPU for both processors average at 5%, memory (as usual) is in plenty, I/O (from a physical view of the server) is non-existent; it takes 45 seconds to move the same 200 MB data that took 20 minutes.
Second Email Server, running only IMAP Clients:
30 users, CPU is pegged at 100% for both processors, I/O is on the high side, but nothing to worry about from the SAN perspective. Users are complaining about performance.

We now think that Microsoft must have thought that all IMAP users were going to have 5 MB mailboxes or something. Exchange can�t processes IMAP mailboxes without crap loads of I/O and CPU. Our first server was I/O bound, so the CPU never became the bottleneck. After placing that data on the SAN, I/O was no longer the bottleneck and now CPU is the problem.

So we are now looking to offload our IMAP users onto a UNIX IMAP server and assume that all will be well form a performance standpoint. (But I will let everyone know what the results of that are)

In short Howard, I would get these users down to tiny mailbox sizes (we couldn�t for political reasons) or get them onto some other platform. (webmail would be a good short-term work-around)

-Zach Malmgren

(in reply to howardphillis)

Post #: 6

RE: Exchange 2003 performance issues - 28.Sep.2005 4:08:40 AM

howardphillis

Posts: 18
Joined: 30.Oct.2001
From: london
Status: offline

Zach

thanks for all the detail, it was really interesting. A couple of points though. You say your OS9 mac users were connecting via MAPI using entourage. I understood that entourage was an IMAP client. I thought that the only mac MAPI client was Outlook 2001 for Mac. Before we moved to OS X we were using a range of clients - Outlook (MAPI), Quickmail (IMAP) and Outlook Express (POP3) and had none of the performance issues. They only began after the move the OS X when everyone started using mail.app set up for exchange which is effectively just IMAP. I am wondering if it might be a problem with mail.app rather that IMAP. Does that sound possible? What client are your mac users currently using?

I think your suggestion of webmail is a sound one. We have been using Outlook Web Access when nothing else would work but were still seeing problems on the Windows Outlook clients. I am thinking of removing IMAP from the server temporarily to see if it makes a difference. Unfortunately I am not able to move all of my IMAP mailboxes onto a new server as I don't have one available.

I'll report back with any developments.

Howard

(in reply to zmalmgren)

Post #: 7

RE: Exchange 2003 performance issues - 29.Sep.2005 8:28:08 AM

howardphillis

Posts: 18
Joined: 30.Oct.2001
From: london
Status: offline

I've just stopped my IMAP virtual server temporarily to see if it made a difference and immediately Outlook started to respond more quickly. Possibly not as quick as I'd like, but still a measurable improvement.

I'm considering moving my mac users to POP3, but now I can't get mail.app to connect over POP3! I can telnet over port 110 and my last remaining OS9 user is on Outlook Express using POP3 wioth no issues so I'm thinking more and more that it's mail.app that's screwing things up. I've heard really good reports about entourage with SP2 as an OS X exchange client so it maybe time to spend some money! Unless anyone can recommend a good free/cheap OS X mail client.

(in reply to howardphillis)

Post #: 8

RE: Exchange 2003 performance issues - 29.Sep.2005 3:33:39 PM

jroch

Posts: 115
Joined: 17.Aug.2005
From: Enterprise Admin
Status: offline

Is there any possibility that implementing a front-end OWA/IMAP/POP3 server would ease those I/O problems?

(in reply to howardphillis)

Post #: 9

RE: Exchange 2003 performance issues - 30.Sep.2005 2:01:44 PM

jiambor

Posts: 243
Joined: 1.Jun.2005
From: Maryland, US
Status: offline

Curious here....

We are having speratic perf problems with some users getting the popup about trying to recieve from the Exchange 2003 server.....We checked everything and could not find any perf issues. The server is a DL380 G3 with onboard gigabit NICs. They were set with HPs teaming software to use Automatic (recommended) settings. A week ago we tried (which should have been done a long time ago) to set the switch and the NICs to static 1000 FULL (I have actually seen this fix a 5.5 Exchange server I had, except it had to be set to 100 HALF to work right....but anyways), but this did not seem to resolve the issue. Now we have switched the teaming to Fault Tolerant Only so only 1 NIC is active at one time......so this leads to my question.....are you using NIC teaming??

(in reply to jroch)

Post #: 10

RE: Exchange 2003 performance issues - 30.Sep.2005 4:40:49 PM

zmalmgren

Posts: 3
Joined: 25.Sep.2005
Status: offline

We are not using NIC teaming.

When we were first having problems (and thought it was a networking issue) we had a 100 Mb card installed set to Auto. I tried the hard sets of 100/full, 100/half, 10/full, 10/half, all with no increase in perf. We actually would have caught the source of the issue (I/O) sooner if the perfmon counters had been working.

I think this is MS's crappy implementation of IMAP, and I would love to hear of someone getting it to work with multiple users, and what their inbox sizes are.

-Zach

(in reply to jiambor)

Post #: 11

RE: Exchange 2003 performance issues - 3.Oct.2005 3:33:11 AM

howardphillis

Posts: 18
Joined: 30.Oct.2001
From: london
Status: offline

We're not using NIC teaming either and we also spent a long time thinking it was a network issue before ruling that out. I'm inclined to agree with Zach here. As soon as IMAP is disabled performance immediately improves. When you switch IMAP back on again performance gradually slows down over the space of a couple of hours.

(in reply to zmalmgren)

Post #: 12