I know it is now IMAP and the way Exchange handles it. Here is an overview of what happened in our environment.
We have an Exchange 5.5 SP4 server running on a Dual 600 MHz box with 1 gig RAM, 15000k RPM disks in RAID5 with Windows NT 4.0 SP6a.
4 Months ago our clients connecting to the server were all using MAPI (either through the outlook client on Windows or entourage on Macintosh [OS9]). We have 300 mailboxes varying in size from almost nothing to 1.5 gigs. There were no complaints about mail performance.
2 months ago, we started upgrading to Mac OSX. Apparently entourage will not connect to the old exchange 5.5 servers anymore, so we had to put them on IMAP. For whatever reason, most of these users have 400+ MB in their inboxes.
1 month ago, performance lags on the server. Users start to complain. Some IMAP users don’t get disconnected, but effectively can’t read their email as it Is so slow as to be unusable.
Troubleshooting starts. Server CPU is about 45-55% on both processors. The page file isn’t being used, so memory is fine. Perfmon is showing “0” for all I/O related counters (but this turns out to be in error), and networking (from perfmon) looks good. We see some runts and alignment errors reported on the switch, and start thinking its networking. We end up replacing the cable, switching to a new port, and replacing the NIC as well. At this point we realize that non-exchange related tasks on the server are suffering as well. It takes 20+ minutes to copy a 200MB directory to the server (this is on a Gb network). We do exchange related performance tasks, ask users to delete unneeded email, defrag and compress the priv.edb as well as the directory services database. Nothing works. (The NIC replacement finally ended the runts and alignment errors we were seeing, but performance was as awful as ever.) Finally someone actually goes and physically looks at the server and notices the hard drive lights are solid on. It turns out that the server is getting killed on I/O, but for whatever reason, the perfmon counters for “physical disk” we not loaded or not working. We decided that the best thing to do is offload the IMAP users to a separate server connected to the SAN. That should be able to take the I/O no problem, and it’s only for about 30 users that we have on IMAP, so we assume there won’t be any problems with this.
New Server stats: Exchange 5.5 w SP4, Windows 2000 SP4, 1 GB RAM, dual 700 Mhz CPU, Mirrored 18GB 10000 rpm drives for the O/S, SAN attached storage for the “D” drive and exchange data.
We disable IMAP on our first exchange server because now it won’t respond to ANY email requests. We complete the email mailbox move of our 30 users. This is now the current state of both boxes:
First Email Server, running only Outlook Clients:
270 users, CPU for both processors average at 5%, memory (as usual) is in plenty, I/O (from a physical view of the server) is non-existent; it takes 45 seconds to move the same 200 MB data that took 20 minutes.
Second Email Server, running only IMAP Clients:
30 users, CPU is pegged at 100% for both processors, I/O is on the high side, but nothing to worry about from the SAN perspective. Users are complaining about performance.
We now think that Microsoft must have thought that all IMAP users were going to have 5 MB mailboxes or something. Exchange can’t processes IMAP mailboxes without crap loads of I/O and CPU. Our first server was I/O bound, so the CPU never became the bottleneck. After placing that data on the SAN, I/O was no longer the bottleneck and now CPU is the problem.
So we are now looking to offload our IMAP users onto a UNIX IMAP server and assume that all will be well form a performance standpoint. (But I will let everyone know what the results of that are)
In short Howard, I would get these users down to tiny mailbox sizes (we couldn’t for political reasons) or get them onto some other platform. (webmail would be a good short-term work-around)