Exchange Server Forums
Forums |
Register |
Login |
My Profile |
Inbox |
RSS 
|
My Subscription |
My Forums |
Address Book |
Member List |
Search |
FAQ |
Ticket List |
Log Out
Mail store size growth out of control
|
Users viewing this topic:
none
|
Logged in as: Guest
|
Login  | |
|
Mail store size growth out of control - 27.Nov.2009 9:18:25 AM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
I have four mail store running under exchange 2003 SP2 under windows 2003,
One of mail store growth out of control. I added all users mailbox size, is about 170G, hasnt change much this week, but the edb of this mail store is growing 20 to 40G a day, right now is 285G, and growing. I saw a similar instance from this forum, but seems no answer,
http://forums.msexchange.org/m_1800493774/tm.htm
can anyone help?
Thank you very much,
Jerry
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 9:25:10 AM
|
|
|
mark@mvps.org
Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
|
A store going crazy would indicate a mail loop, an anti virus scanner not excluding the Exchange directories (why have a/v on a mailbox server in the first place!) or something similar affecting the environment. There isn't anything specifically infrastructure based that would cause a store to grow out of control.
_____________________________
Mark Arnold (Exchange MVP)
List Moderator
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 9:29:54 AM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Thank you, Mark, for quick reply
It jsut happen start this week, one thing interesting is I added the total users mailbox is 170G, but the actually edb file is 285G and growing. On last night event 1221, shows this mail store has 325MB of free space after online defrag has terminated. There is total 4 mail store, and this is the only one has the problem.
I am planning take down the mail storea tonight and run off line defrag, not sure it that will help.
THanks again,
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 9:48:36 AM
|
|
|
mark@mvps.org
Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
|
NO !!!!!
There is only 325MB of space so it's pointless doing that offline defrag; you wouldn't get any space back.
If it just started happening this week there's a good chance some OOF message has gone awry - who is on vacation this week and might have set a message? Did you have some update on the a/v or something else?
Also, take a look at Deleted Item Retention and drop that down. Then give it a day or two and keep an eye on event id 1221.
You could also set up a couple of new stores and move the mailboxes into them then delete the old one.
_____________________________
Mark Arnold (Exchange MVP)
List Moderator
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 1:00:23 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
thanks, Mark,
Its now edb at 320G, but the total mailbox still stay at 170G, and the stm file stays at 29G, we have scanned and scanning 20 suspected PCs, by use inhouse McAfee and downloaded malwarebytes, nothing have been found. I ran a quick malwarebytes scan on the serve two days ago, nothing found, I will plan to move all users(150) to other two mailstoer tonight, I still dont understand
1) 150G defference between edb and actually mailbox, not to mention stm file
2) why its only happen on this mail store, will move mailbox out of store fix this
Would you able to provide a link of the good monitor and anylasis tools for exchagne to find out whats going on on this mail store?
Thansk again,
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 1:04:15 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
One thing I forgot to mention,
This exchange server is out of date for patches for at least three years, windows patches and Exchange patches, Do you know there is any known issue similar to this?
Thanks
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 2:12:53 PM
|
|
|
mark@mvps.org
Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
|
Well, the obvious springs to mind. You're as infected as a very infected thing, spamming, relaying and lord knows what else.
Get the basics fixed and then take stock of where you are.
_____________________________
Mark Arnold (Exchange MVP)
List Moderator
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 4:02:45 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
I'm curious - do you have SMTP logging enabled and/or Message Tracking enabled? If so, are the log files generated suddenly a lot bigger than before you started seeing the increased store size?
I'd also recommend that you download and run the Best Practices Analyzer.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 4:42:00 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Dean,
Yes, this is what we have found out the increase of mail store, when the log partition ran out of space, it generated 20x 5Mb logs every minutes on Monday, then we have to enable circular logging, and we have found that the edb files increasing dramatically, Monday (180G)--Tuesday(203G) - Wednesday(234G)--Thursday(270G)-Friday (320G), but the total size of user mailbox is stay at 170G, and stm stays at 29G, when I compare other mail store( we have total 4 mail store), I have found the total user mailbox size on each mail store =edb+stm files, but not on this mail store,
One of interesting thing I found out is, from messaging log, you can see the same mail have been excuted multiple times, but when ask the user, and look at the sent out item, it only sent out once.
This email is 6.5MB it only sent out once, but it dectect couple of times
ckucher@mycompany.com
6555235
2009-11-27 14:46:27 GMT
Kevin Bereza Crew photos
tolsen@mycompany.com
ckucher@mycompany.com
6555235
2009-11-27 14:46:27 GMT
Kevin Bereza Crew photos
tolsen@mycompany.com
ckucher@mycompany.com
6555235
2009-11-27 14:46:27 GMT
Kevin Bereza Crew photos
tolsen@mycompany.com
ckucher@mycompany.com
6555235
2009-11-27 14:46:27 GMT
Kevin Bereza Crew photos
tolsen@mycompany.com
We have scanned top 10 users pc with malwarebytes, and McAfee, have not found anything, and all of the users commited that they actually sent (forward) those attachment emal to multiple users, but only once. so it looks like not happen on client end, it must something happen on server end.
When look at event 1221, shows only 350MB white space on this mail store, so offline defrag wont help.
Still try to find out what cause this space issue.
Any advice would be appreciated.
Jerrry
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 4:59:58 PM
|
|
|
mark@mvps.org
Posts: 6811
Joined: 9.Jun.2004
From: Philadelphia PA
Status: offline
|
If the STM is staying small that means the traffic is not coming from the Internet and staying in Internet only format (OWA, POP/IMAP etc.) So if it is coming from the net it's being promoted into MAPI.
"Crew Photo's" eh? Interesting. Got any limits on the mailboxes? Got any OOF's going on? Check all OOFs, check all rules, kill everything. Antivirus up to date. Set limits to very small numbers.
_____________________________
Mark Arnold (Exchange MVP)
List Moderator
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 5:01:54 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
As far as advice goes, the first thing I'd recommend is that you find a way to disable circular logging. Circular logging effectively disables an important recovery feature of Exchange by rendering your Transactional Logs useless. The Transactional Logs are created in case you ever have to perform a restore from backup. Any backup is a point-in-time snapshot of your Information Store. The Transactional Logs contain all the inbound and outbound messages since that time and can be "replayed" for recovery purposes.
How often are you backing up your Information Store?
Did anything change on Monday, or possibly the previous weekend?
Do you have SMTP logging enabled? Do you have Message Tracking enabled? If so, what were the sizes of those log files for those same days last week (each creates one file per day)?
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 5:14:03 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Thanks Dean,
1) I will try to find a partition to handle transcation log, 20x5MB/minute =6GB/Hr, 100G should be able to handle one day logs,
2) Weekly full back for Information store
3) Only change we made is move one of mail store to different partition, we are constently struggling spaces, we have to increase SAN space couple of time this year already,
3) these files size didnt change much, around 30 to 40MB per day per file
4) I have change the limit of sent and receive message to 6MB, ( before was 30MB receiving, 10MB senting-------we are contruction company, lots of digital pictures attached through Email) , this change looks like have some effect of this mail store size. it growing much slow now. not sure its because of weekend, or what.
Thank you again , Dean,
I will try to find a partition to put transcation log.
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 27.Nov.2009 6:01:15 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
quote:
1) I will try to find a partition to handle transcation log, 20x5MB/minute =6GB/Hr, 100G should be able to handle one day logs,
2) Weekly full back for Information store Unless you are also taking daily Incremental backups, you'll need to accommodate a week's worth of Transactional Logs. What Backup program are you using?
quote:
3) Only change we made is move one of mail store to different partition, we are constently struggling spaces, we have to increase SAN space couple of time this year already, Perhaps you need to examine your storage policies. For free you can limit mailbox sizes, you can configure retention periods for Deleted Items and/or Deleted Mailboxes, you can configure Mailbox Management to limit the amount of time a message can be stored. At a cost you can implement an archival software system.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 1:27:23 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Dean,
1) daily incremental backup is running as well, we are use Symantec.
The total information store now is 1.02TB, took 36 hours for weekly full backup
2) will in process of puting in archive solution in place
3) still dont understand why
a) only this mail store increase so dramatically,
b) the total user mailbox within this mail store is only increase less than 1G last week, but the edb of this mail store itself is already 320G from 180G a week ago
4) review the log report, and scanned the top 20 users' PC, with McAfee and malwarebytes, nothing has been found, most of users have committed that they actually sent out --- the large emails with attachment sent to mutil reciepents---
Will keep you udpate.
Thanks
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 3:11:53 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
One possibility is that Single Instance Storage is being broken by someone. Exchange only keeps a single copy of each message and each mailbox in that same Mailbox Store points to it. The exception is that if a message is exported and re-imported then Exchange no longer understands it is the same message.
Example, a 5MB message is received by ten (10) people in the same Mailbox Store. Exchange stores one copy of the message and each of the ten mailboxes point to it - effectively taking 5MB of space in the EDB file (plus a handful of bytes for each mailbox pointer). Now, one of those people uses the Outlook Archive function to move the message out of the mailbox and into a PST file. The space taken in the EDB file only decreases by one mailbox pointer - amounting to an insignificant change. Later, that same user moves the message back into the mailbox from the PST file. Exchange no longer recognizes the message as being the same one that the other nine mailboxes still hold, and it takes up another 5MB. If a second person does the same thing, another 5MB is taken. Looking at the reports in the ESM, you won't see any change.
Still, 140GB is a lot of space and when you think about it, the total database file size shouldn't exceed the total of the mailbox sizes.
So what else can it be? Deleted Items and Deleted Mailboxes still take up space until the retention period has expired, and won't show up in the ESM report. What are yours set to for that Mailbox Store?
Look at your Application Log for the following event IDs:
1207 = Deleted Items (those "permanently" deleted, not those still in the Deleted Items folder)
9535 = Deleted Mailboxes
Those events show up for each Mailbox Store, so make sure you're looking at the ones for the store in question.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 4:01:09 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Thanks, Dean,
The retention policy set up was 7 days for deleteld item and 30 days for deleted mailbox, I changed it to 2 days for deleted item and 7 days for mailbox. now it showing this mail store should free about 1GB spcace, I also check the while space is 2.5G on this mail store.
I also changed maximum message size to 6MB, and locate a space so I can disable circular logging, will keep finger crossed on Monday. I still have couple of hundrde GBytes free space on SAN available for this mail store, definitelly want to find the root cause before hit the wall.
Thanks again,
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 5:48:02 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Dean,
Found something interesting,
After I disabled circular logging, I found this happen again :
When an inbound email sent to two users with 5MB attachement, it generate 5 transaction logs( 5Mb each) in one minutes, and from message log, it recorded 7 times this email has been sent to these two recipients, Both of these symptons dont make any sense.
Email firewall logs comfirmed this email only sent once to two users.
I dont know how to upload print screens to explain these two symtons.
Regards,
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 9:48:12 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
Does Message Tracking show a single message being delivered multiple times to the mailboxes, or does the message show up as seven separate messages?
Is there anything else in the Application Log of interest?
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 10:03:26 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
Dean,
Can I send print screen to your yahoo.com account?
Thanks,
Jerry
|
|
|
|
|
RE: Mail store size growth out of control - 29.Nov.2009 10:21:59 PM
|
|
|
prcdiscover
Posts: 19
Joined: 27.Nov.2009
Status: offline
|
IT looks like is a single message been delievered seven times, because the time is exactly , but when check the inbox, it only shows one message
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
Printable Version
