• RSS
  • Twitter
  • FaceBook

Exchange Server Forums

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

Private information store too big even after defrag utility

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [Microsoft Exchange 5.5] >> General >> Private information store too big even after defrag utility Page: [1]
Login
Message << Older Topic   Newer Topic >>
Private information store too big even after defrag uti... - 16.Jan.2004 5:18:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
I'm running Exchange Server 5.5 SP4 on an NT4 SP6a box. Even after running the eseutil to defrag the private information store, it is still 13.4 GB. But when I look at the mailbox resources in Exchange Admin util, they all only add up to ~1.5GB. What am I missing?

When our private IS reached the 16GB limit, it crashed Exchange. I deleted all I could before defragging but I'm still not sure why it's so big when it seems it should be much smaller.

We are also using McAfee VirusScan on the mail server.

Any help is appreciated.

- Rim
Post #: 1
RE: Private information store too big even after defrag... - 17.Jan.2004 5:16:00 PM   
zbnet

 

Posts: 1019
Joined: 25.Sep.2003
From: York, UK
Status: offline
At one time there was a bug in the Admin program that, when a mailbox got bigger than a certain size (I think 2GB), it showed the size as zero. I'm not sure whether that big was fixed, or if it was in what version, but you might be suffering from that.

A good candidate to check would be the IMS administrators mailbox that is set in the IMS - if you have Notifications turned on, you might have got a lot of system messages to that account, which could over time result in the mailbox being too big to see in Admin.

(in reply to rim)
Post #: 2
RE: Private information store too big even after defrag... - 19.Jan.2004 4:57:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
Thanks for the response.

We did experience the bug you posted about. The admin mailbox had over 2 million messages but showed up as 0KB. Attempts to clean the mailbox, including via Clean Mailbox and Outlook client, failed so we deleted the account and recreated it. We ran the defrag utility and did get back 1GB worth, but we're still at 13.4GB.

(in reply to rim)
Post #: 3
RE: Private information store too big even after defrag... - 19.Jan.2004 5:42:00 PM   
ismajor

 

Posts: 27
Joined: 2.Jul.2003
From: Irving, TX
Status: offline
This was an offline defrag you ran with eseutil? I'm just asking since online defrags don't actually recover database whitepace.

(in reply to rim)
Post #: 4
RE: Private information store too big even after defrag... - 19.Jan.2004 8:31:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
Yes, offline - stopped the information store service every time before running defrag.

To give you more details, our Exchange Server crapped out and we discovered it was due to the private IS reaching the 16GB limit. We defragged offline and it got down to about 15.5GB - enough to get Exchange working again. We saw the admin mailbox had over 2 million messages but was showing 0KB mailbox resource. We deleted the account and mailbox and recreated it. I also deleted some non-current employee accounts but it wasn't much - maybe 200MB according to Mailbox Resources list. We did offline defrag again and got it down to 13.4GB. Adding up all the mailbox resources, it looks like it should only be around 1.5GB.

This was late last week and our private IS has actually grown to 13.9GB since then. I'm not sure, but from what I've read, if the actual size needed for the IS is smaller than the allocated size (what it takes up on the hard drive), the file won't get larger unless needed. So, let's say our priv IS really only does need 1.5GB but the file is 13.4GB. When new messages come in, the file shouldn't get bigger until the IS actually becomes larger than 13.4GB. Is this correct? If so, then our priv.edb file shouldn't get larger, but it has. It does seem there's that much info on our IS, but get this, we only have 20 users! And like I said, I've added up all the listed Mailbox Resources and they just add up to 1.5GB.

The only other thing I was thinking of was McAfee GroupShield for Exchange and the quarantine function. I deleted a lot of quarantined messages (with attachments) but I haven't run the defrag since then but priv.edb has grown even after that. From what I've read, the quarantined items are stored in a separate file, not in priv.edb, but I'm not absolutely positive.

I'm scratching my head on this one.

- Rim

[ January 19, 2004, 08:35 PM: Message edited by: rim ]

(in reply to rim)
Post #: 5
RE: Private information store too big even after defrag... - 20.Jan.2004 9:30:00 AM   
zbnet

 

Posts: 1019
Joined: 25.Sep.2003
From: York, UK
Status: offline
Okay, a number of things to say about this.

If there is white space (unused space) in the priv, then yes, the store will use this space first before increasing the size of the file when writing new mail. If your file is growing in size (and you can only really be sure how big it is when you have the IS service stopped), then there is no white space inside the file - you really have that many messages.

Test this by looking for the 1221 event that the online defrag writes each night (database maintenance window). There will be separate events for the priv and the pub.

If you really do have this much mail and it's growing, the big question is why? Do you have a mail loop? Maybe a rule on an unattended mailbox that's bouncing messages out to another rule externally that's bouncing them back in again? Set IMS logging on and look at the traffic being sent. Ask the users to review their rules.

Whilst you're checking users, get them to confirm that the flag is ticked in Outlook to Delete the Delete Items folder when logging out - if they take this flag off, their mailboxes donĂt ever throw away the contents of the deleted items folder.

Also, what's your Deleted Item Retention set to? If this is a really big number, this will effectively stop anything being deleted. Most people set the DIR to somewhere between 7 and 28 days (if they have it set at all, but it's generally a good idea).

(in reply to rim)
Post #: 6
RE: Private information store too big even after defrag... - 20.Jan.2004 7:29:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
I had also created a support request with Microsoft (the $99 kind) and I've included the response in case it may help others.

The diagnostic dump command is:
ESEUTIL.EXE /ms priv.edb > priv.txt

quote:
I carefully checked the eseutil /ms dump and found some abnormal values:

1-24 Tbl 1775866 808341 3790
<Long Values> LV 1775870 264770 54
Msg Tbl 1020 2756981 15442
<Long Values> LV 1021 939346 4923

There are two abnormal <Long Values>. LetĂs see the objectĂs size:
1775870*4096=7.2GB and 939346*4096=3.8GB. They are very huge.
These objects may orphaned or corrupted long values.
As you know, Long-values (LVs) are created when a column is too large to store with the rest of the record. Internally, the Exchange Server database engine breaks large columns into separate parts; these are the long-values. Long-values are stored in a separate binary tree (B-Tree) and each LV is given a table-wide unique identifier (the long-value ID [LID]).
How the orphaned or corrupted long values occurs:
Orphaned Long-Values
To save space, the Exchange Server database engine provides the ability for multiple records to share the same LV (similar to, but not exactly related to the information store concept of single-instance storage for messages). To do this, a reference count is attached to each LV. When the reference count reaches zero, the LV is deleted. If the Exchange Server database engine is shut down (by a crash, power cut, or blue screen error messages) after dereferencing an LV, but before expunging it from the database, the LV will be ˘orphaned,÷ that is, its reference count is set to zero, but it is never removed. Orphaned LVs are invisible to anyone using the database because they are logically deleted, but still take up space because they have not been physically removed.
Corrupted Long-Values
As mentioned above, LVs are stored as chunks of data in the long-value tree of a table. Each LV is prefixed by a header (˘LVROOT÷) that contains the length and reference count of the LV. In rare cases (such as the B-Tree split problem), the LVROOT of an LV can be overwritten. This corrupts the LV, but doesnĂt actually lose any data. An Exchange Server information store (Store.exe) may stop responding or error out trying to access this LV.
To deal with such a problem, renew database is the best way. We can export userĂs mailbox via exmerge tool and renew Exchange database. After that, we can import userĂs mailbox back.
Some questions and answers about the Exmerge Utility for your reference:
http://support.microsoft.com/default.aspx?scid=kb;en-us;265441

So basically there are two objects in priv.edb and they both add up to 11GB. This looks like where the problem lies. I cannot account for what these objects are so they are probably corrupt. I'm looking at EXMERGE now to see if I can just export those two objects somehow. This would give us the free space and then there would be no rush to run the defrag utility, since we are not running out of hard disk space.

[added]
Oh, and our Deleted Item Retention period was already set to 1 day.

[ January 20, 2004, 07:31 PM: Message edited by: rim ]

(in reply to rim)
Post #: 7
RE: Private information store too big even after defrag... - 21.Jan.2004 9:33:00 AM   
zbnet

 

Posts: 1019
Joined: 25.Sep.2003
From: York, UK
Status: offline
A very interesting response from MS.

I think you misunderstand their reference to ExMerge. They recommend using ExMerge to remove all the user mailbox deta from the store (to archive it), then to delete and recreate the store, then to put the mailbox data back again (although this will result inloss of historical single instance storage, it will still be better than having 11GB of rubbish in the store!).

You will not be able to remove these orphaned objects with ExMerge, because they are not messages as such - they are internal database constructs that are not exposed via the MAPI interface (which is what ExMerge uses).

(in reply to rim)
Post #: 8
RE: Private information store too big even after defrag... - 21.Jan.2004 5:33:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
Well, I understood what MS support said. I just never ran EXMERGE before so I didn't know what options were available on that utility. But yes, it cannot do what I hoped it could.

Anyway, running it on all our mailboxes yielded .pst files that totalled 1.91GB - a much more realistic number. Looking at exmerge's logs, there was one mailbox that had 4,000 error messages about being unable to copy a message. Fortunately, we did not need it any longer so I deleted it. If I'm lucky, the two objects were in that mailbox and now they're gone? I'm monitoring our priv.edb file to see if it still grows in size.

If that mailbox was not the problem, then I will have to re-create the private IS.

(in reply to rim)
Post #: 9
RE: Private information store too big even after defrag... - 22.Jan.2004 9:13:00 AM   
zbnet

 

Posts: 1019
Joined: 25.Sep.2003
From: York, UK
Status: offline
Check your overnight event log for 1221 (white space report), this will show whether the 2 objects are gone or not.

(in reply to rim)
Post #: 10
RE: Private information store too big even after defrag... - 22.Jan.2004 8:57:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
I tried looking for that event ID before and have not found any. We have all of our logging set to the default of none, which is basically just critical stuff. What do I need to turn up logging on and to what level?

(in reply to rim)
Post #: 11
RE: Private information store too big even after defrag... - 23.Jan.2004 2:14:00 PM   
zbnet

 

Posts: 1019
Joined: 25.Sep.2003
From: York, UK
Status: offline
I have my logging set to None by default as well - but this doesn't stop the 1221 events being reported.

Are you sure you have Database Maintanence windows defined? It's on the Server properties in Admin.

(in reply to rim)
Post #: 12
RE: Private information store too big even after defrag... - 26.Jan.2004 8:31:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
It looks like it's set. It's set to everyday between midnight and 6 am. [Confused]

(in reply to rim)
Post #: 13
RE: Private information store too big even after defrag... - 28.Jan.2004 12:46:00 PM   
Guest
why don't you upgrade Exchange 5.5 to exchange 5.5 enterprise edition which has an unlimited IS size? it takes about 5 minutes.

you can buy a 2000 enterprise license to do this proceedure then all you need is the disk which microsoft should be able to dig up (i have one) then just run srvmax.exe and there u go.

(in reply to rim)
  Post #: 14
RE: Private information store too big even after defrag... - 28.Jan.2004 10:28:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
We don't have a license for the enterprise version. 16 GB should be plenty for 20 users, for goodness sakes! [Smile] Anyway, we are planning on upgrading to Exch2000 but that is still in the planning stage so this is just interim.

(in reply to rim)
Post #: 15
RE: Private information store too big even after defrag... - 29.Jan.2004 11:02:00 PM   
Velocat

 

Posts: 576
Joined: 6.Jan.2004
From: Tucson, AZ
Status: offline
Hi rim,

So...did you rebuild the server? [Smile]

(in reply to rim)
Post #: 16
RE: Private information store too big even after defrag... - 2.Feb.2004 10:00:00 PM   
rim

 

Posts: 21
Joined: 15.Jan.2004
From: Alexandria, VA
Status: offline
I've just been through hell. [Mad]

Actually, I think the export and import went fine. The priv.edb got down to under 3 GB. I then went ahead and tried to run our backup software but it kept giving me a Dr. Watson (never did it before). Tried other things and thought the mail server may need a reboot - it never booted up to the OS (blank screen with blinking cursor on upper left - after POST).

Long story short, I ended up building another server as the Exchange Server and restored from tape. Unfortunately, we lost a day's worth of data. Yes, I should've backed up just before any of the export business. [Roll Eyes] I worked 37 hours straight on it (including the original export/import).

What's worse is, our priv.edb is back to 14.1 GB in size and our Exchange Server is standard edition again. Even though I installed the enterprise edition when I rebuilt the new server, since what I was restoring from was standard edition, it basically made the server standard edition (confirmed by MS support).

MS Support told me all I need to do is re-run Exchange enterprise setup and choose the re-install option and it would upgrade to enterprise. At this point, though, we've just let our users use it as is for now and maybe looking at other options (outsourcing mail being one of them). Fun, fun, fun.

(in reply to rim)
Post #: 17
RE: Private information store too big even after defrag... - 3.Feb.2004 6:53:00 PM   
Velocat

 

Posts: 576
Joined: 6.Jan.2004
From: Tucson, AZ
Status: offline
Oh my...I feel for you.

Lost a days worth of email, plus all the work you put into it. [Eek!] Probably got nothing but complaints from your users? [Frown]

It downgraded from Enterprise to Standard...oh my goodness, that must have been a surprise.

Keep us posted and good luck with shrinking that DB.

(in reply to rim)
Post #: 18

Page:   [1] << Older Topic    Newer Topic >>
All Forums >> [Microsoft Exchange 5.5] >> General >> Private information store too big even after defrag utility Page: [1]
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Follow TechGenix on Twitter