Exchange Server Forums
Forums |
Register |
Login |
My Profile |
Inbox |
RSS 
|
My Subscription |
My Forums |
Address Book |
Member List |
Search |
FAQ |
Ticket List |
Log Out
Possible Store Corruption but not sure how to tell?
|
Users viewing this topic:
none
|
Logged in as: Guest
|
Login  | |
|
Possible Store Corruption but not sure how to tell? - 4.Nov.2008 12:19:13 PM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Hello everyone!
I am running Windows Exchange Server 2003 on Windows 2000 Server. Mail seems to be flowing and working correctly. The only reason I noticed something is not right is because my log files were never being flushed and looking deeper into my backup, even though it said it completed it gives the error below.
Backup Status
Operation: Backup
Active backup destination: File
Media name: "Media created 11/2/2008 at 9:50 PM"
Backup of "Servername\Microsoft Information Store\First Storage Group"
Backup set #5 on media #1
Backup description: "Set created 11/3/2008 at 11:04 PM"
Backup Type: Normal
Backup started on 11/3/2008 at 11:04 PM.
WARNING: Portions of "\Mailbox Store (Servernam)" cannot be read. The backed up data is corrupt or incomplete.
This file will not restore correctly.
Backup completed on 11/4/2008 at 12:32 AM.
Directories: 4
Files: 5
Bytes: 39,017,021,998
Time: 1 hour, 27 minutes, and 52 seconds
In Event Manager I get the following errors
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 11/4/2008
Time: 12:30:56 AM
User: N/A
Computer: Servername
Description:
Information Store (1436) First Storage Group: Error (-613) during backup of a database (file D:\Exchsrvr\MDBdata\priv1.stm). The database will be unable to restore.
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 222
Date: 11/4/2008
Time: 12:30:56 AM
User: N/A
Computer: Servername
Description:
Information Store (1436) First Storage Group: Ending the backup of the file D:\Exchsrvr\MDBdata\priv1.stm. Not all data in the file has been read (read 28870377472 bytes out of 35534151680 bytes).
Below is what I have done so far. I have never really used ESEUTIL hardly at all so I have been trying to do some non evasive things before I have to get into ESEUTIL.
1. Dismounted public and private stores and ran Eseutil /MH <database_name>
It reported back Clean Shutdown for both and Log Required of 0-0 for both. So I am under the assumption that all of the logs have played into the database. So, I moved all the log files to a seperate drive in case I need them in the future. I kept in place all of the logs from the previous day to make sure I don't mess up the sequence.
What I was going to try next tonight is run a /CHKDSK on the drive that holds the database to see if it is an actual physical drive issue.
Other then that I am looking for some suggestions on what steps I should take next because I do not feel comfortable not having a reliable backup.
Thanks ahead of time!
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 4.Nov.2008 2:29:20 PM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Thanks for the quick response EG. That is kind of what I was afraid of was I was going to have to do run the eseutil /p command, run the eseutil /d command, and then run the isinteg -fix command on the affected store. The second link you gave me was about a rollup for SP 3 for Exchange 2000 and I am running Exchange 2003.
My problem is that I do not have a good backup at this point because this has been running like this for the past month or two. I have multiple tasks here so I don't always get enough time to keep an eye on the Exchange logs like I would like. But, rest assured I am going to set aside alot more time to take care of Exchange from here on out.
Well, this will be my course of action I guess and please let me know if you see anything I should not do or anything you think I should add.
1. Tonight I was going to run a /chkdsk on the drive that holds the database, maybe in hopes that it is a physical disk issue and not a database store issue?
2. If #1 doesn't seem to fix the issue I am going to have all of our users copy their email locally. They are using Entourage so they can just copy all their mail to an mbox folder just in case the repair goes horribly wrong at least our users will still have their email that we can drag up to a brand new store if need be?
3. run the eseutil /p command, run the eseutil /d command, and then run the isinteg -fix command on the affected store and hope for the best?
Is it possible I can run ESEUTIL /G to see if my store is even corrupt at all? With all the warnings of complete database failure I would really want to make sure that is my only option to /p
Thanks again!
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 4.Nov.2008 3:05:14 PM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Wow thanks a ton for your advice. I am by no means an expert Exchange admin or apparently average for that matter, but I am trying to learn what I can as quickly as I can since I was put into this role. Tonight, I will unmount the private and public stores, then stop the IS service and copy the eDB and sTM to a seperate drive. I see you are saying that because if I run the repair over the weekend and something goes horribly wrong I can replace those two files and the database will be back to what it was before I ran the repair.
Thanks again!
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 4.Nov.2008 3:37:17 PM
|
|
|
Exchange_Geek
Posts: 1287
Joined: 31.Dec.2006
Status: offline
|
Do post your results.
Thanks.
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 9.Nov.2008 10:44:57 AM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Ok I have gone through that whole process above. I ran isinteg all tests and the first time through it had no errors. So, I mount the stores and then I try to run an online backup. It still tells me the same error about not being able to use this backup to restore. I think there might be something wrong with the .stm?
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 478
Date: 11/9/2008
Time: 9:15:16 AM
User: N/A
Computer: servername
Description:
Information Store (2216) The streaming page read from the file "D:\Exchsrvr\MDBdata\priv1.stm" at offset 3140882432 (0x00000000bb361000) for 4096 (0x00001000) bytes failed verification due to a page checksum mismatch. The expected checksum was 3585983127 (0x00000000d5bdc297) and the actual checksum was 3590177431 (0x00000000d5fdc297). The read operation will fail with error -613 (0xfffffd9b). If this condition persists then please restore the database from a previous backup.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 217
Date: 11/9/2008
Time: 9:15:27 AM
User: N/A
Computer: servername
Description:
Information Store (2216) First Storage Group: Error (-613) during backup of a database (file D:\Exchsrvr\MDBdata\priv1.stm). The database will be unable to restore.
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8012
Date: 11/9/2008
Time: 9:15:27 AM
User: N/A
Computer: Servername
Description:
The 'Microsoft Information Store' returned 'Error returned from an ESE function call (d).
' from a call to 'HrESEBackupClose()' additional data '-'
It seems it is also trying to backup logs, but I thought after I ran through all of those steps the log sequence was changed so any logs in there were useless after this process?
Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 223
Date: 11/9/2008
Time: 9:16:53 AM
User: N/A
Computer: servername
Description:
Information Store (2216) First Storage Group: Starting the backup of log files (range E:\Exchsrvr\Mail Logs\E0012575.log - E:\Exchsrvr\Mail Logs\E0012579.log).
For more information, click http://www.microsoft.com/contentredirect.asp.
These are the logs currently in the logs area since I remounted the stores and it seems I had the same problem as before where it will not clear the log files after the full backup runs.
Also, my .stm file is 32GB and my .edb is only 8GB, shouldn't the .edb be the larger of the two?
Not sure what to do at this point?
< Message edited by Seppy -- 9.Nov.2008 10:53:17 AM >
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 21.Nov.2008 8:00:12 PM
|
|
|
nageshk
Posts: 2
Joined: 21.Nov.2008
Status: offline
|
Hi,
What is the backup software you use?
Why don't you try moving the database files (edb & stm) to a new location and try backing the data?
Let me know what is the current status.
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 24.Nov.2008 9:54:18 AM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Thank you for your concern. Yes, this issue has not been resolved. I called Microsoft Tech Support 2 weeks ago and I think it might get fixed this upcoming Friday after Thanksgiving. Basically, they had me to a repair and defrag twice. They then had me put in another brand new physical disk even though it was on a mirror. Had me backup again and that did not fix the issue either. They then had me dismount, clear the log files out of their location, remove the .chk file and then remount and try to backup again, that did not work either. They also had me move the system folder location for Exchange to the new drive as well and off of my mirrored set of drives. Their next step for me is to use Exmerge, export all the mailboxes the the bin directory, remove the .edb and .stm files and then mount the stores so it create's brand new .edb and .stm's and then use Exmerge to bring all the mailboxes back in. Supposedly it will take 5-7 hours each way for export and import. We are scheduled to do that this Friday after thanksgiving. I will let eveyone know if that resolved our issue.
On a side note, they said if we had Exchange Enterprise we could just create a new store and move the mailboxes there, but we do not.......so we have to do it this way.
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 24.Nov.2008 6:48:55 PM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
quote:
On a side note, they said if we had Exchange Enterprise we could just create a new store and move the mailboxes there, but we do not.......so we have to do it this way. In light of this suggestion, you could also build another Exchange Standard server and move the mailboxes over. After that completes, you could either remove the original server, or delete the Mailbox Store on the original server, recreate it, move the mailboxes back, then remove the newer server from the Exchange Organization.
I bring up this option because using ExMerge is going to break Single Instance Storage and your resulting database is going to be larger than it was before.
< Message edited by uemurad -- 24.Nov.2008 6:50:08 PM >
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 25.Nov.2008 9:32:21 AM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Ok I will ask Microsoft about that as well. When they first gave me this new plan of attack I asked them if they were going to provide another license key for me to install a second Exchange server, but they said they would not and that I should just use the Exmerge process. That is definately good to know about the size issue though. My guess is I will have the change the registry key that states the max size of our store before I start the Exmerge process?
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 25.Nov.2008 10:01:29 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
quote:
When they first gave me this new plan of attack I asked them if they were going to provide another license key for me to install a second Exchange server, but they said they would not and that I should just use the Exmerge process. I guess that is to be expected. Ask if they have such a thing as a time-limited license. I've never asked, but it would certainly help in cases like this. You'd have to move the mailboxes twice, but at least you could preserve SIS.
The amount the database will grow is dependent upon how much space was originally saved by SIS. It can vary widely. Citing extremes - if every message you received was addressed to a single recipient, then ExMerge will have no effect. If you have 100 users, and the CEO sent out a message with pictures from the corporate picnic which was 10MB, the original database would have taken about 10MB but the new database will take up 10MB x 100 users or approximately 1GB.
It is probably a good idea to increase the database size limit to whatever you can afford (up to the maximum) in disk space.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 25.Nov.2008 10:31:51 AM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Ok thanks I will do that. At this point our .edb and .stm is at 42GB. We have 200GB total size on the mirrored drives we have in there, so I am hoping it doesn't grow 10 times the size or else I don't think I will have enough room.
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 25.Nov.2008 11:12:37 AM
|
|
|
uemurad
Posts: 8232
Joined: 7.Jan.2004
From: California, USA
Status: offline
|
You can get an approximation of how large the new database will be by adding up the sizes of the PST files created by ExMerge.
_____________________________
Regards,
Dean T. Uemura
Microsoft MVP - Exchange (2007-2011)
exchangeguy.blogspot.com
uemurad@yahoo.com
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 1.Dec.2008 2:45:47 PM
|
|
|
Seppy
Posts: 9
Joined: 17.Nov.2006
Status: offline
|
Finally the issue has been resolved! This past Friday after thanksgiving I took one final offline backup of the private store.
Then I
- mounted the store
- Ran Exmerge to export all the mailboxes, calenders, contacts, etc.
- Un-mounted the private store
- Moved the .edb and .stm private store to another location
- Mounted the private store
- Hit yes to create a new database since the existing one was no longer there
- Run Exmerge part 2 to import the mailboxes, etc back in
- Ran an online backup and the logs cleared correctly and the backup completed successfully!
When I talked to Microsoft my .stm file was way larger then the .edb and they said that is because we have all Mac's here and use Microsoft Entourage. But, now it seems all of the data is back into the .edb file. I wonder if over time the data will shift back into the .stm file.
But either way my problems has now been resolved.
Thanks again!
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 9.Dec.2008 7:34:24 AM
|
|
|
ratishnair
Posts: 154
Joined: 29.Jul.2008
Status: offline
|
Hey Seppy,
I should have gone thorugh this post long before.
Let me tell you:
1. Never ever run a checkdisk on an exchange server coz there is an 80 % chance that the stm files get corrupted.
2. Please refer to the kb article: 278958 which talks about rebuilding your exchange database if you have all uyour log files in sequence.
3. If you did, you could have easily done a eseutil /ml and checked if all the log files where healthy and rebuild your database.
4. If you dont have the lof files in sequence or if / ml reports a corrupted log file, saving those log files are absolutely not necessary if eseutil /mh reports the DB to be healthy and no logfile required. Go ahead and delete em all and save disc space.
5. Regarding /p and /d, thats kinda last resort given by MS and you should only try it if you dont have any other options left.
Revert back for queries.
Ratish Sekhar
|
|
|
|
|
RE: Possible Store Corruption but not sure how to tell? - 10.May2012 1:25:25 PM
|
|
|
techinfo
Posts: 72
Joined: 24.Jan.2011
Status: offline
|
Runing an eseutil /p or edbutil /d /r, isinteg command line tools against Exchange database files is called hard repair. These tools analyze critical structures of the database file and checks for damaged pages.
If these utilities encounters a page that is damaged or unable to read, it simply removes the page. Thus, in this situation, critical data may be lost after the repair finishes. This data could be your important emails, contacts, attachments or a part of a system table. It is impossible to find out how much data you've lost?
It is always good to restore from a most recent backup file whenever possible. In case, you have not updated backup file or backup is itself corrupt, then you can use a third party Exchange recovery program like: http://goo.gl/PyCxyQ
This tool can help you to avoid critical data loss problems and you can get upto 100 % accurate data in your database. It is compatible with all versions of MS Exchange Server i.e. 2010, 2007, 2003, 2000, and 5.5
< Message edited by techinfo -- 10.Oct.2015 8:33:00 AM >
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
Printable Version
) - you always have a backup Exchange DB to fall upon to and work with it as you always did. 
