Hi Guys, To start with, i screwed up! and by the time i realised, it was too late. So i hope to share and have your comments on how to get out of this situation.
On a weekend, i got a call that the email server was down. so i connected remotely to the server and realised there was no space on the server. So i compressed the MDBDATA. Okay, i know it was very foolish but then it was a long weekend and i had to come up with a solution for guys who were for some reasons working on a weekend and required emails
So by monday, hell broke loose as below 1. Mailbox wont mount 2. Microsoft support sites mentioned to decompress the files to resolve the situation 3. i tried to decompress, but it failed caus there was no disk space left. 4. The mdbdata was on a mirrored disk (i know database should not be on mirror but thats a different story how i got there)
so i had a compressed EDB file with no space left to decompress it and we were loosing buisness. so by late monday evening the below was my plan of action
1. break the mirror 2. expand the volume using ACU 3. extend the volume 4. decompress the EDB file 5. try to mount the database (i have no time to run ESUTIL)
Will this work ? i reached office at 6:00 AM tuesday and i am still at "step 2 !!! ". the expansion is only 68% complete. what if in the end i find space to decompress the EDB and still the database does not mount. If it does not mount, so i have no choice to RUN ESUTIL. will that utility be a sucess. The EDB looks in a very bad shape. That my story...i am still waiting for people to reach office at 9:00 and blow my brains out( whatever is left of it now). Help !
< Message edited by s_manoj21 -- 22.Sep.2008 11:25:52 PM >
Posts: 8196
Joined: 7.Jan.2004
From: California, USA
Status: online
If I understand correctly, are you converting the two drives of a RAID1 set to a two-drive RAID0 set? If so, do you realize you've converted a redundant drive set into one that will lose all the data if either drive fails?
Is there nothing else on the drives that can be moved somewhere else? What is the capacity of the mirrored drive, and how large is your database?
You might have been better off copying the data somewhere else (e.g. tape) then rebuild the RAID0 set with empty drives, then copy the data back.
_____________________________
Regards,
Dean T. Uemura Microsoft MVP - Exchange exchangeguy.blogspot.com uemurad@yahoo.com
Thanks for the quick response Yes, you are right. I am converting raid1 to raid 0 knowing that its not redundant. But i have no options as i have no space left on the mail server. The database is 134 GB huge (EDB+STM). so it took almost 10 hours to take a backup!!!! (i copied and pasted it to a file server with sufficient space).Now if i rebuid the RAID and try to copy the data back.....again another 10 hours for the data restoration !! and i have no time because everybosy is eagerly waiting for the mail to get alive. I am really hoping that the EDB is in a recoverable shape (after decompression)....or else i am i real trouble....
The capcity fg mirror drive is 146 GB EACH
< Message edited by s_manoj21 -- 23.Sep.2008 12:25:40 AM >
Posts: 3004
Joined: 9.May2007
From: India
Status: offline
hi mate,
This really disaster... scenario
First thing if that is the case: Assuming if the you extract the data (100%) the very first thing you need to check the status of the database by running this command eseutil /mh and check the database status : it should be clean shutdown. for priv.edb and pub.edb
(Note : Keep a copy of the above the data) If it is clean shutdown : Move all the log files from mdbdata folder : keep only those edb and stm and mount the database.
If it is dirty shutdown below the status it will display the logs required ? check whether you have those log files, if so then try eseutil /r command (soft recovery)
If soft recovery fails - restore from the last good known backup.
If you don't have last good known backup and this should be your final solution - Hard repair i.e eseutil /p and then eseutil /d and isinteg -fix.
I would suggest you if the there is dirty shutdown and you want to restore the database then please rename the mdbdata folder (if you are using public folder ) check the status of the PF and put in the mdbdata folder. > create a new folder called as mdbdata > Mount the database with blank store so that they can have their mailflow. > then create a recovery storage group (restore the backup data and the merge the new data with the old one.
Posts: 3004
Joined: 9.May2007
From: India
Status: offline
same for 2000 as well,
Open the command prompt: Redirect to bin directory i.e cd "C:\program files\exchsrvr\bin" and hit enter and then type the following command : eseutil /mh "c:\program files\exchsrvr\mdbdata\priv1.edb" | more
then see the seventh line called as status : either it will clean or dirty shutdown and below the status you will logs required.
Don;t worry about priv1.stm (the above command from priv1.edb will take care of it)
1. when i had copied the EDB file from the mail server to another hard disk, the file was automatically decompressed. i just noticed that. even though the file size for both the original and the backup are the same, atleast the attribute "enable compression" is not checked in the backup.
2. i checked this uncompressed version of the EDB file (that i have in backup) using eseuril /mh and the status is clean shutdown.
so what do you think now....do i have any hope since its a clean shutdown ? i can just move the log file, restart the server and try to mount the database.
WHAT ABOUT THE .IFS FILES in the mdbdata ??? i have a "clean shutdown" copy of the EDB...so can i delete the IFS files too ??
< Message edited by s_manoj21 -- 23.Sep.2008 8:20:28 AM >
Posts: 3004
Joined: 9.May2007
From: India
Status: offline
hi manoj,
Clean shutdown - that's great news
Take a copy of that database.
Now let me know the status of pub1.edb (is it clean shutdown?)
Assuming that you have pub1.edb, pub1.stm, priv1.edb and priv1.stm and other log files (that's it) as far as first storage group is concerned.
Path of the database : C:\program files\exchsrvr\mdbdata
Now you need to rename the mdbdata folder to mdbdata old Create a new folder called as mbddata Copy only the pub1.edb, pub1.stm, priv1.edb and priv1.stm (clean shutdown) Then start the information store service then try to mount the database.
If it is not mounting make a note of the error message you are getting and in the application what is the event id is triggerred.
you mention the following files are required >>>> (pub1.edb, pub1.stm, priv1.edb and priv1.stm and other log files (that's it) as far as first storage group is concerned.....)
what exactly are the "other log files". as per your previous thread, you had advised to move all log files and to keep only the EDB and STM before mounting
< Message edited by s_manoj21 -- 23.Sep.2008 9:02:44 AM >
thanks ismail well i am copying the uncompressed version of the backup with "clean state" and "0-0" logs required back to the email server. this copy will take atleast 5 hrs.....
meanwhile i was going through some sites and there was this guy who mentioned that he had an EDB file with "clean state" and "0-0 logs required". still he was not able to mount the store.....take a look at the link below (even i get the same error like his- C1041724 http://forums.msexchange.org/m_200015500/mpage_1/tm.htm#200015522
in my case, after it takes 5 hrs to move the file back and if the EDB still refuses to mount......i will certainly have to contemplate suicide !!!
Posts: 3004
Joined: 9.May2007
From: India
Status: offline
keep the finger cross,
lets try our level best, if there is a problem then don't worry we have last step solution also... which we will try when we will use all the possible solution.
The problem is solved now !!! after 2 days of compelte downtime.....wow !! what a way to start the week.
The EDB apparantely was rock solid....it survived all the beating....compression...decompression.....copy...paste etc. The EDB was in "clean shutdown" state with "0 Logs" required. so after i moved the old logs restarted the server...the EDB mounted just fine !! I was luckky though caus i was readin in other forums about people with Clean EDB's and still not able to mount.. but i never had to go through eseutil /D or eseuril /P.....luck !!
Now i have to get a new hard drive and migrate to fault tolerant drives (raid 5)...right now there is no fault taolerance cause i have broken the mirror for space !!
Posts: 62
Joined: 14.Apr.2008
From: Sydney
Status: offline
OK,
So the lesson learned in this scenario is to copy the latest transaction logfile and the mailbox database into the default directory and then mount it ?
Please correct me if I'm wrong.
To Manoj: what backup application do you use ? ntbackup.exe ?
Hello. For backup, we are using symantec backup exec 11d (formerly veritas) with exchange option (which means it is an exchange aware backup software)
In my case, the edb had a " clean shutdown" status attached to it. i checked that using the "esutil /mh" command. since it was a clean shutdown, it means that all the transactions had been comitted to the EDB database. so there was no need to copy the log files to the original location. all i needed was the EDB and the STM file with the LOG directory location EMPTY.
In case the EDB had a "dirty shutdown" status, we would have been forced to use esutil /d (for defragmenting) or esutil /p (repair) options. From the research i did, i found that esutil /d is enough to rescue the EDB (it worked for 1 guy who was trying to get exchange up for 52 hours straight!! and finally the esutil /D helped him !! )
The last option would be to restore from a good backup and replay the log files.
And the most important thing learnt was, not to take any decision in a hurry. Even before touching the box, do some research and ask the nice online techie guys for advise.In my case it was Ismail- thanks a lot to him. And ofcourse- never neglect the backups !!!
Posts: 62
Joined: 14.Apr.2008
From: Sydney
Status: offline
Manoj,
You certainly benefits the whole MSExchange.org community here and also me.
thanks for sharing your story and keeping all to the details. By reading your topic today this morning I gain more knowledge not only in Exc. Serv. but also RAID and such.