Very strange behavior , routing & SMTP service crash and restart (Full Version)

All Forums >> [Microsoft Exchange 2003] >> General



Message

mcadek -> Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 1:16:47 PM)

Let me outline what happened:

here is the layout of our servers

- 1 front end server with OWA, our PDA's also use this to connect their activesync with it
- 1 backend server with 1 information store for entire corporation in US
- 1 backend server with 1 information store for our overseas office
- back end in US is 2003 enterprise SP2, overseas BE is 2003 standard SP2 , front-end is 2003 standard SP2,

- this weekend all of a sudden our company PDA's stopped getting emails
- OWA also went down with 503 error , service unavailable I think
- Internally exchange was working , but would not send/receive mail (new message would sit in outbox). I am assuming SMTP service was not functioning right.

- since I did not have much time for daignosis, I chose to reboot both the FE and BE. Everything started working again, but...

- immediately after reboot I was getting IIS crash/illegal operation errors, it was related to the w3... worker process (w3we.exe or something like that, I do not have the screencap right here). It crashed about 10 times and then stopped coming up

-everything works fine since except I get very strange errors.

on the FE:

I get event id 3007 for every user with a PDA/activesync, multiple times at random intervals

: exchange mailbox server response timeout: Server: "servername" , user: "usernaem" , exchange activesync server failed to communicate with the exchange mailbox server in a timely  manner, veryfy that the exchange mailbox server i workign correctly and is not overloaded.

it definitely is NOT overloaded and it seems to work just fine for 150+ users that use it, even the PDA's even though those errors pop up in application event log!

on Back end server I get these ID's that show up in blocks, usually these 5 or 6 Id's pop up all within 1 second in application event log

1005 - msexchangetransport - RE service has been started, Version: 6.5.7638.138.1
1008 - msexchangetransport - RE service instance 1 has been started
332 - msexchagnetransport - SMTP service has been started, initializing queues.
334 - msexchangetransport - SMTP service instance 1 has been started.
10302 - msexchangeactivesync - OMA categorizer successfully started
101 - DAVEX - DAVEX to be shutdown

sometimes followed by DAVEX has successfully started event ID 100

in the system log the following entries provide more general overview of what is happening:

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1013
Description:
A process serving application pool 'ExchangeApplicationPool' exceeded time limits during shut down. The process id was '3132'.

Event Type: Information
Event Source: W3SVC
Event Category: None
Event ID: 1082
Description:
A worker process with pid '3132' that serves application pool 'ExchangeApplicationPool' has been determined to be unhealthy (see previous event log message), but because a debugger is attached to it, the World Wide Web Publishing Service will ignore the error.

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Description:
The IIS Admin Service service terminated unexpectedly.  It has done this 12 time(s).  The following corrective action will be taken in 1 milliseconds: Run the configured recovery program.

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7034
Date:  12/4/2006
Description:
The Microsoft Exchange Routing Engine service terminated unexpectedly.  It has done this 12 time(s).

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7034
Date:  12/4/2006
Description:
The Simple Mail Transfer Protocol (SMTP) service terminated unexpectedly.  It has done this 12 time(s).

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1009
A process serving application pool 'ExchangeApplicationPool' terminated unexpectedly. The process id was '5852'. The process exit code was '0xffffffff'.

then the affected services restart on their own and all is well for another 18 minutes (or more, seems random, last period was 18 minutes...)

I expect my exchange organization will die shortly again, I searched the logs and this behavior started 3 days ago, so possibly in 3 more days it will lock up again...

any ideas?

I would be very grateful for any help you can give me. Thank you!



jchong -> RE: Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 1:39:26 PM)

Virtually anything can cause IIS to hang or crash. You will have to perform some debugging. Download some of the IIS debugger tools.

IIS Crash/Hang Agent & IIS Dump
http://www.microsoft.com/downloads/details.aspx?FamilyID=01c4f89d-cc68-42ba-98d2-0c580437efcf&DisplayLang=en



dvprao -> RE: Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 2:08:02 PM)

Hi :
I want ot report the same peculiar behavior on my server too since this last Friday... All I did was applied wahtever patches thro auto update downlaods on the Win2k Server STD. and  waiting ot reboot.. I rebooted on Friday morning as i saw the inetinfo.exe crash message..
I looked up and did what Ms asked me to try .i.e., renamed the mailroot queue folder and restarted the box, the queue got re-generated...
but still every 80 to 90 minutes thie follwong 5 critical errors are logged in teh EViwer.. Earlier the service was not set to restart but now I have set it to restart but I am not comfortable leaviing it like that.

I do have a 4 versions of metadata back up of IIS Server listed. Is there any risk in restoring the same as suggested by MS ? I am littel wary of doing it sinc eI have not done it.

We use the only sever for pop3, imap4 and OWA as well sitting behind a firewall..

Any help is apprecited. I will post the solution if I get it fomr any other forum also..
I am sure this must be happenign to lot of people...

Thanks
DVP.  
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The World Wide Web Publishing Service service terminated unexpectedly.  It has done this 4 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Simple Mail Transport Protocol (SMTP) service terminated unexpectedly.  It has done this 4 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Microsoft Exchange Routing Engine service terminated unexpectedly.  It has done this 4 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Microsoft Exchange POP3 service terminated unexpectedly.  It has done this 4 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Microsoft Exchange IMAP4 service terminated unexpectedly.  It has done this 4 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The IIS Admin Service service terminated unexpectedly.  It has done this 4 time(s).  The following corrective action will be taken in 0 milliseconds: No action.



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 4:14:00 PM)

my problems started at the same time too! last friday. although I only updated my FE server, BE server I did not as my FE is my "test bed" for windows updates.

I was going to try the queue rename tonight, guess it's not worth it, but I'll give it a shot anyway...

tomorrow night I will be restoring the metadata file

the day after that attempt I'm calling microsoft and raising up a storm :-(

I almost want to update my BE to all windows updates, but I am not sure if I'll just make it worse... at least in its current state it works (question is for how long).




dvprao -> RE: Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 4:44:31 PM)

Yup, Mail queue renaming did not help.
I saw a sub folder called filter wiht lot of *.tmp files in it but that folder did not get generated .... NOw the bad mail folder is getting populated more...

I have just now installed hte IIS crash dump utility from MS and installed i t. But I do not see global filter listed in teh filters portion o fht eIISADMIN !!!. Where is it listed ?

Like I said the services set all of them ( 5 of them) all fail almost same time within 85 minutes... ( is there any sync attempt by GC or something ?)

MS claims that event id 7034 should occur for SMTP service failure but I see only 7031 listed even for that service ....

Should I be careful about anything before restoring meta database ?

Thanks



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 10:10:54 PM)

I am trying to rename the vsi 1 folder like microsoft wants you to, but something in it is still used even though all exchagne & IIS services are stopped :shrug:

(in a middle of a reboot right now...)

my badmail folder has literary thousands and thousands of files in it... whether this helps or not does not matter much, it's time to clean that mess up anyway :)

I say go ahead and restore your metabase.xml file and let me know what happens, and I do not say that lightly just so you can be the lab-rat, but I'll be doing the same tomorrow night :)

WORST case, you end up uninstalling IIS , reinstalling IIS and doing a reinstall of exchange using the "reinstall" option, 1 hour worth of work or so? (I'm assuming that will not wipe the IS, otherwise restoring mine would be a pain and a half and would take several hours).



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (4.Dec.2006 10:12:18 PM)

PS: if you go into IIS you can very easily take a current snapshot of your metabase, so I'd say that is about the only precaution you need to take. Have a good  backup of that and you can always return to this mess we're in right now :)




mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 9:26:18 AM)

Just to post an update , renaming the vsi 1 folder as microsoft suggests in their KB article did nothing (as expected)

I think I will take the following steps tonight

1) restore metabase file that I have
2) reboot and see what happens
3) if problem persists, I will update my back end server to latest MS updates (if you read above I only updated my FE which may or may not have caused this issue). Maybe something my FE/Pda activesync is doing is affecting my back end, I don't know... it's a guess.
4) I have had symantec mail security on my BE for over a month, but I will stop that temporarily and see what happens after reboot.
5) I will then kill my OWA/Activesync FE server and see if IIS still crashes on boot up (I may have to recover the metabase file again).

if anyone has any input I'd love to hear it.

there is a SMTP vulnerability that causes this issue and a hotfix is available but it's from 2004 and another version of it is from 2005, it SEEMS that this has been included in exchange SP2 , I am trying to track the version numbers down to figure out what I have as I always hate to apply older hotfixes



dvprao -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 11:54:39 AM)

I did apply the KB885264 patch..( KB890066)  no luck...
MS is not very clear on this because in the KB article they refer to event ID 7034 for SMTP failure whereas my BOX reports 7031 for that as well --
I di d not try the disable of anti virus and anti spam routines . I will try that now...

The interval seem to be getting reduced by around 10 minutes now for the cycle...
Let me see if MS user forums has any thing to say.. I am sure we two are not the only Admins having this issue today... I wonder....



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 1:21:32 PM)

alright I'm not even going to bother with the patch then, thanks for the update.

What I also noticed that since this started happening my BackupExec that usually backs up VSS / System state on this BE exchange box has been failing, it claims that my IIS metabase writer is in use which makes no sense whatsoever.

when I run command VSSadmin list writers on the BE box they all show up, have no errors and are listed as stable.

They are related somehow as the backup began failing on Friday , same day the exchange headaches started. I can only assume at this point that BackupExec does some sort of intergrity check on metabase/all of VSS ?

I'll be restoring the metabase file from over a month ago tonight if you want to wait for my results, at this point I would rather crash the server completely and had to redo it tomorrow than to keep waiting for it to crash again at worst possible time.




virtus -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 2:19:20 PM)

Hi
Here on my Exchange server exactly same problem. Under 150 users workload same services terminate frequently with no OWA and 40 PDA users without sync until i restart IIS. And... problems started last friday after restart !
All patches applied. On same server Symantec Mail Security for Exchange version 5.0.4.363.
 
I have tried reinstalling Exchange SP2 and renamed mail queue without positive result.
 
Now i wait for your metabase restore results....



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 3:04:59 PM)

I run the same exact version of Sym mail sec. for exchange.

Out of curiosity, when did you install Sym Mail security 5.0 ? I did my upgrade from 4.6 around 3 months ago.

Can you please provide more details on your infrastructure? what OS on the server? what exchange, what service packs on OS/Exchange etc.. .?

Thanks!

I am changing my mind a little, I think I might update my BE with all available updates including .net 2.0 (which went onto my FE just when things broke).



jchong -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 3:15:04 PM)

Take a look at this thread, not sure if it's one of you guys posting, but looks like there's a fix from symantec.

The IIS Admin Service service terminated unexpectedly
http://groups.google.com/group/microsoft.public.exchange.admin/browse_thread/thread/4da19865fae9ff1f



bstanley -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 3:29:40 PM)

Thanks JCHONG...   that did the trick...  Foo on symantec...



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 3:49:46 PM)

what a great find man! I have just stopped premium antispam and will post once I confirm that issue went away.

If it does I will contact symantec and see if they recommend the same fix for me.

I should post some results in a couple of hours, my 7031/7034 ID's pop up about every hour or so on average.




jchong -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 3:57:22 PM)

Looks like a recent issue, I'm sure alot of people are currently having these issues [8|]



mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 4:06:10 PM)

yeah we're not alone in this, I must have found end of internet a few times while searching for a resolution and there's easily 100+ people asking about it today in newsgroups/message boards.

I think it's safe enough to assume it's probably 10,000 admins ripping their hair out if they even know about it.

I was one of the unlucky ones whose exchange server actually locked up and services did not restart on their own, it seems like for most poeple it's hardly noticeable unless they reboot and see IIS crash or look at logs (which lets face it, not everyone reads everyday... even though they probably should).




mcadek -> RE: Very strange behavior , routing & SMTP service crash and restart (5.Dec.2006 9:13:32 PM)

[:D]well... mine is fixed  [:D] no crashes since my previous post after following symantec's instructions outlined in the link above.

what a pain in the ass that was... I figured what the hell and tried the instructions above, my antispam and antivirus still work and nothing crashes anymore, I will restart the server later, probably by Friday to see if IIS still crashes at bootup, but I imagine it will not.  (prior to this I would get the "send/don't send" crash log/report about 4 to 10 times after each reboot as well as the outlined 7031 7034 errors periodically after in system log).

thank you for the help jchong , all those microsoft newsgroup posts popped up today all over the place as I obviously wasn't alone in this, but you surely shortened my search , thanks again.





dvprao -> RE: Very strange behavior , routing & SMTP service crash and restart (6.Dec.2006 2:17:32 PM)

thanks for persisiting this issue.
the symantec fix worked so far !!!
thanks Mr. Chong...
DVP[:)]



cgibson -> RE: Very strange behavior , routing & SMTP service crash and restart (22.Dec.2006 10:23:10 AM)

I've had the same problem for the past week, but I'm not using  Symantec Mail  Security, I'm using GFI mail essentials 11.  However, I do have Symantec Corp. installed on the machine for A/V.



Page: [1] 2   next >   >>