Exchange Server Forums

Very strange behavior , routing & SMTP service crash and restart

Users viewing this topic: none

Logged in as: Guest

Printable Version

All Forums >> [Microsoft Exchange 2003] >> General >> Very strange behavior , routing & SMTP service crash and restart

Page: [1] 2 next > >>

Message

<< Older Topic Newer Topic >>

Limited time MSExchange.org offer! -- 1.Sep.2008 1:00:00 PM

TechGenix and SolarWinds have partnered to provide free copies of SolarWinds Exchange Monitor to all visitors who join the MSExchange.org Forums. SolarWinds Exchange Monitor is a handy desktop dashboard that continuously monitors Microsoft Exchange to deliver real-time insight into Exchange services, mail queue sizes, and host server health. Learn more about Exchange Monitor and the free offer!

Very strange behavior , routing & SMTP service crash an... - 4.Dec.2006 1:16:47 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

Let me outline what happened:

here is the layout of our servers

- 1 front end server with OWA, our PDA's also use this to connect their activesync with it
- 1 backend server with 1 information store for entire corporation in US
- 1 backend server with 1 information store for our overseas office
- back end in US is 2003 enterprise SP2, overseas BE is 2003 standard SP2 , front-end is 2003 standard SP2,

- this weekend all of a sudden our company PDA's stopped getting emails
- OWA also went down with 503 error , service unavailable I think
- Internally exchange was working , but would not send/receive mail (new message would sit in outbox). I am assuming SMTP service was not functioning right.

- since I did not have much time for daignosis, I chose to reboot both the FE and BE. Everything started working again, but...

- immediately after reboot I was getting IIS crash/illegal operation errors, it was related to the w3... worker process (w3we.exe or something like that, I do not have the screencap right here). It crashed about 10 times and then stopped coming up

-everything works fine since except I get very strange errors.

on the FE:

I get event id 3007 for every user with a PDA/activesync, multiple times at random intervals

: exchange mailbox server response timeout: Server: "servername" , user: "usernaem" , exchange activesync server failed to communicate with the exchange mailbox server in a timely manner, veryfy that the exchange mailbox server i workign correctly and is not overloaded.

it definitely is NOT overloaded and it seems to work just fine for 150+ users that use it, even the PDA's even though those errors pop up in application event log!

on Back end server I get these ID's that show up in blocks, usually these 5 or 6 Id's pop up all within 1 second in application event log

1005 - msexchangetransport - RE service has been started, Version: 6.5.7638.138.1
1008 - msexchangetransport - RE service instance 1 has been started
332 - msexchagnetransport - SMTP service has been started, initializing queues.
334 - msexchangetransport - SMTP service instance 1 has been started.
10302 - msexchangeactivesync - OMA categorizer successfully started
101 - DAVEX - DAVEX to be shutdown

sometimes followed by DAVEX has successfully started event ID 100

in the system log the following entries provide more general overview of what is happening:

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1013
Description:
A process serving application pool 'ExchangeApplicationPool' exceeded time limits during shut down. The process id was '3132'.

Event Type: Information
Event Source: W3SVC
Event Category: None
Event ID: 1082
Description:
A worker process with pid '3132' that serves application pool 'ExchangeApplicationPool' has been determined to be unhealthy (see previous event log message), but because a debugger is attached to it, the World Wide Web Publishing Service will ignore the error.

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Description:
The IIS Admin Service service terminated unexpectedly. It has done this 12 time(s). The following corrective action will be taken in 1 milliseconds: Run the configured recovery program.

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7034
Date: 12/4/2006
Description:
The Microsoft Exchange Routing Engine service terminated unexpectedly. It has done this 12 time(s).

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7034
Date: 12/4/2006
Description:
The Simple Mail Transfer Protocol (SMTP) service terminated unexpectedly. It has done this 12 time(s).

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1009
A process serving application pool 'ExchangeApplicationPool' terminated unexpectedly. The process id was '5852'. The process exit code was '0xffffffff'.

then the affected services restart on their own and all is well for another 18 minutes (or more, seems random, last period was 18 minutes...)

I expect my exchange organization will die shortly again, I searched the logs and this behavior started 3 days ago, so possibly in 3 more days it will lock up again...

any ideas?

I would be very grateful for any help you can give me. Thank you!

Post #: 1

Featured Links*

RE: Very strange behavior , routing & SMTP service cras... - 4.Dec.2006 1:39:26 PM

jchong

Posts: 2516
Joined: 1.Dec.2005
From: Centreville, Virginia
Status: offline

Virtually anything can cause IIS to hang or crash. You will have to perform some debugging. Download some of the IIS debugger tools.

IIS Crash/Hang Agent & IIS Dump
http://www.microsoft.com/downloads/details.aspx?FamilyID=01c4f89d-cc68-42ba-98d2-0c580437efcf&DisplayLang=en

_____________________________

James Chong
MCSE | M+, S+, MCTS, Security+
msexchangetips.blogspot.com

(in reply to mcadek)

Post #: 2

RE: Very strange behavior , routing & SMTP service cras... - 4.Dec.2006 2:08:02 PM

dvprao

Posts: 9
Joined: 4.Dec.2006
Status: offline

Hi :
I want ot report the same peculiar behavior on my server too since this last Friday... All I did was applied wahtever patches thro auto update downlaods on the Win2k Server STD. and waiting ot reboot.. I rebooted on Friday morning as i saw the inetinfo.exe crash message..
I looked up and did what Ms asked me to try .i.e., renamed the mailroot queue folder and restarted the box, the queue got re-generated...
but still every 80 to 90 minutes thie follwong 5 critical errors are logged in teh EViwer.. Earlier the service was not set to restart but now I have set it to restart but I am not comfortable leaviing it like that.

I do have a 4 versions of metadata back up of IIS Server listed. Is there any risk in restoring the same as suggested by MS ? I am littel wary of doing it sinc eI have not done it.

We use the only sever for pop3, imap4 and OWA as well sitting behind a firewall..

Any help is apprecited. I will post the solution if I get it fomr any other forum also..
I am sure this must be happenign to lot of people...

Thanks
DVP.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The World Wide Web Publishing Service service terminated unexpectedly. It has done this 4 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Simple Mail Transport Protocol (SMTP) service terminated unexpectedly. It has done this 4 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Microsoft Exchange Routing Engine service terminated unexpectedly. It has done this 4 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Microsoft Exchange POP3 service terminated unexpectedly. It has done this 4 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The Microsoft Exchange IMAP4 service terminated unexpectedly. It has done this 4 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7031
Date:  12/4/2006
Time:  1:35:20 PM
User:  N/A
Computer: CHALMERS10
Description:
The IIS Admin Service service terminated unexpectedly. It has done this 4 time(s). The following corrective action will be taken in 0 milliseconds: No action.

_____________________________

DVP

(in reply to jchong)

Post #: 3

RE: Very strange behavior , routing & SMTP service cras... - 4.Dec.2006 4:14:00 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

my problems started at the same time too! last friday. although I only updated my FE server, BE server I did not as my FE is my "test bed" for windows updates.

I was going to try the queue rename tonight, guess it's not worth it, but I'll give it a shot anyway...

tomorrow night I will be restoring the metadata file

the day after that attempt I'm calling microsoft and raising up a storm :-(

I almost want to update my BE to all windows updates, but I am not sure if I'll just make it worse... at least in its current state it works (question is for how long).

(in reply to dvprao)

Post #: 4

RE: Very strange behavior , routing & SMTP service cras... - 4.Dec.2006 4:44:31 PM

dvprao

Posts: 9
Joined: 4.Dec.2006
Status: offline

Yup, Mail queue renaming did not help.
I saw a sub folder called filter wiht lot of *.tmp files in it but that folder did not get generated .... NOw the bad mail folder is getting populated more...

I have just now installed hte IIS crash dump utility from MS and installed i t. But I do not see global filter listed in teh filters portion o fht eIISADMIN !!!. Where is it listed ?

Like I said the services set all of them ( 5 of them) all fail almost same time within 85 minutes... ( is there any sync attempt by GC or something ?)

MS claims that event id 7034 should occur for SMTP service failure but I see only 7031 listed even for that service ....

Should I be careful about anything before restoring meta database ?

Thanks

(in reply to mcadek)

Post #: 5

RE: Very strange behavior , routing & SMTP service cras... - 4.Dec.2006 10:10:54 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

I am trying to rename the vsi 1 folder like microsoft wants you to, but something in it is still used even though all exchagne & IIS services are stopped :shrug:

(in a middle of a reboot right now...)

my badmail folder has literary thousands and thousands of files in it... whether this helps or not does not matter much, it's time to clean that mess up anyway :)

I say go ahead and restore your metabase.xml file and let me know what happens, and I do not say that lightly just so you can be the lab-rat, but I'll be doing the same tomorrow night :)

WORST case, you end up uninstalling IIS , reinstalling IIS and doing a reinstall of exchange using the "reinstall" option, 1 hour worth of work or so? (I'm assuming that will not wipe the IS, otherwise restoring mine would be a pain and a half and would take several hours).

(in reply to dvprao)

Post #: 6

RE: Very strange behavior , routing & SMTP service cras... - 4.Dec.2006 10:12:18 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

PS: if you go into IIS you can very easily take a current snapshot of your metabase, so I'd say that is about the only precaution you need to take. Have a good backup of that and you can always return to this mess we're in right now :)

(in reply to mcadek)

Post #: 7

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 9:26:18 AM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

Just to post an update , renaming the vsi 1 folder as microsoft suggests in their KB article did nothing (as expected)

I think I will take the following steps tonight

1) restore metabase file that I have
2) reboot and see what happens
3) if problem persists, I will update my back end server to latest MS updates (if you read above I only updated my FE which may or may not have caused this issue). Maybe something my FE/Pda activesync is doing is affecting my back end, I don't know... it's a guess.
4) I have had symantec mail security on my BE for over a month, but I will stop that temporarily and see what happens after reboot.
5) I will then kill my OWA/Activesync FE server and see if IIS still crashes on boot up (I may have to recover the metabase file again).

if anyone has any input I'd love to hear it.

there is a SMTP vulnerability that causes this issue and a hotfix is available but it's from 2004 and another version of it is from 2005, it SEEMS that this has been included in exchange SP2 , I am trying to track the version numbers down to figure out what I have as I always hate to apply older hotfixes

< Message edited by mcadek -- 5.Dec.2006 9:27:58 AM >

(in reply to mcadek)

Post #: 8

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 11:54:39 AM

dvprao

Posts: 9
Joined: 4.Dec.2006
Status: offline

I did apply the KB885264 patch..( KB890066) no luck...
MS is not very clear on this because in the KB article they refer to event ID 7034 for SMTP failure whereas my BOX reports 7031 for that as well --
I di d not try the disable of anti virus and anti spam routines . I will try that now...

The interval seem to be getting reduced by around 10 minutes now for the cycle...
Let me see if MS user forums has any thing to say.. I am sure we two are not the only Admins having this issue today... I wonder....

(in reply to mcadek)

Post #: 9

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 1:21:32 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

alright I'm not even going to bother with the patch then, thanks for the update.

What I also noticed that since this started happening my BackupExec that usually backs up VSS / System state on this BE exchange box has been failing, it claims that my IIS metabase writer is in use which makes no sense whatsoever.

when I run command VSSadmin list writers on the BE box they all show up, have no errors and are listed as stable.

They are related somehow as the backup began failing on Friday , same day the exchange headaches started. I can only assume at this point that BackupExec does some sort of intergrity check on metabase/all of VSS ?

I'll be restoring the metabase file from over a month ago tonight if you want to wait for my results, at this point I would rather crash the server completely and had to redo it tomorrow than to keep waiting for it to crash again at worst possible time.

(in reply to dvprao)

Post #: 10

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 2:19:20 PM

virtus

Posts: 1
Joined: 5.Dec.2006
Status: offline

Hi
Here on my Exchange server exactly same problem. Under 150 users workload same services terminate frequently with no OWA and 40 PDA users without sync until i restart IIS. And... problems started last friday after restart !
All patches applied. On same server Symantec Mail Security for Exchange version 5.0.4.363.

I have tried reinstalling Exchange SP2 and renamed mail queue without positive result.

Now i wait for your metabase restore results....

(in reply to dvprao)

Post #: 11

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 3:04:59 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

I run the same exact version of Sym mail sec. for exchange.

Out of curiosity, when did you install Sym Mail security 5.0 ? I did my upgrade from 4.6 around 3 months ago.

Can you please provide more details on your infrastructure? what OS on the server? what exchange, what service packs on OS/Exchange etc.. .?

Thanks!

I am changing my mind a little, I think I might update my BE with all available updates including .net 2.0 (which went onto my FE just when things broke).

(in reply to virtus)

Post #: 12

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 3:15:04 PM

jchong

Posts: 2516
Joined: 1.Dec.2005
From: Centreville, Virginia
Status: offline

Take a look at this thread, not sure if it's one of you guys posting, but looks like there's a fix from symantec.

The IIS Admin Service service terminated unexpectedly
http://groups.google.com/group/microsoft.public.exchange.admin/browse_thread/thread/4da19865fae9ff1f

_____________________________

James Chong
MCSE | M+, S+, MCTS, Security+
msexchangetips.blogspot.com

(in reply to mcadek)

Post #: 13

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 3:29:40 PM

bstanley

Posts: 1
Joined: 5.Dec.2006
Status: offline

Thanks JCHONG... that did the trick... Foo on symantec...

(in reply to jchong)

Post #: 14

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 3:49:46 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

what a great find man! I have just stopped premium antispam and will post once I confirm that issue went away.

If it does I will contact symantec and see if they recommend the same fix for me.

I should post some results in a couple of hours, my 7031/7034 ID's pop up about every hour or so on average.

(in reply to jchong)

Post #: 15

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 3:57:22 PM

jchong

Posts: 2516
Joined: 1.Dec.2005
From: Centreville, Virginia
Status: offline

Looks like a recent issue, I'm sure alot of people are currently having these issues

_____________________________

James Chong
MCSE | M+, S+, MCTS, Security+
msexchangetips.blogspot.com

(in reply to mcadek)

Post #: 16

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 4:06:10 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

yeah we're not alone in this, I must have found end of internet a few times while searching for a resolution and there's easily 100+ people asking about it today in newsgroups/message boards.

I think it's safe enough to assume it's probably 10,000 admins ripping their hair out if they even know about it.

I was one of the unlucky ones whose exchange server actually locked up and services did not restart on their own, it seems like for most poeple it's hardly noticeable unless they reboot and see IIS crash or look at logs (which lets face it, not everyone reads everyday... even though they probably should).

(in reply to jchong)

Post #: 17

RE: Very strange behavior , routing & SMTP service cras... - 5.Dec.2006 9:13:32 PM

mcadek

Posts: 106
Joined: 25.May2005
From: Boston, MA
Status: offline

well... mine is fixed

no crashes since my previous post after following symantec's instructions outlined in the link above.

what a pain in the ass that was... I figured what the hell and tried the instructions above, my antispam and antivirus still work and nothing crashes anymore, I will restart the server later, probably by Friday to see if IIS still crashes at bootup, but I imagine it will not. (prior to this I would get the "send/don't send" crash log/report about 4 to 10 times after each reboot as well as the outlined 7031 7034 errors periodically after in system log).

thank you for the help jchong , all those microsoft newsgroup posts popped up today all over the place as I obviously wasn't alone in this, but you surely shortened my search , thanks again.

(in reply to mcadek)

Post #: 18

RE: Very strange behavior , routing & SMTP service cras... - 6.Dec.2006 2:17:32 PM

dvprao

Posts: 9
Joined: 4.Dec.2006
Status: offline

thanks for persisiting this issue.
the symantec fix worked so far !!!
thanks Mr. Chong...
DVP

(in reply to mcadek)

Post #: 19

RE: Very strange behavior , routing & SMTP service cras... - 22.Dec.2006 10:23:10 AM

cgibson

Posts: 5
Joined: 22.Dec.2006
Status: offline

I've had the same problem for the past week, but I'm not using Symantec Mail Security, I'm using GFI mail essentials 11. However, I do have Symantec Corp. installed on the machine for A/V.

(in reply to dvprao)

Post #: 20