Hoping someone can assist with this issue? Sorry, this is quite a long post.
We have 3 Exchange Servers that we wish to upgrade from Exchange 2000/2003 to a later version, undecided as yet, but before doing so we have an issue with SMTP that we need to fix that has only started this week.
This is a quick guide to how things are setup.
2 x Exchange 2003 Enterprise SP2 running on Windows 2003 R2 Enterprise
1 x Exchange 2000 running on Windows 2000 SP4 (Ancient I know)
All 3 are backend servers, there are no frontend servers.
We have 1 SMTP connector that all external mail is sent/received on, the Exchange 2000 is the bridgehead server for that connector. This is mainly because it was probably the first server installed in our Exchange site many years ago and hasn't been retired, it doesn't host any mailboxes and could be removed if needed.
All user mailboxes are on the 2 x 2003 Servers
All 3 servers are on the same IP subnet and don't have any issues communicating with each other, there are no access lists of firewalls between them. There are firewalls within the environment, but they are the 'next hop' from this subnet.
Here lies the issue...
Last week all the above was working fine, users could send/receive both internally and externally with no issue, it has worked this way for the past 8 years and nothing on the network has changed recently. Users have reported that they can still send email externally, but they can't receive anything, either externally or internally even from other users within the same mailstore. Internal and external emails are reaching the Exchange 2000 server, but it is unable to send them to either of the 2 Exchange 2003 servers, both of which report an 'SMTP Protocol Error' when you view them within the queues in Exchange System Manager.
To troubleshoot the SMTP Protocol Error we have tried the following with no success:-
Telnet to either of the 2 Exchange 2003 servers results in the error '421 the connection has been dropped due to an error on this server, but that error only appears when you type in the MAIL FROM: command, in fact you can type in just MAIL and press return and it will drop the connection straight away. The SMTP logs are also full of 421 errors, the same as when attempting to send an email via telnet.
We have also created a new mailstore on one of the 2003 servers and moved a couple of mailboxes to it. When we do that users can once again send/receive internally to each other if they are in the same mailstore on the same server and they can send externally too, but they can't receive emails either externally or from users whose mailboxes are on the other server. Prior to this we also ran eseutil to check the integrity of the mailstores, they all checked out okay.
We have checked all the usual suspects for configuration errors within System Manager, but no one has changed anything prior to this issue, so we don't think it is a change anyone has made. Just to rule things out we have deleted the SMTP connector and recreated it, and also made one of the other servers the bridgehead, but it didn't make any difference, so we put it back as it was. Exchange System Manager has always been more or less in its default settings, we don't restict anything in particular.
As a last resort today we built a brand new Exchange 2003 server with SP2, but prior to installing Exchange we tested that SMTP was working okay, which it was. Telnetting to this new Exchange server worked fine, the MAIL FROM: command worked as expected. As soon as we installed Exchange and it added itself to the site with the other Exchange servers, straight away this new server also developed the SMTP Protocol Error and we are unable to do the MAIL FROM: command using telnet like we could before installing Exchange. This server also suffers with users not being able to receive emails either externally or between mailstores on different servers, even though this is a brand new server, brand new mailstore etc.
Now having said all the above, we do have a full backup of our Domain Controllers including Global Catalog servers and the 2 Exchange 2003 servers, is it likely that restoring Active Directory from a week or two ago would fix the issue with Exchange? It seems to be a configuration issue given that the brand new server only developed the symptoms that the other 2 servers have once Exchange was installed, because Exchange uses Active Directory for its configuration it seems logical that restoring that may fix this issue.
Basically, we have ran out of ideas and google isn't leaving us much to go on either. We can't really go another week with this not working given that we have several hundred users who need this to be working, therefore does anyone have any idea what we should do next? With SMTP being so tightly integrated with IIS it is proving difficult if not impossible to troubleshoot and none of the articles on technet have the answer either.
I also forgot to mention that we have turned up the various diagnostic levels, ran the Exchange Troubleshooting Analyzer, and whilst they do report the SMTP Protocol issue there isn't a suggestion as to what the fix may be.