Recently I ran into a problem where emails that were generated using CDOSys were getting stuck in the Microsoft SMTP service outgoing queue. If you are having problems with the Microsoft SMTP service, you may have discovered that it is difficult to troubleshoot. This difficulty is mostly caused by the fact that the service has very little logging available to it. It is true that you can enable logging via the IIS console, but that logging is minimal at best. Errors are not recorded very well in that log.
I did spot in the event viewer some interesting entries that were complaining about internal DNS errors that appeared to be coming from the SMTP service. Upon further research I found that IIS 5.0 running on Windows Server 2000 is not able to make DNS queries via UDP. IIS is only able to query DNS servers using TCP. This poses a problem since TCP is usually only used for zone transfers; therefore most DNS servers have TCP blocked and only allow UDP queries.
This will result in the SMTP service being unable to query DNS at all. In that state SMTP service will initially process mail that is put into the ‘pickup’ directory by CDOSys, and move it into the ‘queue’ directory. From this directory it is normally processed further and sent to the next MTA. However, when SMTP service is unable to query DNS it just leaves the message in the queue directory and begins ‘BadMail’ processing.
How can you test to see if this is the issue you are having? The first step I recommend is using nslookup to verify proper operation of DNS as per the needs of IIS 5.0. Simply run nslookup from the command line (this can be done on either Windows or Linux), and force it to do all queries using TCP. While in nslookup enter:
Now any queries you enter will be done using TCP. If this fails, try turning off forced TCP and run the query again. If this works you know that the IIS server cannot make TCP DNS queries to it’s designated DNS server. You can turn off forced TCP by using this command in nslookup:
Also there is a useful tool from Microsoft that runs various mail sending steps and can help you in debugging this and other issues with SMTP service. This tool is called SMTPDiag. Here is a good article that shows a little on how to use SMTPDiag.
If you have determined that your mail is getting stuck because of TCP DNS queries being rejected or dropped, there are essentially three options for you to fix this:
- If your firewall is preventing TCP DNS queries, modify the rules to allow it.
- If possible, change the configuration of the DNS server to allow TCP queries.
- Change the IIS server to use a DNS server that is configured to allow TCP queries.
Use the nslookup procedure shown above to determine both that the firewall rules are configured to allow TCP queries AND that the DNS server that you’d like to use is allowing TCP queries. Once are sure that you have set IIS to query a DNS server that allows TCP queries, simply restart the SMTP service and all mail in Queue should begin to process.
Remember: On Windows Server 2000, you must reboot the server after you changed the DNS servers for a interface in order for the change to take effect.