SEO is an onion skin, we don't make the rules, we just follow them. What is a 404 code, this is a response code for a page to notify the crawler that the page no longer exists. A page with "Page not found" needs to be created server side, in this example IIS, the only place to set a response code is on an aspx pape. Its not enough to display in the browser, "We are sorry the page you have requested is not found" You must also in codebehind set the response code to 404, otherwise the crawler sees your page not found as a 200 and indexes page not found. In the imitable words of Geico, "Everyone knows that."
The question though, are those 404s hurting your page ranking, and the best MBA answer I gave is "it depends." Part of your page rank is external links to your domain. Let's say bloggers following your link on your site to hotdomain.com/hothotpage.aspx Now you decide to change your site and hothotpage.aspx becomes frickenhotpage.aspx. hothotpage.aspx no longer exists, but all the external links still exist. Because the external links exists, crawlers keep looking for hothotpage.aspx and you keep returning 404 or if you are really intent 410. The crawlers keep asking for hothotpage and anyone that follows the external link is getting "page not found." THIS will effect your pagerank. External links are essential to your page rank. Get creative in your IHttpModule and do a 302 to frickenhotpage.aspx.
When to redirect and when to server.transfer. If the page is truly not found, a typo perhaps, or some bot phishing looking for holes. hotdomain.com/usercodes.php. You do not want a 302 to your 404, you just want a 404 to usercodes.php, server.transfer will insert your pagenotfound.aspx with the header for usercodes.php. But in the case when you are doing a redirect, the case of hothotpage.aspx, you want to notify the crawler of the 302 perm redirect to frickenhotpage.aspx with a normal 200. This will preserve your external links and page ranking instead of 404 to dead links. Best practice is to honor the external links and redirect to the new page.
While on this topic, invariably your site is being crawled. How do you perform system maintenance. Much like a 302 redirect to a new page with a 200 response. Its a good idea to redirect all pages during maintenance, or failures with a 302 to a 503 System Unavailable. This is arguable; server.transfer or response.redirect. I favor the 302 redirect to a 503 page.
The Firebug plugin is a great way to see how this all works.
