Disclaimer

Any opinions expressed here are my own and not necessarily those of my employer (I'm self-employed).

Oct 6, 2010

Keep ASP.NET error pages out of search engines

In a production environment, users should not be presented the default ASP.NET error pages. Instead they should be offered clean, understandable error pages giving them a sensible explanation of the error, along with suggestions to continue their journey on the website. Besides usability concerns, it's also an important security practice to not leak details about application details to those who might tinker with your application!

In ASP.NET, the customErrors configuration element is used to handle error situations. However, the behaviour of the custom errors is somewhat counterintuitive, as you might end up with your error pages indexed by search engines.

How customErrors work
First a quick example of how a customError section might look like in a web.config file (this belongs under system.web):

<customErrors mode="RemoteOnly" 
              defaultRedirect="/StandardError.aspx">
    <error statusCode="404" redirect="/NotFound.aspx"/>
</customErrors>

CustomErrors can be set to: Off | RemoteOnly | On. Off means that you'll get the default (and detailed) ASP.NET error pages when something bad happens. Remoteonly will give the default error pages when the application is accessed from localhost, but will serve your custom pages for requests not originating on the local machine. On will always serve the custom error pages.

Historically, errors would trigger a redirect to en error page — this is still the default behaviour. The path to the page that triggered the error is included as a parameter:

/StandardError.aspx?aspxerrorpath=/my/pagewitherror.aspx

Starting with the .Net framework 3.51, the customErrors element includes the optional redirectMode attribute. When setting this to "ResponseRewrite", the user is no longer redirected to the error page. Instead, the error page is served "in place" on the page where the error occured. This can be advantageous, as the user can simply refresh the page to try again instead of being sent away from the webpage where she's trying to accomplish something.

Though it's important to present a professional looking error page there is also important behaviour invisible to the average end-user: HTTP status codes affecting how search engines index your site.  

HTTP status codes
HTTP status codes are fundamental to the functioning of the web. In short, they're numeric values  describing the outcome of a request. The status codes are included in the first line of the response by a webserver. Here's the most common ones:
  • 200 OK
  • 301 Moved permanently (this is a redirect)
  • 302 Moved temporarily (this is a redirect)
  • 404 Not found
  • 500 Internal server error
The 200, 301, and 404 status codes have a major impact on how search engines index your site. 200 means that they will index the page at will. 301 means that they should replace the source of the redirect with the destination address of the redirect. 404 means that they will remove the page from their index.  Google has an excellent article on the various HTTP status codes, and how they impact the Googlebot crawlers. To inspect HTTP status codes, get a tool such as e.g Fiddler or the Firefox Live HTTP Headers plugin.

When running with customErrors Off, a request for a non-existant aspx will yield a default ASP.NET "Server error, file not found" error page, correctly returned with a 404 HTTP status code. This behaviour is important, as the 404 status code indicates to search engines that the resource did not exist.

However, it all changes when the customErrors are set to RemoteOnly or On! You'll see your custom error page served, but the status code in the response will be 200. This indicates that everything went well! Search engines will consequently index your error page at will — and they will keep returning to the address to check for updates. Why the behaviour changes to return a 200 instead of a 404 is beyond me. In my opinion, it shouldn't. We'll revisit our example to show just how counterintuitive this is, we've now included the ResponseRewrite functionality:

<customErrors mode="RemoteOnly" 
              defaultRedirect="/StandardError.aspx"
              redirectMode="ResponseRewrite">
    <error statusCode="404" redirect="/NotFound.aspx"/>
</customErrors>

Note that we are specific about the 404 errors — we even refer it by it's numerical code — but the error page will still be returned with a 200 OK.

Fixing the problem
For the NotFound.aspx error page in our example, the statuscode can be set programatically in e.g. the Page_Load() event :

Response.StatusCode = 404;

This will override the status code in the response, and make your "file not found" custom error page behave correctly!

As a sidenote, there are several tricks to keep your regular pages out of the search engine indexes. Google has published several articles on how to keep stuff out of their index, check them out!

142 comments:

  1. Thanks for sharing the solution.

    Can we return 404 for path/ alias not found (example: /Path-Not-Exist/ )?

    ReplyDelete
    Replies
    1. یکی از بهترین سایت های موزیک برای دانلود آهنگ سایته: دانلود آهنگ جدید
      می باشد که دارای ارشیو کامل از

      Delete
  2. Thank you for posting the great content.I was looking for something like this.I found it quiet interesting, hopefully you will keep posting such blogs..Keep sharing digital marketing jobs in hyderabad.

    ReplyDelete
  3. Personally I think overjoyed I discovered the blogs.
    technology

    ReplyDelete
  4. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    web design training in chennai

    ReplyDelete
  5. Expected to form you a next to no word to thank you once more with respect to the decent recommendations you've contributed here.
    Dotnet Training in Marathahalli

    ReplyDelete
  6. Thanks for sharing the information. i really enjoyed your blog article and always read your blog.
    Mobile Computing Project Center in Chennai | Mobile Computing Projects in Velachery

    ReplyDelete
  7. I accept there are numerous more pleasurable open doors ahead for people that took a gander at your site.
    R Programming Training in Bangalore

    ReplyDelete
  8. I started to learn something about technologies and computer in high school. That was high time to write college admission essay. I even used https://order-essay-online.net/blog/admission-essay to find some advices about writing.

    ReplyDelete
  9. Very nice blog with nice explanation about ASP.NET error page. Thanks for sharing.
    Austere Technologies Cloud Services | No.1 cloud technologies

    ReplyDelete
  10. wow...nice blog, very help full information. Thanks for sharing.

    BEST APP DEVELOPMENT SERVICES | MASSIL TECHNOLOGIES

    ReplyDelete
  11. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    MASSIL TECHNOLOGIES | ORACLE CLOUD SERVICES FOR APPLICATION DEVELOPMENT

    ReplyDelete
  12. What an interesting informative blog. Thanks for sharing.

    Best Mobility Services | Austere Technologies

    ReplyDelete
  13. Great info! I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have. baba pandey

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    National Caviar Day

    ReplyDelete
  16. Great article, really very helpful content you made. Thank you, keep sharing.

    Best Degree Colleges Hyderabad | Avinash College of Commerce

    ReplyDelete
  17. Thank you for sharing this valuable information. But get out this busy life and find some peace with a beautiful trip. book CHEAP ANDAMAN PACKAGES @9999

    ReplyDelete
  18. Hi Thanks for the nice information its very useful to read your blog. We provide best Find All Isfs Courses

    ReplyDelete
  19. Your website is really cool and this is a great inspiring article. Thank you so much. Internet marketing services

    ReplyDelete
  20. Your article gives lots of information to me. Thanks for sharing.
    cima courses in hyderabad | ISFS

    ReplyDelete
  21. diwali image

    Diwali Images: Hello, users, Diwali is on the way and we are left with just one month left.
    On the precious occasion of Diwali, we do wish our ...

    ReplyDelete
  22. i am for the first time here. I found this board and I in finding It truly helpful & it helped me out a lot. I hope to present something back and help others such as you helped me. buy seo backlinks

    ReplyDelete
  23. It was a very good post indeed. I thoroughly enjoyed reading it in my lunch time. Will surely come and visit this blog more often. Thanks for sharing. university of melbourn

    ReplyDelete
  24. Wow...What an excellent informative blog, really helpful. Thank you. Best Oracle DBA Course Training| orskl

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. I am happy to find this post very useful for me, as it contains lot of information.

    neha kakkar height
    neha kakkar age

    ReplyDelete
  27. Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post. link building agency

    ReplyDelete
  28. Thanks for provide great information and looking beautiful blog. If you wish to learn get connected with Advanced Cyber Security Course. I hope below information will help you.
    Ethical Hacking Training in chennai
    Best Training Institute in Chennai

    ReplyDelete
  29. Hi, Thanks for the great information and it is very useful to read your blog with easily understand for all readers. Well done!
    web design company in velachery

    ReplyDelete
  30. https://www.dotnetnoob.com/2010/10/keep-aspnet-error-pages-out-of-search.html
    https://www.programming-free.com/
    https://www.alliancetek.com/blog/post/2015/09/03/when-and-why-is-tally-integration-required.aspx
    https://www.webdesignphils.com/cited-among-10-awesome-web-developers-in-the-philippines/
    http://cloudepr.blogspot.com/2009/09/cluster-facilities-hardware-and.html
    https://www.sweetromancereads.com/2015/03/aarrggghhh-or-feeling-overloaded-by.html
    https://www.oracleracexpert.com/2009/08/oracle-data-pump-exportimport.html
    http://alonsoyserodio.blogs-r.com/weblog/gl/fotos/mgcolomo/8a8024e7325ae66f01327df990671230
    http://www.directory8.org/Internet/Training/
    https://www.dondir.com/education/online_courses/page-15.html

    ReplyDelete
  31. Thanks for this valuable information sharing, and i learned a lot and cleared my all doubts in this.. keep posting like this useful information.
    post free classified ads in india

    ReplyDelete
  32. Thanks for this valuable information sharing, and i learned a lot and cleared my all doubts in this.. keep posting like this useful information.
    Scaffolding Dealers in Chennai
    Aluminium Scaffolding Dealers in Chennai

    ReplyDelete
  33. Thanks for sharing this wonderful information. I hope you will share more helpful information regarding the content.
    web portal development company in chennai

    ReplyDelete
  34. Good blog thanks for sharing! London Tigers Security Door Supervisor

    ReplyDelete
  35. Good blog post thanks, Providing Us With This Great Knowledge Value Of My Car

    ReplyDelete
  36. Amazing information thanks for sharing good article
    Best management college in Hyderabad

    ReplyDelete
  37. Many of these sites use the power of machine learning. In this article, we are going to talk about the relation between data science and machine learning. Read on. machine learning course in hyderabad

    ReplyDelete
  38. wow...nice blog
    Good article and very useful information
    Best commerce college

    ReplyDelete
  39. Nice blog !!! Keep blogging !!

    https://devu.in/machine-learning-training-in-bangalore/

    ReplyDelete
  40. AB Paving is a Burnaby-based paving contractor that is well-known for offering a wide range of premium Paving Amarillo TX

    ReplyDelete
  41. This is my first time i visit here and I found so many interesting stuff in your blog especially it's discussion, thank you.
    business analytics course

    ReplyDelete
  42. Thanks for always being willing to lend a hand to provide the best knowledge.
    360DigiTMG industrial revolution in malaysia

    ReplyDelete
  43. Tapasya is the Best College of Commerce and Management in Hyderabad with Highly Experienced Faculty for Commerce and Management Courses - MEC, CEC, BCom, BBA & MBA.
    http://tapasyaedu.com/

    ReplyDelete
  44. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck. 안전놀이터

    ReplyDelete
  45. As I website possessor I think the written content here is rattling wonderful, thankyou for your efforts. 메이저놀이터

    ReplyDelete
  46. Your articles are inventive. I am looking forward to reading the plethora of articles that you have linked here. Thumbs up! 먹튀검증

    ReplyDelete
  47. I feel a lot more people need to read this, very good info buying 토토사이트

    ReplyDelete
  48. I would like to say that this blog really convinced me to do it! Thanks, very good post 먹튀검증커뮤니티

    ReplyDelete
  49. Audio began playing any time I opened up this webpage, so annoying! 메이저놀이터

    ReplyDelete
  50. A very informationrmative article and lots of really honest and forthright comments made! This certainly got me thinking about this issue, cheers all. 먹튀검증

    ReplyDelete
  51. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one
    Best Data Science courses in Hyderabad

    ReplyDelete
  52. that will help me grow.먹튀사이트 good of which you’re so good currently

    ReplyDelete
  53. It's great to have a place like this. 먹튀신고 look forward to it.It's always coming

    ReplyDelete
  54. Guess I will just bookmark this site 보증업체 The blog is instructive additionally

    ReplyDelete
  55. Hello to all, how is everything, I think every one is
    getting more from this site, and your views are good for new visitors.부산달리기

    ReplyDelete
  56. This was a really great contest and hopefully I can attend the next one. It was alot of fun and I really enjoyed myself.. marketing1on1.com/professional-seo-services-company

    ReplyDelete
  57. Elisha is what's written on his beginning certificate but he never truly liked that title. Taking treatment of animals has been my occupation for some time but I strategy on changing it. Years in the past he moved to Arkansas. My friends say it's not good for me but what I adore doing is to go to karaoke but I don't have the time recently. Check out her website here: 대구오피


    ReplyDelete
  58. I'm amazed, I have to admit. Rarely do I encounter a blog that?s equally educative and amusing, and without
    a doubt, you've hit the nail on the head. The issue is something not enough men and
    women are speaking intelligently about. I'm very happy I came across this in my
    hunt for something concerning this. 강남오피
    Very good article! We will be linking to this great
    content on our site. Keep up the good writing.


    ReplyDelete
  59. Will really appreciate the writer's choice for choosing this excellent article appropriate to my matter.Here is deep description about the article matter which helped me more휴게텔

    ReplyDelete
  60. I finally found great post here.I will get back here. I just added your blog to my bookmark sites...

    DevOps Training in Hyderabad

    ReplyDelete
  61. I really enjoyed this content.
    Keep posting all these information.
    There are several reasons behind it one of the most common reasons is that your printer and the device from which you are giving commands to it are unable to establish a connection.
    Check into my site and resolve your all problem with the http//ij.start.canon device.

    ReplyDelete
  62. You provide very useful content.
    This post give me lots of advise it is very useful for me.
    Technical assistance is very necessary if you are a non-technical person or don’t have enough http//ij.start.canon knowledge of digital devices.
    CANON printer and other peripheral devices helping users all over the world to complete their daily tasks.

    ReplyDelete
  63. You made some good points there. I did a Google search about the topic and found most people will believe your blog. 먹튀검증


    ReplyDelete
  64. Hello, I am one of the most impressed people in your article. 안전놀이터추천 I'm very curious about how you write such a good article. Are you an expert on this subject? I think so. Thank you again for allowing me to read these posts, and have a nice day today. Thank you.


    ReplyDelete
  65. I accidentally searched and visited your site. I still saw several posts during my visit, but the text was neat and readable. I will quote this post and post it on my blog. Would you like to visit my blog later? 메이저놀이터추천


    ReplyDelete
  66. While looking for articles on these topics, I came across this article on the site here. As I read your article, I felt like an expert in this field. I have several articles on these topics posted on my site. Could you please visit my homepage? 토토사이트모음


    ReplyDelete
  67. Youre so right. Im there with you. Your weblog is definitely worth a read if anyone comes throughout it. Im lucky I did because now Ive received a whole new view of this. 메이저사이트추천


    ReplyDelete
  68. I finally found what I was looking for! I'm so happy. 스포츠토토사이트 I hope you can read my post and let me know what to modify. My writing is in I would like you to visit my blog.


    ReplyDelete
  69. You uploaded a fantastic post. I was extremely touched after I saw your writing, and also I actually suched as the tale of what I was trying to find due to the fact that it consisted of whatever. I'm so interested regarding just how you thought about this info and also exactly how you located it. Consider my writing as well as let me understand. check out this site

    ReplyDelete
  70. Youre so right. Im there with you. Your weblog is definitely worth a read if anyone comes throughout it. Im lucky I did because now Ive received a whole new view of this. 먹튀검증사이트


    ReplyDelete
  71. After reading your article I was amazed. thanks for sharing.
    servicenow training in hyderabad

    ReplyDelete
  72. Thank you for some other informative blog. Where else could I get that type of information written in such an ideal means? I have a mission that I’m just now working on, and I have been at the look out for such information. 먹튀검증 It helped me a lot. If you have time, I hope you come to my site and share your opinions. Have a nice day.


    ReplyDelete
  73. I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at this place
    ServiceNow Training in Pune

    ReplyDelete
  74. 메이저사이트18 November, 2021 09:50

    It has a good meaning. If you always live positively, someday good things will happen. 메이저사이트Let's believe in the power of positivity. Have a nice day.


    ReplyDelete
  75. Why couldn't I have the same or similar opinions as you? T^T I hope you also visit my blog and give us a good opinion.온라인슬롯


    ReplyDelete
  76. wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now i’m a bit clear. I’ve bookmark your site and also add rss. keep us updated. best data science training institute in gurgaon

    ReplyDelete
  77. 메이저놀이터02 December, 2021 03:49

    First of all, thank you for your post. 메이저놀이터 Your posts are neatly organized with the information I want, so there are plenty of resources to reference. I bookmark this site and will find your posts frequently in the future. Thanks again ^^


    ReplyDelete
  78. I'm so happy to finally find a post with what I want. 오공슬롯 You have inspired me a lot. If you are satisfied, please visit my website and leave your feedback.


    ReplyDelete
  79. Hi what an exceptional post I went over and believe me
    I was looking for relative posts last week and hardly at any point found it. Much thanks and we will keep it together for extra messages from you.
    uTorrent Pro Crack

    ReplyDelete
  80. Howdy what an outstanding post I went over and trust me
    I was searching for relative posts last week and scarcely anytime tracked down it. Much thanks and we will maintain some kind of control for additional messages from you.
    uTorrent Pro Crack Howdy what an outstanding post I went over and trust me
    I was searching for relative posts last week and scarcely anytime tracked down it. Much thanks and we will maintain some kind of control for additional messages from you.
    uTorrent Pro Crack

    ReplyDelete
  81. 360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

    ReplyDelete
  82. It's amazing that you have that kind of idea writing this article. Thanks
    Quran Academy
    Online Quran Academy

    ReplyDelete
  83. I guess I am the only one who came here to share my very own experience. Guess what
    I am using my laptop for almost the past 2 years, but I had no idea of solving some basic issues.
    I do not know how to Free Download But thankfully, I recently visited a website named
    Origin Pro Crack

    Free YouTube Download Crack

    VideoProc Crack

    Skype Crack

    IObit Smart Defrag Crack

    Wondershare PDFelement Pro Crack

    Ableton Live Crack

    ReplyDelete
  84. This comment has been removed by the author.

    ReplyDelete
  85. Thanks for supporting us. Parallels Desktop Torrent is a strong
    competitor of virtual machine ware and workstation users.

    ReplyDelete
  86. Are you finding best software for accounting in your business field? If yes i have a software that can definitively help you to do this work more easy. This software is known as Tally Prime Release Crack. It is the world best management tool for business.

    ReplyDelete
  87. 360DigiTMG offers various modules of Data Science course that will help you in understanding the fundamentals of the concept. Start your course today to become the most wanted Data Scientist in the hiring market.


    Data Science Course in Bangalore with Placement

    ReplyDelete
  88. 360DigiTMG offers the best Data Science courses in the market. Join and become a professional Data Scientist.

    Best Data Science Training institute in Bangalore

    ReplyDelete
  89. Learn the fundamentals of Data Science and master the skills to handle epic amounts of data collected by organizations today. Gain expertise in organizing, analyzing, and transforming data to uncover its hidden power. Drive your career forward with a Data Science course in Bangalore and learn to uncover insights to present critical findings using visualization tools. Not only this, avail the best-in-class content delivered by stellar faculty who use a blended approach of theory as well as practical knowledge to ensure all the concepts are crystal clear.

    Data Science Course in Jaipur

    ReplyDelete
  90. Learn to build powerful models to solve business problems by generating useful insights and discover the various scientific processes and methods used to transform the information available in huge datasets into meaningful results. master all the tools and techniques in Data Science and gain domain-specific knowledge which will help you to add more value to your profile. Sign up for the Data Science course in Bangalore with Placements and multiple your chances of working across all industries and job functions.


    Data Science Training in Delhi

    ReplyDelete
  91. The increase in big data has led to a boom in the field of Data Science spiking ample career opportunities. Enroll in the Data Science training in Bangalore and invest in emerging skills and transform any business by wrangling, analyzing, and visualizing data. Give your career a makeover and gain in-depth knowledge on how to extract valuable insights from complex and large sets of data. Get to work on a live project which is designed to give hands-on experience to you along with career guidance and mentorship.


    Data Scientist Course in Delhi

    ReplyDelete
  92. 사설토토 To guard against Son's absence, Bento is taking the 21-year-old forward Oh Hyeon-gyu to Qatar, though he didn't make the team. If everyone on the 26-man roster stays healthy, Bento said Oh will remain out of the list. Should Son or another attacking player can't go due to an injury, Oh will be the insurance. Teams will be allowed to make last-minute roster changes in case of injuries.

    ReplyDelete
  93. 먹튀검증커뮤니티 The injured South Korean football star Son Heung-min was named to the country's World Cup squad Saturday, as the football-crazed nation waits with bated breath to see if the beloved captain will be able to compete in Qatar.

    ReplyDelete
  94. 카지노커뮤니티 Among other notables, RCD Mallorca midfielder Lee Kang-in was selected despite Bento's long-held reluctance to play the 21-year-old.

    ReplyDelete
  95. 슬롯커뮤니티 Bento said he has remained in contact with Tottenham's medical officials, but he wasn't yet certain exactly when Son would be available to train with the team.

    ReplyDelete
  96. 토토사이트 Son, the reigning Premier League Golden Boot winner for Tottenham Hotspur, underwent surgery on Nov. 4 after suffering multiple fractures around his left eye in a collision with an opposing player during a UEFA Champions League match. Son took to social media Wednesday to declare he will be prepared to play at the World Cup wearing a protective mask if that's what it takes.

    ReplyDelete
  97. 먹튀검증사이트 "He's a player with good technique. He's physically strong with good mobility," Bento said. "What we observed in our context made us take this decision.

    ReplyDelete
  98. 토토검증사이트 Bento himself had confirmed Thursday, the day before South Korea's final tuneup against Iceland, that he would select Son, making Saturday's announcement a mere formality. Bento also said then that the national team would do its best to help Son get ready for the big tournament.

    ReplyDelete
  99. 슬롯머신사이트 "We need to wait. We need to analyze this situation day by day," Bento said at a press conference announcing his squad. "We have time to decide. The most important thing is he recovers as well as possible and he feels comfortable. And then, we will take the final decision."

    ReplyDelete
  100. Nice to meet you. I have been very busy these days since I started studying new things. I came to visit to cool my head for a while. This is the best place to rest for a while. I'll visit often

    ReplyDelete
  101. Thanks for sharing this wonderful article. Unlock your full potential with our expert-led English tuition classes at Ziyyara Edutech. Our online English tuition offers personalized learning, interactive sessions, and comprehensive support.
    For more info visit English tuition classes

    ReplyDelete
  102. Thank you for sharing your knowledge on this topic. I've learned a lot from your post.Best AWS Training in Pune with Placement

    ReplyDelete
  103. Great experience I got good information from your blog. Tackle reading challenges head-on with our top-notch online phonics classes designed to eliminate common challenges in literacy development.
    visit best phonics classes online

    ReplyDelete
  104. Thank you for this valuable information. The information you shared is very interesting. Ziyyara’s expert tutors are dedicated to guiding students through AS Level courses, providing targeted support, and fostering a deep understanding of the curriculum.
    Book A Free Demo Today visit AS level online courses

    ReplyDelete

Copyright notice

© André N. Klingsheim and www.dotnetnoob.com, 2009-2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to André N. Klingsheim and www.dotnetnoob.com with appropriate and specific direction to the original content.

Read other popular posts