×
  • remind me tomorrow
  • remind me next week
  • never remind me
Subscribe to the ANN Newsletter • Wake up every Sunday to a curated list of ANN's most interesting posts of the week. read more
You are welcome to look at the talkback but please consider that this article is over 20 years old before posting.

Forum - View topic
NEWS: Proxy Server Pre-Fetch Question




Note: this is the discussion thread for this article
Anime News Network Forum Index -> Site-related -> Talkback
View previous topic :: View next topic  
Author Message
king_micah



Joined: 09 Jun 2003
Posts: 994
Location: OSU
PostPosted: Tue May 25, 2004 7:00 pm Reply with quote
Is this why I keep getting old forum pages loaded? I have to reload to get new pages loaded most of time, otherwise I get the old pages.
Back to top
View user's profile Send private message Send e-mail
RomSteady



Joined: 25 May 2004
Posts: 2
Location: Layton, UT
PostPosted: Tue May 25, 2004 9:51 pm Reply with quote
Most proxy servers respect the robots.txt file with regards to prefetching. If you know the agent string for the proxy causing the problems, you can even tailor the robots.txt to only stop that proxy from caching.

Your only other choice would be to set an expiration to the page in your meta tags.
Back to top
View user's profile Send private message
void



Joined: 13 Apr 2004
Posts: 1
PostPosted: Wed May 26, 2004 12:07 am Reply with quote
The webserver needs to send "Pragma: no-cache" in the HTTP headers.
Back to top
View user's profile Send private message My Anime
lx805



Joined: 26 May 2004
Posts: 2
PostPosted: Wed May 26, 2004 1:43 am Reply with quote
If you're using PHP, you can drop this code into your pages...

Code:
<?php
header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");
header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
header("Cache-Control: no-store, no-cache, must-revalidate");
header("Cache-Control: post-check=0, pre-check=0", false);
header("Pragma: no-cache");
?>


Hopefully proxy servers hitting this site will honor at LEAST one of these pragmas...
Back to top
View user's profile Send private message
Tempest
I Run this place.
ANN Publisher


Joined: 29 Dec 2001
Posts: 10426
Location: Do not message me for support.
PostPosted: Wed May 26, 2004 2:31 pm Reply with quote
lx805 wrote:
If you're using PHP, you can drop this code into your pages...


Will this stop all caching, or just pre-fetching?


Quote:
Most proxy servers respect the robots.txt file with regards to prefetching.


It has become obvious that some don't. Sad

Quote:
If you know the agent string for the proxy causing the problems
Unfortunately our logs are showing that most proxies are passing on the agent string of the proxy client.[/quote]

Let me explain the issue. ANN gets hammerred by errant bots and e-mail harvesters fairly often, sometimes these bots are really bad and cause some problems in terms of server load, bandwidth and banner clickthrough rates.

What we decided to do, was to set up a little script that catches robots that ignore the robots.txt guidelines. Its worked quite well, but unfortunately it seems to have also picked up, and subsequently banned, several proxy servers that pre-fetch.

I'm glad that most proxies follow the robots.txt guidelines for pre-fetching. It'd be a real headache if they didn't. But unfortunately some don't, and we're getting e-mails from people who have been banned from the site because their office proxy prefetched the dissallowed pages.

So, what I'm really looking for is to turn off pre-fetching and only pre-fetching, as opposed to all caching.
Back to top
View user's profile Send private message Send e-mail My Anime My Manga
RomSteady



Joined: 25 May 2004
Posts: 2
Location: Layton, UT
PostPosted: Wed May 26, 2004 6:31 pm Reply with quote
Quote:
Its worked quite well, but unfortunately it seems to have also picked up, and subsequently banned, several proxy servers that pre-fetch.


Can you track back to one of the proxy server's that is prefetching and find out which proxy server they are using?

Because at this point, the only solution I can think of involves CAPTCHA images to "unban" an IP for a brief while.[/quote]
Back to top
View user's profile Send private message
Tempest
I Run this place.
ANN Publisher


Joined: 29 Dec 2001
Posts: 10426
Location: Do not message me for support.
PostPosted: Wed May 26, 2004 11:08 pm Reply with quote
RomSteady wrote:
Because at this point, the only solution I can think of involves CAPTCHA images to "unban" an IP for a brief while.


The thought has ocurred to me. It's actually a pretty good idea and you reminding of it makes me want to do it.

But I'd still like to do whatever I can to minimize the number of people that need to use it.

-t
Back to top
View user's profile Send private message Send e-mail My Anime My Manga
Tempest
I Run this place.
ANN Publisher


Joined: 29 Dec 2001
Posts: 10426
Location: Do not message me for support.
PostPosted: Wed May 26, 2004 11:12 pm Reply with quote
RomSteady wrote:
Can you track back to one of the proxy server's that is prefetching and find out which proxy server they are using?


Working on that. I'll update here when I find out.
Back to top
View user's profile Send private message Send e-mail My Anime My Manga
Display posts from previous:   
Reply to topic    Anime News Network Forum Index -> Site-related -> Talkback All times are GMT - 5 Hours
Page 1 of 1

 


Powered by phpBB © 2001, 2005 phpBB Group