Search | View Active Threads | View New Posts
Current Time: 12:29 PM


+ Reply to Thread
Results 1 to 4 of 4
  1. #1
    Join Date
    Sep 2002

    Blocking Websense spiders

    Websense is filter software that is used by schools, companies etc to filter websites for their users, blocking audlt, gambling, social networks and whatever the sysadmin might want to hide.

    To do this, Websense maintains a big database of websites and classifies them. This means they have to visit these websites. But they don't like to be discovered too easily, so they ignore Robots.txt and vary the user agents. Still, you can quite easily spot them by their IP range.

    Normally they will only visit a site every now and then. However, I have some domains on my name server that used to have websites at third level addresses, like
    Websense does not really know which thrid levels do/did exist and which don't, so it just starts guessing. The result is a bombardment of thousands of domain requests within a few minutes, effectively almost a DDOS attack. This can go on fo hours.

    To avoid this, I am considering to block all Websense spiders. However, I am afraid that Websense then might blacklist all my sites, and it might cut in on my parking revenue for all the domains that run through this server as well.

    Does anyone have any experience with this?

  2. #2
    Join Date
    Jan 2003
    Just block those IP's for the sites they hammer rather than on all sites, or add rate limiting to your Apache server.
    Captain Stupid.
    Jimmy James Inc. fan club membership # 3312

  3. #3
    Join Date
    Sep 2002
    Thanks Drewbert, I'll look into that.

  4. #4
    Join Date
    Sep 2018
    Most web pages contain content from multiple sources (ad servers, streaming video sites, social networking applications, image hosting services, and so on). Some sites aggregate content, pulling pieces from multiple sites into a single presentation.
    In these instances, users may request sites that contain a mix of permitted and blocked content.
    When a frame or iframe within a larger page contains blocked content, Websense software displays a standard or security block page within that frame. When the frame is small, however, the end user might be able to see only a tiny portion of the page (perhaps not even the full block icon), and not understand why the content is blocked.

    Top Astrology Services in Canada | Indian Astrologer in Toronto | Best Indian Astrologer in Toronto | Indian Astrologer in Canada | Best Indian Astrologer in Canada | Astrologer in Canada

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts


Domain Escrow Services Domain Registrar, Marketplace and Revenue Optimizer
Trellian Direct Search Network

Industry Events

11-14 August 2019
Affiliate Summit East
New York, NY

4-6 March 2019
Las Vegas

15-18 September 2018
MERGE 2018
Orlando, FL


How do you invest in domains?
to see the Poll results!
Domain Tools | Domain Directory | Registrar Stats | Domain Glossary | Industry Events | FAQ | Members | Terms | RSS | Link To Us | Advertise | Contact Us
Other Related Trellian Services: Registrar  | Marketplace  | Domain Manager  |  Free Search Toolbar  |  Free Webpage Builder  |  Keyword Research
Direct Search Navigation  |  Search Engine Submission and SEO Tools
Copyright © 2002 a Trellian Company 2014 Bronze Sponsor - Internet Commerce Association