Search | View Active Threads | View New Posts
Current Time: 12:29 PM

 

+ Reply to Thread
Results 1 to 4 of 4
  1. #1
    Join Date
    Sep 2002
    Posts
    996

    Blocking Websense spiders

    Websense is filter software that is used by schools, companies etc to filter websites for their users, blocking audlt, gambling, social networks and whatever the sysadmin might want to hide.

    To do this, Websense maintains a big database of websites and classifies them. This means they have to visit these websites. But they don't like to be discovered too easily, so they ignore Robots.txt and vary the user agents. Still, you can quite easily spot them by their IP range.

    Normally they will only visit a site every now and then. However, I have some domains on my name server that used to have websites at third level addresses, like example.domain.com.
    Websense does not really know which thrid levels do/did exist and which don't, so it just starts guessing. The result is a bombardment of thousands of domain requests within a few minutes, effectively almost a DDOS attack. This can go on fo hours.

    To avoid this, I am considering to block all Websense spiders. However, I am afraid that Websense then might blacklist all my sites, and it might cut in on my parking revenue for all the domains that run through this server as well.

    Does anyone have any experience with this?

  2. #2
    Join Date
    Jan 2003
    Location
    Mexico
    Posts
    3,831
    Just block those IP's for the sites they hammer rather than on all sites, or add rate limiting to your Apache server.
    Captain Stupid.
    Jimmy James Inc. fan club membership # 3312

  3. #3
    Join Date
    Sep 2002
    Posts
    996
    Thanks Drewbert, I'll look into that.

  4. #4
    Join Date
    Sep 2018
    Posts
    107
    Most web pages contain content from multiple sources (ad servers, streaming video sites, social networking applications, image hosting services, and so on). Some sites aggregate content, pulling pieces from multiple sites into a single presentation.
    In these instances, users may request sites that contain a mix of permitted and blocked content.
    When a frame or iframe within a larger page contains blocked content, Websense software displays a standard or security block page within that frame. When the frame is small, however, the end user might be able to see only a tiny portion of the page (perhaps not even the full block icon), and not understand why the content is blocked.

    Top Astrology Services in Canada | Indian Astrologer in Toronto | Best Indian Astrologer in Toronto | Indian Astrologer in Canada | Best Indian Astrologer in Canada | Astrologer in Canada

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  

Sponsors

Domain Escrow Services
Above.com Domain Registrar, Marketplace and Revenue Optimizer
Trellian Direct Search Network

Industry Events

11-14 August 2019
Affiliate Summit East
New York, NY

4-6 March 2019
LeadsCon
Las Vegas

15-18 September 2018
MERGE 2018
Orlando, FL

Poll

How do you invest in domains?
to see the Poll results!
DomainState.com
Domain Tools | Domain Directory | Registrar Stats | Domain Glossary | Industry Events | FAQ | Members | Terms | RSS | Link To Us | Advertise | Contact Us
Other Related Trellian Services:
Above.com Registrar  |  Above.com Marketplace  |  Above.com Domain Manager  |  Free Search Toolbar  |  Free Webpage Builder  |  Keyword Research
Direct Search Navigation  |  Search Engine Submission and SEO Tools
Copyright © 2002 DomainState.com a Trellian Company 2014 Bronze Sponsor - Internet Commerce Association