Bing, Yandex, Baidu, Google, Facebook,
Sogou, Twitter and Yahoo all do the following; they respect and play
by the rules when it comes to crawling sites. When these companies
wish to crawl your site, they ask for permission via a request to
your robots.txt. These companies respect the sites rules and follow
them. Each of these companies identify their crawler with published
user agents. These known agents alerts to their identity and intent.
These user agent are useful determining if the crawlers seeks the
mobile or desktop version of your site. Of course, these user agents
are easy to spoof, which is why each of these companies backs their
user agent with an rdns entry confirming their identify.
Intel Security / McAfee works by their
own set of rules and as a security company pretty much acts like a
rogue agent attacking your site. Intel/McAfee does not respect
robots. Intel/McAfee uses a generic user agent that is common from
bad actor states like the Ukraine, Russia and many countries in Asia;
“Java/1.8.0_51” There is no way to distinguish a rogue crawler
from Intel/McAfee. After much research we have determined that
Intel/McAfee use IP's assigned to such names as pathdefender, or
Digital Ocean. How many strikes does a webmaster need to block
Intel/McAfee as rogue traffic.
After receiving unwanted traffic from
the domain 104.131.0.0 we blocked all of Digital Ocean. Standing up
a new site using our common blocked traffic database, Intel/McAfee
was blocked as a bad actor in this domain, failing to identifying
themselves and using user agents that are known for attacks, they
were justifiably blocked.
Here is the rub and where the hubris
and true arrogance of this purported security company surfaces.
Intel/McAfee is blocked from access to our established sites, yet
denied access, they labeled our new site as suspicious, they said
they scanned our site and found viruses and malware and they
presented a highly inflammatory warning to go back to safety; a load
of excrement. The fact is we blocked them, so the message about
the malware and viruses they found was a flat out lie.
Through inference and trial and error
we determined who was behind the rogue crawls from 104.131.94.252 and
72.5.66.72 with known hacker user agents, spurious DNS; IP's we had blocked as
bad actors; Intel! We had no choice but to succumb to Intel/McAfee's
extortion; either we allow their rogue agents from unknown IP's onto
our site and install their site seal, or they would block our site.
Pay up or get blocked, we had no choice but to allow their rogue
traffic and install their site seal; extortion.
We find Intell/McAfee to be bad actors
that do not properly identify themselves, and if you do not allow
their rogue agents onto your site, they will maliciously and without
merit label your site as dangerous when in fact they are the problem.
We urge Intell/McAfee to follow the
example of Google, Microsoft, Yahoo, Twitter, Baidu, Facebook and
Sogou and establish and publish a branded user agent, then back it up
with published and identifiable ip addresses via rdns. As security
company, shame on you Intel for acting like the very actors you
purport to block with your security software.