This is a fun breakdown of traffic and behaviors :)
One potential improvement: using reverse DNS to check the Agent user-agent-string. User Agent strings are unauthenticated, so can't be attributed to a specific crawler without a check.
I am working on a similar breakdown of traffic understanding, and have found a small but interesting amount of 'forged' agent strings. Mostly from the bots doing 404 knocks for pages that don't exist (vulnerability scanning behavior).
This is a fun breakdown of traffic and behaviors :)
One potential improvement: using reverse DNS to check the Agent user-agent-string. User Agent strings are unauthenticated, so can't be attributed to a specific crawler without a check.
I am working on a similar breakdown of traffic understanding, and have found a small but interesting amount of 'forged' agent strings. Mostly from the bots doing 404 knocks for pages that don't exist (vulnerability scanning behavior).