The case of the missing User Agent

I recently received an email form someone attempting to access one of my client's websites, and at first I was baffled to see what the problem was.

One of the first features of the integrated cookie-tracking system developed by MJB Data was a filter that attempts to block out spurious visitors, such as automated email address harvesters sometimes known as spam-bots.

The User Agent that is sent as part of the headers of a request for a web-page can easily be faked and data gatherers who know what they're doing can easily masquerade as one of the well-known browsers. However, filtering out known spam-bots is still worthwhile.

The MJB system also filters out requests that have no User Agent at all as under normal conditions all the major browsers identify themselves whenever they make a request.

After seeing that someone using Internet Explorer had been blocked from a website for have a blank User Agent I naturally checked my systems, and had the visitor land on a test page a few times, but the cause remained allusive until a few days later when the visitor realised that the culprit was, (drum roll), Zone Alarms which has decided to strip out the User Agent from all HTTP requests being made, presumably as part of a privacy setting.

Private browsing

I can understand why some people might prefer to remain as anonymous as possible when clicking about on the Internet, and to block some cookies, pop-up windows and so on, but I don't really see how removing the User Agent helps anybody. I what way does providing your browser's User Agent relate to security or privacy?

I suppose that if you're using a version of Internet Explorer that is known to have security flaws, then removing the UA might help, and I can only assume that this is a rather blunt security measure of some sort.