[geeks] Spam filter test results

[Chris Byrne]

Okay a couple of weeks ago I started testing two spam filters for those
of us who are afflicted with winblows. I ran each one for as long as I
could stand it. 

I had been using the built in spam filters of MS Outlook. They are based
on a blacklist/whitelist, and word matching. They take basically no
effort to enable, but you do have to keep the word list updated. Spam
filtering tends to get worse/less effective over time because the major
spammers know outlooks word matching scheme and write their messages to
get around it. 

Spam caught: less than 50%
False positives: 0
Overall: Why Bother?

Basically the only good thing was that I didn't have any false
positives. 

The first third party filter I tried was Spam Bully. This is demoware
with a two week trial period and costs $25 to register. The demo is
fully functional. 

Spam caught: over 95%
False positives: several per day
Overall: Don't even think about it

Spam bully was very good at catching spam from the moment I installed
it. It automatically adds words to its analysis table as you delete the
emails, and it seems to have a good prexisting statistical base. It was
well over 90% effective at catching spam. It also had a neat little
charting system to show you how many emails you got, and how many were
spam, broken down by hour, day, week, and month. Cool. 

There were a few major problems though. It caused outlook to slow down
at least a bit at all times, and a hell of a lot whenever there were
messages with complex formatting. It would sometimes render the preview
pane non functional. It used a hell of a lot of memory over and above
what outlook already uses. But the worst problem was the number of false
positives. It falsely identified a lot of messages as spam. As in
several per day, including a very important contract related email.
After four days I removed it because it was just not acceptable. 

So then I installed spam asassin. It's freeware/donationware with no nag

Spam caught: over 90%
False positives: 0
Overall: Get it now

Spammunition didn't catch quite as much spam as Spam bully did but in
every other way it was better. Outlook wasn't slowed at all by it,
though it did use about 10 megs more memory. According to the developers
site he'll be ctting the amount of memory used even more in the next
version.It doesn't have any statistics, but those are apparently coming
in a later version as well.

The biggest disads to spammunition are with how it builds the spam
analysis tables. You have to have a fairly large body of spam before it
becomes effective (at least 20 and its much better with at least 100).
The current version requires you to keep a large email archive, both
good and spam, because it reanalyzes the whole body of email every time
you hit the analyze button. The more email you have, the more accurate
the filter is. Once again, that is scheduled to be fixed in the next
version, which should be out in a few weeks.

The biggest advantage to spammunition was that I havent seen a single
false positive all week. Not one. Yes you trade off a bit of spam
catching effectiveness, but that's a hell of a lot better than losing
email. 

Christopher Byrne IV