SpamAssassin on cPanel2

[Roger]

A consequence (unintended I presume) of the recent disk failure and recovery on cPanel2 (http://www.hostassist.co.uk/forum/showthread.php?p=724#post724) and the software upgrade implemented as part of the recovery is that SpamAssassin has ben disabled on, as far as I can see, all of my accounts.:(

At the same time I think the available features have changed; spam box has gone (this put emails identified as spam into a separate folder) but boxtrapper (a challenge/response system) has been introduced (I think it used to be on the control panel but disabled.)

Could Rob comment on this? Am I right? Would it be possible to restore SpamAssassin settings to how they were before this incident? (I have raised ticket ID: 20070104-1258-73 for that one.) Is boxtrapper a good thing?

[othelloRob]

As far as we can tell, all the files that were there from the backups have been restored.

There have been a couple of permissions issues, where files uploaded by secondary FTP users have now been assigned "ownership" by the main user of the account, which we are fixing as they are reported.

We cannot find anything to indicate SA *shouldn't* be doing exactly what it did before.

A temporary fix is for you to disable SA and then re-enable it for your account(s). The default on the new setup is for SA to be on for all new users.

BoxTrapper has been there for a while but disabled as it was buggy, I personally think the *idea* behind C-R systems is fundamentally flawed, but some users seem to like it.

We are looking into the spambox problem(s).

[Roger]

In a private email to me you said:spamassasin is controlled by the presence of a .spamassassinenable file in the home directory of the cpanel user and this file has appeared in the home directory of all my accounts, with a timestamp of 17:25 today, so that seems to have been fixed. Spambox is there again too (or did I just overlook it?).

I tend to agree with you about Boxtrapper and other challenge-response systems, given the prevalence of spammers forging email headers the result is that each spam message generates a challenge to annoy the victim whose address has been forged, and maybe a response from the ISP if the email account does not exist. So -- a tripling of email traffic, not a good idea.

[othelloRob]

In a private email to me you said: and this file has appeared in the home directory of all my accounts, with a timestamp of 17:25 today, so that seems to have been fixed. Spambox is there again too (or did I just overlook it?).

No, that was Sam, who acted on your support ticket, and manually recreated it for all cpanel directories on your account :D

[othelloRob]

Spambox is there again too (or did I just overlook it?).


SpamBox we've not touched as yet, so its *possibly* not an issue now SA has been sorted out. SpamBox will only be available where SA is enabled for an account and active.

[Roger]

No, that was Sam, who acted on your support ticket, and manually recreated it for all cpanel directories on your account :D
Then thanks to Sam!:)

[othelloRob]

We will shortly be offering an improved version of our corporate mail hosting service. The ever increasing levels of junk, malware and spam are getting to the point where email is no longer as effective as a communications medium as people expect.

The new service will be using a heavily customised combination of:
spamassassin - heuristic spam checking
mailscanner - preparsing of attachments and removal of javascript html
clamav - basic virus checking and removal of incoming files
fprot - realtime enterprise class virus and malware scanning of the servers
vipuls razor - known spam catalogue checks
rzagent consolidation - spam identified on one mailwasher automatically blocked from all of them
adaptive firewalling - automatically blocking ips from the mailsevers which are sending mail to multiple invalid recipients
reverse ip tests - check if the mailserver relaying the mail to us should have received it in the first place
smtp checkback - does the sending machine accept connection to receive mail for the address its sending "from"
escalating tarpit - pausing incoming smtp mail from the same source at increasing intervals to deter spammers

We have this in testing at the moment.

On an "open" catch-all email address yesterday it denied 1804 smtp connections, blocked 2873 spams, allowed 21 emails through, and sadly stopped 2 valid emails, which we are examining to see what rules they triggered and what to change.

On Monday the same domain where mail was delivered to server Plesk7 the account received 5148 emails of which 588 were delivered, the rest tagged as spam by SA.

We hope to have this available as a product for resellers to sell on as well as for our direct clients and our use, however have not yet tried to "front end" it - the ultimate aim is to have this running as an optional "pre-washer" system to all the existing systems, so if your client wants massivley improved spam and virus protection, you simply add their domain or even just certain email addresses into the new mail system, change the MX records to point mail at the new system and voila !

The issues still to resolve are:
false positives - ongoing tweaking of the rules and software to ensure we dont stop valid emails
scalibility - can the systems run in load-balanced parallel states so increases in mail loads are simply a case of adding another box
failover - can mail still be delivered if the service fails for whatever reason
user management - adding each email address to protect is a pain, however there is no obvious way to junk all the mail to invalid addresses unless it knows what the valid adresses are
price - should it be priced by mailbox, by domain, or a fee based on how much it blocks
quarantining - daily reports of mail stopped and the ability for users to collect stoped mail and attachments
anonymity - making the service transparrent to the very end user

None of this belays our commitement to providing as much in the way of virus, spam and malware protection on each hosting server as standard, however the options are always limited by the licencing, o/s integration, control panel subsytems etc, so an "appliance" in front of that seems like the way forward.

With over 3,500,000 incoming emails/day, and (guestimate) 1% of that being possibly valid mail, the cost should be ofset by the time saved wading through the cr@p that clogs up your mailbox ... currently we're thinking of £10 - £12 /month/domain as a price point.

[Roger]

I spend more time than I care to think about dealing with spam, even though I personally have now got SpamAssassin to delete emails automatically if it thinks that they are spam. So any move to reduce its volume must be welcome (complete elimination has to be an unattainable ideal). So please view these comments in that light:

What I could not discern was the number of spam messages that got past the anti-spam cocktail you are testing. If people are paying to stop spam, they will expect it to be much reduced from present levels.

I suspect that many of my clients, who are writers and other creative people and generally sole traders, are on some kind of consumer broadband package and paying in the region of £15-£25 a month for it. Most of them are paying me between £100 and £1000 a year to keep their websites up-to-date. I am not sure how many of them will want to pay an additional fee approaching £200 a year, I think many will keep on deleting spam as they do at present. But I can see the attractiveness of this for larger corporations if it is priced on a per-domain basis.

I have reservations about the smtp checkback. At present I route all my outgoing emails via smtp.btinternet.com but the "from" and "reply-to" addresses are @cornwellinternet.co.uk or @cornwell.org.uk so emails from people like me would no doubt fail this test. Emails sent out by phpList are sent out from nobody@cpanel2.uk.othellotech.net but the reply-to and from addresses are in the domain owning the newsletter.

I appreciate your sharing your thoughts about this service that's still under development, and I hope these comments will help you to fine-tune it and I also hope that others will give you feedback since I suspect I may not be typical of your customer base.

[othelloRob]

What I could not discern was the number of spam messages that got past the anti-spam cocktail you are testing. If people are paying to stop spam, they will expect it to be much reduced from present levels.


Sorry if I missed a vital piece of the statistics - number of spams deliveredto the mailbox - None (i had expected a few to get through) - although as i said, 2 valid emails also got blocked, so by relaxing the rulesets and rule changes to allow those, may well allow mails which had a similar set of characteristics to get through as well.


I am not sure how many of them will want to pay an additional fee approaching £200 a year, I think many will keep on deleting spam as they do at present. But I can see the attractiveness of this for larger corporations if it is priced on a per-domain basis.


Indeed, whilst I would love to be able to roll this out to all hosted clients at no additional cost, that would add about £230,000 in licences and hardware to our outgoings, and *somehow* would affect the pricing - circa £70/year/customer.

For example the the content filter scanning licence is about $1/emailaddress/year - whilst it doesnt sound a lot when said like that, even on our entry level hosting plans that would on average add $7 (£4) to everyones annual hosting bill (12% increase on our starting plan prices).


I have reservations about the smtp checkback. At present I route all my outgoing emails via smtp.btinternet.com but the "from" and "reply-to" addresses are @cornwellinternet.co.uk or @cornwell.org.uk so emails from people like me would no doubt fail this test. Emails sent out by phpList are sent out from nobody@cpanel2.uk.othellotech.net but the reply-to and from addresses are in the domain owning the newsletter.


From and Reply-To are *notational* parts of an email header, and have no direct relationship to the email envelope data and actual sender - exactly as you show, both are checked for validity

I'm not the best person to explain the code, and i'll pass the comments on to Adam to see if the BT example would pass or fail - although I'd be expecting you to relay through the mailserver for your domain, not your ISPs

partial answer technical stuff
Its checking the values in the received lines in the header which get faked or are invalid...

Received: from 195.130.132.43 (neoplus.adsl.tpnet.pl)
fails because the IP given by the mail server in the header (in BE) and the reverse lookup of the actual machine (in PL) resolve to different netblocks entirely - looking at the other headers shows an envelope address in Belgium, so that poor sod is getting all the bounces from the Polish spammer.

Received: from nobody by sm16.lucidityhosting.com with local (Exim 4.63)
From: <00031S@amaeryamsmaa.com>
fails because the domain amaeryamsmaa doesnt accept mail for user 00031S by trying an smtp callback connection to the MX records for that domain.
This is probably abuse of a php script hosted at lucidity - although looking the domain being advertised it is also hosted at lucidity - so most likely one of their adult hosting customers spamming the world to get more visitors to their porn site.

Received: from name-eb5dc284be (pD9E22D7F.dip0.t-ipconnect.de [217.226.45.127])
by spam-man.othellotech.net (8.10.2/8.10.2) with ESMTP id
Received: from 80.65.238.153 (HELO mail.abysse.net)
by othellotech.net with esmtp (D98E7-K5,0Z ?-IZL)
id <'GM70-?>J48Z-/9
fails because the first mail exchange (2nd Received shown) header line is obviously fake - the first server is not the one that passed the message onto the 2nd server - this is a German adsl user claiming to be othellotech.net and sending out spam with us as the from/reply, so all the undeliverables and abuse complaints will come to us.


HTH :D