Re: [gentoo-dev] Anti-spam changes: proposal to drop spammy mail

[Niels Dettenbach]

Am Montag, 11. Mai 2015, 20:36:18 schrieb Robin H. Johnson:
> There are people that still accept mail that violates standards?
yes,
and there are mail sites and/or mail clients sending standard violating emails.

But the more truth is that there are many points within standards which are interpreted differently from different peoples / groups (or even mailer software developers) and there is no real clear / hard "border" what is a violence (and "could be dropped") and what is not - at least if you did not want to loose ham traffic for your users.

The email oecosystem does not dpend from a single RFC today - more and more basic parts of existing internet mail and it's features are defined in further RFCs or are conclusive from each other.

Two very typical examples:

1.) The sender domain has no MX nor abuse contact (i.e. RFC 2142)
Many pro level mass mailers do not have an "working" abuse contact, but there are still many smaller sites out which doesnt have too (because of limited DNS access or lack of knowledge). Dropping mail from such sites will lead you to loosing mails (even if it "just" hits one in thousand ham mails).

2.) BCC Header
Most Mailers today are filtering out BCC recipient headers at some point while this is not defined in the RFCs and still discussed hardly how far the deletion of BCC headers are breaking standards, resulting in possible lost of emails. See i.e. Phillip Hazels (EXIMs) statements in the net.


> My above statement is for mail that we ACCEPTED. If it violates
> standards, it's already denied at SMTP time.
hmmm,
you mean some more to very basic points of the standards.
 
> smtpd_restriction_classes = restrictive,permissive
> restrictive =
>     reject_invalid_hostname
>     reject_non_fqdn_hostname
>     reject_non_fqdn_recipient
>     reject_non_fqdn_sender
>     reject_unknown_sender_domain
>     reject_unknown_recipient_domain
>     check_sender_mx_access cidr:/etc/postfix/bogus_mx_records
>     check_sender_access pcre:/etc/postfix/sender_access_control.pcre
>     check_sender_access pcre:/etc/postfix/sender_access_control-aliases.pcre
> check_helo_access pcre:/etc/postfix/helo_checks
>     reject_unverified_sender
>     check_client_access cidr:/etc/postfix/filter.cidr
>     permit
> permissive =
>     permit


If it helps you a bit further, i can explain the basics of our setup, developed over nearly 20 years now, handling just a few hundredthousands smtp sessions per day and having NO spam folder or similiar (which would not save any time of the email user at the end) - but easily could be ad[a|o]pted per i.e. SIEVE to lead out less hard / more "unclear" spam into folders (i.e. instead of that mail where we make greylisting usually).

Because sendmail and postfix was to ressource inefficient for us sometimes in the early stages, we decided to go to EXIM (Phillip Hazel) - an own build optimized for our needs - including even some own mods today.

We avoided running SA from Amavis because of inefficiency.

Until today our incoming path goes:

 - EXIM with EXIM SA at SMTP time

Means we use Spamassassin directly at SMTP time, which allows us to dynamically "react" or further actively investigate a incoming smtp session if required. SA is only invoked for non authenticated mail over network btw..

Before exim contacts spamassassin at this stage, we run a bunch of checks in EXIM similiar to yours above, but some more (see down) which drop the connection or write data for further processing into the headers. If the connection is still alive, we run a hand full of RDNSBL checks, which "could reject" the session and then a hand full which just writes warnings into headers plus data for further processing steps.

If the sessions still "lives", EXIM contacts Spamassassin over socket and 

Here we have 3 "routes":

	- low spam -> Mails is going trough DIRECTLY
more then SA 2.3/3.0 - possible spam -> Greylisting (3 times TEMP Reject)
more then SA 5.2 - spam -> REJECT
more then SA 33.0 - blackhole

-> This kind of REJECT hits around 5 - 10% of spam connections, all other spam is usually catched before without the full email / mail body recieved.

Greylisting is "remembering" each contact<->contact handle and "quasi whitelists" the sender email after greylisting once to avoid further delays in the future - this helps very well for mail sites and/or clients which uses mail systems with bad reputation while "working OK".

SA EXIM is able to do teergrubing as well, but we did not use it in most situations - except partly in dictionary attacks.

At this point, parts of the mail traffic is going to an AMAVIS-NG for virus filtering only (user decide for it byself here) - no SA or RDNSBL again / at this place.

EXIM SA is no longer maintained officially, so we maintain it byself into actual EXIM source trees (would be nice to get it into Gentoos EXIM ebuild - i.e. by a USE flag - would help here if someone is interested - and if someone has a newer, at least same efficient solution it would be nioce to know).
 


Overview of EXIM checks of incoming SMTP sessions (parts of this are implemented in your postfix rules too):
--- snip ---

= HELO/EHLO required by SMTP RFC  See http://www.syndicat.com/faq/email/no_helo/
= Forged IP detected in HELO (it's mine) - $sender_helo_name  See http://www.syndicat.com/faq/email/forged_ip/
= Forged IP detected in HELO: $sender_helo_name
= Forged IP detected in HELO - $sender_helo_name != $sender_host_address  See http://www.syndicat.com/faq/email/forged_ip/
= Forged hostname detected in HELO - you are not $sender_helo_name See http://www.syndicat.com/faq/email/forged_ip/
= HELO is our IP
= $sender_helo_name is a silly HELO.
= RFC 1918 IP address in HELO ( See http://www.syndicat.com/faq/email/rfc1918-helo/ )
= $sender_address_domain is a silly domain. (i.e. localhost)
= HELO should be hostname but is $sender_helo_name . ( See http://www.syndicat.com/faq/email/helo_nohostname/ )
= HELO should be Fully Qualified Domain Name Host.Domain.Tld ( See RFC821 or http://www.syndicat.com/faq/email/helo_nofqdn/ )
= Forged hostname detected in HELO - $sender_helo_name is one of our domains
= Only one recipient accepted for NULL sender
= (DROP) too many unknown users (${eval:$rcpt_fail_count+1} failed recipients)
= Dictionary attack (${eval:$rcpt_fail_count+1} failed recipients).
=> Teergrube: dictionary attack (${eval:$rcpt_fail_count+1} failed recipients)
= unknown user
= X-Broken-Reverse-DNS: no DNS for IP address $sender_host_address
= acl_mail: (WARN-ONLY) Cannot reverse DNS $sender_host_address
= X-Broken-Reverse-DNS: no DNS for IP address $sender_host_address
= Content Policy Restriction: Mails to undisclosed recipients are not permitted.
= No contact MX - rfc-ignorant host $sender_host_name $sender_host_address . ( See http://www.syndicat.com/faq/email/rfc_ignorant/ )
= (WARN-ONLY, no reliable check possible) No MX abuse contact - rfc-ignorant host $sender_host_name $sender_host_address . ( See http://www.syndicat.com/faq/email/rfc_ignorant/ )
--- snap ---

then

RDNSBL:

deny = sbl-xbl.spamhaus.org : cbl.abuseat.org : zen.spamhaus.org : b.barracudacentral.org : psbl.surriel.com : ix.dnsbl.manitu.net
warn = dnsbl-3.uceprotect.net : ubl.unsubscore.com : dnsbl-1.uceprotect.net : dnsbl.sorbs.net


hth a bit.


cheerioh,


Niels.
-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---