SpamAssassin false positives on DATE_IN_FUTURE_96_Q?
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer. I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date. Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.) From topfivestories@messagent.itworldcanada.com Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.com (mail.messagent.itworldcanada.com [207.112.10.80]) by palladium.cs.umanitoba.ca (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE_96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.) Any ideas? -- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
So you've changed the date manually to be exactly the same, and the rule doesn't trigger? On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
From topfivestories@messagent.**itworldcanada.com<topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.**com<http://mail.messagent.itworldcanada.com>( mail.messagent.itworldcanada.**com<http://mail.messagent.itworldcanada.com>[207.112.10.80]) by palladium.cs.umanitoba.ca (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE_* *96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
-- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609 ______________________________**_________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/**listinfo/roundtable<http://www.muug.mb.ca/mailman/listinfo/roundtable>
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.) So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point. Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
>From topfivestories@messagent.__itworldcanada.com <mailto:topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> (mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE___96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
-- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
Google turned up this: http://lists.nongnu.org/archive/html/spamass-milt-list/2010-05/msg00001.html Looks like the problem is spamass-milter's synthesized Received header, rather than the spamassassin rule. Peter On 11/15/2011 10:45 AM, Gilbert E. Detillieux wrote:
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.)
So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point.
Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
From topfivestories@messagent.__itworldcanada.com <mailto:topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> (mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE___96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
Nice Google-fu! :-) On Fri, Nov 18, 2011 at 10:29 AM, Peter O'Gorman <peter@pogma.com> wrote:
Google turned up this: http://lists.nongnu.org/**archive/html/spamass-milt-** list/2010-05/msg00001.html<http://lists.nongnu.org/archive/html/spamass-milt-list/2010-05/msg00001.html>
Looks like the problem is spamass-milter's synthesized Received header, rather than the spamassassin rule.
Peter
On 11/15/2011 10:45 AM, Gilbert E. Detillieux wrote:
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.)
So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point.
Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux
<gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.**ca<gedetil@cs.umanitoba.ca>>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
From topfivestories@messagent.__itw**orldcanada.com<http://itworldcanada.com> <mailto:topfivestories@**messagent.itworldcanada.com<topfivestories@messagent.itworldcanada.com>> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada._**_com <http://mail.messagent.**itworldcanada.com<http://mail.messagent.itworldcanada.com>
(mail.messagent.itworldcanada.**__com <http://mail.messagent.**itworldcanada.com<http://mail.messagent.itworldcanada.com>> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.**umanitoba.ca <http://palladium.cs.umanitoba.ca>> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.**ca <gedetil@cs.umanitoba.ca>>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE_**__96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.**umanitoba.ca<http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
On 2011-11-18 10:29, Peter O'Gorman wrote:
Google turned up this: http://lists.nongnu.org/archive/html/spamass-milt-list/2010-05/msg00001.html
Looks like the problem is spamass-milter's synthesized Received header, rather than the spamassassin rule.
Well spotted, Peter. I guess I hadn't come up on the right sequence of keywords to bring that Google search result into the first page. Yes, that does seem like the bug I've encountered. I noticed that a lot of my false-positives (though not all) were on messages directly sent by authenticated SMTP clients, i.e. where there was no pre-existing Received header. (I didn't realize that spamass-milter was faking one.) My fix for authenticated clients was much simpler and cleaner than what was proposed on that page, however: spamass-milter has a "-I" option that will skip calling spamc/spamd altogether for authenticated SMTP connections! However, the false-positives still remain for non-authenticated clients. At least I have a better idea of where the problem lies, and what to look for in terms of a solution. Thanks, Gilbert
On 11/15/2011 10:45 AM, Gilbert E. Detillieux wrote:
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.)
So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point.
Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
From topfivestories@messagent.__itworldcanada.com <mailto:topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> (mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE___96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
-- Gilbert E. Detillieux E-mail: <gedetil@cs.umanitoba.ca> Dept. of Computer Science Web: http://www.cs.umanitoba.ca/~gedetil/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609
You do know how Google works, right? It gives you results it *THINKS* you want. Personally, I only use Google as a last resort. I find that DuckDuckGo is more efficient. -----Original Message----- Well spotted, Peter. I guess I hadn't come up on the right sequence of keywords to bring that Google search result into the first page.
On 11/19/2011 08:48 AM, Raymond J. Henry wrote:
You do know how Google works, right? It gives you results it *THINKS* you want. Personally, I only use Google as a last resort. I find that DuckDuckGo is more efficient.
They now offer verbatim search: http://insidesearch.blogspot.com/2011/11/search-using-your-terms-verbatim.ht... But yes, it can be annoying when they give you results for something you didn't search for. Peter
I just tried DuckDuckGo, using the same search keywords I had tried under Google to begin with, and this was the result: No results. Try Google or Bing. ;) On 2011-11-19 08:48, Raymond J. Henry wrote:
You do know how Google works, right? It gives you results it *THINKS* you want. Personally, I only use Google as a last resort. I find that DuckDuckGo is more efficient.
-----Original Message-----
Well spotted, Peter. I guess I hadn't come up on the right sequence of keywords to bring that Google search result into the first page.
-- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
I know this is a really old thread, but it popped up in a search as I've been trying to solve the same problem you ran into over 4 years ago. Like you, I used -I in the EXTRA_FLAGS in /etc/sysconfig/spamass-milter, and also like you I've still been getting a lot of false positives on other mail. Some of these DATE_IN_FUTURE tests have really large scores, so getting them wrong really messes things up. I think I found the answer in this bug report, though: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775183 It seems the problem is adding the b macro to confMILTER_MACROS_ENVRCPT, or even to confMILTER_MACROS_ENVFROM, doesn't work reliably. Sometimes sendmail gives the milter the wrong date - the date that sendmail started, rather than the date of the e-mail. That's why the synthesized Received header is in the past, making it appear that the Date header is in the future. The fix seems to be to have the b macro in confMILTER_MACROS_CONNECT alone, and neither of the others. I've done that in my configuration, and so far so good, but it might be too early to tell for sure. It doesn't look like this is only an issue with RHEL5 & clones, as it seems to affect Debian users using a couple different versions of sendmail as well. I don't know if the latest version of sendmail fixes this or not, but I don't see this config tweak as a problem as long as the CONNECT macro handles b consistently. If this does indeed solve the problem, it beats setting all the DATE_IN_FUTURE scores to 0 in /var/lib/spamass-milter/.spamassassin/user_prefs as I had done earlier today in desperation (and since backed out). On 11/18/2011 03:41 PM, Gilbert E. Detillieux wrote:
On 2011-11-18 10:29, Peter O'Gorman wrote:
Google turned up this: http://lists.nongnu.org/archive/html/spamass-milt-list/2010-05/msg00001.html
Looks like the problem is spamass-milter's synthesized Received header, rather than the spamassassin rule.
Well spotted, Peter. I guess I hadn't come up on the right sequence of keywords to bring that Google search result into the first page.
Yes, that does seem like the bug I've encountered. I noticed that a lot of my false-positives (though not all) were on messages directly sent by authenticated SMTP clients, i.e. where there was no pre-existing Received header. (I didn't realize that spamass-milter was faking one.)
My fix for authenticated clients was much simpler and cleaner than what was proposed on that page, however: spamass-milter has a "-I" option that will skip calling spamc/spamd altogether for authenticated SMTP connections!
However, the false-positives still remain for non-authenticated clients. At least I have a better idea of where the problem lies, and what to look for in terms of a solution.
Thanks, Gilbert
On 11/15/2011 10:45 AM, Gilbert E. Detillieux wrote:
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.)
So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point.
Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
From topfivestories@messagent.__itworldcanada.com <mailto:topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> (mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE___96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
-- Gilles R. Detillieux E-mail: <grdetil@scrc.umanitoba.ca> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ Dept. of Physiology and Pathophysiology, Faculty of Health Sciences, Univ. of Manitoba Winnipeg, MB R3E 0J9 (Canada)
Thanks for the tip. I haven't noticed the DATE_IN_FUTURE_96_Q being a problem lately, but I decided to change my milter macros just in case. Gilbert On 23/02/2016 2:15 PM, Gilles Detillieux wrote:
I know this is a really old thread, but it popped up in a search as I've been trying to solve the same problem you ran into over 4 years ago. Like you, I used -I in the EXTRA_FLAGS in /etc/sysconfig/spamass-milter, and also like you I've still been getting a lot of false positives on other mail. Some of these DATE_IN_FUTURE tests have really large scores, so getting them wrong really messes things up.
I think I found the answer in this bug report, though: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775183
It seems the problem is adding the b macro to confMILTER_MACROS_ENVRCPT, or even to confMILTER_MACROS_ENVFROM, doesn't work reliably. Sometimes sendmail gives the milter the wrong date - the date that sendmail started, rather than the date of the e-mail. That's why the synthesized Received header is in the past, making it appear that the Date header is in the future. The fix seems to be to have the b macro in confMILTER_MACROS_CONNECT alone, and neither of the others. I've done that in my configuration, and so far so good, but it might be too early to tell for sure. It doesn't look like this is only an issue with RHEL5 & clones, as it seems to affect Debian users using a couple different versions of sendmail as well. I don't know if the latest version of sendmail fixes this or not, but I don't see this config tweak as a problem as long as the CONNECT macro handles b consistently. If this does indeed solve the problem, it beats setting all the DATE_IN_FUTURE scores to 0 in /var/lib/spamass-milter/.spamassassin/user_prefs as I had done earlier today in desperation (and since backed out).
On 11/18/2011 03:41 PM, Gilbert E. Detillieux wrote:
On 2011-11-18 10:29, Peter O'Gorman wrote:
Google turned up this: http://lists.nongnu.org/archive/html/spamass-milt-list/2010-05/msg00001.html
Looks like the problem is spamass-milter's synthesized Received header, rather than the spamassassin rule.
Well spotted, Peter. I guess I hadn't come up on the right sequence of keywords to bring that Google search result into the first page.
Yes, that does seem like the bug I've encountered. I noticed that a lot of my false-positives (though not all) were on messages directly sent by authenticated SMTP clients, i.e. where there was no pre-existing Received header. (I didn't realize that spamass-milter was faking one.)
My fix for authenticated clients was much simpler and cleaner than what was proposed on that page, however: spamass-milter has a "-I" option that will skip calling spamc/spamd altogether for authenticated SMTP connections!
However, the false-positives still remain for non-authenticated clients. At least I have a better idea of where the problem lies, and what to look for in terms of a solution.
Thanks, Gilbert
On 11/15/2011 10:45 AM, Gilbert E. Detillieux wrote:
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.)
So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point.
Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
From topfivestories@messagent.__itworldcanada.com <mailto:topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> (mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE___96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
-- Manitoba UNIX User Group E-mail: <gedetil@muug.mb.ca> c/o Gilbert E. Detillieux Web: http://www.muug.mb.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609
It occurred to me that even if this change DIDN'T work, I wouldn't start seeing false positives again until 3 hours after restarting sendmail, because the first DATE_IN_FUTURE rule is 03_06 which kicks in when there's a discrepancy of at least 3 hours (i.e. one more hour from now, as I restarted sendmail 2 hours ago). The 96_Q won't kick in until Saturday. So I'll have to monitor things over the course of the next week or so. Before changing the config & restarting sendmail, I was getting 1-3 of these 96_Q false positives a day in my own e-mail, with sendmail having started 34 days ago at the last reboot. Those would be the most common false-positives because of the large range involved (96-2920 hrs or 4 days-1 quarter), and they stand out because of the high score (3.199). Only 03_06 has a (slightly) higher score (3.399). On 02/23/2016 02:19 PM, Gilbert Detillieux wrote:
Thanks for the tip. I haven't noticed the DATE_IN_FUTURE_96_Q being a problem lately, but I decided to change my milter macros just in case.
Gilbert
On 23/02/2016 2:15 PM, Gilles Detillieux wrote:
I know this is a really old thread, but it popped up in a search as I've been trying to solve the same problem you ran into over 4 years ago. Like you, I used -I in the EXTRA_FLAGS in /etc/sysconfig/spamass-milter, and also like you I've still been getting a lot of false positives on other mail. Some of these DATE_IN_FUTURE tests have really large scores, so getting them wrong really messes things up.
I think I found the answer in this bug report, though: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775183
It seems the problem is adding the b macro to confMILTER_MACROS_ENVRCPT, or even to confMILTER_MACROS_ENVFROM, doesn't work reliably. Sometimes sendmail gives the milter the wrong date - the date that sendmail started, rather than the date of the e-mail. That's why the synthesized Received header is in the past, making it appear that the Date header is in the future. The fix seems to be to have the b macro in confMILTER_MACROS_CONNECT alone, and neither of the others. I've done that in my configuration, and so far so good, but it might be too early to tell for sure. It doesn't look like this is only an issue with RHEL5 & clones, as it seems to affect Debian users using a couple different versions of sendmail as well. I don't know if the latest version of sendmail fixes this or not, but I don't see this config tweak as a problem as long as the CONNECT macro handles b consistently. If this does indeed solve the problem, it beats setting all the DATE_IN_FUTURE scores to 0 in /var/lib/spamass-milter/.spamassassin/user_prefs as I had done earlier today in desperation (and since backed out).
On 11/18/2011 03:41 PM, Gilbert E. Detillieux wrote:
On 2011-11-18 10:29, Peter O'Gorman wrote:
Google turned up this: http://lists.nongnu.org/archive/html/spamass-milt-list/2010-05/msg00001.html
Looks like the problem is spamass-milter's synthesized Received header, rather than the spamassassin rule.
Well spotted, Peter. I guess I hadn't come up on the right sequence of keywords to bring that Google search result into the first page.
Yes, that does seem like the bug I've encountered. I noticed that a lot of my false-positives (though not all) were on messages directly sent by authenticated SMTP clients, i.e. where there was no pre-existing Received header. (I didn't realize that spamass-milter was faking one.)
My fix for authenticated clients was much simpler and cleaner than what was proposed on that page, however: spamass-milter has a "-I" option that will skip calling spamc/spamd altogether for authenticated SMTP connections!
However, the false-positives still remain for non-authenticated clients. At least I have a better idea of where the problem lies, and what to look for in terms of a solution.
Thanks, Gilbert
On 11/15/2011 10:45 AM, Gilbert E. Detillieux wrote:
On 2011-11-14 17:46, Kevin McGregor wrote:
So you've changed the date manually to be exactly the same, and the rule doesn't trigger?
Well... Here's the weird thing: if I pass the exact same message through spamc manually, I don't get the false positive on that rule. So, I tried mailing that message back to myself from a non-local mailer (so that it goes through spamass-milter again), but this generates extra "Received" headers that change the behaviour. (I now get a trigger on the DATE_IN_PAST_24_48 rule, since the message is now that old.)
So, I can't test under exactly the same conditions. Given that running the message through spamc manually didn't trigger the rule, I'm tempted to think it might be something in the spamass-milter configuration, which is causing some information to not be transferred to spamc, or to be transferred incorrectly. Not sure at this point.
Gilbert
On Mon, Nov 14, 2011 at 4:56 PM, Gilbert E. Detillieux <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>> wrote:
I mentioned this problem at the last round-table session, but didn't get a solution, so I thought I'd post it here, just in case anyone has any suggestions to offer.
I'm still seeing a whole bunch of false positives in SpamAssassin, since an update was installed in mid-September on a CentOS 5.7 system, for a rule called DATE_IN_FUTURE_96_Q, which is only supposed to be triggered when the "Date:" header has a date that is 4 days to 4 month ahead of the date in the "Received" header that has the _smallest_ difference in date.
Here are the headers from the latest e-mail I've received with this false-positive. (I've stripped out irrelevant headers, for the sake of clarity and simplicity.)
>From topfivestories@messagent.__itworldcanada.com <mailto:topfivestories@messagent.itworldcanada.com> Mon Nov 14 07:50:13 2011 Received: from mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> (mail.messagent.itworldcanada.__com <http://mail.messagent.itworldcanada.com> [207.112.10.80]) by palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca> (8.13.8/8.13.8) with SMTP id pAEDoAxV028594 for <gedetil@cs.umanitoba.ca <mailto:gedetil@cs.umanitoba.ca>>; Mon, 14 Nov 2011 07:50:12 -0600 Date: Mon, 14 Nov 2011 08:50:13 -0500 X-Spam-Status: No, score=-0.3 required=5.0 tests=BAYES_00,DATE_IN_FUTURE___96_Q, HTML_MESSAGE,RP_MATCHES_RCVD autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on palladium.cs.umanitoba.ca <http://palladium.cs.umanitoba.ca>
Note that I'm calling spamd via the spamass-milter on a system running sendmail. Note also, that in the above example, the only "Received" header was the one generated by my own server. (I've had other false positives, however, with multiple "Received" headers, all of which were within seconds of the time in the "Date" header.)
Any ideas?
-- Gilles R. Detillieux E-mail: <grdetil@scrc.umanitoba.ca> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ Dept. of Physiology and Pathophysiology, Faculty of Health Sciences, Univ. of Manitoba Winnipeg, MB R3E 0J9 (Canada)
participants (6)
-
Gilbert Detillieux -
Gilbert E. Detillieux
-
Gilles Detillieux -
Kevin McGregor -
Peter O'Gorman -
Raymond J. Henry