Enhancement #2954
Avoid fetchmail bounces
Status: | CLOSED | Start date: | ||
---|---|---|---|---|
Priority: | Normal | Due date: | ||
Assignee: | - | % Done: | 100% | |
Category: | nethserver-fetchmail | |||
Target version: | v6.5 | |||
Resolution: | NEEDINFO: | No |
Description
Messages from fetchmail are already delivered to a remote mailbox so their envelope sender is lost. Bouncing such messages is a problem if the sender address does not exist or is forged.
Also a bounce is generated if the destination mailbox does not exist, i.e. if a user has been deleted or similar. To avoid such bounces we must bypass the Postfix queue and talk directly to Dovecot LMTP socket, by means of Amavis. This architecture does not have a queue so Fetchmail knows immediately if a message has been delivered or not and log its activity appropriately.
[RemoteSystem] v v POP/IMAP v v [Fetchmail] >>SMTP>> [127.0.0.100:10024 Amavis] >>LMTP>> [/var/run/dovecot/lmtp Dovecot]
Optionally, if a message is not accepted by Amavis, Fetchmail could bounce a report to the local postmaster account.
Related issues
Associated revisions
/etc/sysconfig/amavisd is now a template. Refs #2954
Start amavisd with vmail additional group. Refs #2954
The vmail group is required to allow amavis talk directly with Dovecot
LMTP server. This happens with fetchmail.
Submit messages to local amavisd at 127.0.0.100:10024. Refs #2924
- Require nethserver-mail-common > 1.4.0-4
Applied word wrap to pop3_connector module documentation. Refs #2954
Added default Postfix mydestination entries to inbound domain list. Refs #2954
Fixes missing spam headers problem, for messages from Fetchmail.
See note 11.
Validate accounts on fetchmailrc template expansion. Refs #2954
Restart fetchmail conditionally when account changes. Refs #2954
Monitored events are:- user-delete
- user-modify
- group-delete
- group-modify
Note: AD accounts can't be monitored. Fetchmail accounts must be
updated manually.
Warning for Active Directory users. Refs #2954
History
#1 Updated by Davide Principi over 6 years ago
- Status changed from TRIAGED to ON_DEV
- Assignee set to Davide Principi
- % Done changed from 20 to 30
#2 Updated by Davide Principi over 6 years ago
- Related to Enhancement #2924: Fetchmail support for AD users added
#3 Updated by Davide Principi over 6 years ago
- Status changed from ON_DEV to MODIFIED
- Assignee deleted (
Davide Principi) - % Done changed from 30 to 60
Note
- Read the test cases. I suggest executing 2 before 1.
- Test Virus and Spam messages are available from
amavisd-new
package. Execute the following commands:mkdir amavis-test cd amavis-test perl -pe 's/./chr(ord($&)^255)/sge' </usr/share/doc/amavisd-new-2.8.0/test-messages/sample.tar.gz.compl | zcat | tar xvf -
Test case 1 - fresh install
In a clean NethServer 6.5 update base packages.
yum --enablerepo=nethserver-testing install nethserver-fetchmail nethserver-mail-filter
- Create an user account
first.user
- In POP3 connector page configure an IMAP or POP account where to fetch messages for first.user
- If a message is delivered to
first.user
you must find in /var/log/fetchmail.logNov 18 15:31:05 [info] 12 messages (11 seen) for davidep2 at nethservice.nethesis.it (15855 octets). Nov 18 15:31:06 [info] reading message davidep2@nethservice.nethesis.it:12 of 12 (7391 octets) not flushed (...or flushed)
In /var/log/maillog:Nov 18 15:31:05 davidep2 dovecot: lmtp(25995): Connect from local Nov 18 15:31:06 davidep2 dovecot: lmtp(25995, first.user): X3IAMClYa1SLZQAA4KLHlg: sieve: msgid=<546b581b.l+j9LuDuzaR2ebOZ%davide.principi@nethesis.it>: stored mail into mailbox 'INBOX' Nov 18 15:31:06 davidep2 dovecot: lmtp(25995): Disconnect from local: Client quit (in reset) Nov 18 15:31:06 davidep2 amavis[25182]: (25182-01) Passed CLEAN {RelayedOpenRelay}, FETCHMAIL [192.168.5.252] <davide.principi@nethesis.it> -> <first.user@localhost>, Message-ID: <546b581b.l+j9LuDuzaR2ebOZ%davide.principi@nethesis.it>, mail_id: qz3X6CmDCqIo, Hits: -, size: 7543, queued_as: 250 2.0.0 <first.user@localhost> X3IAMClYa1SLZQAA4KLHlg Saved, 806 ms
You must not see any Postfix message. - If a message is SPAM it is discarded by amavisd:
In /var/log/maillog: Nov 18 16:20:44 davidep2 amavis[26857]: (26857-01) Blocked SPAM {DiscardedOpenRelay,Quarantined}, FETCHMAIL [192.168.5.252] <davide.principi@nethesis.it> -> <first.user@localhost>, Message-ID: <546b639f.2JA24Xqk2Rj1qm4d%davide.principi@nethesis.it>, mail_id: d0jN0QqJoVwN, Hits: 999.999, size: 1707, 5128 ms In /var/log/fetchmail.log: Nov 18 16:20:44 [info] reading message davidep2@nethservice.nethesis.it:16 of 16 (1555 octets) not flushed
Amavis accepts the message then discards it silently. - If a message is tagged SPAMMY it must be delivered to junkmail folder (if enabled).
Dec 3 11:39:18 davidep2 dovecot: lmtp(17334, first.user): EJf3GiToflS2QwAAoK0gTQ: sieve: msgid=<N1msdrbJXNPfV4wg9>: stored mail into mailbox 'junkmail' Dec 3 11:39:18 davidep2 dovecot: lmtp(17334): Disconnect from local: Client quit (in reset) Dec 3 11:39:18 davidep2 amavis[17306]: (17306-01) Passed SPAMMY {RelayedTaggedInbound}, FETCHMAIL [63.10.249.142] <testx@spammers.gov> -> <first.user@vboxnet0.tld>, Message-ID: <N1msdrbJXNPfV4wg9>, mail_id: 23ntI-0s95qC, Hits: 8.361, size: 4800, queued_as: 250 2.0.0 <first.user+spam@vboxnet0.tld> EJf3GiToflS2QwAAoK0gTQ Saved, 6314 ms
- If a message is VIRUS it is rejected by amavisd:
In /var/log/maillog: Nov 18 16:23:23 davidep2 amavis[26860]: (26860-01) Blocked INFECTED (Eicar-Test-Signature.UNOFFICIAL) {RejectedOpenRelay,Quarantined}, FETCHMAIL [192.168.5.252] <davide.principi@nethesis.it> -> <first.user@localhost>, Message-ID: <546b6463.3KQWNOuvtic9jPJ0%davide.principi@nethesis.it>, mail_id: v8g99-gBhtiF, Hits: -, size: 1083, 463 ms In /var/log/fetchmail.log: Nov 18 16:23:23 [err] reading message davidep2@nethservice.nethesis.it:17 of 17 (931 octets) (log message incomplete) Nov 18 16:23:23 [err] SMTP listener refused delivery Nov 18 16:23:23 [info] not flushed
Test case 2 - fresh install w/o nethserver-mail-filter
Execute test case 1 without installing nethserver-mail-filter. Virus and SPAM messages must be delivered regularly.
Test case 3 - update
Repeat test case 1, updating an existing installation.
- The
yum update nethserver-fetchmail
command must update alsomail-common
andmail-server
#4 Updated by Davide Principi over 6 years ago
- Status changed from MODIFIED to ON_QA
- % Done changed from 60 to 70
In nethserver-testing:
nethserver-mail-server-1.8.2-1.1gitdd2412c.ns6.noarch.rpmnethserver-mail-common-1.4.1-1.1git09556a6.ns6.noarch.rpm
nethserver-fetchmail-1.0.6-1.0git664a2e1d.ns6.noarch.rpm
Packager note
When set to CLOSED merge https://github.com/nethesis/nethserver-docs/pull/44
#5 Updated by Davide Principi over 6 years ago
- Related to Bug #2766: Fetchmail delivers to non-existing email addresses added
#6 Updated by Giacomo Sanchietti over 6 years ago
- Assignee set to Stefano Fancello
#7 Updated by Stefano Fancello over 6 years ago
I'm not able to have a spam hit > 6.x using fetchmail nor with spamtest and messages are delivered anyway
# smtptest --ehlo whitehouse.org --to test1@nethesis.it --from obama@whitehouse.org --addr 213.92.16.101 --port 25 --subject "Enlarge your penis and buy viagra" --input amavis-test/sample-spam.txt ==> /var/log/maillog <== Dec 2 18:17:29 makako transfer/smtpd[18262]: connect from localhost[127.0.0.1] Dec 2 18:17:30 makako transfer/smtpd[18262]: NOQUEUE: client=whitehouse.org[213.92.16.101] Dec 2 18:17:44 makako queue/smtpd[18274]: connect from localhost[127.0.0.1] Dec 2 18:17:44 makako queue/smtpd[18274]: 858DA4255C: client=localhost[127.0.0.1], orig_client=whitehouse.org[213.92.16.101] Dec 2 18:17:44 makako postfix/cleanup[18275]: 858DA4255C: message-id=<N1msdrbJXNPfV4wg9> Dec 2 18:17:44 makako queue/smtpd[18274]: disconnect from localhost[127.0.0.1] Dec 2 18:17:44 makako postfix/qmgr[17734]: 858DA4255C: from=<obama@whitehouse.org>, size=5631, nrcpt=1 (queue active) Dec 2 18:17:45 makako amavis[15668]: (15668-02) Passed SPAMMY {RelayedTaggedInbound,Quarantined}, [213.92.16.101]:46315 [63.10.249.142] <obama@whitehouse.org> -> <test1@nethesis.it>, Message-ID: <N1msdrbJXNPfV4wg9>, mail_id: kJQah4fXQRRw, Hits: 6.235, size: 4959, queued_as: 858DA4255C, 14935 ms Dec 2 18:17:45 makako transfer/smtpd[18262]: proxy-accept: END-OF-MESSAGE: 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 858DA4255C; from=<obama@whitehouse.org> to=<test1@nethesis.it> proto=ESMTP helo=<whitehouse.org> Dec 2 18:17:45 makako transfer/smtpd[18262]: disconnect from whitehouse.org[213.92.16.101] Dec 2 18:17:45 makako dovecot: lmtp(18288): Connect from local Dec 2 18:17:46 makako dovecot: lmtp(18288, test1): nGKTKzn0fVRwRwAAJNjXmA: sieve: msgid=<N1msdrbJXNPfV4wg9>: stored mail into mailbox 'junkmail' Dec 2 18:17:46 makako delivery/lmtp[18281]: 858DA4255C: to=<test1@makako.nethesis.it>, orig_to=<test1+spam@nethesis.it>, relay=makako.nethesis.it[/var/run/dovecot/lmtp], delay=2.3, delays=0.44/0.18/0.81/0.91, dsn=2.0.0, status=sent (250 2.0.0 <test1@makako.nethesis.it> nGKTKzn0fVRwRwAAJNjXmA Saved) Dec 2 18:17:46 makako dovecot: lmtp(18288): Disconnect from local: Client quit (in reset) Dec 2 18:17:46 makako postfix/qmgr[17734]: 858DA4255C: removed
When I fetch an email that I expect to be marked as spam(using sample-spam.txt text):
==> /var/log/maillog <== Dec 2 17:43:06 makako dovecot: lmtp(14907): Connect from local Dec 2 17:43:08 makako dovecot: lmtp(14907, test1): Z153JxrsfVQ7OgAAJNjXmA: sieve: msgid=<4978c5111ab2f63e5a7de454a0a9e9c7264@guerrillamail.com>: stored mail into mailbox 'INBOX ' Dec 2 17:43:08 makako dovecot: lmtp(14907): Disconnect from local: Client quit (in reset) Dec 2 17:43:08 makako amavis[12832]: (12832-01) Passed SPAMMY {RelayedOpenRelay}, FETCHMAIL [198.143.169.10] <2t+0@guerrillamail.com> -> <test1@localhost>, Message-ID: <4978c51 11ab2f63e5a7de454a0a9e9c7264@guerrillamail.com>, mail_id: HAXL08XiL2oT, Hits: 5.004, size: 5453, queued_as: 250 2.0.0 <test1@localhost> Z153JxrsfVQ7OgAAJNjXmA Saved, 16173 ms
When I try to enable junkmail, I expect that "SPAMMY" mails is delivered in there, but what I have is:
==> /var/log/maillog <== Dec 2 18:12:29 makako dovecot: lmtp(17917): Connect from local Dec 2 18:12:30 makako dovecot: lmtp(17917, test1): bJLVK/3yfVT9RQAAJNjXmA: sieve: msgid=<76bf6e0a1840b646076930fb1042b23bf392@guerrillamail.com>: stored mail into mailbox 'INBOX' Dec 2 18:12:30 makako dovecot: lmtp(17917): Disconnect from local: Client quit (in reset) Dec 2 18:12:30 makako amavis[15667]: (15667-02) Passed SPAMMY {RelayedOpenRelay}, FETCHMAIL [198.143.169.10] <2t+0@guerrillamail.com> -> <test1@localhost>, Message-ID: <76bf6e0a1840b646076930fb1042b23bf392@guerrillamail.com>, mail_id: dbU5VBTD7gBP, Hits: 5.004, size: 5562, queued_as: 250 2.0.0 <test1@localhost> bJLVK/3yfVT9RQAAJNjXmA Saved, 11320 ms
Those are packages that I've installed:
# rpm -q nethserver-mail-server nethserver-mail-common nethserver-fetchmail nethserver-antivirus nethserver-mail-server-1.8.2-1.2gitd208942.ns6.noarch nethserver-mail-common-1.4.1-1.4gita8d628a.ns6.noarch nethserver-fetchmail-1.0.6-2.0git7e05d28d.ns6.noarch nethserver-antivirus-1.1.0-1.0gitf33b848b.ns6.noarch
Finally, when I hit "download now" button on interface, I get an error message that tel me to look at /var/log/fetchmail.log for more information, but log file doesn't show anything interesting
I also would like to point out that test cases aren't really clear on how to use amavis test mails to test fetchmail. As an email newbee, it took me a lot of time to test this.
#8 Updated by Stefano Fancello over 6 years ago
- Status changed from ON_QA to TRIAGED
- % Done changed from 70 to 20
#9 Updated by Giacomo Sanchietti over 6 years ago
- Assignee deleted (
Stefano Fancello)
#10 Updated by Davide Principi over 6 years ago
- Status changed from TRIAGED to ON_DEV
- Assignee set to Davide Principi
- % Done changed from 20 to 30
Stefano Fancello wrote:
YourI'm not able to have a spam hit > 6.x using fetchmail nor with spamtest and messages are delivered anyway
[...]
smtptest
is fine, and works as expected:
- connects on port 25
- Postfix acts as SMTP proxy with Amavis
- Amavis tags the message SPAMMY (Hits: 6.235)
- Amavis re-inject the message into the queue
- The message is delivered to dovecot and stores the messages into 'junkmail'
When I fetch an email that I expect to be marked as spam(using sample-spam.txt text):
When I try to enable junkmail, I expect that "SPAMMY" mails is delivered in there, but what I have is:
[...]
This sounds strange: dovecot stores the SPAMMY message into INBOX and needs a deeper test.
Could you attach the spam thresholds values? Please paste here the output of config show amavisd
.
Those are packages that I've installed:
[...]
I assume they were the latest from nethserver-testing.
Finally, when I hit "download now" button on interface, I get an error message that tel me to look at /var/log/fetchmail.log for more information, but log file doesn't show anything interesting
I agree, it's ugly. How would you enhance it? Let's open an enhancement!
I also would like to point out that test cases aren't really clear on how to use amavis test mails to test fetchmail. As an email newbee, it took me a lot of time to test this.
I'm sorry, the mail system is complex and I wrote the test cases with a lot of assumptions. Thanks for your time and effort: I understand your frustration as I used to be a newbie too :) To find help ask here, on IRC or ML: the QA step always offers the chance to learn something new!
#11 Updated by Davide Principi over 6 years ago
Stefano Fancello wrote:
When I try to enable junkmail, I expect that "SPAMMY" mails is delivered in there, but what I have is:
... Dec 2 18:12:30 makako amavis[15667]: (15667-02) Passed SPAMMY {RelayedOpenRelay}
"RelayedOpenRelay" is the evidence of a misconfiguration: I'd expect "RelayedTaggedInbound". Perhaps amavisd.conf is missing a local_domain_maps
entry?
#12 Updated by Davide Principi over 6 years ago
- Status changed from ON_DEV to MODIFIED
- Assignee deleted (
Davide Principi) - % Done changed from 30 to 60
Test case
Repeat previous test cases, adding also the following check condition:- If a message is tagged SPAMMY it must be delivered to junkmail folder (if enabled).
Dec 3 11:39:18 davidep2 dovecot: lmtp(17334, first.user): EJf3GiToflS2QwAAoK0gTQ: sieve: msgid=<N1msdrbJXNPfV4wg9>: stored mail into mailbox 'junkmail' Dec 3 11:39:18 davidep2 dovecot: lmtp(17334): Disconnect from local: Client quit (in reset) Dec 3 11:39:18 davidep2 amavis[17306]: (17306-01) Passed SPAMMY {RelayedTaggedInbound}, FETCHMAIL [63.10.249.142] <testx@spammers.gov> -> <first.user@vboxnet0.tld>, Message-ID: <N1msdrbJXNPfV4wg9>, mail_id: 23ntI-0s95qC, Hits: 8.361, size: 4800, queued_as: 250 2.0.0 <first.user+spam@vboxnet0.tld> EJf3GiToflS2QwAAoK0gTQ Saved, 6314 ms
#13 Updated by Davide Principi over 6 years ago
- Status changed from MODIFIED to ON_QA
- % Done changed from 60 to 70
In nethserver-testing (6.5):
[...]
nethserver-mail-common-1.4.1-1.5gitf0c3400.ns6.noarch.rpm
#14 Updated by Stefano Fancello over 6 years ago
- Status changed from ON_QA to VERIFIED
- % Done changed from 70 to 90
#15 Updated by Giacomo Sanchietti over 6 years ago
- Status changed from VERIFIED to CLOSED
- % Done changed from 90 to 100
- nethserver-mail-server-1.8.3-1.ns6.noarch.rpm
nethserver-fetchmail-1.1.0-1.ns6.noarch.rpm
#16 Updated by Davide Principi over 6 years ago
- Status changed from CLOSED to ON_DEV
- Assignee set to Davide Principi
- % Done changed from 100 to 30
Release of nethserver-fetchmail-1.1.0-1.ns6.noarch.rpm has been delayed: a non-existent user error (SMTP code 550) from Dovecot is ignored by fetchmail that flushes the message from the origin server.
Still TODOs- Validate local recipient in
fetchmailrc
template expansion, to disable invalid fetchmail DB records - Restart fetchmail on
user-delete
andgroup-delete
events, and remove obsolete restarts on pseudonym-modify, domain-delete and domain-modify events. - AD user/group deletion can't be detected: add documentation. Any fetchmail DB record pointing to a deleted AD account must be disabled manually.
#17 Updated by Davide Principi over 6 years ago
- Status changed from ON_DEV to MODIFIED
- Assignee deleted (
Davide Principi) - % Done changed from 30 to 60
Test case
fetchmail
must be restarted on the following conditions- one of
user-modify
,user-delete
,group-modify
,group-delete
events, and - the involved account for the above events is referenced by any of
fetchmail
DB records
- one of
- if an external (AD) account has been removed, the fetchmail service must be restarted manually. Check the removed account (nx here) is skipped by
fetchmailrc
template:perl -Mesmith::templates -e 'print esmith::templates::processTemplate({TEMPLATE_PATH=>"fetchmailrc", OUTPUT_TYPE=>"string"});' WARNING in /etc/e-smith/templates/fetchmailrc/10base: [WARNING] non available account `nx` has been skipped! Check your configuration. [...]
Updated https://github.com/nethesis/nethserver-docs/pull/44
Must be merged on release.
#18 Updated by Davide Principi over 6 years ago
- Status changed from MODIFIED to ON_QA
- % Done changed from 60 to 70
In nethserver-testing (6.5):
nethserver-fetchmail-1.1.0-3.0gitecfd13cc.ns6.noarch.rpm
#19 Updated by Giacomo Sanchietti over 6 years ago
- Assignee set to Giacomo Sanchietti
#20 Updated by Giacomo Sanchietti over 6 years ago
- Status changed from ON_QA to VERIFIED
- Assignee deleted (
Giacomo Sanchietti) - % Done changed from 70 to 90
- user-modify
- group-modify
- user-delete
- group-delete
With a non-existing user:
[root@localhost amavis-test]# db fetchmail show admin@example.tld=fetchmail account=tt active=YES nokeep=YES password=Nethesis,1234 popserver=localhost proto=pop3 ssl=NO username=admin [root@localhost amavis-test]# service fetchmail restart Shutting down fetchmail: [ OK ] WARNING in /etc/e-smith/templates/fetchmailrc/10base: [WARNING] non available account `tt` has been skipped! Check your configuration. WARNING: Template processing succeeded for /fetchmailrc: 1 fragment generated warnings at -e line 1 No active fetchmail accounts. Exiting gracefully.
#21 Updated by Giacomo Sanchietti over 6 years ago
- Status changed from VERIFIED to CLOSED
- % Done changed from 90 to 100
nethserver-fetchmail-1.1.1-1.ns6.noarch.rpm
Documentation has been updated.
#22 Updated by Davide Principi over 6 years ago
- Related to Bug #2978: POP3 connector does not list all groups added
#23 Updated by Davide Principi over 6 years ago
Released an hotfix to fix group alias expansion. In nethserver-updates (6.5):
nethserver-fetchmail-1.1.1-2.ns6.noarch.rpm
Messages accidentally delivered to group Maildir must be moved manually.
sendmail GROUPNAME < FILE
Where
FILE
is the mail file delivered to /var/lib/nethserver/vmail/GROUPNAME/...
Follow #2978