Enhancement #2954

Avoid fetchmail bounces

Added by Davide Principi almost 5 years ago. Updated almost 5 years ago.

Status:CLOSEDStart date:
Priority:NormalDue date:
Assignee:-% Done:

100%

Category:nethserver-fetchmail
Target version:v6.5
Resolution: NEEDINFO:No

Description

Messages from fetchmail are already delivered to a remote mailbox so their envelope sender is lost. Bouncing such messages is a problem if the sender address does not exist or is forged.

Also a bounce is generated if the destination mailbox does not exist, i.e. if a user has been deleted or similar. To avoid such bounces we must bypass the Postfix queue and talk directly to Dovecot LMTP socket, by means of Amavis. This architecture does not have a queue so Fetchmail knows immediately if a message has been delivered or not and log its activity appropriately.

[RemoteSystem]
     v
     v  
    POP/IMAP
     v
     v
[Fetchmail] >>SMTP>> [127.0.0.100:10024 Amavis] >>LMTP>> [/var/run/dovecot/lmtp Dovecot]            

Optionally, if a message is not accepted by Amavis, Fetchmail could bounce a report to the local postmaster account.


Related issues

Related to NethServer 6 - Enhancement #2924: Fetchmail support for AD users CLOSED
Related to NethServer 6 - Bug #2766: Fetchmail delivers to non-existing email addresses CLOSED
Related to NethServer 6 - Bug #2978: POP3 connector does not list all groups CLOSED

Associated revisions

Revision 09556a69
Added by Davide Principi almost 5 years ago

/etc/sysconfig/amavisd is now a template. Refs #2954

Revision dd2412c4
Added by Davide Principi almost 5 years ago

Start amavisd with vmail additional group. Refs #2954

The vmail group is required to allow amavis talk directly with Dovecot
LMTP server. This happens with fetchmail.

Revision 664a2e1d
Added by Davide Principi almost 5 years ago

Submit messages to local amavisd at 127.0.0.100:10024. Refs #2924

- Require nethserver-mail-common > 1.4.0-4

Revision c9fc3326
Added by Davide Principi almost 5 years ago

Applied word wrap to pop3_connector module documentation. Refs #2954

Revision f0c3400c
Added by Davide Principi almost 5 years ago

Added default Postfix mydestination entries to inbound domain list. Refs #2954

Fixes missing spam headers problem, for messages from Fetchmail.
See note 11.

Revision 42d6ea87
Added by Davide Principi almost 5 years ago

Validate accounts on fetchmailrc template expansion. Refs #2954

Revision ecfd13cc
Added by Davide Principi almost 5 years ago

Restart fetchmail conditionally when account changes. Refs #2954

Monitored events are:
  • user-delete
  • user-modify
  • group-delete
  • group-modify

Note: AD accounts can't be monitored. Fetchmail accounts must be
updated manually.

Revision 27d7aebc
Added by Davide Principi almost 5 years ago

Warning for Active Directory users. Refs #2954

Revision 15ee515d
Added by Davide Principi almost 5 years ago

Release 1.1.1-2
  • Route message to postfix, for alias expansion. Refs #2978
  • Fix shown groups in POP3 connector page
  • Partially reverts #2954

History

#1 Updated by Davide Principi almost 5 years ago

  • Status changed from TRIAGED to ON_DEV
  • Assignee set to Davide Principi
  • % Done changed from 20 to 30

#2 Updated by Davide Principi almost 5 years ago

#3 Updated by Davide Principi almost 5 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee deleted (Davide Principi)
  • % Done changed from 30 to 60

Note

  • Read the test cases. I suggest executing 2 before 1.
  • Test Virus and Spam messages are available from amavisd-new package. Execute the following commands:
    mkdir amavis-test
    cd amavis-test
    perl -pe 's/./chr(ord($&)^255)/sge' </usr/share/doc/amavisd-new-2.8.0/test-messages/sample.tar.gz.compl | zcat | tar xvf -
    

Test case 1 - fresh install

In a clean NethServer 6.5 update base packages.

  • yum --enablerepo=nethserver-testing install nethserver-fetchmail nethserver-mail-filter
  • Create an user account first.user
  • In POP3 connector page configure an IMAP or POP account where to fetch messages for first.user
  • If a message is delivered to first.user you must find in /var/log/fetchmail.log
    Nov 18 15:31:05 [info]  12 messages (11 seen) for davidep2 at nethservice.nethesis.it (15855 octets).
    Nov 18 15:31:06 [info]  reading message davidep2@nethservice.nethesis.it:12 of 12 (7391 octets) not flushed (...or flushed)
    
    In /var/log/maillog:
    Nov 18 15:31:05 davidep2 dovecot: lmtp(25995): Connect from local
    Nov 18 15:31:06 davidep2 dovecot: lmtp(25995, first.user): X3IAMClYa1SLZQAA4KLHlg: sieve: msgid=<546b581b.l+j9LuDuzaR2ebOZ%davide.principi@nethesis.it>: stored mail into mailbox 'INBOX'
    Nov 18 15:31:06 davidep2 dovecot: lmtp(25995): Disconnect from local: Client quit (in reset)
    Nov 18 15:31:06 davidep2 amavis[25182]: (25182-01) Passed CLEAN {RelayedOpenRelay}, FETCHMAIL [192.168.5.252] <davide.principi@nethesis.it> -> <first.user@localhost>, Message-ID: <546b581b.l+j9LuDuzaR2ebOZ%davide.principi@nethesis.it>, mail_id: qz3X6CmDCqIo, Hits: -, size: 7543, queued_as: 250 2.0.0 <first.user@localhost> X3IAMClYa1SLZQAA4KLHlg Saved, 806 ms
    
    You must not see any Postfix message.
  • If a message is SPAM it is discarded by amavisd:
    In /var/log/maillog:
    Nov 18 16:20:44 davidep2 amavis[26857]: (26857-01) Blocked SPAM {DiscardedOpenRelay,Quarantined}, FETCHMAIL [192.168.5.252] <davide.principi@nethesis.it> -> <first.user@localhost>, Message-ID: <546b639f.2JA24Xqk2Rj1qm4d%davide.principi@nethesis.it>, mail_id: d0jN0QqJoVwN, Hits: 999.999, size: 1707, 5128 ms
    
    In /var/log/fetchmail.log:
    Nov 18 16:20:44 [info]  reading message davidep2@nethservice.nethesis.it:16 of 16 (1555 octets) not flushed
    
    Amavis accepts the message then discards it silently.
  • If a message is tagged SPAMMY it must be delivered to junkmail folder (if enabled).
    Dec  3 11:39:18 davidep2 dovecot: lmtp(17334, first.user): EJf3GiToflS2QwAAoK0gTQ: sieve: msgid=<N1msdrbJXNPfV4wg9>: stored mail into mailbox 'junkmail'
    Dec  3 11:39:18 davidep2 dovecot: lmtp(17334): Disconnect from local: Client quit (in reset)
    Dec  3 11:39:18 davidep2 amavis[17306]: (17306-01) Passed SPAMMY {RelayedTaggedInbound}, FETCHMAIL [63.10.249.142] <testx@spammers.gov> -> <first.user@vboxnet0.tld>, Message-ID: <N1msdrbJXNPfV4wg9>, mail_id: 23ntI-0s95qC, Hits: 8.361, size: 4800, queued_as: 250 2.0.0 <first.user+spam@vboxnet0.tld> EJf3GiToflS2QwAAoK0gTQ Saved, 6314 ms
  • If a message is VIRUS it is rejected by amavisd:
    In /var/log/maillog:
    Nov 18 16:23:23 davidep2 amavis[26860]: (26860-01) Blocked INFECTED (Eicar-Test-Signature.UNOFFICIAL) {RejectedOpenRelay,Quarantined}, FETCHMAIL [192.168.5.252] <davide.principi@nethesis.it> -> <first.user@localhost>, Message-ID: <546b6463.3KQWNOuvtic9jPJ0%davide.principi@nethesis.it>, mail_id: v8g99-gBhtiF, Hits: -, size: 1083, 463 ms
    
    In /var/log/fetchmail.log:
    Nov 18 16:23:23 [err]  reading message davidep2@nethservice.nethesis.it:17 of 17 (931 octets) (log message incomplete)
    Nov 18 16:23:23 [err]  SMTP listener refused delivery
    Nov 18 16:23:23 [info]   not flushed
    

Test case 2 - fresh install w/o nethserver-mail-filter

Execute test case 1 without installing nethserver-mail-filter. Virus and SPAM messages must be delivered regularly.

Test case 3 - update

Repeat test case 1, updating an existing installation.

  • The yum update nethserver-fetchmail command must update also mail-common and mail-server

#4 Updated by Davide Principi almost 5 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

In nethserver-testing:
nethserver-mail-server-1.8.2-1.1gitdd2412c.ns6.noarch.rpm
nethserver-mail-common-1.4.1-1.1git09556a6.ns6.noarch.rpm
nethserver-fetchmail-1.0.6-1.0git664a2e1d.ns6.noarch.rpm

Packager note

When set to CLOSED merge https://github.com/nethesis/nethserver-docs/pull/44

#5 Updated by Davide Principi almost 5 years ago

  • Related to Bug #2766: Fetchmail delivers to non-existing email addresses added

#6 Updated by Giacomo Sanchietti almost 5 years ago

  • Assignee set to Stefano Fancello

#7 Updated by Stefano Fancello almost 5 years ago

I'm not able to have a spam hit > 6.x using fetchmail nor with spamtest and messages are delivered anyway

# smtptest --ehlo whitehouse.org --to test1@nethesis.it --from obama@whitehouse.org --addr 213.92.16.101 --port 25 --subject "Enlarge your penis and buy viagra" --input amavis-test/sample-spam.txt

==> /var/log/maillog <==
Dec  2 18:17:29 makako transfer/smtpd[18262]: connect from localhost[127.0.0.1]
Dec  2 18:17:30 makako transfer/smtpd[18262]: NOQUEUE: client=whitehouse.org[213.92.16.101]
Dec  2 18:17:44 makako queue/smtpd[18274]: connect from localhost[127.0.0.1]
Dec  2 18:17:44 makako queue/smtpd[18274]: 858DA4255C: client=localhost[127.0.0.1], orig_client=whitehouse.org[213.92.16.101]
Dec  2 18:17:44 makako postfix/cleanup[18275]: 858DA4255C: message-id=<N1msdrbJXNPfV4wg9>
Dec  2 18:17:44 makako queue/smtpd[18274]: disconnect from localhost[127.0.0.1]
Dec  2 18:17:44 makako postfix/qmgr[17734]: 858DA4255C: from=<obama@whitehouse.org>, size=5631, nrcpt=1 (queue active)
Dec  2 18:17:45 makako amavis[15668]: (15668-02) Passed SPAMMY {RelayedTaggedInbound,Quarantined}, [213.92.16.101]:46315 [63.10.249.142] <obama@whitehouse.org> -> <test1@nethesis.it>, Message-ID: <N1msdrbJXNPfV4wg9>, mail_id: kJQah4fXQRRw, Hits: 6.235, size: 4959, queued_as: 858DA4255C, 14935 ms
Dec  2 18:17:45 makako transfer/smtpd[18262]: proxy-accept: END-OF-MESSAGE: 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 858DA4255C; from=<obama@whitehouse.org> to=<test1@nethesis.it> proto=ESMTP helo=<whitehouse.org>
Dec  2 18:17:45 makako transfer/smtpd[18262]: disconnect from whitehouse.org[213.92.16.101]
Dec  2 18:17:45 makako dovecot: lmtp(18288): Connect from local
Dec  2 18:17:46 makako dovecot: lmtp(18288, test1): nGKTKzn0fVRwRwAAJNjXmA: sieve: msgid=<N1msdrbJXNPfV4wg9>: stored mail into mailbox 'junkmail'
Dec  2 18:17:46 makako delivery/lmtp[18281]: 858DA4255C: to=<test1@makako.nethesis.it>, orig_to=<test1+spam@nethesis.it>, relay=makako.nethesis.it[/var/run/dovecot/lmtp], delay=2.3, delays=0.44/0.18/0.81/0.91, dsn=2.0.0, status=sent (250 2.0.0 <test1@makako.nethesis.it> nGKTKzn0fVRwRwAAJNjXmA Saved)
Dec  2 18:17:46 makako dovecot: lmtp(18288): Disconnect from local: Client quit (in reset)
Dec  2 18:17:46 makako postfix/qmgr[17734]: 858DA4255C: removed

When I fetch an email that I expect to be marked as spam(using sample-spam.txt text):

==> /var/log/maillog <==
Dec  2 17:43:06 makako dovecot: lmtp(14907): Connect from local
Dec  2 17:43:08 makako dovecot: lmtp(14907, test1): Z153JxrsfVQ7OgAAJNjXmA: sieve: msgid=<4978c5111ab2f63e5a7de454a0a9e9c7264@guerrillamail.com>: stored mail into mailbox 'INBOX
'
Dec  2 17:43:08 makako dovecot: lmtp(14907): Disconnect from local: Client quit (in reset)
Dec  2 17:43:08 makako amavis[12832]: (12832-01) Passed SPAMMY {RelayedOpenRelay}, FETCHMAIL [198.143.169.10] <2t+0@guerrillamail.com> -> <test1@localhost>, Message-ID: <4978c51
11ab2f63e5a7de454a0a9e9c7264@guerrillamail.com>, mail_id: HAXL08XiL2oT, Hits: 5.004, size: 5453, queued_as: 250 2.0.0 <test1@localhost> Z153JxrsfVQ7OgAAJNjXmA Saved, 16173 ms

When I try to enable junkmail, I expect that "SPAMMY" mails is delivered in there, but what I have is:

==> /var/log/maillog <==
Dec  2 18:12:29 makako dovecot: lmtp(17917): Connect from local
Dec  2 18:12:30 makako dovecot: lmtp(17917, test1): bJLVK/3yfVT9RQAAJNjXmA: sieve: msgid=<76bf6e0a1840b646076930fb1042b23bf392@guerrillamail.com>: stored mail into mailbox 'INBOX'
Dec  2 18:12:30 makako dovecot: lmtp(17917): Disconnect from local: Client quit (in reset)
Dec  2 18:12:30 makako amavis[15667]: (15667-02) Passed SPAMMY {RelayedOpenRelay}, FETCHMAIL [198.143.169.10] <2t+0@guerrillamail.com> -> <test1@localhost>, Message-ID: <76bf6e0a1840b646076930fb1042b23bf392@guerrillamail.com>, mail_id: dbU5VBTD7gBP, Hits: 5.004, size: 5562, queued_as: 250 2.0.0 <test1@localhost> bJLVK/3yfVT9RQAAJNjXmA Saved, 11320 ms

Those are packages that I've installed:

# rpm -q nethserver-mail-server nethserver-mail-common nethserver-fetchmail nethserver-antivirus
nethserver-mail-server-1.8.2-1.2gitd208942.ns6.noarch
nethserver-mail-common-1.4.1-1.4gita8d628a.ns6.noarch
nethserver-fetchmail-1.0.6-2.0git7e05d28d.ns6.noarch
nethserver-antivirus-1.1.0-1.0gitf33b848b.ns6.noarch

Finally, when I hit "download now" button on interface, I get an error message that tel me to look at /var/log/fetchmail.log for more information, but log file doesn't show anything interesting

I also would like to point out that test cases aren't really clear on how to use amavis test mails to test fetchmail. As an email newbee, it took me a lot of time to test this.

#8 Updated by Stefano Fancello almost 5 years ago

  • Status changed from ON_QA to TRIAGED
  • % Done changed from 70 to 20

#9 Updated by Giacomo Sanchietti almost 5 years ago

  • Assignee deleted (Stefano Fancello)

#10 Updated by Davide Principi almost 5 years ago

  • Status changed from TRIAGED to ON_DEV
  • Assignee set to Davide Principi
  • % Done changed from 20 to 30

Stefano Fancello wrote:

I'm not able to have a spam hit > 6.x using fetchmail nor with spamtest and messages are delivered anyway
[...]

Your smtptest is fine, and works as expected:
  • connects on port 25
  • Postfix acts as SMTP proxy with Amavis
  • Amavis tags the message SPAMMY (Hits: 6.235)
  • Amavis re-inject the message into the queue
  • The message is delivered to dovecot and stores the messages into 'junkmail'

When I fetch an email that I expect to be marked as spam(using sample-spam.txt text):
When I try to enable junkmail, I expect that "SPAMMY" mails is delivered in there, but what I have is:
[...]

This sounds strange: dovecot stores the SPAMMY message into INBOX and needs a deeper test.

Could you attach the spam thresholds values? Please paste here the output of config show amavisd.

Those are packages that I've installed:
[...]

I assume they were the latest from nethserver-testing.

Finally, when I hit "download now" button on interface, I get an error message that tel me to look at /var/log/fetchmail.log for more information, but log file doesn't show anything interesting

I agree, it's ugly. How would you enhance it? Let's open an enhancement!

I also would like to point out that test cases aren't really clear on how to use amavis test mails to test fetchmail. As an email newbee, it took me a lot of time to test this.

I'm sorry, the mail system is complex and I wrote the test cases with a lot of assumptions. Thanks for your time and effort: I understand your frustration as I used to be a newbie too :) To find help ask here, on IRC or ML: the QA step always offers the chance to learn something new!

#11 Updated by Davide Principi almost 5 years ago

Stefano Fancello wrote:

When I try to enable junkmail, I expect that "SPAMMY" mails is delivered in there, but what I have is:

...
Dec  2 18:12:30 makako amavis[15667]: (15667-02) Passed SPAMMY {RelayedOpenRelay}

"RelayedOpenRelay" is the evidence of a misconfiguration: I'd expect "RelayedTaggedInbound". Perhaps amavisd.conf is missing a local_domain_maps entry?

#12 Updated by Davide Principi almost 5 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee deleted (Davide Principi)
  • % Done changed from 30 to 60

Test case

Repeat previous test cases, adding also the following check condition:
  • If a message is tagged SPAMMY it must be delivered to junkmail folder (if enabled).
    Dec  3 11:39:18 davidep2 dovecot: lmtp(17334, first.user): EJf3GiToflS2QwAAoK0gTQ: sieve: msgid=<N1msdrbJXNPfV4wg9>: stored mail into mailbox 'junkmail'
    Dec  3 11:39:18 davidep2 dovecot: lmtp(17334): Disconnect from local: Client quit (in reset)
    Dec  3 11:39:18 davidep2 amavis[17306]: (17306-01) Passed SPAMMY {RelayedTaggedInbound}, FETCHMAIL [63.10.249.142] <testx@spammers.gov> -> <first.user@vboxnet0.tld>, Message-ID: <N1msdrbJXNPfV4wg9>, mail_id: 23ntI-0s95qC, Hits: 8.361, size: 4800, queued_as: 250 2.0.0 <first.user+spam@vboxnet0.tld> EJf3GiToflS2QwAAoK0gTQ Saved, 6314 ms
    

#13 Updated by Davide Principi almost 5 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

In nethserver-testing (6.5):
[...]
nethserver-mail-common-1.4.1-1.5gitf0c3400.ns6.noarch.rpm

#14 Updated by Stefano Fancello almost 5 years ago

  • Status changed from ON_QA to VERIFIED
  • % Done changed from 70 to 90

#15 Updated by Giacomo Sanchietti almost 5 years ago

  • Status changed from VERIFIED to CLOSED
  • % Done changed from 90 to 100
Released in nethserver-updates:
  • nethserver-mail-server-1.8.3-1.ns6.noarch.rpm
  • nethserver-fetchmail-1.1.0-1.ns6.noarch.rpm

#16 Updated by Davide Principi almost 5 years ago

  • Status changed from CLOSED to ON_DEV
  • Assignee set to Davide Principi
  • % Done changed from 100 to 30

Release of nethserver-fetchmail-1.1.0-1.ns6.noarch.rpm has been delayed: a non-existent user error (SMTP code 550) from Dovecot is ignored by fetchmail that flushes the message from the origin server.

Still TODOs
  • Validate local recipient in fetchmailrc template expansion, to disable invalid fetchmail DB records
  • Restart fetchmail on user-delete and group-delete events, and remove obsolete restarts on pseudonym-modify, domain-delete and domain-modify events.
  • AD user/group deletion can't be detected: add documentation. Any fetchmail DB record pointing to a deleted AD account must be disabled manually.

#17 Updated by Davide Principi almost 5 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee deleted (Davide Principi)
  • % Done changed from 30 to 60

Test case

  • fetchmail must be restarted on the following conditions
    • one of user-modify, user-delete, group-modify, group-delete events, and
    • the involved account for the above events is referenced by any of fetchmail DB records
  • if an external (AD) account has been removed, the fetchmail service must be restarted manually. Check the removed account (nx here) is skipped by fetchmailrc template:
    perl -Mesmith::templates -e 'print esmith::templates::processTemplate({TEMPLATE_PATH=>"fetchmailrc", OUTPUT_TYPE=>"string"});'
    WARNING in /etc/e-smith/templates/fetchmailrc/10base: [WARNING] non available account `nx` has been skipped! Check your configuration.
    [...]
    

Updated https://github.com/nethesis/nethserver-docs/pull/44
Must be merged on release.

#18 Updated by Davide Principi almost 5 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

In nethserver-testing (6.5):
nethserver-fetchmail-1.1.0-3.0gitecfd13cc.ns6.noarch.rpm

#19 Updated by Giacomo Sanchietti almost 5 years ago

  • Assignee set to Giacomo Sanchietti

#20 Updated by Giacomo Sanchietti almost 5 years ago

  • Status changed from ON_QA to VERIFIED
  • Assignee deleted (Giacomo Sanchietti)
  • % Done changed from 70 to 90
Fetchmail is restarted on:
  • user-modify
  • group-modify
  • user-delete
  • group-delete

With a non-existing user:

[root@localhost amavis-test]# db fetchmail show
admin@example.tld=fetchmail
    account=tt
    active=YES
    nokeep=YES
    password=Nethesis,1234
    popserver=localhost
    proto=pop3
    ssl=NO
    username=admin
[root@localhost amavis-test]# service fetchmail restart
Shutting down fetchmail:                                   [  OK  ]
WARNING in /etc/e-smith/templates/fetchmailrc/10base: [WARNING] non available account `tt` has been skipped! Check your configuration.
WARNING: Template processing succeeded for /fetchmailrc: 1 fragment generated warnings
 at -e line 1
No active fetchmail accounts. Exiting gracefully.

#21 Updated by Giacomo Sanchietti almost 5 years ago

  • Status changed from VERIFIED to CLOSED
  • % Done changed from 90 to 100
Released in nethserver-updates:
  • nethserver-fetchmail-1.1.1-1.ns6.noarch.rpm

Documentation has been updated.

#22 Updated by Davide Principi almost 5 years ago

  • Related to Bug #2978: POP3 connector does not list all groups added

#23 Updated by Davide Principi almost 5 years ago

Released an hotfix to fix group alias expansion. In nethserver-updates (6.5):
nethserver-fetchmail-1.1.1-2.ns6.noarch.rpm

Messages accidentally delivered to group Maildir must be moved manually.

sendmail GROUPNAME < FILE

Where FILE is the mail file delivered to /var/lib/nethserver/vmail/GROUPNAME/...

Follow #2978

Also available in: Atom PDF