Bug #2947

« Previous | Next »

Fetchmail re-download all emails after reboot

Added by Alessio Fattorini over 6 years ago. Updated over 6 years ago.

Status:CLOSEDStart date:
Priority:UrgentDue date:
Assignee:-% Done:

100%

Category:nethserver-fetchmail
Target version:v6.5
Security class: Resolution:
Affected version:v6.5-final NEEDINFO:No

Description

After a reboot fetchmail re-download all emails of pop3 account configured into keep mode (nokeep=NO) in result emails are duplicated two ore three times for every reboot.
With a service restart or stop/start the problem is not reproducible
It's indipendent of the mail ISP used

Related issues

Related to NethServer 6 - Enhancement #2924: Fetchmail support for AD users CLOSED

Associated revisions

Revision bc7a7e21
Added by Davide Principi over 6 years ago

Fetchmail homedir set to /var/lib/nethserver/fetchmail. Refs #2947

- The old home is moved to the new location, if exists
- The .fetchids DB is moved too, if exists
- The .fetchids DB is (automatically) included in data backup
- Fixed primary group of fetchmail account

Revision fb28db5d
Added by Davide Principi over 6 years ago

Fixed RPM %pre scriptlet to exit cleanly. Refs #2947

History

#1 Updated by Alessio Fattorini over 6 years ago

  • Priority changed from Normal to Urgent

I investigated the problem, this file is used to avoids downloading all mails every time:
/var/run/fetchmail/.fetchids
but /var/run is cleared at each reboot then fetchmail wants to re-download every checked email keeped on ISP

A possibile way could be stored fetchid in /var/tmp/ becuase is writable from fecthmail user and preserved between reboots
http://unix.stackexchange.com/questions/30489/what-is-the-difference-between-tmp-and-var-tmp
Logwatch clear it only if remains untouched for 30 days

#2 Updated by Davide Principi over 6 years ago

Alessio Fattorini wrote:

I investigated the problem, this file is used to avoids downloading all mails every time:
/var/run/fetchmail/.fetchids
but /var/run is cleared at each reboot then fetchmail wants to re-download every checked email keeped on ISP

From fetchmail manual:

       -i <pathname> | --idfile <pathname>
              (Keyword: idfile)
              Specify  an alternate name for the .fetchids file used to save message
              UIDs. NOTE: since fetchmail 6.3.0, write access to the directory  con-
              taining  the  idfile is required, as fetchmail writes a temporary file
              and renames it into the place of the real idfile only if the temporary
              file  has  been  written  successfully.  This avoids the truncation of
              idfiles when running out of disk space.

We could set it in /etc/sysconfig/fetchmail, to store the DB under /var/lib/fetchmail/.

We should add a backup-include for /var/lib/fetchmail.

#3 Updated by Alessio Fattorini over 6 years ago

Davide Principi wrote:

We could set it in /etc/sysconfig/fetchmail, to store the DB under /var/lib/fetchmail/.

We should add a backup-include for /var/lib/fetchmail.

It seems a good way, fetchmail user must be able to write on this directory

#4 Updated by Davide Principi over 6 years ago

  • Category set to nethserver-fetchmail
  • Status changed from NEW to TRIAGED
  • Target version set to v6.5
  • % Done changed from 0 to 20
  • Affected version set to v6.5-final

#5 Updated by Davide Principi over 6 years ago

  • Status changed from TRIAGED to ON_DEV
  • Assignee set to Davide Principi
  • % Done changed from 20 to 30

#6 Updated by Davide Principi over 6 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee deleted (Davide Principi)
  • % Done changed from 30 to 60

fetchmail has now a new home dir, where .fetchids DB is stored:
/var/lib/nethserver/fetchmail.

Test cases

The modified version must fix the described bug. Moreover, test

1. Fresh installation of the modified package
2. Update existing 1.0.5, where the fetchmail daemon has never been started (.fetchids DB does not exist)
3. Update a running 1.0.5, with existing /var/run/fetchmail/.fetchids: verify the DB was moved to the new home

On each of the above tests, the deamon must run with fetchmail group:

   # id -g fetchmail
490
   # ps -C fetchmail -o rgid,ruid
 RGID  RUID
  490   490

#7 Updated by Davide Principi over 6 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

In nethserver-testing:
nethserver-fetchmail-1.0.5-1.0gitbc7a7e21.ns6.noarch.rpm

#8 Updated by Alessio Fattorini over 6 years ago

  • Assignee set to Alessio Fattorini

#9 Updated by Alessio Fattorini over 6 years ago

Test 2

No .fetchuids

[root@muflone ~]# ll -la /var/run/fetchmail/
totale 8
drwxr-x---.  2 fetchmail root 4096 15 ott 15:14 .
drwxr-xr-x. 30 root      root 4096 23 ott 15:54 ..

I upgrade

Running Transaction
  Updating   : nethserver-fetchmail-1.0.5-1.0gitbc7a7e21.ns6.noarch                               1/2 
fetchmail è interrotto
mv: impossibile fare stat di `/var/run/fetchmail/.fetchids': No such file or directory
il proprietario di `/var/lib/nethserver/fetchmail' è stato cambiato in fetchmail:fetchmail
  Cleanup    : nethserver-fetchmail-1.0.5-1.ns6.noarch                                            2/2 
  Verifying  : nethserver-fetchmail-1.0.5-1.0gitbc7a7e21.ns6.noarch                               1/2 
  Verifying  : nethserver-fetchmail-1.0.5-1.ns6.noarch

fetchmail configured then it can creates .fetchids

[root@muflone ~]# ll -la /var/lib/nethserver/fetchmail/
totale 12
drwxr-x---.  2 fetchmail fetchmail 4096 23 ott 16:05 .
drwxr-xr-x. 13 root      root      4096 11 nov  2014 ..
-rw-------.  1 fetchmail fetchmail   34 23 ott 16:05 .fetchids

Rebooted, fetchuids is still there

VERIFIED

Test 3

.fetchuid is created 

[root@muflone ~]# ll -a /var/run/fetchmail/
totale 16
drwxr-x---.  2 fetchmail root  4096 11 nov 19:03 .
drwxr-xr-x. 30 root      root  4096 11 nov 19:03 ..
-rw-------.  1 fetchmail users   34 11 nov 19:03 .fetchids
-rw-------.  1 fetchmail users    9 11 nov 19:03 fetchmail.pid

Upgrade

il proprietario di `/var/lib/nethserver/fetchmail/.fetchids' è stato cambiato in fetchmail:fetchmail
il proprietario di `/var/lib/nethserver/fetchmail' è stato cambiato in fetchmail:fetchmail
  Cleanup    : nethserver-fetchmail-1.0.5-1.ns6.noarch                                            2/2 
  Verifying  : nethserver-fetchmail-1.0.5-1.0gitbc7a7e21.ns6.noarch                               1/2 
  Verifying  : nethserver-fetchmail-1.0.5-1.ns6.noarch

.fetchuid is correctly moved

[root@muflone ~]# ll -a /var/lib/nethserver/fetchmail/
totale 12
drwxr-x---.  2 fetchmail fetchmail 4096 11 nov 19:05 .
drwxr-xr-x. 13 root      root      4096 11 nov 11:45 ..
-rw-------.  1 fetchmail fetchmail   34 11 nov 19:05 .fetchids

After reboot .fetchid is persistent

[root@muflone ~]# ll -a /var/lib/nethserver/fetchmail/
totale 12
drwxr-x---.  2 fetchmail fetchmail 4096 11 nov 19:08 .
drwxr-xr-x. 13 root      root      4096 11 nov 11:45 ..
-rw-------.  1 fetchmail fetchmail   34 11 nov 19:08 .fetchids

VERIFIED

Remains only the the first test

#10 Updated by Alessio Fattorini over 6 years ago

Test 1
/var/lib/nethserver/fetchmail/ correctly created

VERIFIED

#11 Updated by Filippo Carletti over 6 years ago

mv: impossibile fare stat di `/var/run/fetchmail/.fetchids': No such file or directory
il proprietario di `/var/lib/nethserver/fetchmail' è stato cambiato in fetchmail:fetchmail

Do we really want the above output?

#12 Updated by Davide Principi over 6 years ago

Filippo Carletti wrote:

Do we really want the above output?

The mv error could be suppressed, as it is a normal condition. I would keep the chown -c output. It keeps track of the scriptlet activity, and may help troubleshooting.

Alessio, what does the yum history say? The transasction was marked with E or similar?

#13 Updated by Davide Principi over 6 years ago

#14 Updated by Alessio Fattorini over 6 years ago

Davide Principi wrote:

Alessio, what does the yum history say? The transasction was marked with E or similar?

It's marked as 2 EE

#15 Updated by Davide Principi over 6 years ago

  • Status changed from ON_QA to TRIAGED
  • Assignee deleted (Alessio Fattorini)
  • % Done changed from 70 to 20

#16 Updated by Davide Principi over 6 years ago

  • Status changed from TRIAGED to ON_DEV
  • Assignee set to Davide Principi
  • % Done changed from 20 to 30

#17 Updated by Davide Principi over 6 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee deleted (Davide Principi)
  • % Done changed from 30 to 60

Alessio Fattorini wrote:

It's marked as 2 EE

The EE says a scriptlet generated some output. It's a warning/non-fatal-error condition. See man yum for more infos.

MODIFIED

Exit code is always 0. Messages are now silenced.

#18 Updated by Davide Principi over 6 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

In nethserver-testing:
nethserver-fetchmail-1.0.5-2.0gitfb28db5d.ns6.noarch.rpm

#19 Updated by Alessio Fattorini over 6 years ago

Tested with a fresh install and upgrade, messages are now silenced.
Verified

#20 Updated by Alessio Fattorini over 6 years ago

  • Status changed from ON_QA to VERIFIED
  • % Done changed from 70 to 90

#21 Updated by Davide Principi over 6 years ago

  • Status changed from VERIFIED to CLOSED
  • % Done changed from 90 to 100

In nethserver-updates:
nethserver-fetchmail-1.0.6-1.ns6.noarch.rpm

Also available in: Atom PDF