Enhancement #2043

Backup: implement and document full restore

Added by Giacomo Sanchietti about 8 years ago. Updated almost 8 years ago.

Status:CLOSEDStart date:
Priority:NormalDue date:
Assignee:-% Done:

100%

Category:nethserver-backup-data
Target version:v6.4-beta2
Resolution: NEEDINFO:No

Description

Implement and document a full restore for a disaster recovery scenario.

The restore should:
  • restore all data
  • restore all configuration
  • restore users from LDAP dump
  • restore network configuration
  • restore custom SSL certificates

Related issues

Related to NethServer 6 - Enhancement #2063: Directory: backup service accounts passwords CLOSED 11/21/2013 11/21/2013
Related to NethServer 6 - Enhancement #2022: Backup data: do not check for already mounted dir CLOSED
Related to NethServer 6 - Enhancement #2118: Backup: force backup of configuration before starting bac... CLOSED

Associated revisions

Revision 282664ce
Added by Giacomo Sanchietti about 8 years ago

restore-data-duplicity: search for a directory as restore point. Refs #2043

Revision 428ca847
Added by Giacomo Sanchietti about 8 years ago

restore-config-duplicity: restore config file from data backup. Refs #2043

Revision fc632a90
Added by Giacomo Sanchietti about 8 years ago

backup: save and restore maildir structure along with sieve filters. Refs #2043

Revision 1d9564e8
Added by Giacomo Sanchietti about 8 years ago

Implement full restore scenario. Refs #2043

Revision 142742d5
Added by Giacomo Sanchietti about 8 years ago

mysql-load-tables: do not empty mysql client configuration. Refs #2043

Revision b97ccc5a
Added by Giacomo Sanchietti about 8 years ago

backup: implement backup and restore for ldap. Refs #2043

Revision bfcb4f02
Added by Giacomo Sanchietti about 8 years ago

createlinks, actions: move nethserver-base-dump-ldap to nethserver-directory package. Refs #2043
nethserver-base.include: add e-smith db directory. Refs #2043

Revision 981d0693
Added by Giacomo Sanchietti about 8 years ago

nethserver-mysql-conf: correctly start mysqld on first install. Refs #2043

Revision d2afb916
Added by Giacomo Sanchietti about 8 years ago

post-restore-config event: reset network. Refs #2043

Revision 9361c5b2
Added by Giacomo Sanchietti almost 8 years ago

backup: save and restore ibay structure. Refs #2043

History

#1 Updated by Giacomo Sanchietti about 8 years ago

  • Description updated (diff)

#2 Updated by Giacomo Sanchietti about 8 years ago

  • Subject changed from Backup data: implement and document full restore to Backup: implement and document full restore
  • Status changed from NEW to TRIAGED
  • Assignee set to Giacomo Sanchietti
  • % Done changed from 0 to 20
Disaster recovery scenario
  • The system is restored in two phases: first configuration, then all data
  • restore-config will search for an existing configuration backup file, if not present the configuration backup can be retrieved from data backup
  • configuration backup will also contains:
    • shadow, passwd and group
    • all maildir directories
    • sieve filters
    • first directory level of ibay
    • ssl certificates
  • At the end of restore-config all update will be launched and the network will be reconfigured

The restore can be performed before or after addon installation.

#3 Updated by Giacomo Sanchietti about 8 years ago

  • Status changed from TRIAGED to ON_DEV
  • % Done changed from 20 to 30

#4 Updated by Giacomo Sanchietti about 8 years ago

  • Status changed from ON_DEV to MODIFIED
  • % Done changed from 30 to 70

Implemented all wished features.

#5 Updated by Giacomo Sanchietti about 8 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 70 to 80
Packages in nethserver-testing repository:
  • nethserver-directory-1.2.2-1
  • nethserver-mail-server-1.4.3-1
  • nethserver-base-1.2.4-1
  • nethserver-backup-config-1.0.3-1
  • nethserver-backup-data-1.0.4-1
  • nethserver-mysql-1.0.3-1

Prepare system for test case

  • Update the system from nethserver-testing
  • Install nethserver-directory nethserver-mail-server nethserver-mysql
  • Create users, groups, databases. Send some mail to users and groups.
  • Configure a backup
  • Execute the backup with: backup-config && backup-data
Test case 1:
  • Install a new machine with the same hostname as the old one
  • Configure a backup
  • Restore configuration backup executing: restore-config
  • Execute bootstrap-console
  • Restore data backup executing: restore-data
  • Install nethserver-directory nethserver-mail-server nethserver-mysql from nethserver-testing repository
  • Check if mails, databases and users are present
Test case 2:
  • Install a new machine with the same hostname as the old one
  • Configure a backup
  • Install nethserver-directory nethserver-mail-server nethserver-mysql from nethserver-testing repository
  • Restore configuration backup executing: restore-config
  • Execute bootstrap-console
  • Check imap server is working and users can authenticate themselves
  • Restore data backup executing: restore-data
  • Check if mails, databases and users are present

#6 Updated by Davide Principi about 8 years ago

  • Assignee changed from Giacomo Sanchietti to Davide Principi

#7 Updated by Davide Principi about 8 years ago

  • Status changed from ON_QA to ON_DEV
  • Assignee deleted (Davide Principi)
  • % Done changed from 80 to 30

Test case 1 OK
No problems with case 1. Tested also with nethserver-sogo-1.2.1-1.ns6.

Test case 2 FAILED
In test case 2 I restored configuration and data on the same VM with changed interface MAC address: this causes interface-update event to fail and host becomes unreachable. I guess the same thing would happen in case 1.

Note: after restore nethserver-sogo-update event must be triggered to reset mysql password for sogo user. We must backup /etc/openldap/*.pw files (or more appropriately, move *.pw files under /var/lib/nethserver/...) see #2063.

#8 Updated by Davide Principi about 8 years ago

Also note that nslcd must be stopped before stopping slapd to avoid log messages like this:

Jul 16 14:27:49 davidep2 nslcd[3666]: [fff902] ldap_search_ext() failed: Can't contact LDAP server
Jul 16 14:27:49 davidep2 nslcd[3666]: [fff902] no available LDAP server found

#9 Updated by Giacomo Sanchietti about 8 years ago

Modifications to nethserver-mysql package break package installation process: nethserver-mysql-conf do not start mysql daemon on first install.

Fix:

--- nethserver-mysql-conf.ori    2013-07-04 14:49:58.147409157 +0000
+++ nethserver-mysql-conf    2013-07-04 14:50:15.614393501 +0000
@@ -42,7 +42,7 @@
 if ( -f "$backup_dir/mysql.dump" ) {
     system("/etc/e-smith/events/actions/mysql-load-tables");
 } else { # or initialize the server
-    if ( system("/sbin/service mysqld status >/dev/null") == 0 ) {
+    if ( system("/sbin/service mysqld status >/dev/null") != 0 ) {
         system("/sbin/service mysqld start");
     }
     my $exp = new Expect;

#10 Updated by Giacomo Sanchietti about 8 years ago

New rpm in nethserver-testing repo:
  • nethserver-mysql-1.0.4
Changes:
  • correctly start mysqld on first install
  • remove unused Runlevels prop, requires:
    • nethserver-base-1.4.0-1.ns6.noarch.rpm
    • nethserver-lib-1.3.0-1.ns6.noarch.rpm
    • nethserver-yum-1.1.1-1.ns6.noarch.rpm
    • nethserver-nethgui-1.2.2-1.ns6.noarch.rpm

#11 Updated by Giacomo Sanchietti about 8 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee set to Giacomo Sanchietti
  • % Done changed from 30 to 60
Changes to post-restore-config event:
  • reset network interface configuration
  • document execution of bootstrap-console after configuration restore (see: Nethserver-backup)
Other changes:
  • Accept '*' character in include and exclude files

#12 Updated by Giacomo Sanchietti about 8 years ago

  • Status changed from MODIFIED to ON_QA
  • Assignee deleted (Giacomo Sanchietti)
  • % Done changed from 60 to 70
New packages in nethserver-testing:
  • nethserver-backup-data-1.0.5
  • nethserver-backup-config-1.0.4
Test case 3
  • Add an exclusion to /etc/backup-data.d/custom.exclude
  • Execute: backup-data
  • Check the excluded file is not in backup

Example of exclusion:

/var/lib/nethserver/vmail/giacomo/
**giacomo**

List files in backup (saved on a NFS share):

/etc/e-smith/events/actions/mount-nfs
/usr/bin/duplicity list-current-files --archive-dir /var/lib/nethserver/backup/duplicity/ file:///mnt/backup/test
/etc/e-smith/events/actions/umount-nfs

#13 Updated by Davide Principi almost 8 years ago

  • Assignee set to Davide Principi

#14 Updated by Davide Principi almost 8 years ago

  • Status changed from ON_QA to TRIAGED
  • Assignee deleted (Davide Principi)
  • % Done changed from 70 to 20

Test case 1, PASSED NOTE

During restore-config the backup-data key configuration DB is overwritten with values from the backup.

Test case 2, FAILED

  1. See note above
  2. After restore-config ldap server is down and passwd backend is not available:
       # ldapsearch -Y EXTERNAL
    ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)
       # id nsstest
    id: nsstest: No such user
    

Test case 3, PASSED


NOTE
In /etc/backup-data.d/custom.exclude empty lines are considered invalid. Can we skip them? Added a note to nethserver-backup.

Extract from log file /var/log/last-backup.log:

Import of duplicity.backends.giobackend Failed: No module named gio
Import of duplicity.backends.sshbackend Failed: No module named paramiko
Reading globbing filelist /tmp/ql4gqwEmxq
Fatal Error: The file specification

cannot match any files in the base directory
    /
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.

#15 Updated by Davide Principi almost 8 years ago

  • Status changed from TRIAGED to MODIFIED
  • % Done changed from 20 to 60

The test case 2 was missing an important step:
before executing restore-data the bootstrap-console command must be issued.

#16 Updated by Davide Principi almost 8 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

#17 Updated by Davide Principi almost 8 years ago

  • NEEDINFO changed from No to Yes

Should backup-data always execute backup-config automatically?

#18 Updated by Giacomo Sanchietti almost 8 years ago

Should backup-data always execute backup-config automatically?

It's a nice enhancement but it's not blocking. I propose to open another issue (#2118).

#19 Updated by Davide Principi almost 8 years ago

  • Status changed from ON_QA to VERIFIED
  • % Done changed from 70 to 90
  • NEEDINFO changed from Yes to No

#20 Updated by Davide Principi almost 8 years ago

  • Status changed from VERIFIED to CLOSED
  • % Done changed from 90 to 100

In nethserver-updates:
nethserver-backup-config-1.0.4-1.ns6.noarch.rpm
nethserver-backup-data-1.0.6-1.ns6.noarch.rpm

Also available in: Atom PDF