Bug #2786

Cannot access Server Manager after migration

Added by Nicola Rauso about 7 years ago. Updated almost 7 years ago.

Status:CLOSEDStart date:
Priority:NormalDue date:
Assignee:-% Done:

100%

Category:nethserver-lib
Target version:v6.5
Security class: Resolution:
Affected version:v6.5-final NEEDINFO:No

Description

After migration from NethService/SME Server you cannot access Server Manager.
You can reach the login form but after the authentication the Server Manager generates an error:


Nethgui:

500 - Internal server error [1366796122]

accounts - accounts db from NethService 8.2 (782 Bytes) Nicola Rauso, 07/01/2014 11:24 AM

messages - messages log from NethServer 6.5 (7.33 KB) Nicola Rauso, 07/01/2014 11:24 AM

0001-Encode-UTF-8-strings-safely.-Refs-2786.patch Magnifier - nethserver-httpd-admin patch (1.12 KB) Davide Principi, 09/29/2014 06:28 AM

0001-esmith-db-safely-decode-DB-data-to-UTF-8-strings.-Re.patch Magnifier - nethserver-lib patch (2.13 KB) Davide Principi, 09/29/2014 06:39 AM

Associated revisions

Revision 3b6ce3a9
Added by Giacomo Sanchietti almost 7 years ago

Migration: encode db in UTF8. Refs #2786

Revision 6a7baf54
Added by Davide Principi almost 7 years ago

Encode UTF-8 strings safely. Refs #2786

Assumes esmith::db layer returns strings marked as UTF-8.

Revision 37b325b2
Added by Davide Principi almost 7 years ago

esmith::db: safely decode DB data to UTF-8 strings. Refs #2786

JSON functions require properly decoded UTF-8 strings to work correctly.

Revision fddcddcc
Added by Davide Principi almost 7 years ago

Revert "Migration: encode db in UTF8. Refs #2786"

This reverts commit 3b6ce3a9a0031a3792e99602af8921c597e9c2b0.

History

#1 Updated by Davide Principi about 7 years ago

  • Subject changed from Cannot access NethGUI after migration to Cannot access Server Manager after migration
  • Description updated (diff)
  • NEEDINFO changed from No to Yes

What's the error message in /var/log/messages?

Please, paste here the lines with EXCEPTION text.

#2 Updated by Nicola Rauso about 7 years ago

  • File accounts added
  • File messages added
  • Subject changed from Cannot access Server Manager after migration to Cannot access NethGUI after migration
  • Description updated (diff)
  • NEEDINFO changed from Yes to No

Trying to access NethGUI generates this exception:

httpd-admin: [EXCEPTION] UnexpectedValueException: Nethgui\System\EsmithDatabase: unexpected json string `[{"props` (in /usr/share/nethesis/Nethgui/System/EsmithDatabase.php:156)

messages log reports many user-modify and group-modify events errors related to migration (see attachment).

Installed packages:

  • nethserver-mysql-1.0.5-1.ns6.noarch
  • nethserver-mail-server-1.7.0-1.ns6.noarch
  • nethserver-backup-data-1.0.9-1.ns6.noarch
  • nethserver-sogo-thunderbird-1.1.1-1.ns6.noarch
  • nethserver-base-2.2.1-1.ns6.noarch
  • nethserver-hosts-1.0.7-1.ns6.noarch
  • nethserver-release-6.5-5.ns6.noarch
  • nethserver-backup-config-1.1.0-1.ns6.noarch
  • nethserver-yum-1.3.1-1.ns6.noarch
  • nethserver-php-1.1.1-1.ns6.noarch
  • nethserver-smartd-1.0.1-1.ns6.noarch
  • nethserver-ntp-1.0.6-1.ns6.noarch
  • nethserver-register-2.0.1-1.ns6.noarch
  • nethserver-httpd-admin-1.2.3-1.ns6.noarch
  • nethserver-httpd-2.3.1-1.ns6.noarch
  • nethserver-memcached-1.0.2-1.ns6.noarch
  • nethserver-sogo-1.4.0-1.ns6.noarch
  • nethserver-samba-1.4.4-1.ns6.noarch
  • nethserver-dnsmasq-1.1.1-1.ns6.noarch
  • nethserver-directory-2.0.3-1.ns6.noarch
  • nethserver-mail-common-1.3.3-1.ns6.noarch
  • nethserver-ibays-2.0.3-1.ns6.noarch
  • nethserver-nethgui-1.5.0-1.ns6.noarch
  • nethserver-openssh-1.0.5-1.ns6.noarch
  • nethserver-nethserverenterprise-branding-1.0.1-1.ns6.noarch
  • nethserver-lib-2.0.3-1.ns6.noarch

#3 Updated by Davide Principi about 7 years ago

  • Subject changed from Cannot access NethGUI after migration to Cannot access Server Manager after migration
  • Description updated (diff)
  • Status changed from NEW to TRIAGED
  • % Done changed from 0 to 20

Maybe, the migration of accounts DB with accented letters causes the problem with Server Manager.

#4 Updated by Giacomo Sanchietti almost 7 years ago

  • NEEDINFO changed from No to Yes

Is it really a problem about db encoding? Anyone can reproduce the problem?

Please report if the issue is reproducible, otherwise close this bug.

#5 Updated by Giacomo Sanchietti almost 7 years ago

  • Status changed from TRIAGED to ON_DEV
  • Assignee set to Giacomo Sanchietti
  • % Done changed from 20 to 30
  • NEEDINFO changed from Yes to No

The bug is reproducible with attached database dump.

To avoid problems, all database should be converted to UTF-8 before starting the real migration.

#6 Updated by Giacomo Sanchietti almost 7 years ago

  • Status changed from ON_DEV to MODIFIED
  • % Done changed from 30 to 60

#7 Updated by Giacomo Sanchietti almost 7 years ago

  • Status changed from MODIFIED to ON_QA
  • Assignee deleted (Giacomo Sanchietti)
  • % Done changed from 60 to 70
Package in nethserver-testing:
  • nethserver-base-2.3.0-2.0git3b6ce3a9.ns6.noarch.rpm
Test case
  • On a clean machine install nethserver-directory
  • Update nethserver-base from testing repository
  • Prepare a fake migration environment:
    mkdir -p /var/lib/migrate/home/e-smith/db
cp /var/lib/nethserver/db/configuration /var/lib/migrate/home/e-smith/db/configuration
mkdir -p /var/lib/migrate/home/e-smith/files/users/admin
mkdir -p /var/lib/migrate/etc
touch /var/lib/migrate/etc/shadow
  • Copy the db dump attached to bug into /var/lib/migrate/home/e-smith/db/ directory
  • Execute the migration:
    /etc/e-smith/events/actions/nethserver-directory-migrate migration-import /var/lib/migrate
  • Check the conversion is completed successfully:
    grep nethserver-base-encode-db /var/log/messages
  • Make sure the new db accounts is in UTF8 format:
    file /var/lib/migrate/home/e-smith/db/accounts
  • Open the web interface and check all is working correctly. You will find some bad characters between user's details but you must be able to correct them and apply modifications

#8 Updated by Davide Principi almost 7 years ago

Giacomo Sanchietti wrote:

To avoid problems, all database should be converted to UTF-8 before starting the real migration.

If the back up set is on a different device than /var/lib/nethserver, files are copied and not moved.

The given solution alters the source database. Maybe this is not always feasible: just think about a read only network filesystem.

#9 Updated by Giacomo Sanchietti almost 7 years ago

The given solution alters the source database. Maybe this is not always feasible: just think about a read only network filesystem.

As far as I remember, there isn't a rule about modifications on source files, neither any requirements about read-only filesystems.
I think converting the original databases is perfectly fine.

By the way, if the error on the Web UI can describe the error, we should simply remove the code and create a simple documentation page.
But I think having all databases in UTF-8 format it's the best solution.

If the modification is not needed, please remove the commit from the master branch.

#10 Updated by Davide Principi almost 7 years ago

Giacomo Sanchietti wrote:

But I think having all databases in UTF-8 format it's the best solution

Me too, if we want to be sure all data is UTF-8 we could check it in esmith::config.

Possible ways to recover:
  • substitute bad sequences
  • try to re-encode
  • ...

I'll dig Perl docs to see if a solution exists.

#11 Updated by Davide Principi almost 7 years ago

The attached patches to nethserver-lib (esmith::db) and nethserver-httpd-admin (smwingsd) ensure JSON strings are formed correctly, replacing bad UTF-8 sequences with the Unicode REPLACEMENT CHARACTER, code point U+FFFD.

See http://perldoc.perl.org/Encode.html#List-of-_CHECK_-values

The problem arises when JSON strings are encoded: esmith::DB libraries treats strings as binary octets, but Perl JSON module requires properly decode()d strings.

The decode() fallback mechanism FB_DEFAULT ensure strings are always decoded safely.

#12 Updated by Davide Principi almost 7 years ago

  • Status changed from ON_QA to TRIAGED
  • % Done changed from 70 to 20

#13 Updated by Davide Principi almost 7 years ago

  • Status changed from TRIAGED to ON_DEV
  • Assignee set to Davide Principi
  • % Done changed from 20 to 30

#14 Updated by Davide Principi almost 7 years ago

  • Status changed from ON_DEV to MODIFIED
  • Assignee deleted (Davide Principi)
  • % Done changed from 30 to 60
  • Reverted commit:3b6ce3a9
  • Applied attached patches

#15 Updated by Davide Principi almost 7 years ago

  • Status changed from MODIFIED to ON_QA
  • % Done changed from 60 to 70

In nethserver-testing:
nethserver-base-2.3.0-5.0gitfddcddcc.ns6.noarch.rpm
nethserver-httpd-admin-1.3.0-3.5git6a7baf5.ns6.noarch.rpm
nethserver-lib-2.1.0-1.0git37b325b2.ns6.noarch.rpm

#16 Updated by Davide Principi almost 7 years ago

  • Category changed from nethserver-directory to nethserver-lib

#17 Updated by Giacomo Sanchietti almost 7 years ago

  • Assignee set to Giacomo Sanchietti

#18 Updated by Giacomo Sanchietti almost 7 years ago

  • Status changed from ON_QA to VERIFIED
  • Assignee deleted (Giacomo Sanchietti)
  • % Done changed from 70 to 90

After migration Server Manager still works and displays special UTF characters for bad encoded data.

#19 Updated by Davide Principi almost 7 years ago

  • Status changed from VERIFIED to CLOSED
  • % Done changed from 90 to 100

In nethserver-updates:
nethserver-base-2.4.0-1.ns6.noarch.rpm
nethserver-lib-2.1.1-1.ns6.noarch.rpm
nethserver-httpd-admin-1.3.1-1.ns6.noarch.rpm

Also available in: Atom PDF