Bug #2786
Cannot access Server Manager after migration
Status: | CLOSED | Start date: | ||
---|---|---|---|---|
Priority: | Normal | Due date: | ||
Assignee: | - | % Done: | 100% | |
Category: | nethserver-lib | |||
Target version: | v6.5 | |||
Security class: | Resolution: | |||
Affected version: | v6.5-final | NEEDINFO: | No |
Description
After migration from NethService/SME Server you cannot access Server Manager.
You can reach the login form but after the authentication the Server Manager generates an error:
Nethgui: 500 - Internal server error [1366796122]
Associated revisions
Migration: encode db in UTF8. Refs #2786
Encode UTF-8 strings safely. Refs #2786
Assumes esmith::db layer returns strings marked as UTF-8.
esmith::db: safely decode DB data to UTF-8 strings. Refs #2786
JSON functions require properly decoded UTF-8 strings to work correctly.
Revert "Migration: encode db in UTF8. Refs #2786"
This reverts commit 3b6ce3a9a0031a3792e99602af8921c597e9c2b0.
History
#1 Updated by Davide Principi about 7 years ago
- Subject changed from Cannot access NethGUI after migration to Cannot access Server Manager after migration
- Description updated (diff)
- NEEDINFO changed from No to Yes
What's the error message in /var/log/messages
?
Please, paste here the lines with EXCEPTION
text.
#2 Updated by Nicola Rauso about 7 years ago
- File accounts added
- File messages added
- Subject changed from Cannot access Server Manager after migration to Cannot access NethGUI after migration
- Description updated (diff)
- NEEDINFO changed from Yes to No
Trying to access NethGUI generates this exception:
httpd-admin: [EXCEPTION] UnexpectedValueException: Nethgui\System\EsmithDatabase: unexpected json string `[{"props` (in /usr/share/nethesis/Nethgui/System/EsmithDatabase.php:156)
messages
log reports many user-modify
and group-modify
events errors related to migration (see attachment).
Installed packages:
- nethserver-mysql-1.0.5-1.ns6.noarch
- nethserver-mail-server-1.7.0-1.ns6.noarch
- nethserver-backup-data-1.0.9-1.ns6.noarch
- nethserver-sogo-thunderbird-1.1.1-1.ns6.noarch
- nethserver-base-2.2.1-1.ns6.noarch
- nethserver-hosts-1.0.7-1.ns6.noarch
- nethserver-release-6.5-5.ns6.noarch
- nethserver-backup-config-1.1.0-1.ns6.noarch
- nethserver-yum-1.3.1-1.ns6.noarch
- nethserver-php-1.1.1-1.ns6.noarch
- nethserver-smartd-1.0.1-1.ns6.noarch
- nethserver-ntp-1.0.6-1.ns6.noarch
- nethserver-register-2.0.1-1.ns6.noarch
- nethserver-httpd-admin-1.2.3-1.ns6.noarch
- nethserver-httpd-2.3.1-1.ns6.noarch
- nethserver-memcached-1.0.2-1.ns6.noarch
- nethserver-sogo-1.4.0-1.ns6.noarch
- nethserver-samba-1.4.4-1.ns6.noarch
- nethserver-dnsmasq-1.1.1-1.ns6.noarch
- nethserver-directory-2.0.3-1.ns6.noarch
- nethserver-mail-common-1.3.3-1.ns6.noarch
- nethserver-ibays-2.0.3-1.ns6.noarch
- nethserver-nethgui-1.5.0-1.ns6.noarch
- nethserver-openssh-1.0.5-1.ns6.noarch
- nethserver-nethserverenterprise-branding-1.0.1-1.ns6.noarch
- nethserver-lib-2.0.3-1.ns6.noarch
#3 Updated by Davide Principi about 7 years ago
- Subject changed from Cannot access NethGUI after migration to Cannot access Server Manager after migration
- Description updated (diff)
- Status changed from NEW to TRIAGED
- % Done changed from 0 to 20
Maybe, the migration of accounts DB with accented letters causes the problem with Server Manager.
#4 Updated by Giacomo Sanchietti almost 7 years ago
- NEEDINFO changed from No to Yes
Is it really a problem about db encoding? Anyone can reproduce the problem?
Please report if the issue is reproducible, otherwise close this bug.
#5 Updated by Giacomo Sanchietti almost 7 years ago
- Status changed from TRIAGED to ON_DEV
- Assignee set to Giacomo Sanchietti
- % Done changed from 20 to 30
- NEEDINFO changed from Yes to No
The bug is reproducible with attached database dump.
To avoid problems, all database should be converted to UTF-8 before starting the real migration.
#6 Updated by Giacomo Sanchietti almost 7 years ago
- Status changed from ON_DEV to MODIFIED
- % Done changed from 30 to 60
#7 Updated by Giacomo Sanchietti almost 7 years ago
- Status changed from MODIFIED to ON_QA
- Assignee deleted (
Giacomo Sanchietti) - % Done changed from 60 to 70
- nethserver-base-2.3.0-2.0git3b6ce3a9.ns6.noarch.rpm
- On a clean machine install nethserver-directory
- Update nethserver-base from testing repository
- Prepare a fake migration environment:
mkdir -p /var/lib/migrate/home/e-smith/db cp /var/lib/nethserver/db/configuration /var/lib/migrate/home/e-smith/db/configuration mkdir -p /var/lib/migrate/home/e-smith/files/users/admin mkdir -p /var/lib/migrate/etc touch /var/lib/migrate/etc/shadow
- Copy the db dump attached to bug into
/var/lib/migrate/home/e-smith/db/ directory
- Execute the migration:
/etc/e-smith/events/actions/nethserver-directory-migrate migration-import /var/lib/migrate
- Check the conversion is completed successfully:
grep nethserver-base-encode-db /var/log/messages
- Make sure the new db accounts is in UTF8 format:
file /var/lib/migrate/home/e-smith/db/accounts
- Open the web interface and check all is working correctly. You will find some bad characters between user's details but you must be able to correct them and apply modifications
#8 Updated by Davide Principi almost 7 years ago
Giacomo Sanchietti wrote:
To avoid problems, all database should be converted to UTF-8 before starting the real migration.
If the back up set is on a different device than /var/lib/nethserver, files are copied and not moved.
The given solution alters the source database. Maybe this is not always feasible: just think about a read only network filesystem.
#9 Updated by Giacomo Sanchietti almost 7 years ago
The given solution alters the source database. Maybe this is not always feasible: just think about a read only network filesystem.
As far as I remember, there isn't a rule about modifications on source files, neither any requirements about read-only filesystems.
I think converting the original databases is perfectly fine.
By the way, if the error on the Web UI can describe the error, we should simply remove the code and create a simple documentation page.
But I think having all databases in UTF-8 format it's the best solution.
If the modification is not needed, please remove the commit from the master branch.
#10 Updated by Davide Principi almost 7 years ago
Giacomo Sanchietti wrote:
But I think having all databases in UTF-8 format it's the best solution
Me too, if we want to be sure all data is UTF-8 we could check it in esmith::config
.
- substitute bad sequences
- try to re-encode
- ...
I'll dig Perl docs to see if a solution exists.
#11 Updated by Davide Principi almost 7 years ago
- File 0001-Encode-UTF-8-strings-safely.-Refs-2786.patch added
- File 0001-esmith-db-safely-decode-DB-data-to-UTF-8-strings.-Re.patch added
The attached patches to nethserver-lib (esmith::db) and nethserver-httpd-admin (smwingsd) ensure JSON strings are formed correctly, replacing bad UTF-8 sequences with the Unicode REPLACEMENT CHARACTER, code point U+FFFD.
See http://perldoc.perl.org/Encode.html#List-of-_CHECK_-values
The problem arises when JSON strings are encoded: esmith::DB
libraries treats strings as binary octets, but Perl JSON module requires properly decode()d strings.
The decode()
fallback mechanism FB_DEFAULT
ensure strings are always decoded safely.
#12 Updated by Davide Principi almost 7 years ago
- Status changed from ON_QA to TRIAGED
- % Done changed from 70 to 20
#13 Updated by Davide Principi almost 7 years ago
- Status changed from TRIAGED to ON_DEV
- Assignee set to Davide Principi
- % Done changed from 20 to 30
#14 Updated by Davide Principi almost 7 years ago
- Status changed from ON_DEV to MODIFIED
- Assignee deleted (
Davide Principi) - % Done changed from 30 to 60
- Reverted commit:3b6ce3a9
- Applied attached patches
#15 Updated by Davide Principi almost 7 years ago
- Status changed from MODIFIED to ON_QA
- % Done changed from 60 to 70
In nethserver-testing:
nethserver-base-2.3.0-5.0gitfddcddcc.ns6.noarch.rpm
nethserver-httpd-admin-1.3.0-3.5git6a7baf5.ns6.noarch.rpm
nethserver-lib-2.1.0-1.0git37b325b2.ns6.noarch.rpm
#16 Updated by Davide Principi almost 7 years ago
- Category changed from nethserver-directory to nethserver-lib
#17 Updated by Giacomo Sanchietti almost 7 years ago
- Assignee set to Giacomo Sanchietti
#18 Updated by Giacomo Sanchietti almost 7 years ago
- Status changed from ON_QA to VERIFIED
- Assignee deleted (
Giacomo Sanchietti) - % Done changed from 70 to 90
After migration Server Manager still works and displays special UTF characters for bad encoded data.
#19 Updated by Davide Principi almost 7 years ago
- Status changed from VERIFIED to CLOSED
- % Done changed from 90 to 100
In nethserver-updates:
nethserver-base-2.4.0-1.ns6.noarch.rpm
nethserver-lib-2.1.1-1.ns6.noarch.rpm
nethserver-httpd-admin-1.3.1-1.ns6.noarch.rpm