Enhancement #2904
Wrong signal-event order under certain circumstance
| Status: | CLOSED | Start date: | ||
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: | - | % Done: | 100%  | |
| Category: | nethserver-yum | |||
| Target version: | v6.5 | |||
| Resolution: | NEEDINFO: | No | 
Description
nethserver_events.py yum plugin (nethserver-yum package) launch package-name-update signal-event when packages are installed. 
If multiple packages are installed, the order of the events matter, because some events may rely on services to be configured. For example, if a package needs to create a mysql database, nethserver-mysql-update signal-event has to be launched before the installation event of this package.
The nethserver_events.py algorithm take the list of packages (that have an -update event) and build the ordered list of signal-event to launch putting required packages signal-event before the package itself
def read_package_list():
    """ 
    Query RPM db for any nethserver* package and return the list of
    package names, preserving the dependency order. RPM 'Require' tags
    are parsed.
    """ 
    packages = []
    ts = rpm.TransactionSet()
    mi = ts.dbMatch()
    for h in mi:
        if not has_update_event(h['name']):
            continue
        for dep in h[rpm.RPMTAG_REQUIRENAME]:
            if not has_update_event(dep):
                continue
            try:
                # insert deps before the package name
                packages.insert(packages.index(h['name']), dep)
            except ValueError:
                packages.append(dep)
        if not h['name'] in packages:
            packages.append(h['name'])
    # Reduce the package list preserving the element order:
    return list_unique(packages)
	The algorithm fails under certain cirmstances: 
if we have the following list of package-> dep1, dep2, dep3...:
A B -> E C D -> C,A E -> D,C,A
At each iteration, dependencies are putted in the final list before the first occurrence of the package
At 1st iteration, resulting list will be the following because A doesn't have dependencies
A B C D E
At 2nd iteration, E will be putted before B (and also D,C and A that are dependencies of E):
A E D C A B C D E
At 3rd nothing changes because C has no deps, but at 4th C and A are putted before D:
A E C A D C A B C D E
At 5th  D,C,A are putted before E but they are putted before the 1st occurrence of E:
A D C A E C A D C A B C D E
And here is the problem, order of D dependencies is defined in 4th iteration, but now they are putted in the top of the list (before E) in an order that isn't defined anywhere.
After list_uniq, the new list will be
A D C E B
D dependencies aren't satisfied Proposed solution:
- in rpm, dependencies order is defined with a tsort function https://en.wikipedia.org/wiki/Tsort 
here https://github.com/tengu/py-tsort a python tsort impelentation (look at description: "Useful for dependency resolution") 
Associated revisions
Use tsort algorithm for sorting -update events after packages installation. Refs #2904
Ensure only a tree of deps exists. Refs #2904
Add a fake "$root" package, then remove it.
History
#1
    
    Updated by Stefano Fancello almost 7 years ago
    - Status changed from NEW to TRIAGED
 - % Done changed from 0 to 20
 
#2
    
    Updated by Stefano Fancello almost 7 years ago
    - Status changed from TRIAGED to ON_DEV
 - Assignee set to Stefano Fancello
 - % Done changed from 20 to 30
 
#3
    
    Updated by Stefano Fancello almost 7 years ago
    here the patch for proposed solution with some extra DEBUG lines
diff --git a/usr/lib/yum-plugins/nethserver_events.py b/usr/lib/yum-plugins/nethserver_events.py
index 947900b..36e8fa8 100644
--- a/usr/lib/yum-plugins/nethserver_events.py
+++ b/usr/lib/yum-plugins/nethserver_events.py
@@ -64,9 +64,11 @@ def read_package_list():
     are parsed.
     """ 
-    packages = []
     ts = rpm.TransactionSet()
     mi = ts.dbMatch()
+    packages = []
+    touples = []
+
     for h in mi:
         if not has_update_event(h['name']):
             continue
@@ -74,18 +76,70 @@ def read_package_list():
         for dep in h[rpm.RPMTAG_REQUIRENAME]:
             if not has_update_event(dep):
                 continue
-
-            try:
-                # insert deps before the package name
-                packages.insert(packages.index(h['name']), dep)
-            except ValueError:
-                packages.append(dep)
-
-        if not h['name'] in packages:
-            packages.append(h['name'])
-
-    # Reduce the package list preserving the element order:
-    return list_unique(packages)
+           print "DEBUG:\t" + dep + " is a dependency of " + h['name']
+            touples.append([dep,h['name']])
+
+    packages = topological_sort(touples)
+
+    print "DEBUG: events: " 
+    for p in packages:
+               print "DEBUG: \t" + p
+    return packages
+
+class GraphError(Exception):
+    pass
+
+def topological_sort(edges):
+    """topologically sort vertices in edges.
+    edges: list of pairs of vertices. Edges must form a DAG.
+           If the graph has a cycle, then GraphError is raised.
+    returns: topologically sorted list of vertices.
+    see http://en.wikipedia.org/wiki/Topological_sorting
+
+    thanks to https://github.com/tengu/py-tsort
+    """ 
+    # resulting list
+    L=[]
+
+    # maintain forward and backward edge maps in parallel.
+    st,ts={},{}
+
+    def prune(s,t):
+        del st[s][t]
+        del ts[t][s]
+
+    def add(s,t):
+        try:
+            st.setdefault(s,{})[t]=1
+        except Exception, e:
+            raise RuntimeError(e, (s,t))
+        ts.setdefault(t,{})[s]=1
+
+    for s,t in edges:
+        add(s,t)
+
+    # frontier
+    S=set(st.keys()).difference(ts.keys())
+
+    while S:
+        s=S.pop()
+        L.append(s)
+        for t in st.get(s,{}).keys():
+            prune(s,t)
+            if not ts[t]:       # new frontier
+                S.add(t)
+
+    if filter(None, st.values()): # we have a cycle. report the cycle.
+        def traverse(vs, seen):
+            for s in vs:
+                if s in seen:
+                    raise GraphError('contains cycle: ', seen)
+                seen.append(s) # xx use ordered set..
+                traverse(st[s].keys(), seen)
+        traverse(st.keys(), list())
+        assert False, 'should not reach..'
+
+    return L
 def list_unique(l):
     s = set()
	patching your /usr/lib/yum-plugins/nethserver_events.py before installing any nethserver package/group, event order will be computed using tsort and will be also printed to stdout
#4
    
    Updated by Stefano Fancello almost 7 years ago
    - Status changed from ON_DEV to MODIFIED
 - Assignee deleted (
Stefano Fancello) - % Done changed from 30 to 60
 
commit fd516c3e12236e1888e47996b180778f2e073051
Test case:- Update nethserver-yum
 - install packages with a lot of dependencies (try "yum groupinstall GROUPS", where GROUPS are one or more groups from yum grouplist)
 - check /var/log/messages for FAIL(s) or ERROR(s). An event that needs another before should fail if the order is wrong. Also make sure that every expected event is executed.
 
#5
    
    Updated by Stefano Fancello almost 7 years ago
    - Status changed from MODIFIED to ON_QA
 - % Done changed from 60 to 70
 
In nethserver-testing:nethserver-yum-1.3.3-1.0gitfd516c3e.ns6.noarch.rpm
nethserver-yum-1.3.3-2.0git31bda416.ns6.noarch.rpm
#6
    
    Updated by Davide Principi almost 7 years ago
    - Assignee set to Davide Principi
 
#7
    
    Updated by Davide Principi almost 7 years ago
    - Assignee deleted (
Davide Principi) 
Tested an everything-install: no errors occurred.
But the nethserver-release-update event was missing. Added a patch that transform a forest of deps in a single tree. Verify again.
#8
    
    Updated by Giacomo Sanchietti over 6 years ago
    - Assignee set to Giacomo Sanchietti
 
#9
    
    Updated by Giacomo Sanchietti over 6 years ago
    - Status changed from ON_QA to VERIFIED
 - Assignee deleted (
Giacomo Sanchietti) - % Done changed from 70 to 90
 
Tested on multiple installations and no problems reported.
Marking as VERIFIED:
#10
    
    Updated by Davide Principi over 6 years ago
    - Status changed from VERIFIED to CLOSED
 - % Done changed from 90 to 100
 
In nethserver-updates:
nethserver-yum-1.3.4-1.ns6.noarch.rpm