Improved the quiet task extraction process by implementing a threaded solution.

Wed, 31 Aug 2016 18:20:40 +0200

author
Detlev Offenbach <detlev@die-offenbachs.de>
date
Wed, 31 Aug 2016 18:20:40 +0200
changeset 5117
1faa0780ae1d
parent 5115
ed2f37c1f6b6
child 5118
4a79a6153485
child 5121
d5d2cbf49daf

Improved the quiet task extraction process by implementing a threaded solution.

APIs/Python3/eric6.api file | annotate | diff | comparison | revisions
APIs/Python3/eric6.bas file | annotate | diff | comparison | revisions
Documentation/Help/source.qch file | annotate | diff | comparison | revisions
Documentation/Help/source.qhp file | annotate | diff | comparison | revisions
Documentation/Source/eric6.Tasks.TaskViewer.html file | annotate | diff | comparison | revisions
Project/Project.py file | annotate | diff | comparison | revisions
Tasks/TaskViewer.py file | annotate | diff | comparison | revisions
--- a/APIs/Python3/eric6.api	Tue Aug 30 20:13:21 2016 +0200
+++ b/APIs/Python3/eric6.api	Wed Aug 31 18:20:40 2016 +0200
@@ -8351,6 +8351,11 @@
 eric6.Tasks.TaskPropertiesDialog.TaskPropertiesDialog.setReadOnly?4()
 eric6.Tasks.TaskPropertiesDialog.TaskPropertiesDialog.setSubTaskMode?4(projectTask)
 eric6.Tasks.TaskPropertiesDialog.TaskPropertiesDialog?1(task=None, parent=None, projectOpen=False)
+eric6.Tasks.TaskViewer.ProjectTaskExtractionThread.requestInterrupt?4()
+eric6.Tasks.TaskViewer.ProjectTaskExtractionThread.run?4()
+eric6.Tasks.TaskViewer.ProjectTaskExtractionThread.scan?4(markers, files)
+eric6.Tasks.TaskViewer.ProjectTaskExtractionThread.taskFound?7
+eric6.Tasks.TaskViewer.ProjectTaskExtractionThread?1(parent=None)
 eric6.Tasks.TaskViewer.TaskViewer.addFileTask?4(summary, filename, lineno, taskType=Task.TypeTodo, description="")
 eric6.Tasks.TaskViewer.TaskViewer.addTask?4(summary, priority=1, filename="", lineno=0, completed=False, _time=0, isProjectTask=False, taskType=Task.TypeTodo, description="", uid="", parentTask=None)
 eric6.Tasks.TaskViewer.TaskViewer.clearFileTasks?4(filename, conditionally=False)
@@ -8364,6 +8369,7 @@
 eric6.Tasks.TaskViewer.TaskViewer.regenerateProjectTasks?4(quiet=False)
 eric6.Tasks.TaskViewer.TaskViewer.saveProjectTasks?4()
 eric6.Tasks.TaskViewer.TaskViewer.setProjectOpen?4(o=False)
+eric6.Tasks.TaskViewer.TaskViewer.stopProjectTaskExtraction?4()
 eric6.Tasks.TaskViewer.TaskViewer?1(parent, project)
 eric6.Templates.TemplateHelpDialog.TemplateHelpDialog?1(parent=None)
 eric6.Templates.TemplateMultipleVariablesDialog.TemplateMultipleVariablesDialog.getVariables?4()
--- a/APIs/Python3/eric6.bas	Tue Aug 30 20:13:21 2016 +0200
+++ b/APIs/Python3/eric6.bas	Wed Aug 31 18:20:40 2016 +0200
@@ -605,6 +605,7 @@
 ProjectReader XMLStreamReaderBase
 ProjectResourcesBrowser ProjectBaseBrowser
 ProjectSourcesBrowser ProjectBaseBrowser
+ProjectTaskExtractionThread QThread
 ProjectTranslationsBrowser ProjectBaseBrowser
 ProjectWriter XMLStreamWriterBase
 PropertiesDialog QDialog Ui_PropertiesDialog
Binary file Documentation/Help/source.qch has changed
--- a/Documentation/Help/source.qhp	Tue Aug 30 20:13:21 2016 +0200
+++ b/Documentation/Help/source.qhp	Wed Aug 31 18:20:40 2016 +0200
@@ -11238,6 +11238,11 @@
       <keyword name="ProjectSourcesBrowser._createPopupMenus" id="ProjectSourcesBrowser._createPopupMenus" ref="eric6.Project.ProjectSourcesBrowser.html#ProjectSourcesBrowser._createPopupMenus" />
       <keyword name="ProjectSourcesBrowser._openItem" id="ProjectSourcesBrowser._openItem" ref="eric6.Project.ProjectSourcesBrowser.html#ProjectSourcesBrowser._openItem" />
       <keyword name="ProjectSourcesBrowser._projectClosed" id="ProjectSourcesBrowser._projectClosed" ref="eric6.Project.ProjectSourcesBrowser.html#ProjectSourcesBrowser._projectClosed" />
+      <keyword name="ProjectTaskExtractionThread" id="ProjectTaskExtractionThread" ref="eric6.Tasks.TaskViewer.html#ProjectTaskExtractionThread" />
+      <keyword name="ProjectTaskExtractionThread (Constructor)" id="ProjectTaskExtractionThread (Constructor)" ref="eric6.Tasks.TaskViewer.html#ProjectTaskExtractionThread.__init__" />
+      <keyword name="ProjectTaskExtractionThread.requestInterrupt" id="ProjectTaskExtractionThread.requestInterrupt" ref="eric6.Tasks.TaskViewer.html#ProjectTaskExtractionThread.requestInterrupt" />
+      <keyword name="ProjectTaskExtractionThread.run" id="ProjectTaskExtractionThread.run" ref="eric6.Tasks.TaskViewer.html#ProjectTaskExtractionThread.run" />
+      <keyword name="ProjectTaskExtractionThread.scan" id="ProjectTaskExtractionThread.scan" ref="eric6.Tasks.TaskViewer.html#ProjectTaskExtractionThread.scan" />
       <keyword name="ProjectTranslationsBrowser" id="ProjectTranslationsBrowser" ref="eric6.Project.ProjectTranslationsBrowser.html#ProjectTranslationsBrowser" />
       <keyword name="ProjectTranslationsBrowser (Constructor)" id="ProjectTranslationsBrowser (Constructor)" ref="eric6.Project.ProjectTranslationsBrowser.html#ProjectTranslationsBrowser.__init__" />
       <keyword name="ProjectTranslationsBrowser (Module)" id="ProjectTranslationsBrowser (Module)" ref="eric6.Project.ProjectTranslationsBrowser.html" />
@@ -13947,6 +13952,7 @@
       <keyword name="TaskViewer.regenerateProjectTasks" id="TaskViewer.regenerateProjectTasks" ref="eric6.Tasks.TaskViewer.html#TaskViewer.regenerateProjectTasks" />
       <keyword name="TaskViewer.saveProjectTasks" id="TaskViewer.saveProjectTasks" ref="eric6.Tasks.TaskViewer.html#TaskViewer.saveProjectTasks" />
       <keyword name="TaskViewer.setProjectOpen" id="TaskViewer.setProjectOpen" ref="eric6.Tasks.TaskViewer.html#TaskViewer.setProjectOpen" />
+      <keyword name="TaskViewer.stopProjectTaskExtraction" id="TaskViewer.stopProjectTaskExtraction" ref="eric6.Tasks.TaskViewer.html#TaskViewer.stopProjectTaskExtraction" />
       <keyword name="Tasks (Package)" id="Tasks (Package)" ref="index-eric6.Tasks.html" />
       <keyword name="TasksPage" id="TasksPage" ref="eric6.Preferences.ConfigurationPages.TasksPage.html#TasksPage" />
       <keyword name="TasksPage (Constructor)" id="TasksPage (Constructor)" ref="eric6.Preferences.ConfigurationPages.TasksPage.html#TasksPage.__init__" />
--- a/Documentation/Source/eric6.Tasks.TaskViewer.html	Tue Aug 30 20:13:21 2016 +0200
+++ b/Documentation/Source/eric6.Tasks.TaskViewer.html	Wed Aug 31 18:20:40 2016 +0200
@@ -34,6 +34,9 @@
 <h3>Classes</h3>
 <table>
 <tr>
+<td><a href="#ProjectTaskExtractionThread">ProjectTaskExtractionThread</a></td>
+<td>Class implementing a thread to extract tasks related to a project.</td>
+</tr><tr>
 <td><a href="#TaskViewer">TaskViewer</a></td>
 <td>Class implementing the task viewer.</td>
 </tr>
@@ -43,6 +46,85 @@
 <tr><td>None</td></tr>
 </table>
 <hr /><hr />
+<a NAME="ProjectTaskExtractionThread" ID="ProjectTaskExtractionThread"></a>
+<h2>ProjectTaskExtractionThread</h2>
+<p>
+    Class implementing a thread to extract tasks related to a project.
+</p><h3>Signals</h3>
+<dl>
+<dt>taskFound(str, str, int, int)</dt>
+<dd>
+emitted with the task description,
+        the file name, the line number and task type to signal the presence of
+        a task
+</dd>
+</dl>
+<h3>Derived from</h3>
+QThread
+<h3>Class Attributes</h3>
+<table>
+<tr><td>None</td></tr>
+</table>
+<h3>Class Methods</h3>
+<table>
+<tr><td>None</td></tr>
+</table>
+<h3>Methods</h3>
+<table>
+<tr>
+<td><a href="#ProjectTaskExtractionThread.__init__">ProjectTaskExtractionThread</a></td>
+<td>Constructor</td>
+</tr><tr>
+<td><a href="#ProjectTaskExtractionThread.requestInterrupt">requestInterrupt</a></td>
+<td>Public method to request iterruption of the thread.</td>
+</tr><tr>
+<td><a href="#ProjectTaskExtractionThread.run">run</a></td>
+<td>Public thread method to scan the given files.</td>
+</tr><tr>
+<td><a href="#ProjectTaskExtractionThread.scan">scan</a></td>
+<td>Public method to scan the given list of files for tasks.</td>
+</tr>
+</table>
+<h3>Static Methods</h3>
+<table>
+<tr><td>None</td></tr>
+</table>
+<a NAME="ProjectTaskExtractionThread.__init__" ID="ProjectTaskExtractionThread.__init__"></a>
+<h4>ProjectTaskExtractionThread (Constructor)</h4>
+<b>ProjectTaskExtractionThread</b>(<i>parent=None</i>)
+<p>
+        Constructor
+</p><dl>
+<dt><i>parent</i></dt>
+<dd>
+reference to the parent object (QObject)
+</dd>
+</dl><a NAME="ProjectTaskExtractionThread.requestInterrupt" ID="ProjectTaskExtractionThread.requestInterrupt"></a>
+<h4>ProjectTaskExtractionThread.requestInterrupt</h4>
+<b>requestInterrupt</b>(<i></i>)
+<p>
+        Public method to request iterruption of the thread.
+</p><a NAME="ProjectTaskExtractionThread.run" ID="ProjectTaskExtractionThread.run"></a>
+<h4>ProjectTaskExtractionThread.run</h4>
+<b>run</b>(<i></i>)
+<p>
+        Public thread method to scan the given files.
+</p><a NAME="ProjectTaskExtractionThread.scan" ID="ProjectTaskExtractionThread.scan"></a>
+<h4>ProjectTaskExtractionThread.scan</h4>
+<b>scan</b>(<i>markers, files</i>)
+<p>
+        Public method to scan the given list of files for tasks.
+</p><dl>
+<dt><i>markers</i> (dict of lists of str)</dt>
+<dd>
+dictionary of defined task markers
+</dd><dt><i>files</i> (list of str)</dt>
+<dd>
+list of file names to be scanned
+</dd>
+</dl>
+<div align="right"><a href="#top">Up</a></div>
+<hr /><hr />
 <a NAME="TaskViewer" ID="TaskViewer"></a>
 <h2>TaskViewer</h2>
 <p>
@@ -171,6 +253,9 @@
 </tr><tr>
 <td><a href="#TaskViewer.setProjectOpen">setProjectOpen</a></td>
 <td>Public slot to set the project status.</td>
+</tr><tr>
+<td><a href="#TaskViewer.stopProjectTaskExtraction">stopProjectTaskExtraction</a></td>
+<td>Public method to stop the project task extraction thread.</td>
 </tr>
 </table>
 <h3>Static Methods</h3>
@@ -488,7 +573,12 @@
 <dd>
 flag indicating the project status
 </dd>
-</dl>
+</dl><a NAME="TaskViewer.stopProjectTaskExtraction" ID="TaskViewer.stopProjectTaskExtraction"></a>
+<h4>TaskViewer.stopProjectTaskExtraction</h4>
+<b>stopProjectTaskExtraction</b>(<i></i>)
+<p>
+        Public method to stop the project task extraction thread.
+</p>
 <div align="right"><a href="#top">Up</a></div>
 <hr />
 </body></html>
\ No newline at end of file
--- a/Project/Project.py	Tue Aug 30 20:13:21 2016 +0200
+++ b/Project/Project.py	Wed Aug 31 18:20:40 2016 +0200
@@ -2896,6 +2896,8 @@
         if not self.checkDirty():
             return False
         
+        e5App().getObject("TaskViewer").stopProjectTaskExtraction()
+        
         # save the user project properties
         if not noSave:
             self.__writeUserProperties()
--- a/Tasks/TaskViewer.py	Tue Aug 30 20:13:21 2016 +0200
+++ b/Tasks/TaskViewer.py	Wed Aug 31 18:20:40 2016 +0200
@@ -15,8 +15,9 @@
 
 import os
 import fnmatch
+import threading
 
-from PyQt5.QtCore import pyqtSignal, Qt
+from PyQt5.QtCore import pyqtSignal, Qt, QThread
 from PyQt5.QtWidgets import QHeaderView, QLineEdit, QTreeWidget, QDialog, \
     QInputDialog, QApplication, QMenu, QAbstractItemView, QTreeWidgetItem
 
@@ -77,6 +78,8 @@
         self.taskFilter.setActive(False)
         
         self.__projectTasksSaveTimer = AutoSaver(self, self.saveProjectTasks)
+        self.__projectTaskExtractionThread = ProjectTaskExtractionThread()
+        self.__projectTaskExtractionThread.taskFound.connect(self.addFileTask)
         
         self.__projectTasksMenu = QMenu(
             self.tr("P&roject Tasks"), self)
@@ -690,56 +693,56 @@
         self.clearProjectTasks(fileOnly=True)
         
         # now process them
-        if not quiet:
+        if quiet:
+            ppath = self.project.getProjectPath()
+            self.__projectTaskExtractionThread.scan(
+                markers, [os.path.join(ppath, f) for f in files])
+        else:
             progress = E5ProgressDialog(
                 self.tr("Extracting project tasks..."),
                 self.tr("Abort"), 0, len(files), self.tr("%v/%m Files"))
             progress.setMinimumDuration(0)
             progress.setWindowTitle(self.tr("Tasks"))
-        count = 0
-        
-        for file in files:
-            if not quiet:
+            count = 0
+            
+            ppath = self.project.getProjectPath()
+            for file in files:
                 progress.setLabelText(
                     self.tr("Extracting project tasks...\n{0}").format(file))
                 progress.setValue(count)
                 QApplication.processEvents()
                 if progress.wasCanceled():
                     break
-            
-            QApplication.processEvents()
-            
-            fn = os.path.join(self.project.ppath, file)
-            # read the file and split it into textlines
-            try:
-                text, encoding = Utilities.readEncodedFile(fn)
-                lines = text.splitlines()
-            except (UnicodeError, IOError):
+                
+                fn = os.path.join(ppath, file)
+                # read the file and split it into textlines
+                try:
+                    text, encoding = Utilities.readEncodedFile(fn)
+                    lines = text.splitlines()
+                except (UnicodeError, IOError):
+                    count += 1
+                    progress.setValue(count)
+                    continue
+                
+                # now search tasks and record them
+                lineIndex = 0
+                for line in lines:
+                    lineIndex += 1
+                    shouldBreak = False
+                    
+                    for taskType, taskMarkers in markers.items():
+                        for taskMarker in taskMarkers:
+                            index = line.find(taskMarker)
+                            if index > -1:
+                                task = line[index:]
+                                self.addFileTask(task, fn, lineIndex, taskType)
+                                shouldBreak = True
+                                break
+                        if shouldBreak:
+                            break
+                
                 count += 1
-                if not quiet:
-                    progress.setValue(count)
-                continue
             
-            # now search tasks and record them
-            lineIndex = 0
-            for line in lines:
-                lineIndex += 1
-                shouldBreak = False
-                
-                for taskType, taskMarkers in markers.items():
-                    for taskMarker in taskMarkers:
-                        index = line.find(taskMarker)
-                        if index > -1:
-                            task = line[index:]
-                            self.addFileTask(task, fn, lineIndex, taskType)
-                            shouldBreak = True
-                            break
-                    if shouldBreak:
-                        break
-            
-            count += 1
-        
-        if not quiet:
             progress.setValue(len(files))
     
     def __configure(self):
@@ -754,3 +757,101 @@
         """
         if self.projectOpen and Preferences.getProject("TasksProjectAutoSave"):
             self.project.writeTasks()
+    
+    def stopProjectTaskExtraction(self):
+        """
+        Public method to stop the project task extraction thread.
+        """
+        self.__projectTaskExtractionThread.requestInterrupt()
+        self.__projectTaskExtractionThread.wait()
+
+
+class ProjectTaskExtractionThread(QThread):
+    """
+    Class implementing a thread to extract tasks related to a project.
+    
+    @signal taskFound(str, str, int, int) emitted with the task description,
+        the file name, the line number and task type to signal the presence of
+        a task
+    """
+    taskFound = pyqtSignal(str, str, int, int)
+    
+    def __init__(self, parent=None):
+        """
+        Constructor
+        
+        @param parent reference to the parent object (QObject)
+        """
+        super(ProjectTaskExtractionThread, self).__init__()
+        
+        self.__lock = threading.Lock()
+        self.__interrupt = False
+    
+    def requestInterrupt(self):
+        """
+        Public method to request iterruption of the thread.
+        """
+        if self.isRunning():
+            self.__interrupt = True
+    
+    def scan(self, markers, files):
+        """
+        Public method to scan the given list of files for tasks.
+        
+        @param markers dictionary of defined task markers
+        @type dict of lists of str
+        @param files list of file names to be scanned
+        @type list of str
+        """
+        with self.__lock:
+            self.__interrupt = False
+            self.__files = files[:]
+            self.__markers = {}
+            for markerType in markers:
+                self.__markers[markerType] = markers[markerType][:]
+            
+            if not self.isRunning():
+                self.start(QThread.LowPriority)
+    
+    def run(self):
+        """
+        Public thread method to scan the given files.
+        """
+        with self.__lock:
+            files = self.__files[:]
+            markers = {}
+            for markerType in self.__markers:
+                markers[markerType] = self.__markers[markerType][:]
+        
+        for fn in files:
+            if self.__interrupt:
+                break
+            
+            # read the file and split it into textlines
+            try:
+                text, encoding = Utilities.readEncodedFile(fn)
+                lines = text.splitlines()
+            except (UnicodeError, IOError):
+                continue
+            
+            # now search tasks and record them
+            lineIndex = 0
+            for line in lines:
+                if self.__interrupt:
+                    break
+                
+                lineIndex += 1
+                found = False
+                
+                for taskType, taskMarkers in markers.items():
+                    for taskMarker in taskMarkers:
+                        index = line.find(taskMarker)
+                        if index > -1:
+                            task = line[index:]
+                            with self.__lock:
+                                self.taskFound.emit(task, fn, lineIndex,
+                                                    taskType)
+                            found = True
+                            break
+                    if found:
+                        break

eric ide

mercurial