eric6/E5Network/E5TldExtractor.py

Thu, 24 Sep 2020 19:51:19 +0200

author
Detlev Offenbach <detlev@die-offenbachs.de>
date
Thu, 24 Sep 2020 19:51:19 +0200
changeset 7716
313e09453306
parent 7360
9190402e4505
child 7717
f32d7965a17e
permissions
-rw-r--r--

Added TODO comments to eliminate the use of Qt resources (prepare for PyQt 6).

4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
2
7360
9190402e4505 Updated copyright for 2020.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7253
diff changeset
3 # Copyright (c) 2016 - 2020 Detlev Offenbach <detlev@die-offenbachs.de>
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
4 #
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
5
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
6 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
7 Module implementing the TLD Extractor.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
8 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
9
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
10 #
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
11 # This is a Python port of the TLDExtractor of Qupzilla
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
12 # Copyright (C) 2014 Razi Alavizadeh <s.r.alavizadeh@gmail.com>
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
13 #
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
14
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
15
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
16 import collections
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
17
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
18 from PyQt5.QtCore import QObject, QUrl, QFile, QFileInfo, QRegExp, qWarning
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
19
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
20 from E5Gui import E5MessageBox
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
21
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
22 from .data import tld_rc # __IGNORE_WARNING__
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
23
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
24
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
25 class E5TldHostParts(object):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
26 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
27 Class implementing the host parts helper.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
28 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
29 def __init__(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
30 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
31 Constructor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
32 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
33 self.host = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
34 self.tld = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
35 self.domain = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
36 self.registrableDomain = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
37 self.subdomain = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
38
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
39
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
40 class E5TldExtractor(QObject):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
41 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
42 Class implementing the TLD Extractor.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
43
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
44 Note: The module function instance() should be used to get a reference
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
45 to a global object to avoid overhead.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
46 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
47 def __init__(self, withPrivate=False, parent=None):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
48 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
49 Constructor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
50
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
51 @param withPrivate flag indicating to load private TLDs as well
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
52 @type bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
53 @param parent reference to the parent object
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
54 @type QObject
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
55 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
56 super(E5TldExtractor, self).__init__(parent)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
57
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
58 self.__withPrivate = withPrivate
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
59 self.__dataFileName = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
60 self.__dataSearchPaths = []
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
61
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
62 self.__tldDict = collections.defaultdict(list)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
63 # dict with list of str as values
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
64
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
65 self.setDataSearchPaths()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
66
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
67 def isDataLoaded(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
68 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
69 Public method to check, if the TLD data ia already loaded.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
70
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
71 @return flag indicating data is loaded
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
72 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
73 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
74 return bool(self.__tldDict)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
75
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
76 def tld(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
77 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
78 Public method to get the top level domain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
79
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
80 @param host host name to get TLD for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
81 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
82 @return TLD for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
83 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
84 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
85 if not host or host.startswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
86 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
87
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
88 cleanHost = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
89
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
90 tldPart = cleanHost[cleanHost.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
91 cleanHost = bytes(QUrl.toAce(cleanHost)).decode("utf-8")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
92
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
93 self.__loadData()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
94
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
95 if tldPart not in self.__tldDict:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
96 return tldPart
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
97
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
98 tldRules = self.__tldDict[tldPart][:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
99
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
100 if tldPart not in tldRules:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
101 tldRules.append(tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
102
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
103 maxLabelCount = 0
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
104 isWildcardTLD = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
105
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
106 for rule in tldRules:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
107 labelCount = rule.count(".") + 1
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
108
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
109 if rule.startswith("!"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
110 rule = rule[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
111
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
112 rule = bytes(QUrl.toAce(rule)).decode("utf-8")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
113
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
114 # matches with exception TLD
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
115 if cleanHost.endswith(rule):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
116 tldPart = rule[rule.find(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
117 break
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
118
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
119 if rule.startswith("*"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
120 rule = rule[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
121
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
122 if rule.startswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
123 rule = rule[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
124
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
125 isWildcardTLD = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
126 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
127 isWildcardTLD = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
128
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
129 rule = bytes(QUrl.toAce(rule)).decode("utf-8")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
130 testRule = "." + rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
131 testUrl = "." + cleanHost
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
132
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
133 if labelCount > maxLabelCount and testUrl.endswith(testRule):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
134 tldPart = rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
135 maxLabelCount = labelCount
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
136
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
137 if isWildcardTLD:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
138 temp = cleanHost
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
139 temp = temp[:temp.rfind(tldPart)]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
140
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
141 if temp.endswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
142 temp = temp[:-1]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
143
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
144 temp = temp[temp.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
145
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
146 if temp:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
147 tldPart = temp + "." + rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
148 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
149 tldPart = rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
150
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
151 temp = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
152 tldPart = ".".join(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
153 temp.split(".")[temp.count(".") - tldPart.count("."):])
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
154
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
155 return tldPart
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
156
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
157 def domain(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
158 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
159 Public method to get the domain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
160
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
161 @param host host name to get the domain for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
162 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
163 @return domain for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
164 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
165 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
166 tldPart = self.tld(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
167
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
168 return self.__domainHelper(host, tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
169
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
170 def registrableDomain(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
171 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
172 Public method to get the registrable domain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
173
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
174 @param host host name to get the registrable domain for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
175 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
176 @return registrable domain for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
177 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
178 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
179 tldPart = self.tld(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
180
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
181 return self.__registrableDomainHelper(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
182 self.__domainHelper(host, tldPart), tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
183
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
184 def subdomain(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
185 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
186 Public method to get the subdomain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
187
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
188 @param host host name to get the subdomain for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
189 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
190 @return subdomain for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
191 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
192 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
193 return self.__subdomainHelper(host, self.registrableDomain(host))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
194
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
195 def splitParts(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
196 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
197 Public method to split a host address into its parts.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
198
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
199 @param host host address to be split
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
200 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
201 @return splitted host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
202 @rtype E5TldHostParts
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
203 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
204 hostParts = E5TldHostParts()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
205 hostParts.host = host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
206 hostParts.tld = self.tld(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
207 hostParts.domain = self.__domainHelper(host, hostParts.tld)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
208 hostParts.registrableDomain = self.__registrableDomainHelper(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
209 hostParts.domain, hostParts.tld)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
210 hostParts.subdomain = self.__subdomainHelper(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
211 host, hostParts.registrableDomain)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
212
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
213 return hostParts
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
214
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
215 def dataSearchPaths(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
216 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
217 Public method to get the search paths for the TLD data file.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
218
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
219 @return search paths for the TLD data file
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
220 @rtype list of str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
221 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
222 return self.__dataSearchPaths[:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
223
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
224 def setDataSearchPaths(self, searchPaths=None):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
225 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
226 Public method to set the search paths for the TLD data file.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
227
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
228 @param searchPaths search paths for the TLD data file or None,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
229 if the default search paths shall be set
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
230 @type list of str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
231 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
232 if searchPaths:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
233 self.__dataSearchPaths = searchPaths[:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
234 self.__dataSearchPaths.extend(self.__defaultDataSearchPaths())
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
235 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
236 self.__dataSearchPaths = self.__defaultDataSearchPaths()[:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
237
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
238 # remove duplicates
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
239 paths = []
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
240 for p in self.__dataSearchPaths:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
241 if p not in paths:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
242 paths.append(p)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
243 self.__dataSearchPaths = paths
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
244
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
245 def __defaultDataSearchPaths(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
246 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
247 Private method to get the default search paths for the TLD data file.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
248
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
249 @return default search paths for the TLD data file
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
250 @rtype list of str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
251 """
7716
313e09453306 Added TODO comments to eliminate the use of Qt resources (prepare for PyQt 6).
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7360
diff changeset
252 # TODO: replace this with path to our data directory to remove .qrc
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
253 return [":"]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
254
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
255 def getTldDownloadUrl(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
256 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
257 Public method to get the TLD data file download URL.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
258
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
259 @return download URL
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
260 @rtype QUrl
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
261 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
262 return QUrl(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
263 "http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
264 "effective_tld_names.dat?raw=1")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
265
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
266 def __loadData(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
267 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
268 Private method to load the TLD data.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
269 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
270 if self.isDataLoaded():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
271 return
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
272
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
273 dataFileName = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
274 parsedDataFileExist = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
275
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
276 for path in self.__dataSearchPaths:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
277 dataFileName = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
278 QFileInfo(path + "/effective_tld_names.dat").absoluteFilePath()
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
279 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
280 if QFileInfo(dataFileName).exists():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
281 parsedDataFileExist = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
282 break
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
283
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
284 if not parsedDataFileExist:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
285 tldDataFileDownloadLink = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
286 "http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/"
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
287 "effective_tld_names.dat?raw=1"
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
288 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
289 E5MessageBox.information(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
290 None,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
291 self.tr("TLD Data File not found"),
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
292 self.tr("""<p>The file 'effective_tld_names.dat' was not"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
293 """ found!<br/>You can download it from """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
294 """'<a href="{0}"><b>here</b></a>' to one of the"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
295 """ following paths:</p><ul>{1}</ul>""").format(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
296 tldDataFileDownloadLink,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
297 "".join(["<li>{0}</li>".format(p)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
298 for p in self.__dataSearchPaths]))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
299 )
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
300 return
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
301
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
302 self.__dataFileName = dataFileName
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
303 if not self.__parseData(dataFileName,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
304 loadPrivateDomains=self.__withPrivate):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
305 qWarning(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
306 "E5TldExtractor: There are some parse errors for file: {0}"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
307 .format(dataFileName))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
308
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
309 def __parseData(self, dataFile, loadPrivateDomains=False):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
310 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
311 Private method to parse TLD data.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
312
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
313 @param dataFile name of the file containing the TLD data
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
314 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
315 @param loadPrivateDomains flag indicating to load private domains
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
316 @type bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
317 @return flag indicating success
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
318 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
319 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
320 # start with a fresh dictionary
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
321 self.__tldDict = collections.defaultdict(list)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
322
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
323 file = QFile(dataFile)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
324
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
325 if not file.open(QFile.ReadOnly | QFile.Text):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
326 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
327
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
328 seekToEndOfPrivateDomains = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
329
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
330 while not file.atEnd():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
331 line = bytes(file.readLine()).decode("utf-8").strip()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
332 if not line:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
333 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
334
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
335 if line.startswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
336 line = line[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
337
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
338 if line.startswith("//"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
339 if "===END PRIVATE DOMAINS===" in line:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
340 seekToEndOfPrivateDomains = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
341
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
342 if (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
343 not loadPrivateDomains and
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
344 "===BEGIN PRIVATE DOMAINS===" in line
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
345 ):
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
346 seekToEndOfPrivateDomains = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
347
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
348 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
349
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
350 if seekToEndOfPrivateDomains:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
351 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
352
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
353 # only data up to the first whitespace is used
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
354 line = line.split(None, 1)[0]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
355
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
356 if "." not in line:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
357 self.__tldDict[line].append(line)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
358 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
359 key = line[line.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
360 self.__tldDict[key].append(line)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
361
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
362 return self.isDataLoaded()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
363
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
364 def __domainHelper(self, host, tldPart):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
365 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
366 Private method to get the domain name without TLD.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
367
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
368 @param host host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
369 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
370 @param tldPart TLD part of the host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
371 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
372 @return domain name
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
373 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
374 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
375 if not host or not tldPart:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
376 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
377
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
378 temp = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
379 temp = temp[:temp.rfind(tldPart)]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
380
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
381 if temp.endswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
382 temp = temp[:-1]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
383
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
384 return temp[temp.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
385
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
386 def __registrableDomainHelper(self, domainPart, tldPart):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
387 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
388 Private method to get the registrable domain (i.e. domain plus TLD).
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
389
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
390 @param domainPart domain part of a host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
391 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
392 @param tldPart TLD part of a host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
393 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
394 @return registrable domain name
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
395 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
396 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
397 if not tldPart or not domainPart:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
398 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
399 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
400 return "{0}.{1}".format(domainPart, tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
401
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
402 def __subdomainHelper(self, host, registrablePart):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
403 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
404 Private method to get the subdomain of a host address (i.e. domain part
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
405 without the registrable domain name).
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
406
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
407 @param host host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
408 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
409 @param registrablePart registrable domain part of the host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
410 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
411 @return subdomain name
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
412 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
413 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
414 if not host or not registrablePart:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
415 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
416
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
417 subdomain = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
418
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
419 subdomain = subdomain[:subdomain.rfind(registrablePart)]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
420
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
421 if subdomain.endswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
422 subdomain = subdomain[:-1]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
423
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
424 return subdomain
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
425
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
426 def __normalizedHost(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
427 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
428 Private method to get the normalized host for a host address.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
429
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
430 @param host host address to be normalized
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
431 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
432 @return normalized host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
433 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
434 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
435 return host.lower()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
436
7716
313e09453306 Added TODO comments to eliminate the use of Qt resources (prepare for PyQt 6).
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7360
diff changeset
437 # TODO: remove this part and delete 'test_psl.txt'
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
438 #################################################################
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
439 ## Methods below are for testing purposes
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
440 #################################################################
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
441
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
442 def test(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
443 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
444 Public method to execute the tests.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
445
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
446 @return flag indicating the test result
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
447 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
448 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
449 self.__withPrivate = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
450 self.__loadData()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
451 if not self.__tldDict:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
452 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
453
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
454 testDataFileName = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
455 testDataFileExist = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
456
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
457 for path in self.__dataSearchPaths:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
458 testDataFileName = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
459 QFileInfo(path + "/test_psl.txt").absoluteFilePath()
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
460 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
461 if QFileInfo(testDataFileName).exists():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
462 testDataFileExist = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
463 break
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
464
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
465 if not testDataFileExist:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
466 testFileDownloadLink = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
467 "http://mxr.mozilla.org/mozilla-central/source/netwerk/test/"
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
468 "unit/data/test_psl.txt?raw=1"
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
469 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
470 E5MessageBox.information(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
471 None,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
472 self.tr("TLD Data File not found"),
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
473 self.tr("""<p>The file 'test_psl.txt' was not found!"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
474 """<br/>You can download it from '<a href="{0}">"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
475 """<b>here</b></a>' to one of the following"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
476 """ paths:</p><ul>{1}</ul>""").format(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
477 testFileDownloadLink,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
478 "".join(["<li>{0}</li>".format(p)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
479 for p in self.__dataSearchPaths]))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
480 )
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
481 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
482
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
483 file = QFile(testDataFileName)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
484
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
485 if not file.open(QFile.ReadOnly | QFile.Text):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
486 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
487
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
488 testRegExp = QRegExp(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
489 "checkPublicSuffix\\(('([^']+)'|null), ('([^']+)'|null)\\);")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
490 allTestSuccess = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
491
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
492 while not file.atEnd():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
493 line = bytes(file.readLine()).decode("utf-8").strip()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
494 if not line or line.startswith("//"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
495 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
496
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
497 if testRegExp.indexIn(line) == -1:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
498 allTestSuccess = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
499 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
500 hostName = testRegExp.cap(2)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
501 registrableName = testRegExp.cap(4)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
502
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
503 if not self.__checkPublicSuffix(hostName, registrableName):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
504 allTestSuccess = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
505
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
506 if allTestSuccess:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
507 qWarning("E5TldExtractor: Test passed successfully.")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
508 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
509 qWarning("E5TldExtractor: Test finished with some errors!")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
510
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
511 # reset the TLD dictionary
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
512 self.__tldDict = collections.defaultdict(list)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
513
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
514 return allTestSuccess
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
515
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
516 def __checkPublicSuffix(self, host, registrableName):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
517 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
518 Private method to test a host name against a registrable name.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
519
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
520 @param host host name to test
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
521 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
522 @param registrableName registrable domain name to test against
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
523 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
524 @return flag indicating the check result
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
525 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
526 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
527 regName = self.registrableDomain(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
528 if regName != registrableName:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
529 qWarning(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
530 "E5TldExtractor Test Error: hostName: {0}\n"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
531 " Correct registrableName: {1}\n"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
532 " Calculated registrableName: {2}".format(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
533 host, registrableName, regName))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
534 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
535
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
536 return True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
537
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
538
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
539 _TLDExtractor = None
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
540
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
541
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
542 def instance(withPrivate=False):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
543 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
544 Global function to get a reference to the TLD extractor and create it, if
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
545 it hasn't been yet.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
546
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
547 @param withPrivate flag indicating to load private TLDs as well
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
548 @type bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
549 @return reference to the zoom manager object
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
550 @rtype E5TldExtractor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
551 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
552 global _TLDExtractor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
553
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
554 if _TLDExtractor is None:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
555 _TLDExtractor = E5TldExtractor(withPrivate=withPrivate)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
556
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
557 return _TLDExtractor

eric ide

mercurial