eric6/E5Network/E5TldExtractor.py

Sat, 11 Apr 2020 14:13:25 +0200

author
Detlev Offenbach <detlev@die-offenbachs.de>
date
Sat, 11 Apr 2020 14:13:25 +0200
changeset 7522
e0e05a1b576f
parent 7360
9190402e4505
child 7716
313e09453306
permissions
-rw-r--r--

Styles: added some Qt style sheets and some highlighter styles.

4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
2
7360
9190402e4505 Updated copyright for 2020.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7253
diff changeset
3 # Copyright (c) 2016 - 2020 Detlev Offenbach <detlev@die-offenbachs.de>
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
4 #
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
5
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
6 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
7 Module implementing the TLD Extractor.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
8 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
9
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
10 #
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
11 # This is a Python port of the TLDExtractor of Qupzilla
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
12 # Copyright (C) 2014 Razi Alavizadeh <s.r.alavizadeh@gmail.com>
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
13 #
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
14
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
15
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
16 import collections
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
17
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
18 from PyQt5.QtCore import QObject, QUrl, QFile, QFileInfo, QRegExp, qWarning
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
19
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
20 from E5Gui import E5MessageBox
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
21
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
22 from .data import tld_rc # __IGNORE_WARNING__
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
23
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
24
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
25 class E5TldHostParts(object):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
26 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
27 Class implementing the host parts helper.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
28 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
29 def __init__(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
30 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
31 Constructor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
32 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
33 self.host = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
34 self.tld = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
35 self.domain = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
36 self.registrableDomain = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
37 self.subdomain = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
38
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
39
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
40 class E5TldExtractor(QObject):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
41 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
42 Class implementing the TLD Extractor.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
43
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
44 Note: The module function instance() should be used to get a reference
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
45 to a global object to avoid overhead.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
46 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
47 def __init__(self, withPrivate=False, parent=None):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
48 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
49 Constructor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
50
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
51 @param withPrivate flag indicating to load private TLDs as well
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
52 @type bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
53 @param parent reference to the parent object
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
54 @type QObject
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
55 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
56 super(E5TldExtractor, self).__init__(parent)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
57
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
58 self.__withPrivate = withPrivate
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
59 self.__dataFileName = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
60 self.__dataSearchPaths = []
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
61
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
62 self.__tldDict = collections.defaultdict(list)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
63 # dict with list of str as values
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
64
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
65 self.setDataSearchPaths()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
66
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
67 def isDataLoaded(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
68 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
69 Public method to check, if the TLD data ia already loaded.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
70
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
71 @return flag indicating data is loaded
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
72 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
73 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
74 return bool(self.__tldDict)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
75
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
76 def tld(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
77 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
78 Public method to get the top level domain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
79
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
80 @param host host name to get TLD for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
81 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
82 @return TLD for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
83 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
84 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
85 if not host or host.startswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
86 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
87
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
88 cleanHost = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
89
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
90 tldPart = cleanHost[cleanHost.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
91 cleanHost = bytes(QUrl.toAce(cleanHost)).decode("utf-8")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
92
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
93 self.__loadData()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
94
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
95 if tldPart not in self.__tldDict:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
96 return tldPart
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
97
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
98 tldRules = self.__tldDict[tldPart][:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
99
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
100 if tldPart not in tldRules:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
101 tldRules.append(tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
102
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
103 maxLabelCount = 0
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
104 isWildcardTLD = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
105
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
106 for rule in tldRules:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
107 labelCount = rule.count(".") + 1
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
108
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
109 if rule.startswith("!"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
110 rule = rule[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
111
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
112 rule = bytes(QUrl.toAce(rule)).decode("utf-8")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
113
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
114 # matches with exception TLD
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
115 if cleanHost.endswith(rule):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
116 tldPart = rule[rule.find(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
117 break
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
118
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
119 if rule.startswith("*"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
120 rule = rule[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
121
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
122 if rule.startswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
123 rule = rule[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
124
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
125 isWildcardTLD = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
126 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
127 isWildcardTLD = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
128
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
129 rule = bytes(QUrl.toAce(rule)).decode("utf-8")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
130 testRule = "." + rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
131 testUrl = "." + cleanHost
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
132
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
133 if labelCount > maxLabelCount and testUrl.endswith(testRule):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
134 tldPart = rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
135 maxLabelCount = labelCount
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
136
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
137 if isWildcardTLD:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
138 temp = cleanHost
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
139 temp = temp[:temp.rfind(tldPart)]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
140
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
141 if temp.endswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
142 temp = temp[:-1]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
143
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
144 temp = temp[temp.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
145
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
146 if temp:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
147 tldPart = temp + "." + rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
148 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
149 tldPart = rule
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
150
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
151 temp = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
152 tldPart = ".".join(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
153 temp.split(".")[temp.count(".") - tldPart.count("."):])
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
154
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
155 return tldPart
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
156
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
157 def domain(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
158 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
159 Public method to get the domain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
160
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
161 @param host host name to get the domain for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
162 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
163 @return domain for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
164 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
165 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
166 tldPart = self.tld(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
167
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
168 return self.__domainHelper(host, tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
169
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
170 def registrableDomain(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
171 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
172 Public method to get the registrable domain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
173
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
174 @param host host name to get the registrable domain for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
175 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
176 @return registrable domain for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
177 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
178 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
179 tldPart = self.tld(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
180
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
181 return self.__registrableDomainHelper(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
182 self.__domainHelper(host, tldPart), tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
183
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
184 def subdomain(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
185 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
186 Public method to get the subdomain for a host.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
187
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
188 @param host host name to get the subdomain for
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
189 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
190 @return subdomain for host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
191 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
192 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
193 return self.__subdomainHelper(host, self.registrableDomain(host))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
194
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
195 def splitParts(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
196 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
197 Public method to split a host address into its parts.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
198
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
199 @param host host address to be split
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
200 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
201 @return splitted host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
202 @rtype E5TldHostParts
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
203 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
204 hostParts = E5TldHostParts()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
205 hostParts.host = host
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
206 hostParts.tld = self.tld(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
207 hostParts.domain = self.__domainHelper(host, hostParts.tld)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
208 hostParts.registrableDomain = self.__registrableDomainHelper(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
209 hostParts.domain, hostParts.tld)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
210 hostParts.subdomain = self.__subdomainHelper(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
211 host, hostParts.registrableDomain)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
212
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
213 return hostParts
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
214
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
215 def dataSearchPaths(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
216 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
217 Public method to get the search paths for the TLD data file.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
218
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
219 @return search paths for the TLD data file
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
220 @rtype list of str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
221 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
222 return self.__dataSearchPaths[:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
223
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
224 def setDataSearchPaths(self, searchPaths=None):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
225 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
226 Public method to set the search paths for the TLD data file.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
227
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
228 @param searchPaths search paths for the TLD data file or None,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
229 if the default search paths shall be set
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
230 @type list of str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
231 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
232 if searchPaths:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
233 self.__dataSearchPaths = searchPaths[:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
234 self.__dataSearchPaths.extend(self.__defaultDataSearchPaths())
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
235 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
236 self.__dataSearchPaths = self.__defaultDataSearchPaths()[:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
237
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
238 # remove duplicates
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
239 paths = []
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
240 for p in self.__dataSearchPaths:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
241 if p not in paths:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
242 paths.append(p)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
243 self.__dataSearchPaths = paths
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
244
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
245 def __defaultDataSearchPaths(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
246 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
247 Private method to get the default search paths for the TLD data file.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
248
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
249 @return default search paths for the TLD data file
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
250 @rtype list of str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
251 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
252 return [":"]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
253
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
254 def getTldDownloadUrl(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
255 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
256 Public method to get the TLD data file download URL.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
257
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
258 @return download URL
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
259 @rtype QUrl
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
260 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
261 return QUrl(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
262 "http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
263 "effective_tld_names.dat?raw=1")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
264
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
265 def __loadData(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
266 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
267 Private method to load the TLD data.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
268 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
269 if self.isDataLoaded():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
270 return
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
271
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
272 dataFileName = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
273 parsedDataFileExist = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
274
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
275 for path in self.__dataSearchPaths:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
276 dataFileName = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
277 QFileInfo(path + "/effective_tld_names.dat").absoluteFilePath()
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
278 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
279 if QFileInfo(dataFileName).exists():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
280 parsedDataFileExist = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
281 break
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
282
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
283 if not parsedDataFileExist:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
284 tldDataFileDownloadLink = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
285 "http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/"
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
286 "effective_tld_names.dat?raw=1"
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
287 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
288 E5MessageBox.information(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
289 None,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
290 self.tr("TLD Data File not found"),
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
291 self.tr("""<p>The file 'effective_tld_names.dat' was not"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
292 """ found!<br/>You can download it from """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
293 """'<a href="{0}"><b>here</b></a>' to one of the"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
294 """ following paths:</p><ul>{1}</ul>""").format(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
295 tldDataFileDownloadLink,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
296 "".join(["<li>{0}</li>".format(p)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
297 for p in self.__dataSearchPaths]))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
298 )
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
299 return
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
300
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
301 self.__dataFileName = dataFileName
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
302 if not self.__parseData(dataFileName,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
303 loadPrivateDomains=self.__withPrivate):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
304 qWarning(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
305 "E5TldExtractor: There are some parse errors for file: {0}"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
306 .format(dataFileName))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
307
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
308 def __parseData(self, dataFile, loadPrivateDomains=False):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
309 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
310 Private method to parse TLD data.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
311
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
312 @param dataFile name of the file containing the TLD data
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
313 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
314 @param loadPrivateDomains flag indicating to load private domains
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
315 @type bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
316 @return flag indicating success
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
317 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
318 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
319 # start with a fresh dictionary
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
320 self.__tldDict = collections.defaultdict(list)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
321
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
322 file = QFile(dataFile)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
323
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
324 if not file.open(QFile.ReadOnly | QFile.Text):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
325 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
326
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
327 seekToEndOfPrivateDomains = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
328
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
329 while not file.atEnd():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
330 line = bytes(file.readLine()).decode("utf-8").strip()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
331 if not line:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
332 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
333
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
334 if line.startswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
335 line = line[1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
336
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
337 if line.startswith("//"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
338 if "===END PRIVATE DOMAINS===" in line:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
339 seekToEndOfPrivateDomains = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
340
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
341 if (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
342 not loadPrivateDomains and
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
343 "===BEGIN PRIVATE DOMAINS===" in line
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
344 ):
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
345 seekToEndOfPrivateDomains = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
346
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
347 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
348
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
349 if seekToEndOfPrivateDomains:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
350 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
351
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
352 # only data up to the first whitespace is used
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
353 line = line.split(None, 1)[0]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
354
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
355 if "." not in line:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
356 self.__tldDict[line].append(line)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
357 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
358 key = line[line.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
359 self.__tldDict[key].append(line)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
360
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
361 return self.isDataLoaded()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
362
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
363 def __domainHelper(self, host, tldPart):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
364 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
365 Private method to get the domain name without TLD.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
366
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
367 @param host host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
368 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
369 @param tldPart TLD part of the host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
370 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
371 @return domain name
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
372 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
373 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
374 if not host or not tldPart:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
375 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
376
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
377 temp = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
378 temp = temp[:temp.rfind(tldPart)]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
379
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
380 if temp.endswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
381 temp = temp[:-1]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
382
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
383 return temp[temp.rfind(".") + 1:]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
384
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
385 def __registrableDomainHelper(self, domainPart, tldPart):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
386 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
387 Private method to get the registrable domain (i.e. domain plus TLD).
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
388
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
389 @param domainPart domain part of a host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
390 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
391 @param tldPart TLD part of a host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
392 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
393 @return registrable domain name
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
394 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
395 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
396 if not tldPart or not domainPart:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
397 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
398 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
399 return "{0}.{1}".format(domainPart, tldPart)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
400
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
401 def __subdomainHelper(self, host, registrablePart):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
402 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
403 Private method to get the subdomain of a host address (i.e. domain part
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
404 without the registrable domain name).
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
405
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
406 @param host host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
407 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
408 @param registrablePart registrable domain part of the host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
409 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
410 @return subdomain name
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
411 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
412 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
413 if not host or not registrablePart:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
414 return ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
415
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
416 subdomain = self.__normalizedHost(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
417
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
418 subdomain = subdomain[:subdomain.rfind(registrablePart)]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
419
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
420 if subdomain.endswith("."):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
421 subdomain = subdomain[:-1]
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
422
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
423 return subdomain
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
424
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
425 def __normalizedHost(self, host):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
426 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
427 Private method to get the normalized host for a host address.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
428
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
429 @param host host address to be normalized
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
430 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
431 @return normalized host address
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
432 @rtype str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
433 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
434 return host.lower()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
435
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
436 #################################################################
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
437 ## Methods below are for testing purposes
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
438 #################################################################
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
439
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
440 def test(self):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
441 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
442 Public method to execute the tests.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
443
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
444 @return flag indicating the test result
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
445 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
446 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
447 self.__withPrivate = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
448 self.__loadData()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
449 if not self.__tldDict:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
450 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
451
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
452 testDataFileName = ""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
453 testDataFileExist = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
454
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
455 for path in self.__dataSearchPaths:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
456 testDataFileName = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
457 QFileInfo(path + "/test_psl.txt").absoluteFilePath()
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
458 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
459 if QFileInfo(testDataFileName).exists():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
460 testDataFileExist = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
461 break
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
462
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
463 if not testDataFileExist:
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
464 testFileDownloadLink = (
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
465 "http://mxr.mozilla.org/mozilla-central/source/netwerk/test/"
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
466 "unit/data/test_psl.txt?raw=1"
7253
50dbe65a1334 Continued to resolve code style issue M841.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 7229
diff changeset
467 )
4971
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
468 E5MessageBox.information(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
469 None,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
470 self.tr("TLD Data File not found"),
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
471 self.tr("""<p>The file 'test_psl.txt' was not found!"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
472 """<br/>You can download it from '<a href="{0}">"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
473 """<b>here</b></a>' to one of the following"""
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
474 """ paths:</p><ul>{1}</ul>""").format(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
475 testFileDownloadLink,
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
476 "".join(["<li>{0}</li>".format(p)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
477 for p in self.__dataSearchPaths]))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
478 )
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
479 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
480
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
481 file = QFile(testDataFileName)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
482
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
483 if not file.open(QFile.ReadOnly | QFile.Text):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
484 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
485
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
486 testRegExp = QRegExp(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
487 "checkPublicSuffix\\(('([^']+)'|null), ('([^']+)'|null)\\);")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
488 allTestSuccess = True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
489
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
490 while not file.atEnd():
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
491 line = bytes(file.readLine()).decode("utf-8").strip()
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
492 if not line or line.startswith("//"):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
493 continue
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
494
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
495 if testRegExp.indexIn(line) == -1:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
496 allTestSuccess = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
497 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
498 hostName = testRegExp.cap(2)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
499 registrableName = testRegExp.cap(4)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
500
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
501 if not self.__checkPublicSuffix(hostName, registrableName):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
502 allTestSuccess = False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
503
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
504 if allTestSuccess:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
505 qWarning("E5TldExtractor: Test passed successfully.")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
506 else:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
507 qWarning("E5TldExtractor: Test finished with some errors!")
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
508
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
509 # reset the TLD dictionary
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
510 self.__tldDict = collections.defaultdict(list)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
511
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
512 return allTestSuccess
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
513
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
514 def __checkPublicSuffix(self, host, registrableName):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
515 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
516 Private method to test a host name against a registrable name.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
517
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
518 @param host host name to test
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
519 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
520 @param registrableName registrable domain name to test against
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
521 @type str
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
522 @return flag indicating the check result
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
523 @rtype bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
524 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
525 regName = self.registrableDomain(host)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
526 if regName != registrableName:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
527 qWarning(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
528 "E5TldExtractor Test Error: hostName: {0}\n"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
529 " Correct registrableName: {1}\n"
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
530 " Calculated registrableName: {2}".format(
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
531 host, registrableName, regName))
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
532 return False
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
533
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
534 return True
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
535
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
536
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
537 _TLDExtractor = None
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
538
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
539
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
540 def instance(withPrivate=False):
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
541 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
542 Global function to get a reference to the TLD extractor and create it, if
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
543 it hasn't been yet.
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
544
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
545 @param withPrivate flag indicating to load private TLDs as well
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
546 @type bool
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
547 @return reference to the zoom manager object
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
548 @rtype E5TldExtractor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
549 """
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
550 global _TLDExtractor
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
551
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
552 if _TLDExtractor is None:
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
553 _TLDExtractor = E5TldExtractor(withPrivate=withPrivate)
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
554
0f21662c0d2d Added a class to extract all top level domains from a file provided by Mozilla.
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
555 return _TLDExtractor

eric ide

mercurial