Mercurial Repositories > eric / annotate

eric6/ThirdParty/CharDet/chardet/jpcntx.py@2d58b9c1a981 (annotated)

eric6/ThirdParty/CharDet/chardet/jpcntx.py

Sat, 07 Sep 2019 17:35:43 +0200

author
Detlev Offenbach <detlev@die-offenbachs.de>
date
Sat, 07 Sep 2019 17:35:43 +0200
branch
without_py2_and_pyqt4
changeset 7223
2d58b9c1a981
parent 6942
2602857055c5
permissions
-rw-r--r--

Closed branch after it was merged into 'default'.

3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
1 ######################## BEGIN LICENSE BLOCK ########################
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
2 # The Original Code is Mozilla Communicator client code.
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
3 #
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
4 # The Initial Developer of the Original Code is
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
5 # Netscape Communications Corporation.
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
6 # Portions created by the Initial Developer are Copyright (C) 1998
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
7 # the Initial Developer. All Rights Reserved.
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
8 #
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
9 # Contributor(s):
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
10 # Mark Pilgrim - port to Python
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
11 #
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
12 # This library is free software; you can redistribute it and/or
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
13 # modify it under the terms of the GNU Lesser General Public
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
14 # License as published by the Free Software Foundation; either
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
15 # version 2.1 of the License, or (at your option) any later version.
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
16 #
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
17 # This library is distributed in the hope that it will be useful,
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
18 # but WITHOUT ANY WARRANTY; without even the implied warranty of
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
19 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
20 # Lesser General Public License for more details.
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
21 #
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
22 # You should have received a copy of the GNU Lesser General Public
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
23 # License along with this library; if not, write to the Free Software
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
24 # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
25 # 02110-1301 USA
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
26 ######################### END LICENSE BLOCK #########################
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
27
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
28
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
29 # This is hiragana 2-char sequence table, the number in each cell represents its frequency category
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
30 jp2CharContext = (
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
31 (0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
32 (2,4,0,4,0,3,0,4,0,3,4,4,4,2,4,3,3,4,3,2,3,3,4,2,3,3,3,2,4,1,4,3,3,1,5,4,3,4,3,4,3,5,3,0,3,5,4,2,0,3,1,0,3,3,0,3,3,0,1,1,0,4,3,0,3,3,0,4,0,2,0,3,5,5,5,5,4,0,4,1,0,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
33 (0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
34 (0,4,0,5,0,5,0,4,0,4,5,4,4,3,5,3,5,1,5,3,4,3,4,4,3,4,3,3,4,3,5,4,4,3,5,5,3,5,5,5,3,5,5,3,4,5,5,3,1,3,2,0,3,4,0,4,2,0,4,2,1,5,3,2,3,5,0,4,0,2,0,5,4,4,5,4,5,0,4,0,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
35 (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
36 (0,3,0,4,0,3,0,3,0,4,5,4,3,3,3,3,4,3,5,4,4,3,5,4,4,3,4,3,4,4,4,4,5,3,4,4,3,4,5,5,4,5,5,1,4,5,4,3,0,3,3,1,3,3,0,4,4,0,3,3,1,5,3,3,3,5,0,4,0,3,0,4,4,3,4,3,3,0,4,1,1,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
37 (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
38 (0,4,0,3,0,3,0,4,0,3,4,4,3,2,2,1,2,1,3,1,3,3,3,3,3,4,3,1,3,3,5,3,3,0,4,3,0,5,4,3,3,5,4,4,3,4,4,5,0,1,2,0,1,2,0,2,2,0,1,0,0,5,2,2,1,4,0,3,0,1,0,4,4,3,5,4,3,0,2,1,0,4,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
39 (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
40 (0,3,0,5,0,4,0,2,1,4,4,2,4,1,4,2,4,2,4,3,3,3,4,3,3,3,3,1,4,2,3,3,3,1,4,4,1,1,1,4,3,3,2,0,2,4,3,2,0,3,3,0,3,1,1,0,0,0,3,3,0,4,2,2,3,4,0,4,0,3,0,4,4,5,3,4,4,0,3,0,0,1,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
41 (1,4,0,4,0,4,0,4,0,3,5,4,4,3,4,3,5,4,3,3,4,3,5,4,4,4,4,3,4,2,4,3,3,1,5,4,3,2,4,5,4,5,5,4,4,5,4,4,0,3,2,2,3,3,0,4,3,1,3,2,1,4,3,3,4,5,0,3,0,2,0,4,5,5,4,5,4,0,4,0,0,5,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
42 (0,5,0,5,0,4,0,3,0,4,4,3,4,3,3,3,4,0,4,4,4,3,4,3,4,3,3,1,4,2,4,3,4,0,5,4,1,4,5,4,4,5,3,2,4,3,4,3,2,4,1,3,3,3,2,3,2,0,4,3,3,4,3,3,3,4,0,4,0,3,0,4,5,4,4,4,3,0,4,1,0,1,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
43 (0,3,1,4,0,3,0,2,0,3,4,4,3,1,4,2,3,3,4,3,4,3,4,3,4,4,3,2,3,1,5,4,4,1,4,4,3,5,4,4,3,5,5,4,3,4,4,3,1,2,3,1,2,2,0,3,2,0,3,1,0,5,3,3,3,4,3,3,3,3,4,4,4,4,5,4,2,0,3,3,2,4,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
44 (0,2,0,3,0,1,0,1,0,0,3,2,0,0,2,0,1,0,2,1,3,3,3,1,2,3,1,0,1,0,4,2,1,1,3,3,0,4,3,3,1,4,3,3,0,3,3,2,0,0,0,0,1,0,0,2,0,0,0,0,0,4,1,0,2,3,2,2,2,1,3,3,3,4,4,3,2,0,3,1,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
45 (0,4,0,4,0,3,0,3,0,4,4,4,3,3,3,3,3,3,4,3,4,2,4,3,4,3,3,2,4,3,4,5,4,1,4,5,3,5,4,5,3,5,4,0,3,5,5,3,1,3,3,2,2,3,0,3,4,1,3,3,2,4,3,3,3,4,0,4,0,3,0,4,5,4,4,5,3,0,4,1,0,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
46 (0,2,0,3,0,3,0,0,0,2,2,2,1,0,1,0,0,0,3,0,3,0,3,0,1,3,1,0,3,1,3,3,3,1,3,3,3,0,1,3,1,3,4,0,0,3,1,1,0,3,2,0,0,0,0,1,3,0,1,0,0,3,3,2,0,3,0,0,0,0,0,3,4,3,4,3,3,0,3,0,0,2,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
47 (2,3,0,3,0,2,0,1,0,3,3,4,3,1,3,1,1,1,3,1,4,3,4,3,3,3,0,0,3,1,5,4,3,1,4,3,2,5,5,4,4,4,4,3,3,4,4,4,0,2,1,1,3,2,0,1,2,0,0,1,0,4,1,3,3,3,0,3,0,1,0,4,4,4,5,5,3,0,2,0,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
48 (0,2,0,1,0,3,1,3,0,2,3,3,3,0,3,1,0,0,3,0,3,2,3,1,3,2,1,1,0,0,4,2,1,0,2,3,1,4,3,2,0,4,4,3,1,3,1,3,0,1,0,0,1,0,0,0,1,0,0,0,0,4,1,1,1,2,0,3,0,0,0,3,4,2,4,3,2,0,1,0,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
49 (0,1,0,4,0,5,0,4,0,2,4,4,2,3,3,2,3,3,5,3,3,3,4,3,4,2,3,0,4,3,3,3,4,1,4,3,2,1,5,5,3,4,5,1,3,5,4,2,0,3,3,0,1,3,0,4,2,0,1,3,1,4,3,3,3,3,0,3,0,1,0,3,4,4,4,5,5,0,3,0,1,4,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
50 (0,2,0,3,0,3,0,0,0,2,3,1,3,0,4,0,1,1,3,0,3,4,3,2,3,1,0,3,3,2,3,1,3,0,2,3,0,2,1,4,1,2,2,0,0,3,3,0,0,2,0,0,0,1,0,0,0,0,2,2,0,3,2,1,3,3,0,2,0,2,0,0,3,3,1,2,4,0,3,0,2,2,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
51 (2,4,0,5,0,4,0,4,0,2,4,4,4,3,4,3,3,3,1,2,4,3,4,3,4,4,5,0,3,3,3,3,2,0,4,3,1,4,3,4,1,4,4,3,3,4,4,3,1,2,3,0,4,2,0,4,1,0,3,3,0,4,3,3,3,4,0,4,0,2,0,3,5,3,4,5,2,0,3,0,0,4,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
52 (0,3,0,4,0,1,0,1,0,1,3,2,2,1,3,0,3,0,2,0,2,0,3,0,2,0,0,0,1,0,1,1,0,0,3,1,0,0,0,4,0,3,1,0,2,1,3,0,0,0,0,0,0,3,0,0,0,0,0,0,0,4,2,2,3,1,0,3,0,0,0,1,4,4,4,3,0,0,4,0,0,1,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
53 (1,4,1,5,0,3,0,3,0,4,5,4,4,3,5,3,3,4,4,3,4,1,3,3,3,3,2,1,4,1,5,4,3,1,4,4,3,5,4,4,3,5,4,3,3,4,4,4,0,3,3,1,2,3,0,3,1,0,3,3,0,5,4,4,4,4,4,4,3,3,5,4,4,3,3,5,4,0,3,2,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
54 (0,2,0,3,0,1,0,0,0,1,3,3,3,2,4,1,3,0,3,1,3,0,2,2,1,1,0,0,2,0,4,3,1,0,4,3,0,4,4,4,1,4,3,1,1,3,3,1,0,2,0,0,1,3,0,0,0,0,2,0,0,4,3,2,4,3,5,4,3,3,3,4,3,3,4,3,3,0,2,1,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
55 (0,2,0,4,0,3,0,2,0,2,5,5,3,4,4,4,4,1,4,3,3,0,4,3,4,3,1,3,3,2,4,3,0,3,4,3,0,3,4,4,2,4,4,0,4,5,3,3,2,2,1,1,1,2,0,1,5,0,3,3,2,4,3,3,3,4,0,3,0,2,0,4,4,3,5,5,0,0,3,0,2,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
56 (0,3,0,4,0,3,0,1,0,3,4,3,3,1,3,3,3,0,3,1,3,0,4,3,3,1,1,0,3,0,3,3,0,0,4,4,0,1,5,4,3,3,5,0,3,3,4,3,0,2,0,1,1,1,0,1,3,0,1,2,1,3,3,2,3,3,0,3,0,1,0,1,3,3,4,4,1,0,1,2,2,1,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
57 (0,1,0,4,0,4,0,3,0,1,3,3,3,2,3,1,1,0,3,0,3,3,4,3,2,4,2,0,1,0,4,3,2,0,4,3,0,5,3,3,2,4,4,4,3,3,3,4,0,1,3,0,0,1,0,0,1,0,0,0,0,4,2,3,3,3,0,3,0,0,0,4,4,4,5,3,2,0,3,3,0,3,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
58 (0,2,0,3,0,0,0,3,0,1,3,0,2,0,0,0,1,0,3,1,1,3,3,0,0,3,0,0,3,0,2,3,1,0,3,1,0,3,3,2,0,4,2,2,0,2,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,2,1,2,0,1,0,1,0,0,0,1,3,1,2,0,0,0,1,0,0,1,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
59 (0,3,0,3,0,5,0,1,0,2,4,3,1,3,3,2,1,1,5,2,1,0,5,1,2,0,0,0,3,3,2,2,3,2,4,3,0,0,3,3,1,3,3,0,2,5,3,4,0,3,3,0,1,2,0,2,2,0,3,2,0,2,2,3,3,3,0,2,0,1,0,3,4,4,2,5,4,0,3,0,0,3,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
60 (0,3,0,3,0,3,0,1,0,3,3,3,3,0,3,0,2,0,2,1,1,0,2,0,1,0,0,0,2,1,0,0,1,0,3,2,0,0,3,3,1,2,3,1,0,3,3,0,0,1,0,0,0,0,0,2,0,0,0,0,0,2,3,1,2,3,0,3,0,1,0,3,2,1,0,4,3,0,1,1,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
61 (0,4,0,5,0,3,0,3,0,4,5,5,4,3,5,3,4,3,5,3,3,2,5,3,4,4,4,3,4,3,4,5,5,3,4,4,3,4,4,5,4,4,4,3,4,5,5,4,2,3,4,2,3,4,0,3,3,1,4,3,2,4,3,3,5,5,0,3,0,3,0,5,5,5,5,4,4,0,4,0,1,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
62 (0,4,0,4,0,3,0,3,0,3,5,4,4,2,3,2,5,1,3,2,5,1,4,2,3,2,3,3,4,3,3,3,3,2,5,4,1,3,3,5,3,4,4,0,4,4,3,1,1,3,1,0,2,3,0,2,3,0,3,0,0,4,3,1,3,4,0,3,0,2,0,4,4,4,3,4,5,0,4,0,0,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
63 (0,3,0,3,0,3,1,2,0,3,4,4,3,3,3,0,2,2,4,3,3,1,3,3,3,1,1,0,3,1,4,3,2,3,4,4,2,4,4,4,3,4,4,3,2,4,4,3,1,3,3,1,3,3,0,4,1,0,2,2,1,4,3,2,3,3,5,4,3,3,5,4,4,3,3,0,4,0,3,2,2,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
64 (0,2,0,1,0,0,0,0,0,1,2,1,3,0,0,0,0,0,2,0,1,2,1,0,0,1,0,0,0,0,3,0,0,1,0,1,1,3,1,0,0,0,1,1,0,1,1,0,0,0,0,0,2,0,0,0,0,0,0,0,0,1,1,2,2,0,3,4,0,0,0,1,1,0,0,1,0,0,0,0,0,1,1),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
65 (0,1,0,0,0,1,0,0,0,0,4,0,4,1,4,0,3,0,4,0,3,0,4,0,3,0,3,0,4,1,5,1,4,0,0,3,0,5,0,5,2,0,1,0,0,0,2,1,4,0,1,3,0,0,3,0,0,3,1,1,4,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
66 (1,4,0,5,0,3,0,2,0,3,5,4,4,3,4,3,5,3,4,3,3,0,4,3,3,3,3,3,3,2,4,4,3,1,3,4,4,5,4,4,3,4,4,1,3,5,4,3,3,3,1,2,2,3,3,1,3,1,3,3,3,5,3,3,4,5,0,3,0,3,0,3,4,3,4,4,3,0,3,0,2,4,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
67 (0,1,0,4,0,0,0,0,0,1,4,0,4,1,4,2,4,0,3,0,1,0,1,0,0,0,0,0,2,0,3,1,1,1,0,3,0,0,0,1,2,1,0,0,1,1,1,1,0,1,0,0,0,1,0,0,3,0,0,0,0,3,2,0,2,2,0,1,0,0,0,2,3,2,3,3,0,0,0,0,2,1,0),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
68 (0,5,1,5,0,3,0,3,0,5,4,4,5,1,5,3,3,0,4,3,4,3,5,3,4,3,3,2,4,3,4,3,3,0,3,3,1,4,4,3,4,4,4,3,4,5,5,3,2,3,1,1,3,3,1,3,1,1,3,3,2,4,5,3,3,5,0,4,0,3,0,4,4,3,5,3,3,0,3,4,0,4,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
69 (0,5,0,5,0,3,0,2,0,4,4,3,5,2,4,3,3,3,4,4,4,3,5,3,5,3,3,1,4,0,4,3,3,0,3,3,0,4,4,4,4,5,4,3,3,5,5,3,2,3,1,2,3,2,0,1,0,0,3,2,2,4,4,3,1,5,0,4,0,3,0,4,3,1,3,2,1,0,3,3,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
70 (0,4,0,5,0,5,0,4,0,4,5,5,5,3,4,3,3,2,5,4,4,3,5,3,5,3,4,0,4,3,4,4,3,2,4,4,3,4,5,4,4,5,5,0,3,5,5,4,1,3,3,2,3,3,1,3,1,0,4,3,1,4,4,3,4,5,0,4,0,2,0,4,3,4,4,3,3,0,4,0,0,5,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
71 (0,4,0,4,0,5,0,1,1,3,3,4,4,3,4,1,3,0,5,1,3,0,3,1,3,1,1,0,3,0,3,3,4,0,4,3,0,4,4,4,3,4,4,0,3,5,4,1,0,3,0,0,2,3,0,3,1,0,3,1,0,3,2,1,3,5,0,3,0,1,0,3,2,3,3,4,4,0,2,2,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
72 (2,4,0,5,0,4,0,3,0,4,5,5,4,3,5,3,5,3,5,3,5,2,5,3,4,3,3,4,3,4,5,3,2,1,5,4,3,2,3,4,5,3,4,1,2,5,4,3,0,3,3,0,3,2,0,2,3,0,4,1,0,3,4,3,3,5,0,3,0,1,0,4,5,5,5,4,3,0,4,2,0,3,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
73 (0,5,0,4,0,4,0,2,0,5,4,3,4,3,4,3,3,3,4,3,4,2,5,3,5,3,4,1,4,3,4,4,4,0,3,5,0,4,4,4,4,5,3,1,3,4,5,3,3,3,3,3,3,3,0,2,2,0,3,3,2,4,3,3,3,5,3,4,1,3,3,5,3,2,0,0,0,0,4,3,1,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
74 (0,1,0,3,0,3,0,1,0,1,3,3,3,2,3,3,3,0,3,0,0,0,3,1,3,0,0,0,2,2,2,3,0,0,3,2,0,1,2,4,1,3,3,0,0,3,3,3,0,1,0,0,2,1,0,0,3,0,3,1,0,3,0,0,1,3,0,2,0,1,0,3,3,1,3,3,0,0,1,1,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
75 (0,2,0,3,0,2,1,4,0,2,2,3,1,1,3,1,1,0,2,0,3,1,2,3,1,3,0,0,1,0,4,3,2,3,3,3,1,4,2,3,3,3,3,1,0,3,1,4,0,1,1,0,1,2,0,1,1,0,1,1,0,3,1,3,2,2,0,1,0,0,0,2,3,3,3,1,0,0,0,0,0,2,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
76 (0,5,0,4,0,5,0,2,0,4,5,5,3,3,4,3,3,1,5,4,4,2,4,4,4,3,4,2,4,3,5,5,4,3,3,4,3,3,5,5,4,5,5,1,3,4,5,3,1,4,3,1,3,3,0,3,3,1,4,3,1,4,5,3,3,5,0,4,0,3,0,5,3,3,1,4,3,0,4,0,1,5,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
77 (0,5,0,5,0,4,0,2,0,4,4,3,4,3,3,3,3,3,5,4,4,4,4,4,4,5,3,3,5,2,4,4,4,3,4,4,3,3,4,4,5,5,3,3,4,3,4,3,3,4,3,3,3,3,1,2,2,1,4,3,3,5,4,4,3,4,0,4,0,3,0,4,4,4,4,4,1,0,4,2,0,2,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
78 (0,4,0,4,0,3,0,1,0,3,5,2,3,0,3,0,2,1,4,2,3,3,4,1,4,3,3,2,4,1,3,3,3,0,3,3,0,0,3,3,3,5,3,3,3,3,3,2,0,2,0,0,2,0,0,2,0,0,1,0,0,3,1,2,2,3,0,3,0,2,0,4,4,3,3,4,1,0,3,0,0,2,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
79 (0,0,0,4,0,0,0,0,0,0,1,0,1,0,2,0,0,0,0,0,1,0,2,0,1,0,0,0,0,0,3,1,3,0,3,2,0,0,0,1,0,3,2,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,4,0,2,0,0,0,0,0,0,2),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
80 (0,2,1,3,0,2,0,2,0,3,3,3,3,1,3,1,3,3,3,3,3,3,4,2,2,1,2,1,4,0,4,3,1,3,3,3,2,4,3,5,4,3,3,3,3,3,3,3,0,1,3,0,2,0,0,1,0,0,1,0,0,4,2,0,2,3,0,3,3,0,3,3,4,2,3,1,4,0,1,2,0,2,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
81 (0,3,0,3,0,1,0,3,0,2,3,3,3,0,3,1,2,0,3,3,2,3,3,2,3,2,3,1,3,0,4,3,2,0,3,3,1,4,3,3,2,3,4,3,1,3,3,1,1,0,1,1,0,1,0,1,0,1,0,0,0,4,1,1,0,3,0,3,1,0,2,3,3,3,3,3,1,0,0,2,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
82 (0,0,0,0,0,0,0,0,0,0,3,0,2,0,3,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,3,0,3,0,3,1,0,1,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,2,0,2,3,0,0,0,0,0,0,0,0,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
83 (0,2,0,3,1,3,0,3,0,2,3,3,3,1,3,1,3,1,3,1,3,3,3,1,3,0,2,3,1,1,4,3,3,2,3,3,1,2,2,4,1,3,3,0,1,4,2,3,0,1,3,0,3,0,0,1,3,0,2,0,0,3,3,2,1,3,0,3,0,2,0,3,4,4,4,3,1,0,3,0,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
84 (0,2,0,1,0,2,0,0,0,1,3,2,2,1,3,0,1,1,3,0,3,2,3,1,2,0,2,0,1,1,3,3,3,0,3,3,1,1,2,3,2,3,3,1,2,3,2,0,0,1,0,0,0,0,0,0,3,0,1,0,0,2,1,2,1,3,0,3,0,0,0,3,4,4,4,3,2,0,2,0,0,2,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
85 (0,0,0,1,0,1,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,2,2,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,3,1,0,0,0,0,0,0,0,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
86 (0,3,0,3,0,2,0,3,0,3,3,3,2,3,2,2,2,0,3,1,3,3,3,2,3,3,0,0,3,0,3,2,2,0,2,3,1,4,3,4,3,3,2,3,1,5,4,4,0,3,1,2,1,3,0,3,1,1,2,0,2,3,1,3,1,3,0,3,0,1,0,3,3,4,4,2,1,0,2,1,0,2,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
87 (0,1,0,3,0,1,0,2,0,1,4,2,5,1,4,0,2,0,2,1,3,1,4,0,2,1,0,0,2,1,4,1,1,0,3,3,0,5,1,3,2,3,3,1,0,3,2,3,0,1,0,0,0,0,0,0,1,0,0,0,0,4,0,1,0,3,0,2,0,1,0,3,3,3,4,3,3,0,0,0,0,2,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
88 (0,0,0,1,0,0,0,0,0,0,2,0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,1,0,0,0,0,0,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
89 (0,1,0,3,0,4,0,3,0,2,4,3,1,0,3,2,2,1,3,1,2,2,3,1,1,1,2,1,3,0,1,2,0,1,3,2,1,3,0,5,5,1,0,0,1,3,2,1,0,3,0,0,1,0,0,0,0,0,3,4,0,1,1,1,3,2,0,2,0,1,0,2,3,3,1,2,3,0,1,0,1,0,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
90 (0,0,0,1,0,3,0,3,0,2,2,1,0,0,4,0,3,0,3,1,3,0,3,0,3,0,1,0,3,0,3,1,3,0,3,3,0,0,1,2,1,1,1,0,1,2,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,2,2,1,2,0,0,2,0,0,0,0,2,3,3,3,3,0,0,0,0,1,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
91 (0,0,0,3,0,3,0,0,0,0,3,1,1,0,3,0,1,0,2,0,1,0,0,0,0,0,0,0,1,0,3,0,2,0,2,3,0,0,2,2,3,1,2,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,2,0,0,0,0,2,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
92 (2,4,0,5,0,5,0,4,0,3,4,3,3,3,4,3,3,3,4,3,4,4,5,4,5,5,5,2,3,0,5,5,4,1,5,4,3,1,5,4,3,4,4,3,3,4,3,3,0,3,2,0,2,3,0,3,0,0,3,3,0,5,3,2,3,3,0,3,0,3,0,3,4,5,4,5,3,0,4,3,0,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
93 (0,3,0,3,0,3,0,3,0,3,3,4,3,2,3,2,3,0,4,3,3,3,3,3,3,3,3,0,3,2,4,3,3,1,3,4,3,4,4,4,3,4,4,3,2,4,4,1,0,2,0,0,1,1,0,2,0,0,3,1,0,5,3,2,1,3,0,3,0,1,2,4,3,2,4,3,3,0,3,2,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
94 (0,3,0,3,0,1,0,0,0,1,4,3,3,2,3,1,3,1,4,2,3,2,4,2,3,4,3,0,2,2,3,3,3,0,3,3,3,0,3,4,1,3,3,0,3,4,3,3,0,1,1,0,1,0,0,0,4,0,3,0,0,3,1,2,1,3,0,4,0,1,0,4,3,3,4,3,3,0,2,0,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
95 (0,3,0,4,0,1,0,3,0,3,4,3,3,0,3,3,3,1,3,1,3,3,4,3,3,3,0,0,3,1,5,3,3,1,3,3,2,5,4,3,3,4,5,3,2,5,3,4,0,1,0,0,0,0,0,2,0,0,1,1,0,4,2,2,1,3,0,3,0,2,0,4,4,3,5,3,2,0,1,1,0,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
96 (0,5,0,4,0,5,0,2,0,4,4,3,3,2,3,3,3,1,4,3,4,1,5,3,4,3,4,0,4,2,4,3,4,1,5,4,0,4,4,4,4,5,4,1,3,5,4,2,1,4,1,1,3,2,0,3,1,0,3,2,1,4,3,3,3,4,0,4,0,3,0,4,4,4,3,3,3,0,4,2,0,3,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
97 (1,4,0,4,0,3,0,1,0,3,3,3,1,1,3,3,2,2,3,3,1,0,3,2,2,1,2,0,3,1,2,1,2,0,3,2,0,2,2,3,3,4,3,0,3,3,1,2,0,1,1,3,1,2,0,0,3,0,1,1,0,3,2,2,3,3,0,3,0,0,0,2,3,3,4,3,3,0,1,0,0,1,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
98 (0,4,0,4,0,4,0,0,0,3,4,4,3,1,4,2,3,2,3,3,3,1,4,3,4,0,3,0,4,2,3,3,2,2,5,4,2,1,3,4,3,4,3,1,3,3,4,2,0,2,1,0,3,3,0,0,2,0,3,1,0,4,4,3,4,3,0,4,0,1,0,2,4,4,4,4,4,0,3,2,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
99 (0,0,0,1,0,4,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,3,2,0,0,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,2),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
100 (0,2,0,3,0,4,0,4,0,1,3,3,3,0,4,0,2,1,2,1,1,1,2,0,3,1,1,0,1,0,3,1,0,0,3,3,2,0,1,1,0,0,0,0,0,1,0,2,0,2,2,0,3,1,0,0,1,0,1,1,0,1,2,0,3,0,0,0,0,1,0,0,3,3,4,3,1,0,1,0,3,0,2),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
101 (0,0,0,3,0,5,0,0,0,0,1,0,2,0,3,1,0,1,3,0,0,0,2,0,0,0,1,0,0,0,1,1,0,0,4,0,0,0,2,3,0,1,4,1,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,1,0,0,0,0,0,0,0,2,0,0,3,0,0,0,0,0,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
102 (0,2,0,5,0,5,0,1,0,2,4,3,3,2,5,1,3,2,3,3,3,0,4,1,2,0,3,0,4,0,2,2,1,1,5,3,0,0,1,4,2,3,2,0,3,3,3,2,0,2,4,1,1,2,0,1,1,0,3,1,0,1,3,1,2,3,0,2,0,0,0,1,3,5,4,4,4,0,3,0,0,1,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
103 (0,4,0,5,0,4,0,4,0,4,5,4,3,3,4,3,3,3,4,3,4,4,5,3,4,5,4,2,4,2,3,4,3,1,4,4,1,3,5,4,4,5,5,4,4,5,5,5,2,3,3,1,4,3,1,3,3,0,3,3,1,4,3,4,4,4,0,3,0,4,0,3,3,4,4,5,0,0,4,3,0,4,5),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
104 (0,4,0,4,0,3,0,3,0,3,4,4,4,3,3,2,4,3,4,3,4,3,5,3,4,3,2,1,4,2,4,4,3,1,3,4,2,4,5,5,3,4,5,4,1,5,4,3,0,3,2,2,3,2,1,3,1,0,3,3,3,5,3,3,3,5,4,4,2,3,3,4,3,3,3,2,1,0,3,2,1,4,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
105 (0,4,0,5,0,4,0,3,0,3,5,5,3,2,4,3,4,0,5,4,4,1,4,4,4,3,3,3,4,3,5,5,2,3,3,4,1,2,5,5,3,5,5,2,3,5,5,4,0,3,2,0,3,3,1,1,5,1,4,1,0,4,3,2,3,5,0,4,0,3,0,5,4,3,4,3,0,0,4,1,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
106 (1,3,0,4,0,2,0,2,0,2,5,5,3,3,3,3,3,0,4,2,3,4,4,4,3,4,0,0,3,4,5,4,3,3,3,3,2,5,5,4,5,5,5,4,3,5,5,5,1,3,1,0,1,0,0,3,2,0,4,2,0,5,2,3,2,4,1,3,0,3,0,4,5,4,5,4,3,0,4,2,0,5,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
107 (0,3,0,4,0,5,0,3,0,3,4,4,3,2,3,2,3,3,3,3,3,2,4,3,3,2,2,0,3,3,3,3,3,1,3,3,3,0,4,4,3,4,4,1,1,4,4,2,0,3,1,0,1,1,0,4,1,0,2,3,1,3,3,1,3,4,0,3,0,1,0,3,1,3,0,0,1,0,2,0,0,4,4),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
108 (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
109 (0,3,0,3,0,2,0,3,0,1,5,4,3,3,3,1,4,2,1,2,3,4,4,2,4,4,5,0,3,1,4,3,4,0,4,3,3,3,2,3,2,5,3,4,3,2,2,3,0,0,3,0,2,1,0,1,2,0,0,0,0,2,1,1,3,1,0,2,0,4,0,3,4,4,4,5,2,0,2,0,0,1,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
110 (0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,1,1,0,0,1,1,0,0,0,4,2,1,1,0,1,0,3,2,0,0,3,1,1,1,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,1,0,0,0,2,0,0,0,1,4,0,4,2,1,0,0,0,0,0,1),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
111 (0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,0,0,1,0,1,0,0,0,0,3,1,0,0,0,2,0,2,1,0,0,1,2,1,0,1,1,0,0,3,0,0,0,0,0,0,0,0,0,0,0,1,3,1,0,0,0,0,0,1,0,0,2,1,0,0,0,0,0,0,0,0,2),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
112 (0,4,0,4,0,4,0,3,0,4,4,3,4,2,4,3,2,0,4,4,4,3,5,3,5,3,3,2,4,2,4,3,4,3,1,4,0,2,3,4,4,4,3,3,3,4,4,4,3,4,1,3,4,3,2,1,2,1,3,3,3,4,4,3,3,5,0,4,0,3,0,4,3,3,3,2,1,0,3,0,0,3,3),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
113 (0,4,0,3,0,3,0,3,0,3,5,5,3,3,3,3,4,3,4,3,3,3,4,4,4,3,3,3,3,4,3,5,3,3,1,3,2,4,5,5,5,5,4,3,4,5,5,3,2,2,3,3,3,3,2,3,3,1,2,3,2,4,3,3,3,4,0,4,0,2,0,4,3,2,2,1,2,0,3,0,0,4,1),
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
114 )
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
115
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
116 class JapaneseContextAnalysis(object):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
117 NUM_OF_CATEGORY = 6
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
118 DONT_KNOW = -1
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
119 ENOUGH_REL_THRESHOLD = 100
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
120 MAX_REL_THRESHOLD = 1000
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
121 MINIMUM_DATA_THRESHOLD = 4
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
122
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
123 def __init__(self):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
124 self._total_rel = None
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
125 self._rel_sample = None
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
126 self._need_to_skip_char_num = None
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
127 self._last_char_order = None
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
128 self._done = None
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
129 self.reset()
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
130
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
131 def reset(self):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
132 self._total_rel = 0 # total sequence received
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
133 # category counters, each integer counts sequence in its category
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
134 self._rel_sample = [0] * self.NUM_OF_CATEGORY
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
135 # if last byte in current buffer is not the last byte of a character,
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
136 # we need to know how many bytes to skip in next buffer
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
137 self._need_to_skip_char_num = 0
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
138 self._last_char_order = -1 # The order of previous char
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
139 # If this flag is set to True, detection is done and conclusion has
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
140 # been made
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
141 self._done = False
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
142
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
143 def feed(self, byte_str, num_bytes):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
144 if self._done:
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
145 return
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
146
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
147 # The buffer we got is byte oriented, and a character may span in more than one
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
148 # buffers. In case the last one or two byte in last buffer is not
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
149 # complete, we record how many byte needed to complete that character
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
150 # and skip these bytes here. We can choose to record those bytes as
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
151 # well and analyse the character once it is complete, but since a
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
152 # character will not make much difference, by simply skipping
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
153 # this character will simply our logic and improve performance.
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
154 i = self._need_to_skip_char_num
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
155 while i < num_bytes:
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
156 order, char_len = self.get_order(byte_str[i:i + 2])
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
157 i += char_len
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
158 if i > num_bytes:
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
159 self._need_to_skip_char_num = i - num_bytes
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
160 self._last_char_order = -1
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
161 else:
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
162 if (order != -1) and (self._last_char_order != -1):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
163 self._total_rel += 1
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
164 if self._total_rel > self.MAX_REL_THRESHOLD:
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
165 self._done = True
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
166 break
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
167 self._rel_sample[jp2CharContext[self._last_char_order][order]] += 1
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
168 self._last_char_order = order
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
169
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
170 def got_enough_data(self):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
171 return self._total_rel > self.ENOUGH_REL_THRESHOLD
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
172
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
173 def get_confidence(self):
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
174 # This is just one way to calculate confidence. It works well for me.
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
175 if self._total_rel > self.MINIMUM_DATA_THRESHOLD:
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
176 return (self._total_rel - self._rel_sample[0]) / self._total_rel
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
177 else:
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
178 return self.DONT_KNOW
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
179
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
180 def get_order(self, byte_str):
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
181 return -1, 1
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
182
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
183 class SJISContextAnalysis(JapaneseContextAnalysis):
5310
f2b774d78b4a Updated chardet to version 2.3.0.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 3537
diff changeset
184 def __init__(self):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
185 super(SJISContextAnalysis, self).__init__()
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
186 self._charset_name = "SHIFT_JIS"
5310
f2b774d78b4a Updated chardet to version 2.3.0.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 3537
diff changeset
187
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
188 @property
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
189 def charset_name(self):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
190 return self._charset_name
5310
f2b774d78b4a Updated chardet to version 2.3.0.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 3537
diff changeset
191
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
192 def get_order(self, byte_str):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
193 if not byte_str:
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
194 return -1, 1
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
195 # find out current char's byte length
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
196 first_char = byte_str[0]
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
197 if (0x81 <= first_char <= 0x9F) or (0xE0 <= first_char <= 0xFC):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
198 char_len = 2
5310
f2b774d78b4a Updated chardet to version 2.3.0.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 3537
diff changeset
199 if (first_char == 0x87) or (0xFA <= first_char <= 0xFC):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
200 self._charset_name = "CP932"
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
201 else:
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
202 char_len = 1
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
203
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
204 # return its order if it is hiragana
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
205 if len(byte_str) > 1:
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
206 second_char = byte_str[1]
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
207 if (first_char == 202) and (0x9F <= second_char <= 0xF1):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
208 return second_char - 0x9F, char_len
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
209
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
210 return -1, char_len
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
211
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
212 class EUCJPContextAnalysis(JapaneseContextAnalysis):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
213 def get_order(self, byte_str):
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
214 if not byte_str:
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
215 return -1, 1
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
216 # find out current char's byte length
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
217 first_char = byte_str[0]
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
218 if (first_char == 0x8E) or (0xA1 <= first_char <= 0xFE):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
219 char_len = 2
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
220 elif first_char == 0x8F:
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
221 char_len = 3
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
222 else:
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
223 char_len = 1
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
224
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
225 # return its order if it is hiragana
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
226 if len(byte_str) > 1:
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
227 second_char = byte_str[1]
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
228 if (first_char == 0xA4) and (0xA1 <= second_char <= 0xF3):
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
229 return second_char - 0xA1, char_len
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
230
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
231 return -1, char_len
3537
7662053c3906 updated CharDet to 2.2.1, updated changelog
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 12
diff changeset
232
5714
90c57b50600f Updated chardet to 3.0.2.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 5310
diff changeset
233

Mercurial Repository: eric

eric ide

mercurial