ThirdParty/CharDet/chardet/codingstatemachine.py

Sat, 02 Jan 2010 15:11:35 +0000

author
Detlev Offenbach <detlev@die-offenbachs.de>
date
Sat, 02 Jan 2010 15:11:35 +0000
changeset 12
1d8dd9706f46
parent 0
de9c2efb9d02
child 3537
7662053c3906
permissions
-rw-r--r--

First commit after changing to Python 3.1.

12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
1 ######################## BEGIN LICENSE BLOCK ########################
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
2 # The Original Code is mozilla.org code.
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
3 #
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
4 # The Initial Developer of the Original Code is
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
5 # Netscape Communications Corporation.
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
6 # Portions created by the Initial Developer are Copyright (C) 1998
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
7 # the Initial Developer. All Rights Reserved.
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
8 #
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
9 # Contributor(s):
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
10 # Mark Pilgrim - port to Python
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
11 #
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
12 # This library is free software; you can redistribute it and/or
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
13 # modify it under the terms of the GNU Lesser General Public
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
14 # License as published by the Free Software Foundation; either
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
15 # version 2.1 of the License, or (at your option) any later version.
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
16 #
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
17 # This library is distributed in the hope that it will be useful,
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
18 # but WITHOUT ANY WARRANTY; without even the implied warranty of
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
19 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
20 # Lesser General Public License for more details.
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
21 #
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
22 # You should have received a copy of the GNU Lesser General Public
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
23 # License along with this library; if not, write to the Free Software
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
24 # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
25 # 02110-1301 USA
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
26 ######################### END LICENSE BLOCK #########################
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
27
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
28 from .constants import eStart, eError, eItsMe
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
29
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
30 class CodingStateMachine:
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
31 def __init__(self, sm):
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
32 self._mModel = sm
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
33 self._mCurrentBytePos = 0
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
34 self._mCurrentCharLen = 0
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
35 self.reset()
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
36
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
37 def reset(self):
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
38 self._mCurrentState = eStart
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
39
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
40 def next_state(self, c):
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
41 # for each byte we get its class
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
42 # if it is first byte, we also get byte length
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
43 # PY3K: aBuf is a byte stream, so c is an int, not a byte
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
44 byteCls = self._mModel['classTable'][c]
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
45 if self._mCurrentState == eStart:
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
46 self._mCurrentBytePos = 0
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
47 self._mCurrentCharLen = self._mModel['charLenTable'][byteCls]
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
48 # from byte's class and stateTable, we get its next state
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
49 self._mCurrentState = self._mModel['stateTable'][self._mCurrentState * self._mModel['classFactor'] + byteCls]
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
50 self._mCurrentBytePos += 1
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
51 return self._mCurrentState
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
52
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
53 def get_current_charlen(self):
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
54 return self._mCurrentCharLen
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
55
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
56 def get_coding_state_machine(self):
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
57 return self._mModel['name']

eric ide

mercurial