ThirdParty/Pygments/pygments/filters/__init__.py

Wed, 20 Oct 2010 08:39:56 +0200

author
Detlev Offenbach <detlev@die-offenbachs.de>
date
Wed, 20 Oct 2010 08:39:56 +0200
changeset 684
2f29a0b6e1c7
parent 12
1d8dd9706f46
child 1705
b0fbc9300f2b
permissions
-rw-r--r--

Updated Pygments to version 1.3.1.

0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
2 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
3 pygments.filters
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
4 ~~~~~~~~~~~~~~~~
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
5
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
6 Module containing filter lookup functions and default
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
7 filters.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
8
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
9 :copyright: Copyright 2006-2010 by the Pygments team, see AUTHORS.
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
10 :license: BSD, see LICENSE for details.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
11 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
12
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
13 import re
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
14
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
15 from pygments.token import String, Comment, Keyword, Name, Error, Whitespace, \
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
16 string_to_tokentype
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
17 from pygments.filter import Filter
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
18 from pygments.util import get_list_opt, get_int_opt, get_bool_opt, \
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
19 get_choice_opt, ClassNotFound, OptionError
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
20 from pygments.plugin import find_plugin_filters
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
21
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
22
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
23 def find_filter_class(filtername):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
24 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
25 Lookup a filter by name. Return None if not found.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
26 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
27 if filtername in FILTERS:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
28 return FILTERS[filtername]
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
29 for name, cls in find_plugin_filters():
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
30 if name == filtername:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
31 return cls
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
32 return None
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
33
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
34
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
35 def get_filter_by_name(filtername, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
36 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
37 Return an instantiated filter. Options are passed to the filter
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
38 initializer if wanted. Raise a ClassNotFound if not found.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
39 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
40 cls = find_filter_class(filtername)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
41 if cls:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
42 return cls(**options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
43 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
44 raise ClassNotFound('filter %r not found' % filtername)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
45
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
46
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
47 def get_all_filters():
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
48 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
49 Return a generator of all filter names.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
50 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
51 for name in FILTERS:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
52 yield name
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
53 for name, _ in find_plugin_filters():
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
54 yield name
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
55
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
56
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
57 def _replace_special(ttype, value, regex, specialttype,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
58 replacefunc=lambda x: x):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
59 last = 0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
60 for match in regex.finditer(value):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
61 start, end = match.start(), match.end()
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
62 if start != last:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
63 yield ttype, value[last:start]
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
64 yield specialttype, replacefunc(value[start:end])
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
65 last = end
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
66 if last != len(value):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
67 yield ttype, value[last:]
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
68
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
69
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
70 class CodeTagFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
71 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
72 Highlight special code tags in comments and docstrings.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
73
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
74 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
75
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
76 `codetags` : list of strings
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
77 A list of strings that are flagged as code tags. The default is to
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
78 highlight ``XXX``, ``TODO``, ``BUG`` and ``NOTE``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
79 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
80
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
81 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
82 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
83 tags = get_list_opt(options, 'codetags',
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
84 ['XXX', 'TODO', 'BUG', 'NOTE'])
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
85 self.tag_re = re.compile(r'\b(%s)\b' % '|'.join([
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
86 re.escape(tag) for tag in tags if tag
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
87 ]))
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
88
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
89 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
90 regex = self.tag_re
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
91 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
92 if ttype in String.Doc or \
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
93 ttype in Comment and \
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
94 ttype not in Comment.Preproc:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
95 for sttype, svalue in _replace_special(ttype, value, regex,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
96 Comment.Special):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
97 yield sttype, svalue
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
98 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
99 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
100
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
101
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
102 class KeywordCaseFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
103 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
104 Convert keywords to lowercase or uppercase or capitalize them, which
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
105 means first letter uppercase, rest lowercase.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
106
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
107 This can be useful e.g. if you highlight Pascal code and want to adapt the
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
108 code to your styleguide.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
109
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
110 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
111
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
112 `case` : string
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
113 The casing to convert keywords to. Must be one of ``'lower'``,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
114 ``'upper'`` or ``'capitalize'``. The default is ``'lower'``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
115 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
116
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
117 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
118 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
119 case = get_choice_opt(options, 'case', ['lower', 'upper', 'capitalize'], 'lower')
12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
120 self.convert = getattr(str, case)
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
121
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
122 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
123 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
124 if ttype in Keyword:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
125 yield ttype, self.convert(value)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
126 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
127 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
128
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
129
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
130 class NameHighlightFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
131 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
132 Highlight a normal Name token with a different token type.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
133
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
134 Example::
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
135
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
136 filter = NameHighlightFilter(
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
137 names=['foo', 'bar', 'baz'],
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
138 tokentype=Name.Function,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
139 )
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
140
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
141 This would highlight the names "foo", "bar" and "baz"
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
142 as functions. `Name.Function` is the default token type.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
143
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
144 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
145
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
146 `names` : list of strings
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
147 A list of names that should be given the different token type.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
148 There is no default.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
149 `tokentype` : TokenType or string
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
150 A token type or a string containing a token type name that is
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
151 used for highlighting the strings in `names`. The default is
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
152 `Name.Function`.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
153 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
154
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
155 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
156 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
157 self.names = set(get_list_opt(options, 'names', []))
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
158 tokentype = options.get('tokentype')
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
159 if tokentype:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
160 self.tokentype = string_to_tokentype(tokentype)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
161 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
162 self.tokentype = Name.Function
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
163
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
164 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
165 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
166 if ttype is Name and value in self.names:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
167 yield self.tokentype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
168 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
169 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
170
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
171
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
172 class ErrorToken(Exception):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
173 pass
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
174
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
175 class RaiseOnErrorTokenFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
176 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
177 Raise an exception when the lexer generates an error token.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
178
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
179 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
180
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
181 `excclass` : Exception class
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
182 The exception class to raise.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
183 The default is `pygments.filters.ErrorToken`.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
184
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
185 *New in Pygments 0.8.*
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
186 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
187
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
188 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
189 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
190 self.exception = options.get('excclass', ErrorToken)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
191 try:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
192 # issubclass() will raise TypeError if first argument is not a class
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
193 if not issubclass(self.exception, Exception):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
194 raise TypeError
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
195 except TypeError:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
196 raise OptionError('excclass option is not an exception class')
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
197
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
198 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
199 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
200 if ttype is Error:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
201 raise self.exception(value)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
202 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
203
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
204
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
205 class VisibleWhitespaceFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
206 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
207 Convert tabs, newlines and/or spaces to visible characters.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
208
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
209 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
210
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
211 `spaces` : string or bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
212 If this is a one-character string, spaces will be replaces by this string.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
213 If it is another true value, spaces will be replaced by ``·`` (unicode
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
214 MIDDLE DOT). If it is a false value, spaces will not be replaced. The
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
215 default is ``False``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
216 `tabs` : string or bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
217 The same as for `spaces`, but the default replacement character is ``»``
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
218 (unicode RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK). The default value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
219 is ``False``. Note: this will not work if the `tabsize` option for the
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
220 lexer is nonzero, as tabs will already have been expanded then.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
221 `tabsize` : int
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
222 If tabs are to be replaced by this filter (see the `tabs` option), this
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
223 is the total number of characters that a tab should be expanded to.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
224 The default is ``8``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
225 `newlines` : string or bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
226 The same as for `spaces`, but the default replacement character is ``¶``
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
227 (unicode PILCROW SIGN). The default value is ``False``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
228 `wstokentype` : bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
229 If true, give whitespace the special `Whitespace` token type. This allows
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
230 styling the visible whitespace differently (e.g. greyed out), but it can
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
231 disrupt background colors. The default is ``True``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
232
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
233 *New in Pygments 0.8.*
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
234 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
235
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
236 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
237 Filter.__init__(self, **options)
12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
238 for name, default in list({'spaces': '·', 'tabs': '»', 'newlines': '¶'}.items()):
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
239 opt = options.get(name, False)
12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
240 if isinstance(opt, str) and len(opt) == 1:
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
241 setattr(self, name, opt)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
242 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
243 setattr(self, name, (opt and default or ''))
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
244 tabsize = get_int_opt(options, 'tabsize', 8)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
245 if self.tabs:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
246 self.tabs += ' '*(tabsize-1)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
247 if self.newlines:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
248 self.newlines += '\n'
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
249 self.wstt = get_bool_opt(options, 'wstokentype', True)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
250
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
251 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
252 if self.wstt:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
253 spaces = self.spaces or ' '
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
254 tabs = self.tabs or '\t'
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
255 newlines = self.newlines or '\n'
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
256 regex = re.compile(r'\s')
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
257 def replacefunc(wschar):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
258 if wschar == ' ':
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
259 return spaces
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
260 elif wschar == '\t':
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
261 return tabs
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
262 elif wschar == '\n':
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
263 return newlines
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
264 return wschar
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
265
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
266 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
267 for sttype, svalue in _replace_special(ttype, value, regex,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
268 Whitespace, replacefunc):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
269 yield sttype, svalue
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
270 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
271 spaces, tabs, newlines = self.spaces, self.tabs, self.newlines
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
272 # simpler processing
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
273 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
274 if spaces:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
275 value = value.replace(' ', spaces)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
276 if tabs:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
277 value = value.replace('\t', tabs)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
278 if newlines:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
279 value = value.replace('\n', newlines)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
280 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
281
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
282
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
283 class GobbleFilter(Filter):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
284 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
285 Gobbles source code lines (eats initial characters).
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
286
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
287 This filter drops the first ``n`` characters off every line of code. This
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
288 may be useful when the source code fed to the lexer is indented by a fixed
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
289 amount of space that isn't desired in the output.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
290
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
291 Options accepted:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
292
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
293 `n` : int
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
294 The number of characters to gobble.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
295
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
296 *New in Pygments 1.2.*
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
297 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
298 def __init__(self, **options):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
299 Filter.__init__(self, **options)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
300 self.n = get_int_opt(options, 'n', 0)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
301
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
302 def gobble(self, value, left):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
303 if left < len(value):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
304 return value[left:], 0
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
305 else:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
306 return '', left - len(value)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
307
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
308 def filter(self, lexer, stream):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
309 n = self.n
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
310 left = n # How many characters left to gobble.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
311 for ttype, value in stream:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
312 # Remove ``left`` tokens from first line, ``n`` from all others.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
313 parts = value.split('\n')
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
314 (parts[0], left) = self.gobble(parts[0], left)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
315 for i in range(1, len(parts)):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
316 (parts[i], left) = self.gobble(parts[i], n)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
317 value = '\n'.join(parts)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
318
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
319 if value != '':
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
320 yield ttype, value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
321
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
322
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
323 class TokenMergeFilter(Filter):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
324 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
325 Merges consecutive tokens with the same token type in the output stream of a
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
326 lexer.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
327
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
328 *New in Pygments 1.2.*
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
329 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
330 def __init__(self, **options):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
331 Filter.__init__(self, **options)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
332
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
333 def filter(self, lexer, stream):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
334 output = []
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
335 current_type = None
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
336 current_value = None
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
337 for ttype, value in stream:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
338 if ttype is current_type:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
339 current_value += value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
340 else:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
341 if current_type is not None:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
342 yield current_type, current_value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
343 current_type = ttype
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
344 current_value = value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
345 if current_type is not None:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
346 yield current_type, current_value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
347
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
348
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
349 FILTERS = {
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
350 'codetagify': CodeTagFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
351 'keywordcase': KeywordCaseFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
352 'highlight': NameHighlightFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
353 'raiseonerror': RaiseOnErrorTokenFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
354 'whitespace': VisibleWhitespaceFilter,
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
355 'gobble': GobbleFilter,
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
356 'tokenmerge': TokenMergeFilter,
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
357 }

eric ide

mercurial