ThirdParty/Pygments/pygments/filters/__init__.py

Mon, 25 Mar 2013 03:11:06 +0100

author
T.Rzepka <Tobias.Rzepka@gmail.com>
date
Mon, 25 Mar 2013 03:11:06 +0100
branch
Py2 comp.
changeset 2525
8b507a9a2d40
parent 2426
da76c71624de
child 3145
a9de05d4a22f
permissions
-rw-r--r--

Script changes: Future import added, super calls modified and unicode behavior for str.

0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
2 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
3 pygments.filters
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
4 ~~~~~~~~~~~~~~~~
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
5
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
6 Module containing filter lookup functions and default
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
7 filters.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
8
2426
da76c71624de Updated to Pygments 1.6.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 1705
diff changeset
9 :copyright: Copyright 2006-2013 by the Pygments team, see AUTHORS.
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
10 :license: BSD, see LICENSE for details.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
11 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
12
2525
8b507a9a2d40 Script changes: Future import added, super calls modified and unicode behavior for str.
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 2426
diff changeset
13 from __future__ import unicode_literals # __IGNORE_WARNING__
8b507a9a2d40 Script changes: Future import added, super calls modified and unicode behavior for str.
T.Rzepka <Tobias.Rzepka@gmail.com>
parents: 2426
diff changeset
14
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
15 import re
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
16
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
17 from pygments.token import String, Comment, Keyword, Name, Error, Whitespace, \
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
18 string_to_tokentype
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
19 from pygments.filter import Filter
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
20 from pygments.util import get_list_opt, get_int_opt, get_bool_opt, \
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
21 get_choice_opt, ClassNotFound, OptionError
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
22 from pygments.plugin import find_plugin_filters
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
23
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
24
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
25 def find_filter_class(filtername):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
26 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
27 Lookup a filter by name. Return None if not found.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
28 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
29 if filtername in FILTERS:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
30 return FILTERS[filtername]
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
31 for name, cls in find_plugin_filters():
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
32 if name == filtername:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
33 return cls
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
34 return None
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
35
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
36
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
37 def get_filter_by_name(filtername, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
38 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
39 Return an instantiated filter. Options are passed to the filter
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
40 initializer if wanted. Raise a ClassNotFound if not found.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
41 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
42 cls = find_filter_class(filtername)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
43 if cls:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
44 return cls(**options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
45 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
46 raise ClassNotFound('filter %r not found' % filtername)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
47
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
48
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
49 def get_all_filters():
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
50 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
51 Return a generator of all filter names.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
52 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
53 for name in FILTERS:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
54 yield name
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
55 for name, _ in find_plugin_filters():
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
56 yield name
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
57
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
58
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
59 def _replace_special(ttype, value, regex, specialttype,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
60 replacefunc=lambda x: x):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
61 last = 0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
62 for match in regex.finditer(value):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
63 start, end = match.start(), match.end()
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
64 if start != last:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
65 yield ttype, value[last:start]
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
66 yield specialttype, replacefunc(value[start:end])
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
67 last = end
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
68 if last != len(value):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
69 yield ttype, value[last:]
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
70
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
71
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
72 class CodeTagFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
73 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
74 Highlight special code tags in comments and docstrings.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
75
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
76 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
77
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
78 `codetags` : list of strings
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
79 A list of strings that are flagged as code tags. The default is to
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
80 highlight ``XXX``, ``TODO``, ``BUG`` and ``NOTE``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
81 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
82
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
83 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
84 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
85 tags = get_list_opt(options, 'codetags',
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
86 ['XXX', 'TODO', 'BUG', 'NOTE'])
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
87 self.tag_re = re.compile(r'\b(%s)\b' % '|'.join([
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
88 re.escape(tag) for tag in tags if tag
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
89 ]))
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
90
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
91 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
92 regex = self.tag_re
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
93 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
94 if ttype in String.Doc or \
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
95 ttype in Comment and \
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
96 ttype not in Comment.Preproc:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
97 for sttype, svalue in _replace_special(ttype, value, regex,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
98 Comment.Special):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
99 yield sttype, svalue
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
100 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
101 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
102
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
103
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
104 class KeywordCaseFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
105 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
106 Convert keywords to lowercase or uppercase or capitalize them, which
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
107 means first letter uppercase, rest lowercase.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
108
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
109 This can be useful e.g. if you highlight Pascal code and want to adapt the
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
110 code to your styleguide.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
111
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
112 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
113
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
114 `case` : string
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
115 The casing to convert keywords to. Must be one of ``'lower'``,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
116 ``'upper'`` or ``'capitalize'``. The default is ``'lower'``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
117 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
118
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
119 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
120 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
121 case = get_choice_opt(options, 'case', ['lower', 'upper', 'capitalize'], 'lower')
12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
122 self.convert = getattr(str, case)
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
123
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
124 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
125 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
126 if ttype in Keyword:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
127 yield ttype, self.convert(value)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
128 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
129 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
130
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
131
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
132 class NameHighlightFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
133 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
134 Highlight a normal Name token with a different token type.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
135
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
136 Example::
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
137
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
138 filter = NameHighlightFilter(
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
139 names=['foo', 'bar', 'baz'],
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
140 tokentype=Name.Function,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
141 )
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
142
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
143 This would highlight the names "foo", "bar" and "baz"
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
144 as functions. `Name.Function` is the default token type.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
145
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
146 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
147
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
148 `names` : list of strings
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
149 A list of names that should be given the different token type.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
150 There is no default.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
151 `tokentype` : TokenType or string
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
152 A token type or a string containing a token type name that is
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
153 used for highlighting the strings in `names`. The default is
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
154 `Name.Function`.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
155 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
156
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
157 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
158 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
159 self.names = set(get_list_opt(options, 'names', []))
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
160 tokentype = options.get('tokentype')
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
161 if tokentype:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
162 self.tokentype = string_to_tokentype(tokentype)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
163 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
164 self.tokentype = Name.Function
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
165
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
166 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
167 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
168 if ttype is Name and value in self.names:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
169 yield self.tokentype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
170 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
171 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
172
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
173
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
174 class ErrorToken(Exception):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
175 pass
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
176
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
177 class RaiseOnErrorTokenFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
178 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
179 Raise an exception when the lexer generates an error token.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
180
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
181 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
182
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
183 `excclass` : Exception class
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
184 The exception class to raise.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
185 The default is `pygments.filters.ErrorToken`.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
186
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
187 *New in Pygments 0.8.*
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
188 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
189
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
190 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
191 Filter.__init__(self, **options)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
192 self.exception = options.get('excclass', ErrorToken)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
193 try:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
194 # issubclass() will raise TypeError if first argument is not a class
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
195 if not issubclass(self.exception, Exception):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
196 raise TypeError
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
197 except TypeError:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
198 raise OptionError('excclass option is not an exception class')
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
199
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
200 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
201 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
202 if ttype is Error:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
203 raise self.exception(value)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
204 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
205
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
206
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
207 class VisibleWhitespaceFilter(Filter):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
208 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
209 Convert tabs, newlines and/or spaces to visible characters.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
210
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
211 Options accepted:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
212
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
213 `spaces` : string or bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
214 If this is a one-character string, spaces will be replaces by this string.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
215 If it is another true value, spaces will be replaced by ``·`` (unicode
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
216 MIDDLE DOT). If it is a false value, spaces will not be replaced. The
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
217 default is ``False``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
218 `tabs` : string or bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
219 The same as for `spaces`, but the default replacement character is ``»``
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
220 (unicode RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK). The default value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
221 is ``False``. Note: this will not work if the `tabsize` option for the
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
222 lexer is nonzero, as tabs will already have been expanded then.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
223 `tabsize` : int
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
224 If tabs are to be replaced by this filter (see the `tabs` option), this
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
225 is the total number of characters that a tab should be expanded to.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
226 The default is ``8``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
227 `newlines` : string or bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
228 The same as for `spaces`, but the default replacement character is ``¶``
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
229 (unicode PILCROW SIGN). The default value is ``False``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
230 `wstokentype` : bool
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
231 If true, give whitespace the special `Whitespace` token type. This allows
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
232 styling the visible whitespace differently (e.g. greyed out), but it can
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
233 disrupt background colors. The default is ``True``.
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
234
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
235 *New in Pygments 0.8.*
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
236 """
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
237
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
238 def __init__(self, **options):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
239 Filter.__init__(self, **options)
12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
240 for name, default in list({'spaces': '·', 'tabs': '»', 'newlines': '¶'}.items()):
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
241 opt = options.get(name, False)
12
1d8dd9706f46 First commit after changing to Python 3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 0
diff changeset
242 if isinstance(opt, str) and len(opt) == 1:
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
243 setattr(self, name, opt)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
244 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
245 setattr(self, name, (opt and default or ''))
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
246 tabsize = get_int_opt(options, 'tabsize', 8)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
247 if self.tabs:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
248 self.tabs += ' '*(tabsize-1)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
249 if self.newlines:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
250 self.newlines += '\n'
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
251 self.wstt = get_bool_opt(options, 'wstokentype', True)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
252
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
253 def filter(self, lexer, stream):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
254 if self.wstt:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
255 spaces = self.spaces or ' '
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
256 tabs = self.tabs or '\t'
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
257 newlines = self.newlines or '\n'
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
258 regex = re.compile(r'\s')
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
259 def replacefunc(wschar):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
260 if wschar == ' ':
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
261 return spaces
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
262 elif wschar == '\t':
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
263 return tabs
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
264 elif wschar == '\n':
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
265 return newlines
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
266 return wschar
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
267
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
268 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
269 for sttype, svalue in _replace_special(ttype, value, regex,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
270 Whitespace, replacefunc):
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
271 yield sttype, svalue
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
272 else:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
273 spaces, tabs, newlines = self.spaces, self.tabs, self.newlines
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
274 # simpler processing
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
275 for ttype, value in stream:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
276 if spaces:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
277 value = value.replace(' ', spaces)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
278 if tabs:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
279 value = value.replace('\t', tabs)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
280 if newlines:
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
281 value = value.replace('\n', newlines)
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
282 yield ttype, value
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
283
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
284
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
285 class GobbleFilter(Filter):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
286 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
287 Gobbles source code lines (eats initial characters).
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
288
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
289 This filter drops the first ``n`` characters off every line of code. This
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
290 may be useful when the source code fed to the lexer is indented by a fixed
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
291 amount of space that isn't desired in the output.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
292
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
293 Options accepted:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
294
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
295 `n` : int
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
296 The number of characters to gobble.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
297
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
298 *New in Pygments 1.2.*
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
299 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
300 def __init__(self, **options):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
301 Filter.__init__(self, **options)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
302 self.n = get_int_opt(options, 'n', 0)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
303
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
304 def gobble(self, value, left):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
305 if left < len(value):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
306 return value[left:], 0
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
307 else:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
308 return '', left - len(value)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
309
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
310 def filter(self, lexer, stream):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
311 n = self.n
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
312 left = n # How many characters left to gobble.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
313 for ttype, value in stream:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
314 # Remove ``left`` tokens from first line, ``n`` from all others.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
315 parts = value.split('\n')
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
316 (parts[0], left) = self.gobble(parts[0], left)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
317 for i in range(1, len(parts)):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
318 (parts[i], left) = self.gobble(parts[i], n)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
319 value = '\n'.join(parts)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
320
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
321 if value != '':
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
322 yield ttype, value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
323
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
324
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
325 class TokenMergeFilter(Filter):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
326 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
327 Merges consecutive tokens with the same token type in the output stream of a
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
328 lexer.
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
329
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
330 *New in Pygments 1.2.*
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
331 """
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
332 def __init__(self, **options):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
333 Filter.__init__(self, **options)
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
334
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
335 def filter(self, lexer, stream):
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
336 current_type = None
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
337 current_value = None
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
338 for ttype, value in stream:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
339 if ttype is current_type:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
340 current_value += value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
341 else:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
342 if current_type is not None:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
343 yield current_type, current_value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
344 current_type = ttype
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
345 current_value = value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
346 if current_type is not None:
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
347 yield current_type, current_value
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
348
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
349
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
350 FILTERS = {
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
351 'codetagify': CodeTagFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
352 'keywordcase': KeywordCaseFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
353 'highlight': NameHighlightFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
354 'raiseonerror': RaiseOnErrorTokenFilter,
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
355 'whitespace': VisibleWhitespaceFilter,
684
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
356 'gobble': GobbleFilter,
2f29a0b6e1c7 Updated Pygments to version 1.3.1.
Detlev Offenbach <detlev@die-offenbachs.de>
parents: 12
diff changeset
357 'tokenmerge': TokenMergeFilter,
0
de9c2efb9d02 Started porting eric4 to Python3
Detlev Offenbach <detlev@die-offenbachs.de>
parents:
diff changeset
358 }

eric ide

mercurial