20 </style> |
20 </style> |
21 </head> |
21 </head> |
22 <body><a NAME="top" ID="top"></a> |
22 <body><a NAME="top" ID="top"></a> |
23 <h1>eric5.Plugins.CheckerPlugins.Tabnanny.Tabnanny</h1> |
23 <h1>eric5.Plugins.CheckerPlugins.Tabnanny.Tabnanny</h1> |
24 <p> |
24 <p> |
25 The Tab Nanny despises ambiguous indentation. She knows no mercy. |
25 The Tab Nanny despises ambiguous indentation. She knows no mercy. |
26 </p><p> |
26 </p><p> |
27 tabnanny -- Detection of ambiguous indentation |
27 tabnanny -- Detection of ambiguous indentation |
28 </p><p> |
28 </p><p> |
29 For the time being this module is intended to be called as a script. |
29 For the time being this module is intended to be called as a script. |
30 However it is possible to import it into an IDE and use the function |
30 However it is possible to import it into an IDE and use the function |
31 check() described below. |
31 check() described below. |
32 </p><p> |
32 </p><p> |
33 Warning: The API provided by this module is likely to change in future |
33 Warning: The API provided by this module is likely to change in future |
34 releases; such changes may not be backward compatible. |
34 releases; such changes may not be backward compatible. |
35 </p><p> |
35 </p><p> |
36 This is a modified version to make the original tabnanny better suitable |
36 This is a modified version to make the original tabnanny better suitable |
37 for being called from within the eric5 IDE. |
37 for being called from within the eric5 IDE. |
38 </p><dl> |
38 </p><dl> |
39 <dt>Raises <b>ValueError</b>:</dt> |
39 <dt>Raises <b>ValueError</b>:</dt> |
40 <dd> |
40 <dd> |
41 The tokenize module is too old. |
41 The tokenize module is too old. |
42 </dd> |
42 </dd> |
43 </dl> |
43 </dl> |
44 <h3>Global Attributes</h3> |
44 <h3>Global Attributes</h3> |
45 <table> |
45 <table> |
46 <tr><td>__all__</td></tr><tr><td>__version__</td></tr> |
46 <tr><td>__all__</td></tr><tr><td>__version__</td></tr> |
47 </table> |
47 </table> |
48 <h3>Classes</h3> |
48 <h3>Classes</h3> |
49 <table> |
49 <table> |
50 <tr> |
50 <tr> |
51 <td><a href="#NannyNag">NannyNag</a></td> |
51 <td><a href="#NannyNag">NannyNag</a></td> |
52 <td>Raised by tokeneater() if detecting an ambiguous indent.</td> |
52 <td>Raised by tokeneater() if detecting an ambiguous indent.</td> |
53 </tr><tr> |
53 </tr><tr> |
54 <td><a href="#Whitespace">Whitespace</a></td> |
54 <td><a href="#Whitespace">Whitespace</a></td> |
55 <td>Class implementing the whitespace checker.</td> |
55 <td>Class implementing the whitespace checker.</td> |
56 </tr> |
56 </tr> |
57 </table> |
57 </table> |
58 <h3>Functions</h3> |
58 <h3>Functions</h3> |
59 <table> |
59 <table> |
60 <tr> |
60 <tr> |
61 <td><a href="#check">check</a></td> |
61 <td><a href="#check">check</a></td> |
62 <td>Private function to check one Python source file for whitespace related problems.</td> |
62 <td>Private function to check one Python source file for whitespace related problems.</td> |
63 </tr><tr> |
63 </tr><tr> |
64 <td><a href="#format_witnesses">format_witnesses</a></td> |
64 <td><a href="#format_witnesses">format_witnesses</a></td> |
65 <td>Function to format the witnesses as a readable string.</td> |
65 <td>Function to format the witnesses as a readable string.</td> |
66 </tr><tr> |
66 </tr><tr> |
67 <td><a href="#process_tokens">process_tokens</a></td> |
67 <td><a href="#process_tokens">process_tokens</a></td> |
68 <td>Function processing all tokens generated by a tokenizer run.</td> |
68 <td>Function processing all tokens generated by a tokenizer run.</td> |
69 </tr> |
69 </tr> |
70 </table> |
70 </table> |
71 <hr /><hr /> |
71 <hr /><hr /> |
72 <a NAME="NannyNag" ID="NannyNag"></a> |
72 <a NAME="NannyNag" ID="NannyNag"></a> |
73 <h2>NannyNag</h2> |
73 <h2>NannyNag</h2> |
74 <p> |
74 <p> |
75 Raised by tokeneater() if detecting an ambiguous indent. |
75 Raised by tokeneater() if detecting an ambiguous indent. |
76 Captured and handled in check(). |
76 Captured and handled in check(). |
77 </p> |
77 </p> |
78 <h3>Derived from</h3> |
78 <h3>Derived from</h3> |
79 Exception |
79 Exception |
80 <h3>Class Attributes</h3> |
80 <h3>Class Attributes</h3> |
81 <table> |
81 <table> |
83 </table> |
83 </table> |
84 <h3>Methods</h3> |
84 <h3>Methods</h3> |
85 <table> |
85 <table> |
86 <tr> |
86 <tr> |
87 <td><a href="#NannyNag.__init__">NannyNag</a></td> |
87 <td><a href="#NannyNag.__init__">NannyNag</a></td> |
88 <td>Constructor</td> |
88 <td>Constructor</td> |
89 </tr><tr> |
89 </tr><tr> |
90 <td><a href="#NannyNag.get_line">get_line</a></td> |
90 <td><a href="#NannyNag.get_line">get_line</a></td> |
91 <td>Method to retrieve the offending line.</td> |
91 <td>Method to retrieve the offending line.</td> |
92 </tr><tr> |
92 </tr><tr> |
93 <td><a href="#NannyNag.get_lineno">get_lineno</a></td> |
93 <td><a href="#NannyNag.get_lineno">get_lineno</a></td> |
94 <td>Method to retrieve the line number.</td> |
94 <td>Method to retrieve the line number.</td> |
95 </tr><tr> |
95 </tr><tr> |
96 <td><a href="#NannyNag.get_msg">get_msg</a></td> |
96 <td><a href="#NannyNag.get_msg">get_msg</a></td> |
97 <td>Method to retrieve the message.</td> |
97 <td>Method to retrieve the message.</td> |
98 </tr> |
98 </tr> |
99 </table> |
99 </table> |
100 <a NAME="NannyNag.__init__" ID="NannyNag.__init__"></a> |
100 <a NAME="NannyNag.__init__" ID="NannyNag.__init__"></a> |
101 <h4>NannyNag (Constructor)</h4> |
101 <h4>NannyNag (Constructor)</h4> |
102 <b>NannyNag</b>(<i>lineno, msg, line</i>) |
102 <b>NannyNag</b>(<i>lineno, msg, line</i>) |
103 <p> |
103 <p> |
104 Constructor |
104 Constructor |
105 </p><dl> |
105 </p><dl> |
106 <dt><i>lineno</i></dt> |
106 <dt><i>lineno</i></dt> |
107 <dd> |
107 <dd> |
108 Line number of the ambiguous indent. |
108 Line number of the ambiguous indent. |
109 </dd><dt><i>msg</i></dt> |
109 </dd><dt><i>msg</i></dt> |
110 <dd> |
110 <dd> |
111 Descriptive message assigned to this problem. |
111 Descriptive message assigned to this problem. |
112 </dd><dt><i>line</i></dt> |
112 </dd><dt><i>line</i></dt> |
113 <dd> |
113 <dd> |
114 The offending source line. |
114 The offending source line. |
115 </dd> |
115 </dd> |
116 </dl><a NAME="NannyNag.get_line" ID="NannyNag.get_line"></a> |
116 </dl><a NAME="NannyNag.get_line" ID="NannyNag.get_line"></a> |
117 <h4>NannyNag.get_line</h4> |
117 <h4>NannyNag.get_line</h4> |
118 <b>get_line</b>(<i></i>) |
118 <b>get_line</b>(<i></i>) |
119 <p> |
119 <p> |
120 Method to retrieve the offending line. |
120 Method to retrieve the offending line. |
121 </p><dl> |
121 </p><dl> |
122 <dt>Returns:</dt> |
122 <dt>Returns:</dt> |
123 <dd> |
123 <dd> |
124 The line of code (string) |
124 The line of code (string) |
125 </dd> |
125 </dd> |
126 </dl><a NAME="NannyNag.get_lineno" ID="NannyNag.get_lineno"></a> |
126 </dl><a NAME="NannyNag.get_lineno" ID="NannyNag.get_lineno"></a> |
127 <h4>NannyNag.get_lineno</h4> |
127 <h4>NannyNag.get_lineno</h4> |
128 <b>get_lineno</b>(<i></i>) |
128 <b>get_lineno</b>(<i></i>) |
129 <p> |
129 <p> |
130 Method to retrieve the line number. |
130 Method to retrieve the line number. |
131 </p><dl> |
131 </p><dl> |
132 <dt>Returns:</dt> |
132 <dt>Returns:</dt> |
133 <dd> |
133 <dd> |
134 The line number (integer) |
134 The line number (integer) |
135 </dd> |
135 </dd> |
136 </dl><a NAME="NannyNag.get_msg" ID="NannyNag.get_msg"></a> |
136 </dl><a NAME="NannyNag.get_msg" ID="NannyNag.get_msg"></a> |
137 <h4>NannyNag.get_msg</h4> |
137 <h4>NannyNag.get_msg</h4> |
138 <b>get_msg</b>(<i></i>) |
138 <b>get_msg</b>(<i></i>) |
139 <p> |
139 <p> |
140 Method to retrieve the message. |
140 Method to retrieve the message. |
141 </p><dl> |
141 </p><dl> |
142 <dt>Returns:</dt> |
142 <dt>Returns:</dt> |
143 <dd> |
143 <dd> |
144 The error message (string) |
144 The error message (string) |
145 </dd> |
145 </dd> |
146 </dl> |
146 </dl> |
147 <div align="right"><a href="#top">Up</a></div> |
147 <div align="right"><a href="#top">Up</a></div> |
148 <hr /><hr /> |
148 <hr /><hr /> |
149 <a NAME="Whitespace" ID="Whitespace"></a> |
149 <a NAME="Whitespace" ID="Whitespace"></a> |
150 <h2>Whitespace</h2> |
150 <h2>Whitespace</h2> |
151 <p> |
151 <p> |
152 Class implementing the whitespace checker. |
152 Class implementing the whitespace checker. |
153 </p> |
153 </p> |
154 <h3>Derived from</h3> |
154 <h3>Derived from</h3> |
155 object |
155 object |
156 <h3>Class Attributes</h3> |
156 <h3>Class Attributes</h3> |
157 <table> |
157 <table> |
159 </table> |
159 </table> |
160 <h3>Methods</h3> |
160 <h3>Methods</h3> |
161 <table> |
161 <table> |
162 <tr> |
162 <tr> |
163 <td><a href="#Whitespace.__init__">Whitespace</a></td> |
163 <td><a href="#Whitespace.__init__">Whitespace</a></td> |
164 <td>Constructor</td> |
164 <td>Constructor</td> |
165 </tr><tr> |
165 </tr><tr> |
166 <td><a href="#Whitespace.equal">equal</a></td> |
166 <td><a href="#Whitespace.equal">equal</a></td> |
167 <td>Method to compare the indentation levels of two Whitespace objects for equality.</td> |
167 <td>Method to compare the indentation levels of two Whitespace objects for equality.</td> |
168 </tr><tr> |
168 </tr><tr> |
169 <td><a href="#Whitespace.indent_level">indent_level</a></td> |
169 <td><a href="#Whitespace.indent_level">indent_level</a></td> |
170 <td>Method to determine the indentation level.</td> |
170 <td>Method to determine the indentation level.</td> |
171 </tr><tr> |
171 </tr><tr> |
172 <td><a href="#Whitespace.less">less</a></td> |
172 <td><a href="#Whitespace.less">less</a></td> |
173 <td>Method to compare the indentation level against another Whitespace objects to be smaller.</td> |
173 <td>Method to compare the indentation level against another Whitespace objects to be smaller.</td> |
174 </tr><tr> |
174 </tr><tr> |
175 <td><a href="#Whitespace.longest_run_of_spaces">longest_run_of_spaces</a></td> |
175 <td><a href="#Whitespace.longest_run_of_spaces">longest_run_of_spaces</a></td> |
176 <td>Method to calculate the length of longest contiguous run of spaces.</td> |
176 <td>Method to calculate the length of longest contiguous run of spaces.</td> |
177 </tr><tr> |
177 </tr><tr> |
178 <td><a href="#Whitespace.not_equal_witness">not_equal_witness</a></td> |
178 <td><a href="#Whitespace.not_equal_witness">not_equal_witness</a></td> |
179 <td>Method to calculate a tuple of witnessing tab size.</td> |
179 <td>Method to calculate a tuple of witnessing tab size.</td> |
180 </tr><tr> |
180 </tr><tr> |
181 <td><a href="#Whitespace.not_less_witness">not_less_witness</a></td> |
181 <td><a href="#Whitespace.not_less_witness">not_less_witness</a></td> |
182 <td>Method to calculate a tuple of witnessing tab size.</td> |
182 <td>Method to calculate a tuple of witnessing tab size.</td> |
183 </tr> |
183 </tr> |
184 </table> |
184 </table> |
185 <a NAME="Whitespace.__init__" ID="Whitespace.__init__"></a> |
185 <a NAME="Whitespace.__init__" ID="Whitespace.__init__"></a> |
186 <h4>Whitespace (Constructor)</h4> |
186 <h4>Whitespace (Constructor)</h4> |
187 <b>Whitespace</b>(<i>ws</i>) |
187 <b>Whitespace</b>(<i>ws</i>) |
188 <p> |
188 <p> |
189 Constructor |
189 Constructor |
190 </p><dl> |
190 </p><dl> |
191 <dt><i>ws</i></dt> |
191 <dt><i>ws</i></dt> |
192 <dd> |
192 <dd> |
193 The string to be checked. |
193 The string to be checked. |
194 </dd> |
194 </dd> |
195 </dl><a NAME="Whitespace.equal" ID="Whitespace.equal"></a> |
195 </dl><a NAME="Whitespace.equal" ID="Whitespace.equal"></a> |
196 <h4>Whitespace.equal</h4> |
196 <h4>Whitespace.equal</h4> |
197 <b>equal</b>(<i>other</i>) |
197 <b>equal</b>(<i>other</i>) |
198 <p> |
198 <p> |
199 Method to compare the indentation levels of two Whitespace objects for equality. |
199 Method to compare the indentation levels of two Whitespace objects for equality. |
200 </p><dl> |
200 </p><dl> |
201 <dt><i>other</i></dt> |
201 <dt><i>other</i></dt> |
202 <dd> |
202 <dd> |
203 Whitespace object to compare against. |
203 Whitespace object to compare against. |
204 </dd> |
204 </dd> |
205 </dl><dl> |
205 </dl><dl> |
206 <dt>Returns:</dt> |
206 <dt>Returns:</dt> |
207 <dd> |
207 <dd> |
208 True, if we compare equal against the other Whitespace object. |
208 True, if we compare equal against the other Whitespace object. |
209 </dd> |
209 </dd> |
210 </dl><a NAME="Whitespace.indent_level" ID="Whitespace.indent_level"></a> |
210 </dl><a NAME="Whitespace.indent_level" ID="Whitespace.indent_level"></a> |
211 <h4>Whitespace.indent_level</h4> |
211 <h4>Whitespace.indent_level</h4> |
212 <b>indent_level</b>(<i>tabsize</i>) |
212 <b>indent_level</b>(<i>tabsize</i>) |
213 <p> |
213 <p> |
214 Method to determine the indentation level. |
214 Method to determine the indentation level. |
215 </p><dl> |
215 </p><dl> |
216 <dt><i>tabsize</i></dt> |
216 <dt><i>tabsize</i></dt> |
217 <dd> |
217 <dd> |
218 The length of a tab stop. (integer) |
218 The length of a tab stop. (integer) |
219 </dd> |
219 </dd> |
220 </dl><dl> |
220 </dl><dl> |
221 <dt>Returns:</dt> |
221 <dt>Returns:</dt> |
222 <dd> |
222 <dd> |
223 indentation level (integer) |
223 indentation level (integer) |
224 </dd> |
224 </dd> |
225 </dl><a NAME="Whitespace.less" ID="Whitespace.less"></a> |
225 </dl><a NAME="Whitespace.less" ID="Whitespace.less"></a> |
226 <h4>Whitespace.less</h4> |
226 <h4>Whitespace.less</h4> |
227 <b>less</b>(<i>other</i>) |
227 <b>less</b>(<i>other</i>) |
228 <p> |
228 <p> |
229 Method to compare the indentation level against another Whitespace objects to |
229 Method to compare the indentation level against another Whitespace objects to |
230 be smaller. |
230 be smaller. |
231 </p><dl> |
231 </p><dl> |
232 <dt><i>other</i></dt> |
232 <dt><i>other</i></dt> |
233 <dd> |
233 <dd> |
234 Whitespace object to compare against. |
234 Whitespace object to compare against. |
235 </dd> |
235 </dd> |
236 </dl><dl> |
236 </dl><dl> |
237 <dt>Returns:</dt> |
237 <dt>Returns:</dt> |
238 <dd> |
238 <dd> |
239 True, if we compare less against the other Whitespace object. |
239 True, if we compare less against the other Whitespace object. |
240 </dd> |
240 </dd> |
241 </dl><a NAME="Whitespace.longest_run_of_spaces" ID="Whitespace.longest_run_of_spaces"></a> |
241 </dl><a NAME="Whitespace.longest_run_of_spaces" ID="Whitespace.longest_run_of_spaces"></a> |
242 <h4>Whitespace.longest_run_of_spaces</h4> |
242 <h4>Whitespace.longest_run_of_spaces</h4> |
243 <b>longest_run_of_spaces</b>(<i></i>) |
243 <b>longest_run_of_spaces</b>(<i></i>) |
244 <p> |
244 <p> |
245 Method to calculate the length of longest contiguous run of spaces. |
245 Method to calculate the length of longest contiguous run of spaces. |
246 </p><dl> |
246 </p><dl> |
247 <dt>Returns:</dt> |
247 <dt>Returns:</dt> |
248 <dd> |
248 <dd> |
249 The length of longest contiguous run of spaces (whether or not |
249 The length of longest contiguous run of spaces (whether or not |
250 preceding a tab) |
250 preceding a tab) |
251 </dd> |
251 </dd> |
252 </dl><a NAME="Whitespace.not_equal_witness" ID="Whitespace.not_equal_witness"></a> |
252 </dl><a NAME="Whitespace.not_equal_witness" ID="Whitespace.not_equal_witness"></a> |
253 <h4>Whitespace.not_equal_witness</h4> |
253 <h4>Whitespace.not_equal_witness</h4> |
254 <b>not_equal_witness</b>(<i>other</i>) |
254 <b>not_equal_witness</b>(<i>other</i>) |
255 <p> |
255 <p> |
256 Method to calculate a tuple of witnessing tab size. |
256 Method to calculate a tuple of witnessing tab size. |
257 </p><p> |
257 </p><p> |
258 Intended to be used after not self.equal(other) is known, in which |
258 Intended to be used after not self.equal(other) is known, in which |
259 case it will return at least one witnessing tab size. |
259 case it will return at least one witnessing tab size. |
260 </p><dl> |
260 </p><dl> |
261 <dt><i>other</i></dt> |
261 <dt><i>other</i></dt> |
262 <dd> |
262 <dd> |
263 Whitespace object to calculate against. |
263 Whitespace object to calculate against. |
264 </dd> |
264 </dd> |
265 </dl><dl> |
265 </dl><dl> |
266 <dt>Returns:</dt> |
266 <dt>Returns:</dt> |
267 <dd> |
267 <dd> |
268 A list of tuples (ts, i1, i2) such that |
268 A list of tuples (ts, i1, i2) such that |
269 i1 == self.indent_level(ts) != other.indent_level(ts) == i2. |
269 i1 == self.indent_level(ts) != other.indent_level(ts) == i2. |
270 </dd> |
270 </dd> |
271 </dl><a NAME="Whitespace.not_less_witness" ID="Whitespace.not_less_witness"></a> |
271 </dl><a NAME="Whitespace.not_less_witness" ID="Whitespace.not_less_witness"></a> |
272 <h4>Whitespace.not_less_witness</h4> |
272 <h4>Whitespace.not_less_witness</h4> |
273 <b>not_less_witness</b>(<i>other</i>) |
273 <b>not_less_witness</b>(<i>other</i>) |
274 <p> |
274 <p> |
275 Method to calculate a tuple of witnessing tab size. |
275 Method to calculate a tuple of witnessing tab size. |
276 </p><p> |
276 </p><p> |
277 Intended to be used after not self.less(other is known, in which |
277 Intended to be used after not self.less(other is known, in which |
278 case it will return at least one witnessing tab size. |
278 case it will return at least one witnessing tab size. |
279 </p><dl> |
279 </p><dl> |
280 <dt><i>other</i></dt> |
280 <dt><i>other</i></dt> |
281 <dd> |
281 <dd> |
282 Whitespace object to calculate against. |
282 Whitespace object to calculate against. |
283 </dd> |
283 </dd> |
284 </dl><dl> |
284 </dl><dl> |
285 <dt>Returns:</dt> |
285 <dt>Returns:</dt> |
286 <dd> |
286 <dd> |
287 A list of tuples (ts, i1, i2) such that |
287 A list of tuples (ts, i1, i2) such that |
288 i1 == self.indent_level(ts) >= other.indent_level(ts) == i2. |
288 i1 == self.indent_level(ts) >= other.indent_level(ts) == i2. |
289 </dd> |
289 </dd> |
290 </dl> |
290 </dl> |
291 <div align="right"><a href="#top">Up</a></div> |
291 <div align="right"><a href="#top">Up</a></div> |
292 <hr /><hr /> |
292 <hr /><hr /> |
293 <a NAME="check" ID="check"></a> |
293 <a NAME="check" ID="check"></a> |
294 <h2>check</h2> |
294 <h2>check</h2> |
295 <b>check</b>(<i>file</i>) |
295 <b>check</b>(<i>file</i>) |
296 <p> |
296 <p> |
297 Private function to check one Python source file for whitespace related problems. |
297 Private function to check one Python source file for whitespace related problems. |
298 </p><dl> |
298 </p><dl> |
299 <dt><i>file</i></dt> |
299 <dt><i>file</i></dt> |
300 <dd> |
300 <dd> |
301 source filename (string) |
301 source filename (string) |
302 </dd> |
302 </dd> |
303 </dl><dl> |
303 </dl><dl> |
304 <dt>Returns:</dt> |
304 <dt>Returns:</dt> |
305 <dd> |
305 <dd> |
306 A tuple indicating status (True = an error was found), the |
306 A tuple indicating status (True = an error was found), the |
307 filename, the linenumber and the error message |
307 filename, the linenumber and the error message |
308 (boolean, string, string, string). The values are only |
308 (boolean, string, string, string). The values are only |
309 valid, if the status is True. |
309 valid, if the status is True. |
310 </dd> |
310 </dd> |
311 </dl> |
311 </dl> |
312 <div align="right"><a href="#top">Up</a></div> |
312 <div align="right"><a href="#top">Up</a></div> |
313 <hr /><hr /> |
313 <hr /><hr /> |
314 <a NAME="format_witnesses" ID="format_witnesses"></a> |
314 <a NAME="format_witnesses" ID="format_witnesses"></a> |
315 <h2>format_witnesses</h2> |
315 <h2>format_witnesses</h2> |
316 <b>format_witnesses</b>(<i>w</i>) |
316 <b>format_witnesses</b>(<i>w</i>) |
317 <p> |
317 <p> |
318 Function to format the witnesses as a readable string. |
318 Function to format the witnesses as a readable string. |
319 </p><dl> |
319 </p><dl> |
320 <dt><i>w</i></dt> |
320 <dt><i>w</i></dt> |
321 <dd> |
321 <dd> |
322 A list of witnesses |
322 A list of witnesses |
323 </dd> |
323 </dd> |
324 </dl><dl> |
324 </dl><dl> |
325 <dt>Returns:</dt> |
325 <dt>Returns:</dt> |
326 <dd> |
326 <dd> |
327 A formated string of the witnesses. |
327 A formated string of the witnesses. |
328 </dd> |
328 </dd> |
329 </dl> |
329 </dl> |
330 <div align="right"><a href="#top">Up</a></div> |
330 <div align="right"><a href="#top">Up</a></div> |
331 <hr /><hr /> |
331 <hr /><hr /> |
332 <a NAME="process_tokens" ID="process_tokens"></a> |
332 <a NAME="process_tokens" ID="process_tokens"></a> |
333 <h2>process_tokens</h2> |
333 <h2>process_tokens</h2> |
334 <b>process_tokens</b>(<i>tokens</i>) |
334 <b>process_tokens</b>(<i>tokens</i>) |
335 <p> |
335 <p> |
336 Function processing all tokens generated by a tokenizer run. |
336 Function processing all tokens generated by a tokenizer run. |
337 </p><dl> |
337 </p><dl> |
338 <dt><i>tokens</i></dt> |
338 <dt><i>tokens</i></dt> |
339 <dd> |
339 <dd> |
340 list of tokens |
340 list of tokens |
341 </dd> |
341 </dd> |
342 </dl> |
342 </dl> |
343 <div align="right"><a href="#top">Up</a></div> |
343 <div align="right"><a href="#top">Up</a></div> |
344 <hr /> |
344 <hr /> |
345 </body></html> |
345 </body></html> |