VK8 G d Z dZdZddlmZ ddlmZmZ ddl Z ddl
Z
ddl m Z ddl
ZddlZddlZddlT dd lmZ ej d
ej Z ej d ej ZddlZej g d z Z[ G d
d e j dd Zd Zd Zd Z dZ dZ e ede z z e e z Z dZ dZ!dZ"dZ#dZ$ ee!e"e#e$ Z%dZ& ed d e e& z Z'd e&z Z( ee'e( Z) ed e)d z Z* ee*e)e% Z+d! Z,e
j- d" Z. e e, Z/d#Z0d$Z1d%Z2d&Z3 ee/d'z e/d(z Z4 ee/d)z e/d*z Z5 e e6ej7 e8ed+, Z9 ed-e9 Z: ee+e:e5e Z;e e;z Z< ee/d.z ed/d z e/d0z ed1d z Z= ed2e e4 Z>e ee>e+e:e=e z Z?i Z@ e, D ]"ZAe0e@eAd/z < e1e@eAd1z < e2e@eAd'z < e3e@eAd(z < #[A eB ZC eB ZD e, D ]FZEeEd1z eEd/z fD ]ZFeCG eF eEd(z eEd'z fD ]ZFeDG eF G[E[Fd3ZH G d4 d5eI ZJ G d6 d7eI ZK G d8 d9 ZLd: ZMd; ZNd< ZOd= Zd> ZPd? ZQd@ ZRdA ZSdB ZTeUdCk r eS dS dS )Dao Tokenization help for Python programs.
tokenize(readline) is a generator that breaks a stream of bytes into
Python tokens. It decodes the bytes according to PEP-0263 for
determining source file encoding.
It accepts a readline-like method which is called repeatedly to get the
next line of input (or b"" for EOF). It generates 5-tuples with these
members:
the token type (see token.py)
the token (a string)
the starting (row, column) indices of the token (a 2-tuple of ints)
the ending (row, column) indices of the token (a 2-tuple of ints)
the original line (string)
It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators. Additionally, all token lists start with an ENCODING token
which tells you which encoding was used to decode the bytes stream.
zKa-Ping Yee