| Server IP : 172.67.201.108 / Your IP : 216.73.217.99 Web Server : Apache/2.4.68 (Amazon Linux) OpenSSL/3.5.5 System : Linux ip-172-31-69-123.ec2.internal 6.1.176-223.369.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 24 13:34:27 UTC 2026 x86_64 User : ec2-user ( 1000) PHP Version : 8.4.23 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : ON Directory : /lib/python3.9/site-packages/elementpath/__pycache__/ |
Upload File : |
a
���c�| � @ s d Z ddlZddlZddlmZ ddlmZmZ ddl m
Z
ddl m Z ddl
mZ ddlmZmZmZmZmZmZmZmZmZmZmZmZ e d �Z e� d
�Z G d d � d e �Z!e"e#d
�dd�Z$e"e"d�dd�Z%G dd� d�Z&G dd� de�Z'G dd� de(�Z)G dd� de)d�Z*dS ) zT
This module contains base classes and helper functions for defining Pratt parsers.
� N)�name)�Decimal�DecimalException)� takewhile)�ABCMeta)�MutableSequence) �Any�cast�ClassVar�Callable�Dict�List�Optional�Union�Tuple�Type�Pattern�Match) �(start)�(end)�(string)�(float)� (decimal)� (integer)�(name)� (invalid)� (unknown)z\sc @ s e Zd ZdZdS )�
ParseErrorz.An error when parsing source with TDOP parser.N)�__name__�
__module__� __qualname__�__doc__� r" r" �4/usr/lib/python3.9/site-packages/elementpath/tdop.pyr 4 s r )�s�returnc C s t dd� ttj| �D ��S )Nc s s | ]
}d V qdS )� Nr" )�.0�_r" r" r# � <genexpr>9 � z'count_leading_spaces.<locals>.<genexpr>)�sumr �str�isspace�r$ r" r" r# �count_leading_spaces8 s r/ )�symbolr% c s� dd� � | � � r| �� S | tv r0| dd� �� S tdd� | D ��rvd�dd� | D ��}|�� �dd ��d
d ��d d �S | �d
d �}|�� r�|�� �d d �S d �� fd d�| D ��}|�dd ��d
d ��d d �S )
zP
Converts a symbol string to an identifier (only alphanumeric and '_').
c S s, | � � s| dkr| S dtt| ���� S d S )Nr( z%s_)�isalnum� unicode_namer, �title)�cr" r" r# � get_id_name@ s z(symbol_to_classname.<locals>.get_id_namer& ���c s s | ]}|d v V qdS )z-_Nr" �r' r4 r" r" r# r) J r* z&symbol_to_classname.<locals>.<genexpr>� c s s | ]}t |�V qd S �N)r2 r7 r" r" r# r) K r* � �-r( c 3 s | ]}� |�V qd S r9 r" r7 �r5 r" r# r) R r* )r1 r3 �SPECIAL_SYMBOLS�all�join�replace� isidentifier)r0 �valuer" r< r# �symbol_to_classname< s rC c @ sp e Zd ZdZed�dd�Zdd� Zdd� Zd d
� Zd d � Z d
d� Z
ed�dd�Z ed�dd�Z ed�dd�Z
dS )�
MultiLabelaV
Helper class for defining multi-value label for tokens. Useful when a symbol has more roles.
A label of this type has equivalence with each of its values.
Example:
label = MultiLabel('function', 'operator')
label == 'symbol' # False
label == 'function' # True
label == 'operator' # True
��valuesc G s
|| _ d S r9 rE )�selfrF r" r" r# �__init__a s zMultiLabel.__init__c s t � fdd�| jD ��S )Nc 3 s | ]}� |kV qd S r9 r" �r' �v��otherr" r# r) e r* z$MultiLabel.__eq__.<locals>.<genexpr>��anyrF �rG rL r" rK r# �__eq__d s zMultiLabel.__eq__c s t � fdd�| jD ��S )Nc 3 s | ]}� |kV qd S r9 r" rI rK r" r# r) h r* z$MultiLabel.__ne__.<locals>.<genexpr>)r> rF rO r" rK r# �__ne__g s zMultiLabel.__ne__c C s d| j j| jf S )Nz%s%s)� __class__r rF �rG r" r" r# �__repr__j s zMultiLabel.__repr__c C s d� | j��dd�S )N�__r8 r( )r? rF r@ rS r" r" r# �__str__m s zMultiLabel.__str__c C s
t | j�S r9 )�hashrF rS r" r" r# �__hash__p s zMultiLabel.__hash__��itemc s t � fdd�| jD ��S )Nc 3 s | ]}� |v V qd S r9 r" rI rY r" r# r) t r* z*MultiLabel.__contains__.<locals>.<genexpr>rM )rG rZ r" rY r# � __contains__s s zMultiLabel.__contains__r. c s t � fdd�| jD ��S )Nc 3 s | ]}|� � �V qd S r9 )�
startswithrI r. r" r# r) w r* z(MultiLabel.startswith.<locals>.<genexpr>rM �rG r$ r" r. r# r\ v s zMultiLabel.startswithc s t � fdd�| jD ��S )Nc 3 s | ]}|� � �V qd S r9 )�endswithrI r. r" r# r) z r* z&MultiLabel.endswith.<locals>.<genexpr>rM r] r" r. r# r^ y s zMultiLabel.endswithN)r r r r! r, rH rP rQ rT rV rX r[ r\ r^ r" r" r" r# rD V s
rD c @ s� e Zd ZU dZdZeed< dZeed< dZe ed< dZ
e ed< dZ e e ed < d
Z
ee ed < d ed
< eed< e ed< eeef ed< dBd e e dd�dd�Zdd� Zdd� Zdd� Zdd� Zd d � Ze d �d d �Zd!d"� Zd#d$� Zeed �d%d&��Zee d �d'd(��Z ee d �d)d*��Z eeeef d �d+d,��Z d-d.� Z d/d0� Z d1d2� Z!d3d4� Z"dd5�d6d7�Z#dd5�d8d9�Z$dCd:d;�Z%dDd=d>�Z&dEd@dA�Z'dS )F�Tokena
Token base class for defining a parser based on Pratt's method.
Each token instance is a list-like object. The number of token's items is
the arity of the represented operator, where token's items are the operands.
Nullary operators are used for symbols, names and literals. Tokens with items
represent the other operators (unary, binary and so on).
Each token class has a *symbol*, a lbp (left binding power) value and a rbp
(right binding power) value, that are used in the sense described by the
Pratt's method. This implementation of Pratt tokens includes two extra
attributes, *pattern* and *label*, that can be used to simplify the parsing
of symbols in a concrete parser.
:param parser: The parser instance that creates the token instance.
:param value: The token value. If not provided defaults to token symbol.
:cvar symbol: the symbol of the token class.
:cvar lbp: Pratt's left binding power, defaults to 0.
:cvar rbp: Pratt's right binding power, defaults to 0.
:cvar pattern: the regex pattern used for the token class. Defaults to the escaped symbol. Can be customized to match more detailed conditions (eg. a function with its left round bracket), in order to simplify the related code.
:cvar label: defines the typology of the token class. Its value is used in representations of the token instance and can be used to restrict code choices without more complicated analysis. The label value can be set as needed by the parser implementation (eg. 'function', 'axis', 'constructor function' are used by the XPath parsers). In the base parser class defaults to 'symbol' with 'literal' and 'operator' as possible alternatives. If set by a tuple of values the token class label is transformed to a multi-value label, that means the token class can covers multiple roles (eg. as XPath function or axis). In those cases the definitive role is defined at parse time (nud and/or led methods) after the token instance creation.
r �lbp�rbpr: r0 �labelN�pattern)�_items�parserrB �_source�spanrd �Parserre rB rf rg )re rB r% c C sF g | _ || _|d ur|n| j| _|j| _|jd u r6dn|j�� | _d S )N)r r )rd re r0 rB �sourcerf �
next_matchrg )rG re rB r" r" r# rH � s
zToken.__init__c C s
| j | S r9 �rd �rG �ir" r" r# � __getitem__� s zToken.__getitem__c C s || j |< d S r9 rk �rG rm rZ r" r" r# � __setitem__� s zToken.__setitem__c C s | j |= d S r9 rk rl r" r" r# � __delitem__� s zToken.__delitem__c C s
t | j�S r9 )�lenrd rS r" r" r# �__len__� s z
Token.__len__c C s | j �||� d S r9 )rd �insertro r" r" r# rt � s z Token.insert�r% c C s6 | j tv r"d| j| j dd� f S d| j | jf S d S )Nz%r %sr& r6 )r0 r= rB rb rS r" r" r# rV � s
z
Token.__str__c C s6 | j | j }}||kr&d| jj|f S d| jj S d S )Nz %s(value=%r)z%s())r0 rB rR r )rG r0 rB r" r" r# rT � s zToken.__repr__c C s4 z| j |j ko| j|jkW S ty. Y dS 0 d S )NF)r0 rB �AttributeErrorrO r" r" r# rP � s z Token.__eq__c C s t | �S r9 �rr rS r" r" r# �arity� s z Token.arityc C s� | j dkrd| j S | j tv r(d| j S | j dkrh| s:dS t| �dkrP| d jS dd�d d
� | D �� S | svd| j S d | j d�d d
� | D ��f S d
S )z%Returns a tree representation string.r z(%s)z(%r)�(z()r& r r8 c s s | ] }|j V qd S r9 ��tree�r' rZ r" r" r# r) � r* z Token.tree.<locals>.<genexpr>z(%s %s)c s s | ] }|j V qd S r9 rz r| r" r" r# r) � r* N)r0 rB r= rr r{ r? rS r" r" r# r{ � s
z
Token.treec C s� | j }|dkr| jS |dkr&t| j�S |tv r8t| j�S t| �}|sH|S |dkr~d| jv rld| d j|f S d|| d jf S |dkr�d| d j|| d jf S d|d �d
d � | D ��f S d S )
z)Returns the source representation string.r r r& �postfixz%s %sr � z%s %s %sr8 c s s | ] }|j V qd S r9 )ri r| r" r" r# r) r* z Token.source.<locals>.<genexpr>N) r0 rB r, r= �reprrr rb ri r? )rG r0 Zlengthr" r" r# ri � s"
z Token.sourcec C sR | j d }| jd|� �d�d }|dkr6d|d fS ||| jd|� �d� fS )zCA tuple with the position of the token in terms of line and column.r N�
r& )rg rf �count�rindex)rG � token_index�liner" r" r# �position s
zToken.positionc C s | � � �dS )z Pratt's null denotation methodN�� wrong_syntaxrS r" r" r# �nud s z Token.nudc C s | � � �dS )z Pratt's left denotation methodNr� �rG �leftr" r" r# �led s z Token.ledc C s | j S )zEvaluation method�rB rS r" r" r# �evaluate s zToken.evaluatec g s� | s|r| j |v r�| V n�t| �dkrP|r4| j |v r:| V | d j|� E dH nL| d j|� E dH |rr| j |v rx| V | jdd� D ]}|j|� E dH q�dS )z3Returns a generator for iterating the token's tree.r& r N)r0 rr �iterrd )rG �symbols�tr" r" r# r� s z
Token.iter��messagec G s |r| j |vr| �|��d S r9 �r0 r� �rG r� r� r" r" r# �expected( s zToken.expectedc G s |r| j |v r| �|��d S r9 r� r� r" r" r# �
unexpected, s zToken.unexpectedc C s� |r t |�S | jtvr"t d| �S | jdkr:t d| j �S | jdkrRt d| j �S | jdkrjt d| j �S | jdkr�t d | j �S | jjjd
kr�t d �S t d �S d S )
Nz
unexpected %sr zinvalid literal %rr zunknown symbol %rr zunexpected name %rr zunexpected literal %rr zsource is emptyzunexpected end of source)r r0 r= rB re �token�rG r� r" r" r# r� 0 s
zToken.wrong_syntax� invalid typec C s t |�S r9 )� TypeErrorr� r" r" r# �
wrong_typeB s zToken.wrong_type�
invalid valuec C s t |�S r9 )�
ValueErrorr� r" r" r# � wrong_valueE s zToken.wrong_value)N)N)r� )r� )(r r r r! r` �int�__annotations__ra r0 r, rb rc r � __slots__r
r r rH rn rp rq rs rt rV rT rP �propertyrx r{ ri r� r� r� r� r� r� r� r� r� r� r" r" r" r# r_ } sL
!
r_ c s e Zd Z� fdd�Z� ZS )�
ParserMetac s t t| ��| |||�}tj|j j�� D ]&\}}t|t�r(|j|jkr(t d��q(t
|d�s`t |_ t
|d�svt
�d�|_t
|d�s�t
�d�|_d|vr�d |_d|vr�t� |_|D ] }t
|d�r�|j�|j� q�q�d |v�ri |_|D ]"}t
|d �r�|j�|j� �qq�|S )
Nz<Multiple parser class definitions per module are not allowed�token_base_class�literals_patternz8'[^']*'|"[^"]*"|(?:\d+|\.\d+)(?:\.\d*)?(?:[Ee][+-]?\d+)?� name_patternz
[A-Za-z0-9_]+� tokenizer�SYMBOLS� symbol_table)�superr� �__new__�sys�modulesr �__dict__�items�
isinstance� RuntimeError�hasattrr_ r� �re�compiler� r� r� �setr� �updater� )Zmcsr �bases� namespace�cls�krJ Z
base_class�rR r" r# r� K s6
�
zParserMeta.__new__)r r r r� �
__classcell__r" r" r� r# r� I s r� c @ s e Zd ZU dZeZeZdZe e
e d< i Z e
eeee f e d< ee d< ee d< ee d< e e e d< e
e d < e
e d
< d Zd d
� Zdd� Zeed�dd�Zeed�dd�Zeed�dd�ZdFeed�dd �Zed d � �Zd d � Z d!d"� Z dGd$d%�Z e d&d'� �Z e!ee d(�d)d*��Z"e!ed+�d,d-��Z#e!eeee d.�d/d0��Z$e!dHeed1�d2d3��Z%e!dIeed1�d4d5��Z&e!dJeed1�d6d7��Z'e!dKeed1�d8d9��Z(e!dLeed1�d:d;��Z)e!dMeed1�d<d=��Z*e!dNe+eee f ee,d>�d?d@��Z-e!dAdB� �Z.e.Z/e!eeee f e
dC�dDdE��Z0dS )Orh a�
Parser class for implementing a Top Down Operator Precedence parser.
:cvar SYMBOLS: the symbols of the definable tokens for the parser. In the base class it's an immutable set that contains the symbols for special tokens (literals, names and end-token). Has to be extended in a concrete parser adding all the symbols of the language.
:cvar symbol_table: a dictionary that stores the token classes defined for the language.
:type symbol_table: dict
:cvar token_base_class: the base class for creating language's token classes.
:type token_base_class: Token
:cvar tokenizer: the language tokenizer compiled regexp.
Nr� r� � _start_tokenr� �
next_tokenrj r� r� )ri �tokensrj r� r� r� c C sJ | j d u r| �� d| _td�| _d | _| jd | �| _| j | _| _ d S )Nr: r" r )
r� �buildri r� r� rj r� r� r� r� rS r" r" r# rH � s
zParser.__init__c C s$ | j |j u o"| j|jko"| j|jkS r9 )r� r� r� rO r" r" r# rP � s
�
�z
Parser.__eq__)ri r% c
C s� z�zt | j�|��| _W nH ty` } z0| jd | t|��}|�d�|���W Y d}~n
d}~0 0 || _ | �
� | � � }| j �
d� |W t d�| _d| _| j | _| _ S t d�| _d| _| j | _| _ 0 dS )z�
Parses a source code of the formal language. This is the main method that has to be
called for a parser's instance.
:param source: The source string.
:return: The root of the token's tree that parse the source.
r zinvalid source type, {}Nr r" )r� r� �finditerr� r� r� �typer� �formatri �advance�
expressionr� r� rj r� r� )rG ri �errr� Z
root_tokenr" r" r# �parse� s* &
� �
z Parser.parse)r� r% c
G s� | j jdks |r&| j j|vr&| j �� �| j | _zttt| j��| _W n( t yl | j
d | �| _ Y �q�Y q.0 | j� � \}}}}|dur�z| j
| | �| _ W nP t y� | j
�|�du r�| j
d | |�| _ | j �� �| j
d | |�| _ Y n0 �q�q.|du�r>|d dv �r,| �|�}| j
d | |�| _ �q�d|v �s@d |v �r�z t|�}W nJ t�y� } z0| j
d
| |�| _ | j jt|�d ��W Y d}~n d}~0 0 | j
d | |�| _ n�d
|v �r"z t|�}W nJ t�y } z0| j
d
| |�| _ | j jt|�d ��W Y d}~n d}~0 0 | j
d | |�| _ n| j
d | t|��| _ �q�q.|du�r`| j
d | |�| _ �q�q.|du�r�| j
d | |�| _ | j �� �q.t| j�� ��� r.d}t|| j�� ��q.| j S )a*
The Pratt's function for advancing to next token.
:param symbols: Optional arguments tuple. If not empty one of the provided symbols is expected. If the next token's symbol differs the parser raises a parse error.
:return: The next token instance.
r Nr r r z'"r �e�Er r� r �.r r z.unexpected matching %r: incompatible tokenizer)r� r0 r� r� r r �nextr� rj �
StopIterationr� �groups�KeyErrorr� �match�unescape�floatr� r, r r r� �group�stripr� ) rG r� �literalr0 r �unknownrB r� �msgr" r" r# r� � sh �
�
(
(
zParser.advance)� stop_symbolsr% c G s |s| j �d��n| j jdkr(| j �� �| j | _g }zttt| j��| _ W n& t
yp | j d | �| _ Y q�Y q40 | j � d�}|dur�|�
� }||vr�|�|� q�z| j | | �| _ W q�W q� ty� | j d | �| _ | j �� �Y q�0 q4|�| j � � � q4d�|�S )a�
Advances until one of the symbols is found or the end of source is reached,
returning the raw source string placed before. Useful for raw parsing of
comments and references enclosed between specific symbols.
:param stop_symbols: The symbols that have to be found for stopping advance.
:return: The source string chunk enclosed between the initial position and the first stop symbol.
z at least a stop symbol required!r r~ Nr r: )r� r� r0 r� r� r r r� r� rj r� r� r� r� �appendr� r? )rG r� Z source_chunkr0 r" r" r# �
advance_until� s0
zParser.advance_untilr )ra r% c C s@ | j }| �� |�� }|| j jk r<| j }| �� |�|�}q|S )aP
Pratt's function for parsing an expression. It calls token.nud() and then advances
until the right binding power is less the left binding power of the next
token, invoking the led() method on the following token.
:param rbp: right binding power for the expression.
:return: left token.
)r� r� r� r` r� )rG ra r� r� r" r" r# r� s zParser.expressionc C s | j jS )z:Property that returns the current line and column indexes.)r� r� rS r" r" r# r� * s zParser.positionc C s t | jd| jjd � �� � S )zu
Returns `True` if the parser is positioned at the start
of the source, ignoring the spaces.
r )�boolri r� rg r� rS r" r" r# �is_source_start/ s zParser.is_source_startc C sp | j jd }z | jd|� �d�d }W n* tyR t| jd|� �� � Y S 0 t| j||� �� � S dS )zx
Returns `True` if the parser is positioned at the start
of a source line, ignoring the spaces.
r Nr� r& )r� rg ri r� r� r� r� )rG r� Z
line_startr" r" r# �
is_line_start6 s zParser.is_line_startTc C s^ | j j\}}z8|r2|dkr2| j|d dv r2W dS |oB| j| dv W S tyX Y dS 0 dS )ay
Returns `True` if the source has an extra space (whitespace, tab or newline)
immediately before or after the current position of the parser.
:param before: if `True` considers also the extra spaces before the current token symbol.
:param after: if `True` considers also the extra spaces after the current token symbol.
r r& z
TFN)r� rg ri �
IndexError)rG ZbeforeZafter�start�endr" r" r# � is_spacedC s
zParser.is_spacedc C s | dd� � dd�� dd�S )Nr& r6 z\'�'z\"�")r@ )Zstring_literalr" r" r# r� U s zParser.unescaperu c
K s� zFzd|v rt d| ��W n ty8 |j| }}Y n 0 | j| }W n� t�y, || jvrx|dkrxtd|| f ��||d< |�dd�}t|t �r�t
|� }|d< d� t |�t
|��� �dd��}|�d | jf�}|�| j|d
d �� ttt t|||��}|| j|< t�|� ttj| j ||� Y nh0 |� � D ]\\}}|d k�r\||j k�r\||_ n4|d
k�rz||j k�rz||_ nt |��r6t|||� �q6|S )a
Register/update a token class in the symbol table.
:param symbol: The identifier symbol for a new class or an existent token class.
:param kwargs: Optional attributes/methods for the token class.
:return: A token class.
r8 z&%r: a symbol can't contain whitespacesr z$%r is not a symbol of the parser %r.r0 rb z_{}{}r: r� N)r r Z
__return__r` ra ) r� r� r0 r� r� r� � NameError�getr� �tuplerD r� rC r, r3 r@ r� r� r r r r_ r r �register�setattrr� r� r� r` ra �callable) r� r0 �kwargs� token_classrb Ztoken_class_nameZtoken_class_bases�keyrB r" r" r# r� Y sJ
��
zParser.register)r0 c C s | j |�� = dS )z/Unregister a token class from the symbol table.N)r� r� )r� r0 r" r" r# �
unregister� s zParser.unregister)r0 �
new_symbolr% c K s\ | j | }| j|fi |��}|j�� D ]0\}}||v s&|dv s&|�d�rJq&t|||� q&|S )z*Duplicate a token class with a new symbol.)r0 rc r( )r� r� r� r� r\ r� )r� r0 r� r� r� Znew_token_classr� rB r" r" r# � duplicate� s
zParser.duplicate)r0 �bpc C s$ dd� }dd� }| j |d|||d�S )z:Register a token for a symbol that represents a *literal*.c S s | S r9 r" rS r" r" r# r� � s zParser.literal.<locals>.nudc _ s | j S r9 r� )rG Z_argsZ_kwargsr" r" r# r� � s z Parser.literal.<locals>.evaluater� )rb r` r� r� �r� )r� r0 r� r� r� r" r" r# r� � s zParser.literalc C s dd� }| j |d||d�S )zCRegister a token for a symbol that represents a *nullary* operator.c S s | S r9 r" rS r" r" r# r� � s zParser.nullary.<locals>.nud�operator)rb r` r� r� �r� r0 r� r� r" r" r# �nullary� s zParser.nullaryc s � fdd�}| j |d� � |d�S )zHRegister a token for a symbol that represents a *prefix* unary operator.c s | j j� d�f| d d �< | S �N�ra �re r� rS �r� r" r# r� � s zParser.prefix.<locals>.nudzprefix operator)rb r` ra r� r� r� r" r� r# �prefix� s z
Parser.prefixc C s dd� }| j |d|||d�S )zIRegister a token for a symbol that represents a *postfix* unary operator.c S s |f| d d �< | S r9 r" r� r" r" r# r� � s zParser.postfix.<locals>.ledzpostfix operator�rb r` ra r� r� �r� r0 r� r� r" r" r# r} � s zParser.postfixc s � fdd�}| j |d� � |d�S )zIRegister a token for a symbol that represents an *infix* binary operator.c s || j j� d�f| d d �< | S r� r� r� r� r" r# r� � s zParser.infix.<locals>.ledr� r� r� r� r" r� r# �infix� s z Parser.infixc s$ � fdd�}| j |d� � d |d�S )zJRegister a token for a symbol that represents an *infixr* binary operator.c s" || j j� d d�f| d d �< | S )Nr& r� r� r� r� r" r# r� � s zParser.infixr.<locals>.ledr� r& r� r� r� r" r� r# �infixr� s z
Parser.infixr)r0 r� r% c s" | j |d||d�� � fdd�}|S )z�
Register a token for a symbol that represents a custom operator or redefine
a method for an existing token.
r� )rb r` ra c sB | j �d�d }tt� |��s2td|�d� � ���t� || � | S )Nr( r zThe attribute z is not a callable of )r � partitionr� �getattrr� r� )�funcZ method_name�r� r" r# �bind� s
zParser.method.<locals>.bindr� )r� r0 r� r r" r r# �method� s z
Parser.methodc s\ d� j vr� �d� � j�� j �� �sJ� fdd�� jD �}td� |f ��� �� j �� _dS )z�
Builds the parser class. Checks if all declared symbols are defined
and builds a the regex tokenizer using the symbol related patterns.
r c s g | ]}|� j vr|�qS r" )r� )r' r$ �r� r" r# �
<listcomp>� r* z Parser.build.<locals>.<listcomp>z*The parser %r has unregistered symbols: %rN)r� r� r� �issubset�keysr� �create_tokenizerr� )r� Z unregisteredr" r r# r� � s
z Parser.build)r� r% c
C s8 g }g }g }t � }|�� D ]|\}}|tv r.qq|jdurF|�|j� q| j�|�durh|�t� |�� qt
|�dkr�|�t� |�� q|�t� |�� qg }|r�|�d� t |dd� d��� |r�|�d�
d� |��� |r�|�d �
d� t |d
d� d���� |�r|�d� |�� d �
| jjd� |�| jj�} t�| �S ) a�
Returns a regex based tokenizer built from a symbol table of token classes.
The returned tokenizer skips extra spaces between symbols.
A regular expression is created from the symbol table of the parser using a template.
The symbols are inserted in the template putting the longer symbols first. Symbols and
their patterns can't contain spaces.
:param symbol_table: a dictionary containing the token classes of the formal language.
Nr& �|c S s
t | � S r9 rw ��xr" r" r# �<lambda> r* z)Parser.create_tokenizer.<locals>.<lambda>)r� z[{}]r: z\b(?:{})\b(?![\-\.])c S s
t | � S r9 rw r r" r" r# r r* z({})|({})|({})|(\S)|\s+)r� r� r= rc �addr� r� r� r� �escaperr r? �sortedr� r� r� )
r� r� Zcharacter_patternsZstring_patternsZ
name_patternsZcustom_patternsr0 r� Zsymbols_patternsZtokenizer_patternr" r" r# r � s>
��zParser.create_tokenizer)r )TT)r )r )r )r )r )r )r )1r r r r! r= r� r_ r� r� r r r� r� r
r r, r r r� rH rP r� r� r� r� r� r� r� r� r� r� � staticmethodr� � classmethodr� r� r� r� r� r� r} r� r� r r r r� Zbuild_tokenizerr r" r" r" r# rh m sd
A(
:
"
rh )� metaclass)+r! r� r� � unicodedatar r2 Zdecimalr r � itertoolsr �abcr �collections.abcr �typingr r r
r r r
r r r r r r � frozensetr= r� Z
SPACE_PATTERN� SyntaxErrorr r, r� r/ rC rD r_ r� r� rh r" r" r" r# �<module>
s$ 8
' M$