Current File : //proc/self/root/opt/alt/python35/lib64/python3.5/urllib/__pycache__/robotparser.cpython-35.pyc
�q�\4�@sddZddlZddlZdgZGdd�d�ZGdd�d�ZGdd�d�ZdS) a% robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
�N�RobotFileParserc@s�eZdZdZddd�Zdd�Zdd�Zd d
�Zdd�Zd
d�Z dd�Z
dd�Zdd�ZdS)rzs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
�cCs>g|_d|_d|_d|_|j|�d|_dS)NFr)�entries�
default_entry�disallow_all� allow_all�set_url�last_checked)�self�url�r�7/opt/alt/python35/lib64/python3.5/urllib/robotparser.py�__init__s
zRobotFileParser.__init__cCs|jS)z�Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r )r
rrr
�mtimeszRobotFileParser.mtimecCsddl}|j�|_dS)zYSets the time the robots.txt file was last fetched to the
current time.
rN)�timer )r
rrrr
�modified(szRobotFileParser.modifiedcCs5||_tjj|�dd�\|_|_dS)z,Sets the URL referring to a robots.txt file.��N)r�urllib�parse�urlparse�host�path)r
rrrr
r0s zRobotFileParser.set_urlcCs�ytjj|j�}Wnmtjjk
r�}zG|jdkrOd|_n'|jdkrv|jdkrvd|_WYdd}~Xn)X|j �}|j
|jd�j��dS) z4Reads the robots.txt URL and feeds it to the parser.��Ti�i�Nzutf-8)rr)
rZrequestZurlopenr�errorZ HTTPError�coderr�readr�decode�
splitlines)r
�f�err�rawrrr
r5szRobotFileParser.readcCs>d|jkr*|jdkr:||_n|jj|�dS)N�*)�
useragentsrr�append)r
�entryrrr
�
_add_entryBszRobotFileParser._add_entrycCsd}t�}|j�x�|D]�}|sr|dkrJt�}d}n(|dkrr|j|�t�}d}|jd�}|dkr�|d|�}|j�}|s�q |jdd�}t|�dkr |dj�j�|d<tj j
|dj��|d<|ddkr_|dkrB|j|�t�}|jj|d�d}q |ddkr�|dkr�|j
jt|dd ��d}q |dd
kr |dkr |j
jt|dd��d}q W|dkr�|j|�dS)z�Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
rr��#N�:z
user-agentZdisallowFZallowT)�Entryrr'�find�strip�split�len�lowerrr�unquoter$r%� rulelines�RuleLine)r
�lines�stater&�line�irrr
rKsL
zRobotFileParser.parsecCs�|jr
dS|jrdS|js'dStjjtjj|��}tjjdd|j|j |j
|jf�}tjj|�}|s�d}x-|j
D]"}|j|�r�|j|�Sq�W|jr�|jj|�SdS)z=using the parsed robots.txt decide if useragent can fetch urlFTr�/)rrr rrrr1�
urlunparserZparamsZqueryZfragment�quoter�
applies_to� allowancer)r
� useragentrZ
parsed_urlr&rrr
� can_fetchs$ zRobotFileParser.can_fetchcCsdjdd�|jD��S)NrcSs g|]}t|�d�qS)�
)�str)�.0r&rrr
�
<listcomp>�s z+RobotFileParser.__str__.<locals>.<listcomp>)�joinr)r
rrr
�__str__�szRobotFileParser.__str__N)
�__name__�
__module__�__qualname__�__doc__rrrrrr'rr>rDrrrr
rs
4c@s:eZdZdZdd�Zdd�Zdd�ZdS) r3zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.cCsY|dkr|rd}tjjtjj|��}tjj|�|_||_dS)NrT)rrr9rr:rr<)r
rr<rrr
r�s
zRuleLine.__init__cCs|jdkp|j|j�S)Nr#)r�
startswith)r
�filenamerrr
r;�szRuleLine.applies_tocCs|jrdndd|jS)NZAllowZDisallowz: )r<r)r
rrr
rD�szRuleLine.__str__N)rErFrGrHrr;rDrrrr
r3�sr3c@sFeZdZdZdd�Zdd�Zdd�Zdd �Zd
S)r+z?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_dS)N)r$r2)r
rrr
r�s zEntry.__init__cCsjg}x'|jD]}|jd|dg�qWx*|jD]}|jt|�dg�q:Wdj|�S)NzUser-agent: r?r)r$�extendr2r@rC)r
Zret�agentr6rrr
rD�sz
Entry.__str__cCs]|jd�dj�}x=|jD]2}|dkr9dS|j�}||kr#dSq#WdS)z2check if this entry applies to the specified agentr8rr#TF)r.r0r$)r
r=rLrrr
r;�szEntry.applies_tocCs.x'|jD]}|j|�r
|jSq
WdS)zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)r2r;r<)r
rJr6rrr
r<�szEntry.allowanceN)rErFrGrHrrDr;r<rrrr
r+�s
r+)rHZurllib.parserZurllib.request�__all__rr3r+rrrr
�<module>s
�