| Current File : //usr/lib64/python3.6/site-packages/borg/__pycache__/remote.cpython-36.pyc |
3
y��b�� � @ s� d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlZd dlZd dl Z d dl
Z
d dl Z d dl Z d dl
Z
d dlZd dlZd dlmZmZ ddlmZ ddlmZ ddlT ddlmZmZ ddlmZ dd lm Z dd
lm Z dd lm Z dd lm Z dd
lm Z ddlm!Z! ddlm"Z" ddlm#Z# ddl$m%Z%m&Z& ddlm'Z' ddl(m)Z) ddl*m+Z+m,Z, ddl-m.Z. e%e/�Z$dZ0e+e�Z1dP\Z2Z3Z4Z5dZ6d Z7d d � Z8G d d � d e�Z9G d!d"� d"e9�Z:G d#d$� d$e�Z;G d%d&� d&e�Z<G d'd(� d(e�Z=G d)d*� d*e�Z>dQdRf f f dSdTdUdVdWf f dXdYf dZd;�Z?d<d=� Z@G d>d?� d?�ZAG d@dA� dA�ZBdBdC� ZCG dDdE� dE�ZDdFdG� ZEG dHdI� dI�ZFG dJdK� dKeF�ZGdLddddLdM�dNdO�ZHdS )[� N)�Popen�PIPE� )� __version__)�LZ4)�*)�Error�IntegrityError)�
bin_to_hex)� get_base_dir)�get_limited_unpacker)�hostname_is_unique)�replace_placeholders)�sysinfo)�format_file_size)� safe_unlink)�prepare_subprocess_env)�
create_logger�
setup_logging)�msgpack)�
Repository)�
parse_version�format_version)�xxh64� � i� m� a� r�d g�������?c C sN t |� }}x<|rHtj| |�}||8 }|s,P ||d� }tj|d � qW |S )z;os.write wrapper so we do not lose data for partial writes.Ng��&� .>)�len�os�write�time�sleep)�fd�dataZamountZ remaining�count� r( � /usr/lib64/python3.6/remote.py�os_write1 s r* c @ s e Zd ZdZdS )�ConnectionClosedz Connection closed by remote hostN)�__name__�
__module__� __qualname__�__doc__r( r( r( r) r+ D s r+ c @ s e Zd ZdZdS )�ConnectionClosedWithHintz$Connection closed by remote host. {}N)r, r- r. r/ r( r( r( r) r0 H s r0 c @ s e Zd ZdZdS )�PathNotAllowedz Repository path not allowed: {}N)r, r- r. r/ r( r( r( r) r1 L s r1 c @ s e Zd ZdZdS )�InvalidRPCMethodzRPC method {} is not validN)r, r- r. r/ r( r( r( r) r2 P s r2 c @ s e Zd ZdZdS )�!UnexpectedRPCDataFormatFromClientz4Borg {}: Got unexpected RPC data format from client.N)r, r- r. r/ r( r( r( r) r3 T s r3 c s e Zd ZdZ� fdd�Z� ZS )�!UnexpectedRPCDataFormatFromServerz.Got unexpected RPC data format from server:
{}c sf y|j � d d� }W n@ tk
rT |d d� }dd� |D �}tjdj|�d�}Y nX t� j|� d S )N� c S s g | ] }d | �qS )z%02Xr( )�.0Zbyter( r( r) �
<listcomp>` s z>UnexpectedRPCDataFormatFromServer.__init__.<locals>.<listcomp>� � � �0 )�decode�UnicodeDecodeError�textwrapZfill�join�super�__init__)�selfr& )� __class__r( r) rA [ s z*UnexpectedRPCDataFormatFromServer.__init__)r, r- r. r/ rA �
__classcell__r( r( )rC r) r4 X s r4 �repair�
save_space�limit�marker�idr& �keydata� client_data�path�create� lock_wait�lock� exclusive� append_only�next_unreserved� start_nonce)�check�commit�rollback�destroy�__len__�list�put�get�delete�save_key�load_key�
break_lock� negotiate�open�get_free_nonce�commit_nonce_reservationc s � fdd�� D �S )Nc s i | ]}� | |j � �qS r( )r< )r6 �k)�dr( r) �
<dictcomp>� s z decode_keys.<locals>.<dictcomp>r( )re r( )re r) � decode_keys� s rg c @ sR e Zd Zd&Zdd� Zdd� Zdd� Zdd� Zdd � Zd d � Z d'd"d#�Z
d$d%� Z d S )(�RepositoryServerrX rT rU r\ rW r[ rY �scanr` ra rZ rV r] r^ r_ rb rc �inject_exceptionc C s, d | _ || _|| _|| _|| _td�| _d S )Nz1.0.8)�
repository�restrict_to_paths�restrict_to_repositoriesrQ �
storage_quotar �client_version)rB rl rm rQ rn r( r( r) rA � s zRepositoryServer.__init__c sn y� fdd�t t| �D �S tk
rh |dkrPt� �dkrPt| dd� }n� � fdd�t |�D �S X dS )z5Translate from positional protocol to named protocol.c s i | ]\}}� | |�qS r( r( )r6 �pos�name)�argvr( r) rf � s z8RepositoryServer.positional_to_named.<locals>.<dictcomp>ra � Nc s i | ]\}}� | |�qS r( r( )r6 rp rq )rr r( r) rf � s )� enumerate� compatMap�
IndexErrorr )rB �methodrr �mappingr( )rr r) �positional_to_named� s z$RepositoryServer.positional_to_namedc s$ t tj|�j�� � �fdd��D �S )zWRemove unknown named parameters from call, because client did (implicitly) say it's ok.c s i | ]}|� kr�| |�qS r( r( )r6 rq )�known�kwargsr( r) rf � s z0RepositoryServer.filter_args.<locals>.<dictcomp>)�set�inspect� signature�
parameters)rB �fr{ r( )rz r{ r) � filter_args� s z RepositoryServer.filter_argsc " C s t jj� }t jj� }t jj� }tj|tj�}tj|tj|tjB � tj|tj�}tj|tj|tj @ � tj|tj�}tj|tj|tj @ � t d�}�xvt
j
|gg g d�\}}}|�r�tj |t �} | s�| j
d k r�| j
j� nt|djt�j� � d S |j| � �x�|D �]�}
t|
t��r@d} |
t } |
t j� }
t|
t �}n^t|
t��r�t |
�dk�r�d} |
\}} }
}|
j� }
| j |
|�}n | j
d k �r�| j
j� t t��yb|
| j k�r�t |
��yt!| |
�}W n" t"k
�r� t!| j
|
�}Y nX | j#||�}|f |�}W �n� t$k
�r� } �z�| �rt%j&|j'|�}t%j(t j)� � }d}t|t*��rX|j+� g}|j%}t|t,j-t,j.t/f��rpnt0j1dj2|�� y2t3j4t| d|j'j5d |j6d
|d |d |d
t7� i�}W nN t8k
�r t3j4t| d|j'j5d dd� |j6D �d
|d |d |d
t7� i�}Y nX t||� n�t|t,j-t,j.t/f��r&ndt|t*��rP|j%�r@t0j9nt0j:}|j+� }nt0j9}d|j'j5 }dt%j;� t7� f }t0j<|� t0j=||� d}t|t3j4d| |j'j5|f�� W Y d d }~X n:X | �r�t|t3j4t| t>|i�� nt|t3j4d| d |f�� �qW |r�| j
j� d S q�W d S )NZserver�
z<Borg {}: Got connection close before repository was opened.
Trs F�
s exception_classs exception_argss exception_fulls exception_shorts exception_traces sysinfoc S s$ g | ] }t |tttf�r |nd �qS )N)�
isinstance�str�bytes�int)r6 �xr( r( r) r7 s z*RepositoryServer.serve.<locals>.<listcomp>z%s Exception in RPC callz%s
%sz3Remote Exception (see remote log for the traceback)r )?�sys�stdin�fileno�stdout�fcntl�F_GETFL�F_SETFLr! �
O_NONBLOCKr �select�read�BUFSIZErk �closer* �formatr �encode�feedr� �dict�MSGID�MSGr< rg �ARGS�tupler ry r3 � rpc_methodsr2 �getattr�AttributeErrorr� �
BaseException� traceback�format_exception_onlyrC �format_exception�exc_infor � get_messager � DoesNotExist�
AlreadyExistsr1 �logging�debugr? r �packbr, �argsr � TypeError�ERROR�DEBUG�
format_exc�error�log�RESULT)rB �stdin_fd� stdout_fd� stderr_fdZfl�unpacker�r�wZesr& �unpacked�
dictFormat�msgidrw r� �_r� �res�eZex_shortZex_fullZex_trace�msgZ tb_log_level�tb�excr( r( r) �serve� s�
.
zRepositoryServer.servec C sZ |t kr t S t|t�rL|d | _tjtjd�j�}tdd|d� t j
d� nt | _dt iS )Ns client_version� T)Zis_serve�json�levelz2Initialized logging system for JSON-based protocol�server_version) �RPC_PROTOCOL_VERSIONr� r� ro r� Z getLevelName� getLoggerr� r �loggerr� � BORG_VERSION)rB rK r� r( r( r) r` 6 s
zRepositoryServer.negotiatec C s� t |t�rtj|�}|jd�r8tjjt� |dd � �}nr|jd�r\tjjt� |dd � �}nN|jd�r|tjj|dd � �}n.|jd�r�tjj|�}n|jd�r�|dd � }tjj |�S ) Nz/~/r: z~/r z/~r �~z/./)
r� r� r! �fsdecode�
startswithrL r? r �
expanduser�realpath)rB rL r( r( r) �
_resolve_pathG s
z RepositoryServer._resolve_pathFNTc
C s� t jd|� | j|�}t jd|� tjj|d�}| jrpx8| jD ]&} tjjtjj| �d�}
|j|
�r>P q>W t |��| j
r�x6| j
D ]$} tjjtjj| �d�} | |kr~P q~W t |��| r�| j p�|}t |||||| j
||d�| _| jj� | jjS )Nz Resolving repository path %rz Resolved repository path to %rr� )rN rO rQ rn rP �make_parent_dirs)r� r� r� r! rL r? rl r� r� r1 rm rQ r rn rk � __enter__rI )
rB rL rM rN rO rP rQ r� Z
path_with_sepZrestrict_to_pathZrestrict_to_path_with_sepZrestrict_to_repositoryZ restrict_to_repository_with_sepr( r( r) ra W s0
zRepositoryServer.openc C s� |j � }d}d}|dkr$tj|��n�|dkr8tj|��np|dkrLtj|��n\|dkr^t|��nJ|dkrptd��n8|d kr�tj||��n"|d
kr�t|��n|d kr�d d d S )
Nz test stringz test string2r� r� � CheckNeededr r1 Zfoo�ObjectNotFoundr2 Zdivider ) r< r r� r� r� r r1 r� r2 )rB Zkind�s1�s2r( r( r) rj z s&
z!RepositoryServer.inject_exception)rX rT rU r\ rW r[ rY ri r` ra rZ rV r] r^ r_ rb rc rj )FNTNFF) r, r- r. r� rA ry r� r� r` r� ra rj r( r( r( r) rh � s6 q
"rh c @ s e Zd Zdd� Zdd� ZdS )�SleepingBandwidthLimiterc C s0 |r&t |t �| _tj� | _| j| _nd | _d S )N)r� �RATELIMIT_PERIOD� ratelimitr# � monotonic�ratelimit_last�ratelimit_quota)rB rG r( r( r) rA � s
z!SleepingBandwidthLimiter.__init__c C s� | j r�tj� }| jt |krN| j| j 7 _| jd| j krHd| j | _|| _| jdkr�| jt | }tj|� | j| j 7 _tj� | _t|�| jkr�|d | j� }tj ||�}| j r�| j|8 _|S )Nr r )
r� r# r� r� r� r� r$ r r! r" )rB r% �to_sendZnowZtosleep�writtenr( r( r) r" � s$
z SleepingBandwidthLimiter.writeN)r, r- r. rA r" r( r( r( r) r� � s r� c s � �fdd�}|S )a~ Check version requirements and use self.call to do the remote method call.
<since> specifies the version in which borg introduced this method,
calling this method when connected to an older version will fail without transmiting
anything to the server.
Further kwargs can be used to encode version specific restrictions.
If a previous hardcoded behaviour is parameterized in a version, this allows calls that
use the previously hardcoded behaviour to pass through and generates an error if another
behaviour is requested by the client.
e.g. when 'append_only' was introduced in 1.0.7 the previous behaviour was what now is append_only=False.
Thus @api(..., append_only={'since': parse_version('1.0.7'), 'previously': False}) allows calls
with append_only=False for all version but rejects calls using append_only=True on versions older than 1.0.7.
c s t j� �� ��fdd��}|S )Nc
s t j� �}|j| f|�|�}i }i }xh|jj� D ]Z\}}|dkrBq0||jkrt|dkrd|j| ||< q�|j| ||< q0|j|jk r0|j||< q0W | j�k r�| j � j
t ����xb�j� D ]V\}} | d | jkr�q�d| kr�|| | d kr�q�| j dj � j
||| �t | d ���q�W | j
� j
|f|�S )NrB �wait�since�
previouslyz
{0} {1}={2!s})r} r~ Zbindr �itemsZ arguments�default�emptyr� �RPCServerOutdatedr, r r� �call)
rB r� r{ ZsigZ
bound_argsZnamedZextrarq ZparamZ restriction)r� �kwargs_decoratorr� r( r) �do_rpc� s.
z&api.<locals>.decorator.<locals>.do_rpc)� functools�wraps)r� r� )r� r� )r� r) � decorator� s zapi.<locals>.decoratorr( )r� r� r� r( )r� r� r) �api� s !r� c @ s0 e Zd Zg ZG dd� de�ZG dd� de�ZdZdMdd �Z d
d � Z
d d
� Z dd� Z dd� Z
edd� �Zdd� Zdd� Zdd� Zdd� ZdNd d �Zeed �ed �dd �ed!�dd �d"�dOd#d$��Zeed �d%�dPd&d'��Zeed �d%�dQd(d)��Zeed �d%�d*d+� �Zeed �d%�d,d-� �Zeed �d%�d.d/� �Z eed �d%�dRd0d1��Z eed2�d%�dSd3d4��Z d5d6� Z dTd7d8�Z eed �d%�dUd9d:��Z!eed �d%�dVd;d<��Z"eed �d%�d=d>� �Z#eed �d%�d?d@� �Z$eed �d%�dAdB� �Z%eed �d%�dCdD� �Z&eed �d%�dEdF� �Z'dGdH� Z(dWdIdJ�Z)dKdL� Z*dS )X�RemoteRepositoryc @ sL e Zd Zdd� Zdd� Zedd� �Zedd� �Zed d
� �Zed d � �Z d
S )zRemoteRepository.RPCErrorc C s
|| _ d S )N)r� )rB r� r( r( r) rA � s z"RemoteRepository.RPCError.__init__c C s( d| j kr dj| j d �j� S | jS d S )Ns exception_short�
)r� r? r< �exception_class)rB r( r( r) r� � s
z%RemoteRepository.RPCError.get_messagec C s | j jdd�S )Ns exception_traceT)r� r[ )rB r( r( r) r� � s z#RemoteRepository.RPCError.tracebackc C s | j d j� S )Ns exception_class)r� r< )rB r( r( r) r� � s z)RemoteRepository.RPCError.exception_classc C s. d| j kr dj| j d �j� S | j� d S d S )Ns exception_fullr� z4
Remote Exception (see remote log for the traceback))r� r? r< r� )rB r( r( r) �exception_full� s
z(RemoteRepository.RPCError.exception_fullc C s d| j kr| j d j� S dS d S )Ns sysinfor� )r� r< )rB r( r( r) r s
z!RemoteRepository.RPCError.sysinfoN)
r, r- r. rA r� �propertyr� r� r� r r( r( r( r) �RPCError� s r� c @ s( e Zd ZdZedd� �Zedd� �ZdS )z"RemoteRepository.RPCServerOutdatedz2Borg server is too old for {}. Required version {}c C s
| j d S )Nr )r� )rB r( r( r) rw
s z)RemoteRepository.RPCServerOutdated.methodc C s
| j d S )Nr )r� )rB r( r( r) �required_version s z3RemoteRepository.RPCServerOutdated.required_versionN)r, r- r. r/ r� rw r� r( r( r( r) r� s r� FNTc s� | �_ �_g �_d�_d�_d�_d�_d�_i �_t � �_
i �_ i �_ d �_
t|rf|jrf|jd nd��_td��_td��_d �_|�_|jdk} t| d�}
�j|| �} | sj|�| } tj d| � t | dt t t |
d ��_�jj j � �_!�jj"j � �_#�jj$j � �_%t&j&�j!t&j't&j&�j!t&j(�t)j*B � t&j&�j#t&j't&j&�j#t&j(�t)j*B � t&j&�j%t&j't&j&�j%t&j(�t)j*B � �j#�j%g�_+�j!�j#�j%g�_,�yy�j-d
d d t.ii�} W n t/k
�r� t0d
�d �Y nX | t1k�r�d�_2n4t3| t4��rd| k�rd�_2| d �_n t5d| ��� ������fdd�}
�j2�rF|
� nhy
|
� W n\ �j6k
�r� } z<|j7dk�rt� d}t8j$j9|� td��_d t:d< |
� W Y d d }~X nX W n t5k
�r� �j;� � Y nX d S ) Nr � i Zclientz1.0.8Z
__testsuite__)�systemzSSH command line: %s)�bufsizer� r� �stderr�envr` rK s client_versionz Is borg working on the server?Fs server_versionTz8Server insisted on using unsupported protocol version %sc s" �j �jj����� �d��_d S )N)rL rM rN rO rP rQ r� )ra �locationrL rI r( )rQ rM rP rO rN r� rB r( r) �do_openI s z*RemoteRepository.__init__.<locals>.do_openr� ao Please note:
If you see a TypeError complaining about the number of positional arguments
given to open(), you can ignore it if it comes from a borg version < 1.0.7.
This TypeError is a cosmetic side effect of the compatibility code borg
clients >= 1.0.7 have to support older borg servers.
This problem will go away as soon as the server has been upgraded to 1.0.7+.
z1.0.6rL rM rN rO ra )rL rM rN rO )<r� Z _location� preload_idsr� �rx_bytes�tx_bytesr� �stderr_received�chunkid_to_msgidsr| �ignore_responses� responses�async_responses�
shutdown_timer� Zremote_ratelimitr� r r� r r� �p�_args�hostr �borg_cmd�ssh_cmdr� r� r r r� r� r� r� r� r� r� r� r� r� r! r� �r_fds�x_fdsr� r� r+ r0 r� r� r� r� � Exceptionr� r� r� r" ru r� )rB r� rM rP rN rO rQ r� r� �testingr� r �versionr� �errr� r( )rQ rM rP rO rN r� rB r) rA sx
$$$
zRemoteRepository.__init__c C s> t | j�r tjdt | j�f � | jr:| j� ds:td��d S )Nz2still %d cached responses left in RemoteRepositoryFz&cleanup happened in Repository.__del__)r r r� r� r r� �AssertionError)rB r( r( r) �__del__i s
zRemoteRepository.__del__c C s d| j j| jj� f S )Nz<%s %s>)rC r, r� Zcanonical_path)rB r( r( r) �__repr__p s zRemoteRepository.__repr__c C s | S )Nr( )rB r( r( r) r� s s zRemoteRepository.__enter__c C sP z"|d k r t j� d | _| j� W d tjdt| j�t| j�| j � | j
� X d S )N� zDRemoteRepository: %s bytes sent, %s bytes received, %d messages sent) r# r� r rV r� r� r r� r� r� r� )rB �exc_type�exc_val�exc_tbr( r( r) �__exit__v s zRemoteRepository.__exit__c C s
t | j�S )N)r
rI )rB r( r( r) �id_str� s zRemoteRepository.id_strc C sX g }|dk r�|j d|j � tj� }|jtj�r<|j d� n^|jtj�rT|j d� nF|jtj�rbn8|jtj�rz|j d� n |jtj �r�|j d� nt
d��x4|j D ]*}d|kr�d | }d
|kr�|j d | � q�W d |kr�|j r�|j d
|j � g }t
� �s|j d� |�r&|tjdddg | | j S |j�p:tjjdd�}t|�}||dg | S dS )z return a borg serve command lineNz --umask=%03oz--debugz--infoz--errorz
--criticalz log level missing, fix this code�.z borg.debug.rk z--debug-topic=%srn z--storage-quota=%szBORG_HOSTNAME_IS_UNIQUE=noz-mz
borg.archiverr� ZBORG_REMOTE_PATH�borg)�append�umaskr� r� Z isEnabledForr� �INFOZWARNINGr� �CRITICAL�
ValueErrorZ debug_topicsrn r
r� �
executable�extra_test_args� remote_pathr! �environr[ r )rB r� r ZoptsZ root_loggerZtopicZenv_varsr! r( r( r) r � s<
zRemoteRepository.borg_cmdc C sj | j jptjjdd�}tj|�}|jr8|dt|j�g7 }|j rV|j
d|j |j f � n|j
d|j � |S )zEreturn a ssh command line that can be prefixed to a borg command lineZBORG_RSHZsshz-pz%s@%sz%s) r �rshr! r" r[ �shlex�splitZportr� �userr r )rB r� r# r� r( r( r) r � s
zRemoteRepository.ssh_cmdc s � fdd�t | D �S )Nc s g | ] }� | �qS r( r( )r6 rq )r{ r( r) r7 � s z8RemoteRepository.named_to_positional.<locals>.<listcomp>)ru )rB rw r{ r( )r{ r) �named_to_positional� s z$RemoteRepository.named_to_positionalc K s" x | j ||gf|�D ]}|S W d S )N)� call_many)rB �cmdr� �kw�respr( r( r) r� � s zRemoteRepository.callc # s� | r|dkrd S � fdd�}� fdd�}t |�}g }�x>|sD|�rv� jrttj� � jkrttjdt|�t� j�� d S xj|r�yJ� jj |d �} |j d� d| kr�|| � n| t
V | r�| r�d S W qv t k
r� P Y qvX qvW |dk�rNxby� jj � \}
} W n, t k
�r* |�r"� j
�r"P nd S Y q�X d| k�r@|| � q�| t
V q�W � j�sr|�sd� j�r|t|�tk �r|� jg} ng } tj� j| � jd �\} }
}|�r�td
���x�| D �]�}|� jk�r�tj|t�}|�s�t� �� jt|�7 _� j j |� xȈ j D ]�} t | t ��r| t }
nPt | t!��rbt| �d k�rb| \}}
}}|�rTt |
d|i} n t |
t
|i} nt"|��|
� j
k�r�� j
j#|
� d| k�r�| � j|
< n| t
d k �r�| � j|
< n
| � j|
< �q W n�|� j$k�r�tj|d �}|�s�t� �� jt|�7 _� j%�r� j%| }d
� _%|j&dd�}|�r@|d j'd� �r@|j � � _%x |D ]}t(|j)d�� �qFW �q�W |
r:�x�� j �r|�s�� j�rt|�tk �r|�rr|�r�|dk�s�t*d��|d d � j+k�rr|j,||j d�d �� n�|j d�}|dk�r|d � j+k�r|j,||d �� n\� j-d 7 _-|j,� j-� � j.�rTt/j0t � j-t1|t2|i�� _n t/j0d � j-|� j3||�f�� _� j �rl� j�rl� jj d�}d|i}� j-d 7 _-� j+j4|g �j,� j-� � j.�r�t/j0t � j-t1dt2|i�� _n t/j0d � j-d� j3d|�f�� _�qlW � jr:y4� j5j6� j� j�}� j7|7 _7� j|d � � _W q: t8k
�rr } z|j9t9j:k�rb� W Y d d }~X q:X q:W � j
t;|�O _
d S )Nr c s& � j | jd�}� j | s"� j | = |S )Nr )r� �pop)Zchunkidr� )rB r( r) �pop_preload_msgid� s
z5RemoteRepository.call_many.<locals>.pop_preload_msgidc s| | d j � }d| k}| jd�}|dkr8tj� jj���n@|dkrRtj� jj���n&|dkrltj� jj���n |dkr�|r�td��nt|d j � ��n�|d kr�|r�tj d��ntj |d j � ��n�|d
kr�|r�t
d ��nt
|d j � ��n�|d k�rtj |d j � ��nr|d
k�rB|�r(tj d� jj��ntj |d j � � jj��n6|dk�rn|�r\t
d��nt
|d j � ��n
� j| ��d S )Ns exception_classs exception_argsr� r� r� r z(not available)r �AtticRepositoryr1 z (unknown)�ParentPathDoesNotExistr� r2 )r< r[ r r� r� Z processedr� r� r r. r1 r/ r� r2 r� )r� r� Z
old_serverr� )rB r( r) � handle_error� s@
z0RemoteRepository.call_many.<locals>.handle_errorzPshutdown_time reached, shutting down with %d waiting_for and %d async_responses.r s exception_classr zFD exception occurredrs i � r� T)�keepends�
r� zutf-8r[ z&is_preload is only supported for 'get'rI ���)r2 r� )<rY r r# r� r� r� r r r r, r� �KeyError�popitemr r� r� � MAX_INFLIGHTr� r� r r
r r� r! r� r� r+ r� r� r� r� r� r� r� r4 �remover� r� �
splitlines�endswith�handle_remote_liner< r r� r r� r� r r� r� r� r' �
setdefaultr� r" r� �OSError�errnoZEAGAINr| )rB r) �callsr� � is_preloaded�
async_waitr- r0 Z waiting_forr� r� Zw_fdsr� r� r� r% r& r� r� r� �lines�liner� Zchunk_idr� r� r( )rB r) r( � s� )
$
*
$zRemoteRepository.call_manyz1.0.0z1.0.7)r� r� z1.1.9)r� rQ r� c C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rL rM rN rO rP rQ r� r( r( r) ra � s zRemoteRepository.open)r� c C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rE rF r( r( r) rT � s zRemoteRepository.checkc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rF r( r( r) rU � s zRemoteRepository.commitc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB r( r( r) rV � s zRemoteRepository.rollbackc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB r( r( r) rW � s zRemoteRepository.destroyc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB r( r( r) rX � s zRemoteRepository.__len__c C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rG rH r( r( r) rY � s zRemoteRepository.listz1.1.0b3c C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rG rH r( r( r) ri � s zRemoteRepository.scanc C s x| j |g�D ]}|S W d S )N)�get_many)rB rI r+ r( r( r) r[ � s zRemoteRepository.getc c s. x(| j ddd� |D �|d�D ]
}|V q W d S )Nr[ c S s g | ] }d |i�qS )rI r( )r6 rI r( r( r) r7 � s z-RemoteRepository.get_many.<locals>.<listcomp>)r? )r( )rB �idsr? r+ r( r( r) rC � s zRemoteRepository.get_manyc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rI r& r� r( r( r) rZ � s zRemoteRepository.putc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rI r� r( r( r) r\ � s zRemoteRepository.deletec C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rJ r( r( r) r] � s zRemoteRepository.save_keyc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB r( r( r) r^ � s zRemoteRepository.load_keyc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB r( r( r) rb � s z RemoteRepository.get_free_noncec C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB rR rS r( r( r) rc � s z)RemoteRepository.commit_nonce_reservationc C s dS )z;actual remoting is done via self.call in the @api decoratorNr( )rB r( r( r) r_ � s zRemoteRepository.break_lockc C s2 | j r.| j jj� | j jj� | j j� d | _ d S )N)r r� r� r� r� )rB r( r( r) r� � s
zRemoteRepository.closec C s" x | j dg d|d�D ]}|S W d S )Nr T)r> r� r@ )r( )rB r� r+ r( r( r) �async_response� s z RemoteRepository.async_responsec C s | j |7 _ d S )N)r� )rB rD r( r( r) �preload� s zRemoteRepository.preload)FFNTFFN)TFT)FNTFFF)FF)F)NN)NN)F)T)T)T)+r, r- r. r r r� r r� r� rA r r r� r r� r r r r' r� r( r� r ra rT rU rV rW rX rY ri r[ rC rZ r\ r] r^ rb rc r_ r� rE rF r( r( r( r) r� � s^ "
S
1
7
r� c C s | j d�st�| jd��rbtj| �}|d dkrFtjd|d | � d S |d dkr�tt|d
tj �}t
|t �srt�tj |d �}d |d
|d
< tj d�jr�||j
� kr�tjjtj|�d � n|j|d|d
� n�|d jd��r tj d�}|j
� tjk�r tj d�j�r>d
|k�r&d |d
|d
< tjjtj|�d � n"d
|k�r tjjd |d
d � n�| jd��r�| jdd�\}}}tt|tj �}|jd��r�tj||j� � n&|jdd�\}}tj |�j||j� � n8tj d�j�r�tj d�jd | j� � ntjjd | � d S )zv
Handle a remote log line.
This function is remarkably complex because it handles multiple wire formats.
�
r� �{�type�progress_message�progress_percent� log_messagez3Dropped remote log message with unknown type %r: %sNZ levelnamerq zRemote: �messager z%sZ progress_zborg.output.progressz$LOG r8 r zRemote:r r� )rG r� )rJ rK rL )r9 r r� r� �loadsr� Zwarningr� r� r r� r� r� ZgetEffectiveLevelr� r� r" �dumpsr� r r% �rstrip�strip)rB r� r� Z
target_loggerZprogress_loggerr� Zlognamer( r( r) r: � sB
r: c @ sL e Zd ZdZddd�Zdd� Zdd� Zd d
� Zd d � Zddd�Z dd� Z
dS )�RepositoryNoCacheaR A not caching Repository wrapper, passes through to repository.
Just to have same API (including the context manager) as RepositoryCache.
*transform* is a callable taking two arguments, key and raw repository data.
The return value is returned from get()/get_many(). By default, the raw
repository data is returned.
Nc C s || _ |pdd� | _d S )Nc S s |S )Nr( )�keyr& r( r( r) �<lambda>, s z,RepositoryNoCache.__init__.<locals>.<lambda>)rk � transform)rB rk rU r( r( r) rA * s zRepositoryNoCache.__init__c C s d S )Nr( )rB r( r( r) r� . s zRepositoryNoCache.closec C s | S )Nr( )rB r( r( r) r� 1 s zRepositoryNoCache.__enter__c C s | j � d S )N)r� )rB r r r r( r( r) r 4 s zRepositoryNoCache.__exit__c C s t | j|gdd��S )NF)�cache)�nextrC )rB rS r( r( r) r[ 7 s zRepositoryNoCache.getTc c s2 x,t || jj|��D ]\}}| j||�V qW d S )N)�ziprk rC rU )rB �keysrV rS r& r( r( r) rC : s zRepositoryNoCache.get_manyc C s d S )Nr( )rB r( r( r) �log_instrumentation> s z%RepositoryNoCache.log_instrumentation)N)T) r, r- r. r/ rA r� r� r r[ rC rZ r( r( r( r) rR ! s
rR c s\ e Zd ZdZd� fdd� Zdd� Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
ddd�Z � Z S )�RepositoryCacheaf
A caching Repository wrapper.
Caches Repository GET operations locally.
*pack* and *unpack* complement *transform* of the base class.
*pack* receives the output of *transform* and should return bytes,
which are stored in the cache. *unpack* receives these bytes and
should return the initial data (as returned by *transform*).
Nc sv t � j||� |pdd� | _|p&dd� | _t� | _tjdd�| _| j � d| _
d| _ d| _ d| _
d| _d| _d| _d S )Nc S s | S )Nr( )r& r( r( r) rT P s z*RepositoryCache.__init__.<locals>.<lambda>c S s | S )Nr( )r& r( r( r) rT Q s z borg-cache-)�prefixr g )r@ rA �pack�unpackr| rV �tempfileZmkdtemp�basedir�query_size_limit�size�hits�misses� slow_misses�slow_lat� evictions�enospc)rB rk r] r^ rU )rC r( r) rA N s zRepositoryCache.__init__c C s0 t j| j�}|j|j }tt|d d��| _d S )Ng �?r � l )r! �statvfsr` �f_bavail�f_frsizer� �min�
size_limit)rB Zstat_fsZavailable_spacer( r( r) ra ^ s z RepositoryCache.query_size_limitc C s t jj| jt|��S )N)r! rL r? r` r
)rB rS r( r( r) � key_filenamec s z RepositoryCache.key_filenamec C sr | j � td| j �}xV| j|krl| jrl| jj� }| j|�}| jtj|�j 8 _tj
|� | j d7 _ qW d S )Ng�������?r ) ra r� rn rb rV r, ro r! �stat�st_size�unlinkrg )rB Z target_sizerS �filer( r( r) �backofff s
zRepositoryCache.backoffc C s� | j ||�}|s|S | j|�}| j|�}y$t|d��}|j|� W d Q R X W nj tk
r� } zNy t|� W n tk
r~ Y nX |jtj kr�| j
d7 _
| j � n� W Y d d }~X n4X | j t
|�7 _ | jj|� | j | jkr�| j � |S )N�wbr )rU r] ro ra r" r< r �FileNotFoundErrorr= ZENOSPCrh rt rb r rV �addrn ) rB rS r& rV � transformedZpackedrs r% Zos_errorr( r( r) � add_entryp s,
zRepositoryCache.add_entryc C s> t jdt| j�t| j�t| j�| j| j| j | j
| j | j �
d S )NzyRepositoryCache: current items %d, size %s / %s, %d hits, %d misses, %d slow misses (+%.1fs), %d evictions, %d ENOSPC hit)
r� r� r rV r rb rn rc rd re rf rg rh )rB r( r( r) rZ � s z#RepositoryCache.log_instrumentationc C s" | j � | jj� tj| j� d S )N)rZ rV �clear�shutilZrmtreer` )rB r( r( r) r� � s
zRepositoryCache.closeTc
# s � fdd�|D �}t |� jj|��}x�|D ]�}|� jkrx� j|�}t|d��$}� jd7 _� j|j� �V W d Q R X q*x�|D ]4\}} ||kr~� j || |�}
� j
d7 _
|
V P q~W t j � } � jj
|�} � jt j � | 7 _� j || |�}
� jd7 _|
V q*W x|D ]} �q W d S )Nc s g | ]}|� j kr|�qS r( )rV )r6 rS )rB r( r) r7 � s z,RepositoryCache.get_many.<locals>.<listcomp>�rbr )rX rk rC rV ro ra rc r^ r� ry rd r# Z perf_counterr[ rf re )
rB rY rV Z unknown_keysZrepository_iteratorrS rs r% Zkey_r& rx Zt0r� r( )rB r) rC � s,
zRepositoryCache.get_many)NNN)T)
r, r- r. r/ rA ra ro rt ry rZ r� rC rD r( r( )rC r) r[ B s
r[ F)�decrypted_cacher] r^ rU � force_cachec s� |r|s|s|rt d��n@|rZ|�tjd�� t� �� �fdd�}� �fdd�}�fdd�}t| t�sh|rvt| |||�S t| |�S d S )
a�
Return a Repository(No)Cache for *repository*.
If *decrypted_cache* is a key object, then get and get_many will return a tuple
(csize, plaintext) instead of the actual data in the repository. The cache will
store decrypted data, which increases CPU efficiency (by avoiding repeatedly decrypting
and more importantly MAC and ID checking cached objects).
Internally, objects are compressed with LZ4.
z:decrypted_cache and pack/unpack/transform are incompatiblez=I8sc s& | \}}�j |�}� j|t|��| S )N)�compressr] r )r& �csize� decrypted�
compressed)� cache_struct�
compressorr( r) r] � s
z cache_if_remote.<locals>.packc sP t | �} � j| d � j� �\}}| � jd � }|t|�krBtd��|�j|�fS )Nz)detected corrupted data in metadata cache)�
memoryviewr^ rb r r Z
decompress)r& r� Zchecksumr� )r� r� r( r) r^ � s z cache_if_remote.<locals>.unpackc s t |�}� j| |�}||fS )N)r Zdecrypt)Zid_r& r� r� )rS r( r) rU � s z"cache_if_remote.<locals>.transformN)r �struct�Structr r� r� r[ rR )rk r} r] r^ rU r~ r( )r� r� rS r) �cache_if_remote� s
r� )r r r r )rE rF )rF )rG rH )rI r& )rI )rI )rJ )rK )rL rM rN rO rP rQ )rR rS )Ir= r� r� r} r� r� r! r� r$ r{ r� r� r_ r> r# r� �
subprocessr r r� r r r Z constantsZhelpersr r r
r r r
r r r r r r� r r r rk r r
r r Zalgorithms.checksumsr r, r� r� r� r� r� r� r6 r� r* r+ r0 r1 r2 r3 r4 ru rg rh r� r� r� r: rR r[ r� r( r( r( r) �<module> s� # 4 xJ!q