| Current File : //usr/lib/python3/dist-packages/s3transfer/__pycache__/manager.cpython-312.pyc |
�
�<�e[o � �0 � d dl Z d dlZd dlZd dlZd dlmZmZ d dlmZm Z m
Z
d dl m Z d dl
mZ d dlmZ d dlmZmZ d dlmZmZmZmZmZmZ d d lm Z d d
l m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z% ejL e'� Z( G d � d � Z) G d
� d� Z* G d� d� Z+y)� N)�BandwidthLimiter� LeakyBucket)�ALLOWED_DOWNLOAD_ARGS�KB�MB)�CopySubmissionTask)�DeleteSubmissionTask)�DownloadSubmissionTask)�CancelledError�
FatalError)�IN_MEMORY_DOWNLOAD_TAG�IN_MEMORY_UPLOAD_TAG�BoundedExecutor�TransferCoordinator�TransferFuture� TransferMeta)�UploadSubmissionTask)�CallArgs�OSUtils�SlidingWindowSemaphore�
TaskSemaphore�add_s3express_defaults�
get_callbacks�signal_not_transferring�signal_transferringc �D � e Zd Zdez dez ddddddez ddddf d�Zd� Zy) �TransferConfig� �
� i� � Nc
�� � || _ || _ || _ || _ || _ || _ || _ || _ | | _ |
| _ | | _
| | _ | j � y)as Configurations for the transfer manager
:param multipart_threshold: The threshold for which multipart
transfers occur.
:param max_request_concurrency: The maximum number of S3 API
transfer-related requests that can happen at a time.
:param max_submission_concurrency: The maximum number of threads
processing a call to a TransferManager method. Processing a
call usually entails determining which S3 API requests that need
to be enqueued, but does **not** entail making any of the
S3 API data transferring requests needed to perform the transfer.
The threads controlled by ``max_request_concurrency`` is
responsible for that.
:param multipart_chunksize: The size of each transfer if a request
becomes a multipart transfer.
:param max_request_queue_size: The maximum amount of S3 API requests
that can be queued at a time.
:param max_submission_queue_size: The maximum amount of
TransferManager method calls that can be queued at a time.
:param max_io_queue_size: The maximum amount of read parts that
can be queued to be written to disk per download. The default
size for each elementin this queue is 8 KB.
:param io_chunksize: The max size of each chunk in the io queue.
Currently, this is size used when reading from the downloaded
stream as well.
:param num_download_attempts: The number of download attempts that
will be tried upon errors with downloading an object in S3. Note
that these retries account for errors that occur when streaming
down the data from s3 (i.e. socket errors and read timeouts that
occur after receiving an OK response from s3).
Other retryable exceptions such as throttling errors and 5xx errors
are already retried by botocore (this default is 5). The
``num_download_attempts`` does not take into account the
number of exceptions retried by botocore.
:param max_in_memory_upload_chunks: The number of chunks that can
be stored in memory at a time for all ongoing upload requests.
This pertains to chunks of data that need to be stored in memory
during an upload if the data is sourced from a file-like object.
The total maximum memory footprint due to a in-memory upload
chunks is roughly equal to:
max_in_memory_upload_chunks * multipart_chunksize
+ max_submission_concurrency * multipart_chunksize
``max_submission_concurrency`` has an affect on this value because
for each thread pulling data off of a file-like object, they may
be waiting with a single read chunk to be submitted for upload
because the ``max_in_memory_upload_chunks`` value has been reached
by the threads making the upload request.
:param max_in_memory_download_chunks: The number of chunks that can
be buffered in memory and **not** in the io queue at a time for all
ongoing download requests. This pertains specifically to file-like
objects that cannot be seeked. The total maximum memory footprint
due to a in-memory download chunks is roughly equal to:
max_in_memory_download_chunks * multipart_chunksize
:param max_bandwidth: The maximum bandwidth that will be consumed
in uploading and downloading file content. The value is in terms of
bytes per second.
N)
�multipart_threshold�multipart_chunksize�max_request_concurrency�max_submission_concurrency�max_request_queue_size�max_submission_queue_size�max_io_queue_size� io_chunksize�num_download_attempts�max_in_memory_upload_chunks� max_in_memory_download_chunks�
max_bandwidth�_validate_attrs_are_nonzero)
�selfr# r$ r% r& r' r( r) r* r+ r, r- r. s
�4/usr/lib/python3/dist-packages/s3transfer/manager.py�__init__zTransferConfig.__init__0 sq � �l $7�� �#6�� �'>��$�*D��'�&<��#�)B��&�!2�� � (���%:��"�+F��(�-J��*� *��� �(�(�*� c �| � | j j � D ] \ }}|�� |dk s�t d|�d|�d�� � y )Nr zProvided parameter z
of value z must be greater than 0.)�__dict__�items�
ValueError)r0 �attr�attr_vals r1 r/ z*TransferConfig._validate_attrs_are_nonzero� sE � � "�m�m� 1� 1� 3� �N�D�(��#��A�
� � �(�,�� � r3 )�__name__�
__module__� __qualname__r r r2 r/ � r3 r1 r r / s@ � � ��F� ��F� "�#$� #�"&� ��2�X� �$&�&(��b+�Hr3 r c � � e Zd ZeZg d�Zeg d�z Zg d�ZdZd ej d� iZ
dd�Z e d � � Z
e d
� � Zdd �Z dd �Z dd
�Zdd�Zd� Zd� Zd� Z d d�Zd� Zd� Zd� Zd� Zd� Zd d�Z e fd�Z y) �TransferManager)�ACL� CacheControl�ChecksumAlgorithm�ContentDisposition�ContentEncoding�ContentLanguage� ContentType�ExpectedBucketOwner�Expires�GrantFullControl� GrantRead� GrantReadACP�
GrantWriteACP�Metadata�ObjectLockLegalHoldStatus�ObjectLockMode�ObjectLockRetainUntilDate� RequestPayer�ServerSideEncryption� StorageClass�SSECustomerAlgorithm�SSECustomerKey�SSECustomerKeyMD5� SSEKMSKeyId�SSEKMSEncryptionContext�Tagging�WebsiteRedirectLocation) �CopySourceIfMatch�CopySourceIfModifiedSince�CopySourceIfNoneMatch�CopySourceIfUnmodifiedSince� CopySourceSSECustomerAlgorithm�CopySourceSSECustomerKey�CopySourceSSECustomerKeyMD5�MetadataDirective�TaggingDirective)�MFA� VersionIdrQ rG TzS3 Object LambdazV^arn:(aws).*:s3-object-lambda:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}Nc
�r � || _ || _ |�t � | _ || _ |�t � | _ t � | _ d| _ t | j j | j j t t | j j � t t | j j � i|�� | _ t | j j$ | j j& |�� | _ t | j j* d|�� | _ d| _ | j j0 �Yt2 j5 d| j j0 � t7 | j j0 � }t9 |� | _ | j; � y)a A transfer manager interface for Amazon S3
:param client: Client to be used by the manager
:param config: TransferConfig to associate specific configurations
:param osutil: OSUtils object to use for os-related behavior when
using with transfer manager.
:type executor_cls: s3transfer.futures.BaseExecutor
:param executor_cls: The class of executor to use with the transfer
manager. By default, concurrent.futures.ThreadPoolExecutor is used.
Nr )�max_size�max_num_threads�tag_semaphores� executor_cls)rg rh rj � zSetting max_bandwidth to %s) �_client�_configr �_osutilr � TransferCoordinatorController�_coordinator_controller� _id_counterr r' r% r r r, r
r r- �_request_executorr( r& �_submission_executorr) � _io_executor�_bandwidth_limiterr. �logger�debugr r �_register_handlers)r0 �client�config�osutilrj � leaky_buckets r1 r2 zTransferManager.__init__� s_ � � �� � �� � �>�)�+�D�L� �� � �>�"�9�D�L�'D�'F��$� ��� "1��\�\�8�8� �L�L� @� @�$�m��L�L�<�<�'� '�(>��L�L�>�>�)� � &� "
�� � %4��\�\�;�;� �L�L� C� C�%�%
��!� ,��\�\�3�3� �%�
��� #'�� � �<�<� %� %� 1� �L�L�-�t�|�|�/I�/I�
� '�t�|�|�'A�'A�B�L�&6�|�&D�D� #� � � �!r3 c � � | j S �N)rl �r0 s r1 ry zTransferManager.client � � ��|�|�r3 c � � | j S r~ )rm r s r1 rz zTransferManager.config r� r3 c � � |�i }|�g }| j || j � | j |� | j ||� t |||||�� }i }| j
r| j
|d< | j
|t |� S )a� Uploads a file to S3
:type fileobj: str or seekable file-like object
:param fileobj: The name of a file to upload or a seekable file-like
object to upload. It is recommended to use a filename because
file-like objects may result in higher memory usage.
:type bucket: str
:param bucket: The name of the bucket to upload to
:type key: str
:param key: The name of the key to upload to
:type extra_args: dict
:param extra_args: Extra arguments that may be passed to the
client operation
:type subscribers: list(s3transfer.subscribers.BaseSubscriber)
:param subscribers: The list of subscribers to be invoked in the
order provided based on the event emit during the process of
the transfer request.
:rtype: s3transfer.futures.TransferFuture
:returns: Transfer future representing the upload
)�fileobj�bucket�key�
extra_args� subscribers�bandwidth_limiter)�_validate_all_known_args�ALLOWED_UPLOAD_ARGS� _validate_if_bucket_supported�_add_operation_defaultsr ru �_submit_transferr )r0 r� r� r� r� r� � call_args�extra_main_kwargss r1 �uploadzTransferManager.upload$ s� � �4 � ��J� � � �K� �%�%�j�$�2J�2J�K� �*�*�6�2� �$�$�V�Z�8� ����!�#�
� � �� � "� "�59�5L�5L� � 1� 2��$�$� �+�->�
�
r3 c � � |�i }|�g }| j || j � | j |� t |||||�� }d| j i}| j
r| j
|d< | j
|t |� S )a� Downloads a file from S3
:type bucket: str
:param bucket: The name of the bucket to download from
:type key: str
:param key: The name of the key to download from
:type fileobj: str or seekable file-like object
:param fileobj: The name of a file to download or a seekable file-like
object to download. It is recommended to use a filename because
file-like objects may result in higher memory usage.
:type extra_args: dict
:param extra_args: Extra arguments that may be passed to the
client operation
:type subscribers: list(s3transfer.subscribers.BaseSubscriber)
:param subscribers: The list of subscribers to be invoked in the
order provided based on the event emit during the process of
the transfer request.
:rtype: s3transfer.futures.TransferFuture
:returns: Transfer future representing the download
)r� r� r� r� r� � io_executorr� )r� r r� r rt ru r� r
)r0 r� r� r� r� r� r� r� s r1 �downloadzTransferManager.downloadS s� � �8 � ��J� � � �K� �%�%�j�$�2L�2L�M� �*�*�6�2� ����!�#�
� � +�D�,=�,=� >�� � "� "�59�5L�5L� � 1� 2��$�$� �-�/@�
�
r3 c �6 � |�i }|�g }|� | j }| j || j � t |t � r | j |j
d� � | j |� t ||||||�� }| j |t � S )a� Copies a file in S3
:type copy_source: dict
:param copy_source: The name of the source bucket, key name of the
source object, and optional version ID of the source object. The
dictionary format is:
``{'Bucket': 'bucket', 'Key': 'key', 'VersionId': 'id'}``. Note
that the ``VersionId`` key is optional and may be omitted.
:type bucket: str
:param bucket: The name of the bucket to copy to
:type key: str
:param key: The name of the key to copy to
:type extra_args: dict
:param extra_args: Extra arguments that may be passed to the
client operation
:type subscribers: a list of subscribers
:param subscribers: The list of subscribers to be invoked in the
order provided based on the event emit during the process of
the transfer request.
:type source_client: botocore or boto3 Client
:param source_client: The client to be used for operation that
may happen at the source object. For example, this client is
used for the head_object that determines the size of the copy.
If no client is provided, the transfer manager's client is used
as the client for the source object.
:rtype: s3transfer.futures.TransferFuture
:returns: Transfer future representing the copy
�Bucket)� copy_sourcer� r� r� r� �
source_client)
rl r� �ALLOWED_COPY_ARGS�
isinstance�dictr� �getr r� r )r0 r� r� r� r� r� r� r� s r1 �copyzTransferManager.copy� s� � �V � ��J� � � �K� � � �L�L�M� �%�%�j�$�2H�2H�I� �k�4� (� � .� .�{���x�/H� I� �*�*�6�2� �#���!�#�'�
� � �$�$�Y�0B�C�Cr3 c � � |�i }|�g }| j || j � | j |� t ||||�� }| j |t
� S )a� Delete an S3 object.
:type bucket: str
:param bucket: The name of the bucket.
:type key: str
:param key: The name of the S3 object to delete.
:type extra_args: dict
:param extra_args: Extra arguments that may be passed to the
DeleteObject call.
:type subscribers: list
:param subscribers: A list of subscribers to be invoked during the
process of the transfer request. Note that the ``on_progress``
callback is not invoked during object deletion.
:rtype: s3transfer.futures.TransferFuture
:return: Transfer future representing the deletion.
)r� r� r� r� )r� �ALLOWED_DELETE_ARGSr� r r� r )r0 r� r� r� r� r� s r1 �deletezTransferManager.delete� sj � �, � ��J� � � �K� �%�%�j�$�2J�2J�K� �*�*�6�2� ���!�#�
� � �$�$�Y�0D�E�Er3 c � � | j rD| j j � D ]& \ }}|j |� }|s�t d|z � � y y )NzTTransferManager methods do not support %s resource. Use direct client calls instead.)� VALIDATE_SUPPORTED_BUCKET_VALUES� _UNSUPPORTED_BUCKET_PATTERNSr6 �matchr7 )r0 r� �resource�patternr� s r1 r� z-TransferManager._validate_if_bucket_supported� sb � � � 0� 0�%)�%F�%F�%L�%L�%N�
�!��'� �
�
�f�-���$�E�GO�P�� �
� 1r3 c �Z � |D ]& }||vs�t d|�ddj |� ��� � y )NzInvalid extra_args key 'z', must be one of: z, )r7 �join)r0 �actual�allowed�kwargs r1 r� z(TransferManager._validate_all_known_args� s: � �� �E��G�#� �,1�4�9�9�W�3E�G�� � r3 c � � t ||� y r~ )r )r0 r� r� s r1 r� z'TransferManager._add_operation_defaults� s � � �v�z�2r3 c � � |si }| j |� \ }}t |d� D ] }|d j |� � | j ||� }| j j ||d |�� � | xj dz
c_ |S )N�done� coordinator)�transfer_coordinator� main_kwargsrk )�_get_future_with_componentsr �add_done_callback� _get_submission_task_main_kwargsrs �submitrq )r0 r� �submission_task_clsr� �transfer_future�
components�callbackr� s r1 r� z TransferManager._submit_transfer� s� � � !� "� � '+�&F�&F� �'
�#��� &�o�v�>� B�H� �}� %� 7� 7�� A� B� �;�;� � .�
� �
�!�!�(�(� �%/�
�%>� '�
�
�
���A� �� � r3 c �� � | j }t |�� }| j j |� |j | j j
|� t
||�� |d�}t di |��}||fS )N)� transfer_id)�metar� r= )rq r rp �add_transfer_coordinatorr� �remove_transfer_coordinatorr r )r0 r� r� r� r� r� s r1 r� z+TransferManager._get_future_with_components! s� � ��&�&� � 2�{� K� � �$�$�=�=� �
�
�.�.� � (� (� D� D� �
�
!�� �D�/�
�
� )�6�:�6�� �
�*�*r3 c � � | j | j | j | j |d�}|j |� |S )N)ry rz r{ �request_executorr� )rl rm rn rr �update)r0 r� r� r� s r1 r� z0TransferManager._get_submission_task_main_kwargs6 sC � � �l�l��l�l��l�l� $� 6� 6� .�
� � ���,�-��r3 c �� � d}| j j j j |t d�� | j j j j |t d�� y )Nzrequest-created.s3zs3upload-not-transferring)� unique_idzs3upload-transferring)rl r� �events�register_firstr �
register_lastr )r0 �
event_names r1 rx z"TransferManager._register_handlersC sa � �)�
� � � ��� � �/�/� � #�1� 0�
�
� � ��� � �.�.� �+�7N� /�
r3 c � � | S r~ r= r s r1 � __enter__zTransferManager.__enter__O s � �� r3 c � � d}d}t }|r0d}t |� }|s t |� }t |t � rt
}| j
|||� y )NF� T)r �str�reprr� �KeyboardInterruptr � _shutdown)r0 �exc_type� exc_value�args�cancel�
cancel_msg�cancel_exc_types r1 �__exit__zTransferManager.__exit__R sR � ����
�$�� ��F� �Y� �J� � !�)�_�
� �)�%6�7�"0�� ���v�z�?�;r3 c �* � | j |||� y)a� Shutdown the TransferManager
It will wait till all transfers complete before it completely shuts
down.
:type cancel: boolean
:param cancel: If True, calls TransferFuture.cancel() for
all in-progress in transfers. This is useful if you want the
shutdown to happen quicker.
:type cancel_msg: str
:param cancel_msg: The message to specify if canceling all in-progress
transfers.
N)r� )r0 r� r� s r1 �shutdownzTransferManager.shutdownc s � �
���v�v�z�2r3 c � � |r | j j ||� | j j � | j j � | j j � | j j � y # t $ r | j j d� � w xY w# | j j � | j j � | j j � w xY w)NzKeyboardInterrupt())rp r� �waitr� rs r� rr rt )r0 r� r� r� s r1 r� zTransferManager._shutdownt s� � � �
� (� (� /� /�
�H� E� )�
� (� (� -� -� /�
� %� %� .� .� 0� � "� "� +� +� -� � � � &� &� (�� !� �
� (� (� /� /�0E� F� � ��
� %� %� .� .� 0� � "� "� +� +� -� � � � &� &� (�s �B
�
&B0�0B3 �3AD)NNN)NNr~ )Fr� ) r: r; r< r r� r� r� r� �re�compiler� r2 �propertyry rz r� r� r� r� r� r� r� r� r� r� rx r� r� r� r r� r= r3 r1 r? r? � s� � � 1�� ��< ,�
/�
���� (,�$� �J�B�J�J�
2�
�$� �@"�D � � � � � � � �-
�` BF�.
�j ���=D�~"F�H ��3� AE�! �F+�* �
��<�"3�" 6D� )r3 r? c �@ � e Zd Zd� Zed� � Zd� Zd� Zdefd�Z d� Z
y) ro c �T � t j � | _ t � | _ y)z�Abstraction to control all transfer coordinators
This abstraction allows the manager to wait for inprogress transfers
to complete and cancel all inprogress transfers.
N)� threading�Lock�_lock�set� _tracked_transfer_coordinatorsr s r1 r2 z&TransferCoordinatorController.__init__� s � � �^�^�%��
�.1�e��+r3 c � � | j 5 t j | j � cddd� S # 1 sw Y yxY w)z.The set of transfer coordinators being trackedN)r� r� r� r s r1 � tracked_transfer_coordinatorsz;TransferCoordinatorController.tracked_transfer_coordinators� s8 � � �Z�Z� B� �9�9�T� @� @�A� B� B� B�s � 6�?c �| � | j 5 | j j |� ddd� y# 1 sw Y yxY w)z�Adds a transfer coordinator of a transfer to be canceled if needed
:type transfer_coordinator: s3transfer.futures.TransferCoordinator
:param transfer_coordinator: The transfer coordinator for the
particular transfer
N)r� r� �add�r0 r� s r1 r� z6TransferCoordinatorController.add_transfer_coordinator� s9 � � �Z�Z� J� � /� /� 3� 3�4H� I� J� J� J�� � 2�;c �| � | j 5 | j j |� ddd� y# 1 sw Y yxY w)a� Remove a transfer coordinator from cancellation consideration
Typically, this method is invoked by the transfer coordinator itself
to remove its self when it completes its transfer.
:type transfer_coordinator: s3transfer.futures.TransferCoordinator
:param transfer_coordinator: The transfer coordinator for the
particular transfer
N)r� r� �remover� s r1 r� z9TransferCoordinatorController.remove_transfer_coordinator� s9 � � �Z�Z� M� � /� /� 6� 6�7K� L� M� M� M�r� r� c �J � | j D ] }|j ||� � y)aP Cancels all inprogress transfers
This cancels the inprogress transfers by calling cancel() on all
tracked transfer coordinators.
:param msg: The message to pass on to each transfer coordinator that
gets cancelled.
:param exc_type: The type of exception to set for the cancellation
N)r� r� )r0 �msgr� r� s r1 r� z$TransferCoordinatorController.cancel� s* � � %)�$F�$F� 7� � � '� '��X� 6� 7r3 c �� � d}| j D ] }|j � � y# t $ r/ t j d� rt j d|� � t
$ r Y yw xY w)z�Wait until there are no more inprogress transfers
This will not stop when failures are encountered and not propagate any
of these errors from failed transfers, but it can be interrupted with
a KeyboardInterrupt.
Nz$Received KeyboardInterrupt in wait()z'On KeyboardInterrupt was waiting for %s)r� �resultr� rv rw � Exceptionr� s r1 r� z"TransferCoordinatorController.wait� st � � �#'� �(,�(J�(J�
.�$�$�+�+�-�
.�� �
� �L�L�?� @� $�� � �=�(��
�� �
� �s �#& �?A(�'A(N) r: r; r< r2 r� r� r� r� r r� r� r= r3 r1 ro ro � s: � �4� �B� �B�J� M� �n� 7� r3 ro ),r� �loggingr� r� �s3transfer.bandwidthr r �s3transfer.constantsr r r �s3transfer.copiesr �s3transfer.deleter �s3transfer.downloadr
�s3transfer.exceptionsr r �s3transfer.futuresr
r r r r r �s3transfer.uploadr �s3transfer.utilsr r r r r r r r � getLoggerr: rv r r? ro r= r3 r1 �<module>r� s� �� � � � � >� >� >� 0� 2� 6� <�� � 3� � � �
�� � �8� $��k� k�\m)� m)�`S� Sr3