| Current File : //usr/lib/python3/dist-packages/boto3/s3/__pycache__/transfer.cpython-312.pyc |
�
��e8> � �R � d Z ddlZddlZddlmZmZmZ ddlmZ ddl m
Z
ddl m Z
ddlmZ ddlmZ dd lmZ dd
lmZ dd lmZ ddlmc mZ dd lm Z m Z er
ddl Z dd
l m Z dZ!e!e!z Z" ejF e$� Z%dd�Z&d� Z'd� Z(d� Z) G d� de� Z G d� d� Z* G d� de� Z+y)az Abstractions over S3's upload/download operations.
This module provides high level abstractions for efficient
uploads/downloads. It handles several things for the user:
* Automatically switching to multipart transfers when
a file is over a specific size threshold
* Uploading/downloading a file in parallel
* Progress callbacks to monitor transfers
* Retries. While botocore handles retries for streaming uploads,
it is not possible for it to handle retries for streaming
downloads. This module handles retries for both cases so
you don't need to implement any retry logic yourself.
This module has a reasonable set of defaults. It also allows you
to configure many aspects of the transfer process including:
* Multipart threshold size
* Max parallel downloads
* Socket timeouts
* Retry amounts
There is no support for s3->s3 multipart copies at this
time.
.. _ref_s3transfer_usage:
Usage
=====
The simplest way to use this module is:
.. code-block:: python
client = boto3.client('s3', 'us-west-2')
transfer = S3Transfer(client)
# Upload /tmp/myfile to s3://bucket/key
transfer.upload_file('/tmp/myfile', 'bucket', 'key')
# Download s3://bucket/key to /tmp/myfile
transfer.download_file('bucket', 'key', '/tmp/myfile')
The ``upload_file`` and ``download_file`` methods also accept
``**kwargs``, which will be forwarded through to the corresponding
client operation. Here are a few examples using ``upload_file``::
# Making the object public
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
extra_args={'ACL': 'public-read'})
# Setting metadata
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
extra_args={'Metadata': {'a': 'b', 'c': 'd'}})
# Setting content type
transfer.upload_file('/tmp/myfile.json', 'bucket', 'key',
extra_args={'ContentType': "application/json"})
The ``S3Transfer`` class also supports progress callbacks so you can
provide transfer progress to users. Both the ``upload_file`` and
``download_file`` methods take an optional ``callback`` parameter.
Here's an example of how to print a simple progress percentage
to the user:
.. code-block:: python
class ProgressPercentage(object):
def __init__(self, filename):
self._filename = filename
self._size = float(os.path.getsize(filename))
self._seen_so_far = 0
self._lock = threading.Lock()
def __call__(self, bytes_amount):
# To simplify we'll assume this is hooked up
# to a single filename.
with self._lock:
self._seen_so_far += bytes_amount
percentage = (self._seen_so_far / self._size) * 100
sys.stdout.write(
"
%s %s / %s (%.2f%%)" % (
self._filename, self._seen_so_far, self._size,
percentage))
sys.stdout.flush()
transfer = S3Transfer(boto3.client('s3', 'us-west-2'))
# Upload /tmp/myfile to s3://bucket/key and print upload progress.
transfer.upload_file('/tmp/myfile', 'bucket', 'key',
callback=ProgressPercentage('/tmp/myfile'))
You can also provide a TransferConfig object to the S3Transfer
object that gives you more fine grained control over the
transfer. For example:
.. code-block:: python
client = boto3.client('s3', 'us-west-2')
config = TransferConfig(
multipart_threshold=8 * 1024 * 1024,
max_concurrency=10,
num_download_attempts=10,
)
transfer = S3Transfer(client, config)
transfer.upload_file('/tmp/foo', 'bucket', 'key')
� N)�PathLike�fspath�getpid)�HAS_CRT)� ClientError)�RetriesExceededError)�NonThreadedExecutor)�TransferConfig)�TransferManager)�BaseSubscriber)�OSUtils)r �S3UploadFailedError)�create_crt_transfer_manageri c �& � t |� rEt | |� }|�7t j dt � � dt j � � �� |S t j dt � � dt j � � �� t | ||� S )a� Creates a transfer manager based on configuration
:type client: boto3.client
:param client: The S3 client to use
:type config: boto3.s3.transfer.TransferConfig
:param config: The transfer config to use
:type osutil: s3transfer.utils.OSUtils
:param osutil: The os utility to use
:rtype: s3transfer.manager.TransferManager
:returns: A transfer manager based on parameters provided
zUsing CRT client. pid: z
, thread: zUsing default client. pid: )�_should_use_crtr �logger�debugr � threading� get_ident� _create_default_transfer_manager)�client�config�osutil�crt_transfer_managers �3/usr/lib/python3/dist-packages/boto3/s3/transfer.py�create_transfer_managerr � s� � � �v� � :�6�6� J� � � +� �L�L�)�&�(��:�i�>Q�>Q�>S�=T�U�
� (� '� �L�L�
%�f�h�Z�z�)�:M�:M�:O�9P�Q�� ,�F�F�F� C�C� c �8 � t r*t d� r t j j � }nd}| j
j
� }|r)|t j k( rt j d� yt j d|� dt � d|� d�� y) N)r � � FzEAttempting to use CRTTransferManager. Config settings may be ignored.Tz6Opting out of CRT Transfer Manager. Preferred client: z, CRT available: z, Instance Optimized: �.) r �has_minimum_crt_version�awscrt�s3�is_optimized_for_system�preferred_transfer_client�lower� constants� AUTO_RESOLVE_TRANSFER_CLIENTr r )r �is_optimized_instance�pref_transfer_clients r r r � s� � ��*�;�7� &� � � A� A� C� � %� �!�;�;�A�A�C�� � �I�$J�$J� J�� � � S�
� �
�L�L�@� �
� 1�'�� ; � 4� 5�Q� 8��
r c � � t syt j } t t |j d� � }t
|� }|| k\ S # t t f$ r Y yw xY w)z#Not intended for use outside boto3.Fr! ) r r# � __version__�map�int�split�tuple� TypeError�
ValueError)�minimum_version�crt_version_str�crt_version_ints�crt_version_tuples r r" r" � sb � � �� �(�(�O�� �s�O�$9�$9�#�$>�?�� !�"2� 3�� �� /�/��
�z� "� ���s �*A �A�Ac �F � d}|j st }t | |||� S )zACreate the default TransferManager implementation for s3transfer.N)� use_threadsr r )r r r � executor_clss r r r � s& � ��L� � � �*� � �6�6�6�<� @�@r c �l � � e Zd Zddd�Zdez ddez dddez d d
ej f � fd � Z� fd �Z � xZ
S )
r
�max_request_concurrency�max_io_queue_size)�max_concurrency� max_io_queue� �
� �d � TNc
� �� t � | � |||||||�� | j D ]& }
t | |
t | | j |
� � �( || _ | | _ y)a� Configuration object for managed S3 transfers
:param multipart_threshold: The transfer size threshold for which
multipart uploads, downloads, and copies will automatically be
triggered.
:param max_concurrency: The maximum number of threads that will be
making requests to perform a transfer. If ``use_threads`` is
set to ``False``, the value provided is ignored as the transfer
will only ever use the main thread.
:param multipart_chunksize: The partition size of each part for a
multipart transfer.
:param num_download_attempts: The number of download attempts that
will be retried upon errors with downloading an object in S3.
Note that these retries account for errors that occur when
streaming down the data from s3 (i.e. socket errors and read
timeouts that occur after receiving an OK response from s3).
Other retryable exceptions such as throttling errors and 5xx
errors are already retried by botocore (this default is 5). This
does not take into account the number of exceptions retried by
botocore.
:param max_io_queue: The maximum amount of read parts that can be
queued in memory to be written for a download. The size of each
of these read parts is at most the size of ``io_chunksize``.
:param io_chunksize: The max size of each chunk in the io queue.
Currently, this is size used when ``read`` is called on the
downloaded stream as well.
:param use_threads: If True, threads will be used when performing
S3 transfers. If False, no threads will be used in
performing transfers; all logic will be run in the main thread.
:param max_bandwidth: The maximum bandwidth that will be consumed
in uploading and downloading file content. The value is an integer
in terms of bytes per second.
:param preferred_transfer_client: String specifying preferred transfer
client for transfer operations.
Current supported settings are:
* auto (default) - Use the CRTTransferManager when calls
are made with supported environment and settings.
* classic - Only use the origin S3TransferManager with
requests. Disables possible CRT upgrade on requests.
)�multipart_thresholdr< �multipart_chunksize�num_download_attemptsr= � io_chunksize�
max_bandwidthN)�super�__init__�ALIAS�setattr�getattrr9 r& ) �selfrF r> rG rH r? rI r9 rJ r& �alias� __class__s �r rL zTransferConfig.__init__� sp �� �z ��� 3�$3� 3�"7� *�%�'� �
� �Z�Z� C�E� �D�%���t�z�z�%�/@�!A� B� C�&���)B��&r c �| �� || j v r t �| � | j | |� t �| � ||� y �N)rM rK � __setattr__)rP �name�valuerR s �r rU zTransferConfig.__setattr__< s8 �� � �4�:�:� � �G� ��
�
�4� 0�%� 8�
���D�%�(r ) �__name__�
__module__� __qualname__rM �MB�KBr( r) rL rU �
__classcell__)rR s @r r
r
� sQ �� �4�+�
�E� ��F�� ��F� ���2�X���"+�"H�"H�LC�\)� )r r
c �j � e Zd Zej Zej
Zdd�Z d d�Z d d�Zd� Z d� Z
d� Z y)
�
S3TransferNc � � |s
|s t d� �|rt |||g� r t d� �|�
t � }|�
t � }|r|| _ y t |||� | _ y )NzLEither a boto3.Client or s3transfer.manager.TransferManager must be providedzdManager cannot be provided with client, config, nor osutil. These parameters are mutually exclusive.)r3 �anyr
r
�_managerr )rP r r r �managers r rL zS3Transfer.__init__I st � ��g� �#��
� �s�F�F�F�3�4� �G��
� �>�#�%�F� �>� �Y�F� � #�D�M� 3�F�F�F� K�D�Mr c
�n � t |t � r t |� }t |t � s t d� �| j |� }| j j |||||� } |j � y# t $ r2}t dj |dj ||g� |� � �d}~ww xY w)a( Upload a file to an S3 object.
Variants have also been injected into S3 client, Bucket and Object.
You don't have to use S3Transfer.upload_file() directly.
.. seealso::
:py:meth:`S3.Client.upload_file`
:py:meth:`S3.Client.upload_fileobj`
�/Filename must be a string or a path-like objectz Failed to upload {} to {}: {}�/N)
�
isinstancer r �strr3 �_get_subscribersrb �upload�resultr r �format�join) rP �filename�bucket�key�callback�
extra_args� subscribers�future�es r � upload_filezS3Transfer.upload_file] s� � � �h�� )� �h�'�H��(�C�(� � N�O� O��+�+�H�5� �� � �%�%� �f�c�:�{�
�� � �M�M�O��
� �%�/�6�6� �c�h�h���}� 5�q���
�� �s �(A9 �9 B4�-B/�/B4c �> � t |t � r t |� }t |t � s t d� �| j |� }| j j |||||� } |j � y# t $ r}t |j � �d}~ww xY w)a0 Download an S3 object to a file.
Variants have also been injected into S3 client, Bucket and Object.
You don't have to use S3Transfer.download_file() directly.
.. seealso::
:py:meth:`S3.Client.download_file`
:py:meth:`S3.Client.download_fileobj`
re N) rg r r rh r3 ri rb �downloadrk � S3TransferRetriesExceededErrorr �last_exception) rP ro rp rn rr rq rs rt ru s r �
download_filezS3Transfer.download_file s� � � �h�� )� �h�'�H��(�C�(� � N�O� O��+�+�H�5� �� � �'�'� �C��:�{�
�� 9� �M�M�O�� .� 9�&�q�'7�'7�8� 8�� 9�s �(A9 �9 B �B�B c � � |sy t |� gS rT )�ProgressCallbackInvoker�rP rq s r ri zS3Transfer._get_subscribers� s � ���'��1�2�2r c � � | S rT � )rP s r � __enter__zS3Transfer.__enter__� s � �� r c �6 � | j j |� y rT )rb �__exit__)rP �argss r r� zS3Transfer.__exit__� s � � ��
�
� � ��%r )NNNN)NN) rX rY rZ r �ALLOWED_DOWNLOAD_ARGS�ALLOWED_UPLOAD_ARGSrL rv r{ ri r� r� r� r r r_ r_ E sH � � +� A� A��)�=�=��L�* @D� �F @D� 9�>3�
�&r r_ c � � e Zd ZdZd� Zd� Zy)r} z�A back-compat wrapper to invoke a provided callback via a subscriber
:param callback: A callable that takes a single positional argument for
how many bytes were transferred.
c � � || _ y rT �� _callbackr~ s r rL z ProgressCallbackInvoker.__init__� s � �!��r c �& � | j |� y rT r� )rP �bytes_transferred�kwargss r � on_progressz#ProgressCallbackInvoker.on_progress� s � � ���(�)r N)rX rY rZ �__doc__rL r� r� r r r} r} � s � �� "�*r r} rT ),r� �loggingr �osr r r �botocore.compatr �botocore.exceptionsr �s3transfer.exceptionsr ry �s3transfer.futuresr �s3transfer.managerr
�S3TransferConfigr �s3transfer.subscribersr �s3transfer.utilsr
�boto3.s3.constantsr$ r( �boto3.exceptionsr � awscrt.s3r# � boto3.crtr r\ r[ � getLoggerrX r r r r" r r_ r} r� r r �<module>r� s� ��o�` � � '� '� #� +�� 3� A� .� 1� $� &� &� F�
��5� ���"�W�� �� � �8� $��D�<�2 0� A�Z)�%� Z)�zb&� b&�J *�n� *r