commit | 7a02120643f5afb3cddd5befa3f85d7a2d4f8ba1 | [log] [tgz] |
---|---|---|
author | Rohan McGovern <rohan@mcgovern.id.au> | Wed Dec 14 09:09:01 2022 +1000 |
committer | Rohan McGovern <rohan@mcgovern.id.au> | Wed Dec 14 09:09:01 2022 +1000 |
tree | 77b8209ad59653b31ae82121b792c63ca6d5b78d | |
parent | 6048bbfeadbfb06b62a2897bcad0413da8108a35 [diff] |
Release 2.11.3
A library of composable Python executors and futures.
This library is intended for use with the concurrent.futures
module. It includes a collection of Executor
implementations in order to extend the behavior of Future
objects.
concurrent.futures
with asyncio
See the API documentation for detailed information on usage.
This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.
import requests from concurrent.futures import as_completed from more_executors import Executors def get_json(response): response.raise_for_status() return (response.url, response.json()) def fetch_urls(urls): # Configure an executor: # - run up to 4 requests concurrently, in separate threads # - run get_json on each response # - retry up to several minutes on any errors executor = Executors.\ thread_pool(max_workers=4).\ with_map(get_json).\ with_retry() # Submit requests for each given URL futures = [executor.submit(requests.get, url) for url in urls] # Futures API works as normal; we can block on the completed # futures and map/retry happens implicitly for future in as_completed(futures): (url, data) = future.result() do_something(url, data)
virtualenv
and pip
may be used to locally install this project from source:
virtualenv ~/dev/python . ~/dev/python/bin/activate git clone https://github.com/rohanpm/more-executors cd more-executors pip install --editable .
Autotests may be run with pytest:
pip install -r test-requirements.txt py.test
Submit pull requests against https://github.com/rohanpm/more-executors.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.