Google Git
Sign in
third-party-mirror / rohanpm / more-executors / e2ae563af056c7297df6ad6953dc92e7d54e0e6d
commite2ae563af056c7297df6ad6953dc92e7d54e0e6d[log] [tgz]
authorRohan McGovern <rohan@mcgovern.id.au>Wed Feb 01 14:44:29 2023 +1000
committerRohan McGovern <rohan@mcgovern.id.au>Wed Feb 01 14:54:48 2023 +1000
treeff36be420f997f6d9e50896f6db424048ff5c963
parentd0b5e379c7679313c75982214b0c32dd338634c2 [diff]
Reduce reference cycles to improve memory usage

Previously, a MapFuture and its delegate would always form a reference
cycle. If the futures point to large objects, this can result in
significant memory pressure as the cycle will not be freed until GC gets
around to it. Since GC is only triggered according to object counts and
not object sizes, this can increase apparent memory usage a lot in some
cases.

The cycle can be broken as soon as the delegate future has resolved, so
let's do that. (RetryFuture and PollFuture were already doing this, so
it seems there was an oversight specifically for MapFuture.)

Fixes #344
1 file changed
tree: ff36be420f997f6d9e50896f6db424048ff5c963
  1. .github/
  2. docs/
  3. examples/
  4. more_executors/
  5. scripts/
  6. tests/
  7. .gitignore
  8. .pre-commit-config.yaml
  9. .pylintrc
  10. CHANGELOG.md
  11. docs-requirements.in
  12. docs-requirements.txt
  13. LICENSE
  14. Makefile
  15. MANIFEST.in
  16. mypy.ini
  17. pidiff-requirements.in
  18. pidiff-requirements.txt
  19. README.md
  20. requirements.in
  21. requirements.txt
  22. setup.cfg
  23. setup.py
  24. test-requirements.in
  25. test-requirements.txt
  26. tox.ini
README.md

more-executors

A library of composable Python executors and futures.

Build Status Maintainability Test Coverage

This library is intended for use with the concurrent.futures module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

  • Futures with implicit retry
  • Futures with implicit cancel on executor shutdown
  • Futures with implicit cancel after timeout
  • Futures with transformed output values (sync & async)
  • Futures resolved by a caller-provided polling function
  • Throttle the number of futures running at once
  • Synchronous executor
  • Bridge concurrent.futures with asyncio
  • Convenience API for creating executors
  • Instrumented with Prometheus

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

Development

virtualenv and pip may be used to locally install this project from source:

virtualenv ~/dev/python
. ~/dev/python/bin/activate

git clone https://github.com/rohanpm/more-executors
cd more-executors

pip install --editable .

Autotests may be run with pytest:

pip install -r test-requirements.txt
py.test

Submit pull requests against https://github.com/rohanpm/more-executors.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.