~linaro-infrastructure/linaro-license-protection/master

  • Committer: Benjamin Copeland
  • Author(s): Kelley Spoon
  • Date: 2019-10-15 15:00:38 UTC
  • Revision ID: git-v1:453ce66dcac200d6124d4341ea580e044e533507
s3_purge: rewrite to use Bucket.list_versions()

The current s3_purge script uses the Bucket.list() method
to get a listing of all s3 objects to work on.  The list()
method unfortunately filters out files that have been marked
for deleted, and this creates a race condition where
s3_purge is unable to see the files in the list that it needs
to delete in a --forcedelete run.

This change uses the list_versions() method to get a list of
everything.  It groups keys by filename, and uses the first
key in the list (the lastest version of the file) to determine
when deciding what action to take.  This should allow for the
script to run slightly faster as it process the files as all
of its keys come in instead of waiting for the list() method
to complete.

This change drops the needs for the --clean-releases option
as the method for deleting files is working on the s3 object's
name and the scope of the operation should have been limited
by the --prefix option already (default 'snapshots/').

Change-Id: If30f0dd9c7a6155442ac87bf44ef2f609802bf3c
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33011
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
Filename Latest Rev Last Changed Committer Comment Size
..
css 131 11 years ago Stevan Radakovic Predefined textfiles rendering support. Reviewed b Diff
js 131 11 years ago Stevan Radakovic Predefined textfiles rendering support. Reviewed b Diff
license_protected_downloads 86 11 years ago Danilo Segan Merge in django-refactoring. Diff
sampleroot 105 11 years ago Danilo Šegan Clean up unused files and READMEs (provide INSTALL Diff
scripts 29 12 years ago Danilo Segan Add uncommitted stuff: licenses/nolicense.html and Diff
templates 86 11 years ago Danilo Segan Merge in django-refactoring. Diff
testplans 75 12 years ago Stevan Radakovic Renamed directory with doctests for the new produc Diff
tests 76 12 years ago Danilo Šegan Clean up the Python code to adhere to PEP-8, do a Diff
.gitignore 435 6 years ago Benjamin Copeland Dockerfile: Include a dockerfile for debugging No 108 bytes Diff Download File
.gitreview 233 10 years ago Milo Casagrande Add .gitreview file. 93 bytes Diff Download File
.testr.conf 75.1.2 12 years ago Danilo Šegan Rename "testing" subdirectory to "tests". 91 bytes Diff Download File
__init__.py 84.1.2 11 years ago James Tunnicliffe Moved files out od django-rewrite. Combines .bzrig Empty Diff Download File
Dockerfile 465 5 years ago Benjamin Copeland Dockerfile: Update dockerfile docs Missing port n 1 KB Diff Download File
HACKING 193.1.1 10 years ago James Tunnicliffe Updated buildinfo logic to be much faster at machi 3.8 KB Diff Download File
INSTALL 208.1.1 10 years ago Milo Casagrande Merged James branch. 1.6 KB Diff Download File
File manage.py 295 9 years ago Andy Doan fix import issues with recent versions of django 699 bytes Diff Download File
manage.sh 284 9 years ago Paul Sokolovsky manage.sh: Add helper script to call manage.py wit 527 bytes Diff Download File
README 468 4 years ago Riku Voipio Improve documentation Signed-off-by: Riku Voipio 7.8 KB Diff Download File
reload.sh 449 6 years ago Kelley Spoon Docker: add a script to reload gunicorn from outsi 96 bytes Diff Download File
requirements.txt 467 5 years ago Kelley Spoon textile: revert change to lockdown version of Text 178 bytes Diff Download File
settings.py 456 5 years ago Kelley Spoon Settings: import proxy_settings file if present C 10.5 KB Diff Download File
settings_production.py 454 6 years ago Kelley Spoon Settings: Move report csv to prevent overwrites in 1.1 KB Diff Download File
File unit-test.sh 397 8 years ago Ben Copeland unit-test: define python version Due to some OS'e 519 bytes Diff Download File
urls.py 439 6 years ago Benjamin Copeland URLS: Add missing view Our reports page is missin 4.5 KB Diff Download File
version.py 262 9 years ago Andy Doan bug #322: fix version information for snapshots T 642 bytes Diff Download File
wsgi_production.py 393 8 years ago Ben Copeland allowed_hosts: allow llp code to use a hosts file 776 bytes Diff Download File