~linaro-infrastructure/linaro-license-protection/master

  • Committer: Benjamin Copeland
  • Author(s): Kelley Spoon
  • Date: 2019-10-15 15:00:38 UTC
  • Revision ID: git-v1:453ce66dcac200d6124d4341ea580e044e533507
s3_purge: rewrite to use Bucket.list_versions()

The current s3_purge script uses the Bucket.list() method
to get a listing of all s3 objects to work on.  The list()
method unfortunately filters out files that have been marked
for deleted, and this creates a race condition where
s3_purge is unable to see the files in the list that it needs
to delete in a --forcedelete run.

This change uses the list_versions() method to get a list of
everything.  It groups keys by filename, and uses the first
key in the list (the lastest version of the file) to determine
when deciding what action to take.  This should allow for the
script to run slightly faster as it process the files as all
of its keys come in instead of waiting for the list() method
to complete.

This change drops the needs for the --clean-releases option
as the method for deleting files is working on the s3 object's
name and the scope of the operation should have been limited
by the --prefix option already (default 'snapshots/').

Change-Id: If30f0dd9c7a6155442ac87bf44ef2f609802bf3c
Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33011
Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>
Filename Latest Rev Last Changed Committer Comment Size
..
api 271 9 years ago Linaro Code Review Merge "API: refactor "uploads" into more common "a Diff
artifact 346 8 years ago Andy Doan break Artifact logic out of common.py Now that we Diff
management 111 11 years ago Stevan Radakovic Add setsuperuser command. Reviewed by danilo. Diff
migrations 267 9 years ago Andy Doan add support for DB migrations This is a bit of a Diff
static 86 11 years ago Danilo Segan Merge in django-refactoring. Diff
templatetags 296 9 years ago Andy Doan Remove need for two versions of header.html This Diff
tests 86 11 years ago Danilo Segan Merge in django-refactoring. Diff
__init__.py 84.1.2 11 years ago James Tunnicliffe Moved files out od django-rewrite. Combines .bzrig Empty Diff Download File
admin.py 289 9 years ago Andy Doan auto populate api key in admin panel I decided no 808 bytes Diff Download File
buildinfo.py 335 8 years ago Andy Doan buildinfo: remove need for search-path and full na 6.7 KB Diff Download File
common.py 427 7 years ago Andy Doan fix last commit The last commit caused a big issu 5.8 KB Diff Download File
config.py 464 5 years ago Benjamin Copeland Whitelist: Add cambridge lab IP Change-Id: I0704e 1.6 KB Diff Download File
context_processors.py 450 6 years ago Benjamin Copeland site_name: Set hostname using settings file Setti 196 bytes Diff Download File
group_auth_common.py 192.2.19 11 years ago Paul Sokolovsky Handle errors during group auth properly. 115 bytes Diff Download File
group_auth_crowd.py 236 10 years ago Milo Casagrande group_auth_crowd: Add log statement. * Log crow 1.8 KB Diff Download File
group_auth_openid.py 203 11 years ago Paul Sokolovsky Add warning which will help finding discrepancies 1 KB Diff Download File
models.py 445 6 years ago Benjamin Copeland License: Make Digest field unique In very rare ca 6 KB Diff Download File
OrderedDict.py 142.1.4 11 years ago Georgy Redkozubov Fixed pyflakes 'redefenition of unused import' err 8.7 KB Diff Download File
render_text_files.py 466 5 years ago Kelley Spoon textile: use utf-8 Assume source text is utf-8, i 4.9 KB Diff Download File
splice_build_infos.py 182.1.1 11 years ago Stevan Radakovic Fix problem with non found files in BUILD-INFO.txt 1.8 KB Diff Download File
views.py 469 4 years ago Benjamin Copeland views.py: check X_FORWARDED_FOR if set for whiteli 12.7 KB Diff Download File
wsgi.py 89.4.1 11 years ago Georgy Redkozubov Added lost wsgi.py and excepted hosts. 1.5 KB Diff Download File