~linaro-infrastructure/linaro-license-protection/master : changes

~linaro-infrastructure/linaro-license-protection/master » Changes from revision 522

From Revision 476 to 457

Rev	Summary	Authors	Date
476	s3_flatten: fix prefix default s3_flatten: fix prefix default The default prefix was set to "snapshots/aarch64laptops" as part of testing out the script. Let's set that properly to "snapshots/" Change-Id: I84521c4a43497a1911eb4df743a35d57e8062740 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33581 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>	Kelley Spoon	4 years ago
475	s3_flatten: break bucket listing up by delimiters s3_flatten: break bucket listing up by delimiters Boto's bucket.list_version() function has the option to specify a delimiter for the key names that are returned. Delimiters will treat any occurance of the character as if it was a new directory and will not "descend" into it. This change uses delimiters to traverse through all s3 file keys in the given --prefix, and will clean up the objects table before descending into a subdirectory in order to prevent wasting memory storing s3 key data for objects that are no longer going to be accessed. Change-Id: I294937c99c60e89a87439789b9031f50e2e5cefc Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33359 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Kelley Spoon	4 years ago
474	s3_flatten: add s3_flatten script s3_flatten: add s3_flatten script This script is intended to replace the s3_purge command. It simplifies the logic by breaking the clean up process into 3 parts: 1 - "flatten" all file objects to have no more than 1 version object and optionally 1 deletemarker regardless of prefix 2 - delete any expired files from prefixes that have not been excluded via S3_PURGE_EXCLUDES 3 - place deletemarkers on files that are older than MARK_DAYS that have not been excluded via S3_PURGE_EXCLUDES The other difference with this script is that it attempts to determine what objects to delete as it iterates through the versioned bucket listing, and will collect multiple delete operations into a single multidelete request. Change-Id: I25a227f574504bb4e828f29670aec366e87952c9 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33268 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Kelley Spoon	4 years ago
473	Docker: Fix broken production command Docker: Fix broken production command We need to mount the secrets.py file for the "S3" mode to work. Change-Id: I78817f961200f8e171d6f8587625b80473f5ca19 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33094 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>	Benjamin Copeland	4 years ago
472	s3_purge: rewrite to use Bucket.list_versions() s3_purge: rewrite to use Bucket.list_versions() The current s3_purge script uses the Bucket.list() method to get a listing of all s3 objects to work on. The list() method unfortunately filters out files that have been marked for deleted, and this creates a race condition where s3_purge is unable to see the files in the list that it needs to delete in a --forcedelete run. This change uses the list_versions() method to get a list of everything. It groups keys by filename, and uses the first key in the list (the lastest version of the file) to determine when deciding what action to take. This should allow for the script to run slightly faster as it process the files as all of its keys come in instead of waiting for the list() method to complete. This change drops the needs for the --clean-releases option as the method for deleting files is working on the s3 object's name and the scope of the operation should have been limited by the --prefix option already (default 'snapshots/'). Change-Id: If30f0dd9c7a6155442ac87bf44ef2f609802bf3c Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/33011 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Kelley Spoon	4 years ago
471	s3_purge: enable delete_key to set delete marker s3_purge: enable delete_key to set delete marker Previously the s3_purge script had the line to delete an object commented out. Since the publishing buckets are using versioning, this seems to have prevented setting the DeleteMarker on a file object. As a result, a check in handle_permanent_deletes() never removed anything since nothing had DeleteMarker set, and old versions of files were allowed to stack up. Change-Id: I34524d3b0b7776d9d41603d8e0e9f34ef1b526ec Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/32999 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>	Kelley Spoon	4 years ago
470	Token_cleanup: Add a token cleanup script Token_cleanup: Add a token cleanup script We have a large amount of expired API tokens sitting the database, this script will run as a crontab and clean them up. Change-Id: Ibbae225fe45f69956bc40bddcd7d76cf69cd6aff Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/32892 Reviewed-by: Riku Voipio <riku.voipio@linaro.org>	Benjamin Copeland	4 years ago
469	views.py: check X_FORWARDED_FOR if set for whitelisted IP views.py: check X_FORWARDED_FOR if set for whitelisted IP If run behind a proxy, LLP will always see a request as coming from the proxy's IP address. This change updates the IP check to use the X_FORWARDED_FOR header to check the client's IP for whitelisting internal hosts. Change-Id: I1f220301d7ac8658141407d930f255fc0d4f2e13 Reviewed-on: https://review.linaro.org/c/infrastructure/linaro-license-protection/+/31458 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Kelley Spoon	4 years ago
468	Improve documentation Improve documentation Signed-off-by: Riku Voipio <riku.voipio@linaro.org> Change-Id: I9219c7f8b0fb3ef555fe8a51cb01129e367710cf Reviewed-on: https://review.linaro.org/c/31829 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>	Riku Voipio	4 years ago
467	textile: revert change to lockdown version of Textile given textile: revert change to lockdown version of Textile given Riku's codecs solution. This reverts commit b1b6dd426f114e7f3c2d385ddca751fd6dc39084. Change-Id: I1b8adb2fe3fccf792d76f3a7146ce11830166d50 Reviewed-on: https://review.linaro.org/c/31476 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>	Kelley Spoon	4 years ago
466	textile: use utf-8 textile: use utf-8 Assume source text is utf-8, it's 2019. Should fix LSS-604. Same codecs.decode() trick fixes the validation script on the ticket, but I can't reproduce the problem with llp unit-tests. Signed-off-by: Riku Voipio <riku.voipio@linaro.org> Change-Id: Ia41a15630a71a8c2a982d797c4738a6af4bf594c Reviewed-on: https://review.linaro.org/c/31474 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Riku Voipio	4 years ago
465	Dockerfile: Update dockerfile docs Dockerfile: Update dockerfile docs Missing port needs to be added and make documentation clearer. Change-Id: I85b1418dc5a333e354b7b136b0deb541c3be8963 Reviewed-on: https://review.linaro.org/c/31353 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Benjamin Copeland	4 years ago
464	Whitelist: Add cambridge lab IP Whitelist: Add cambridge lab IP Change-Id: I0704e9e99b66d58651cee22b6e1af2feda306bf5 Reviewed-on: https://review.linaro.org/c/31269 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org>	Benjamin Copeland	5 years ago
463	textile: lock to known working version textile: lock to known working version More recent versions of the textile library have a problem processing the unicode present in some of our textile templates. In order to temporarily avoid having to find and update affected files this change pins the version of textile to a version that is known to work. Change-Id: Ide2d6f45cb0fff3054afb1be49c4965b2a749a9a Reviewed-on: https://review.linaro.org/c/31194 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Kelley Spoon	5 years ago
462	Django: Update to latest LTS Django: Update to latest LTS Due to the latest CVE-2019-6975 ( https://security.archlinux.org/CVE-2019-6975 ) lets update to latest version of django Change-Id: Ied909078b46d024779c29f5a8784476336f70f4b Reviewed-on: https://review.linaro.org/30292 Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Benjamin Copeland	5 years ago
461	Update text about Auth-Groups Update text about Auth-Groups * README: - Explain what groups should be used for Auth-Groups. Change-Id: Ic7b7e1dc3e36f8dd287f4dea94c1ad49f3b2f8fe Signed-off-by: Luis Machado <luis.machado@linaro.org> Reviewed-on: https://review.linaro.org/29380 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Luis Machado	5 years ago
460	s3_purge: fix the releases deletemarker bug and code cleanup s3_purge: fix the releases deletemarker bug and code cleanup When testing this the releases/ bucket wasn't checking that the deletemarker was being set. Obviously this is very dangerous so a fix has been put in to ensure a deletemarker is set before deleting the file. Since we handle the same call a couple of times, I have put handle_permanent_deletes, into its own function which has reduced a lot of reputation. Along with those fix, the code has been cleaned up to be a bit more readable Change-Id: Iffd2f55d04bc889050bfd7a19e6f62fe205de0c6 Reviewed-on: https://review.linaro.org/29085 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Benjamin Copeland	5 years ago
459	s3_purge: Delete S3 versioned objects s3_purge: Delete S3 versioned objects Our bucket has grown too big over time, this change will allow S3 objects to be fully deleted from S3. The script is split into two ways (ish): 1. No options will mark the objects that are over --markdays as deleted. This is only a version marker in S3 and as such will now delete the file. 2. --forcedelete is used to permanently delete that marked file. This is only ran within the deletedays period. 2a. --cleanup_releases. This cleans up all marked objects in the releases/ prefix. This needs to be used with --forcedelete since it will permanently delete objects. Since we don't have a life time policy for releases/ this is only ever going to delete files that are older than deletedays. Change-Id: I438526978c186f96af4ee689bd0eb2340bc94c01 Reviewed-on: https://review.linaro.org/28796 Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Benjamin Copeland	5 years ago
458	Whitelist: Add lkft.validation.linaro.org Whitelist: Add lkft.validation.linaro.org Change-Id: I4308a9c9feb5ca1ae7ec4064f9cfbbc89d8c8657 Reviewed-on: https://review.linaro.org/28709 Reviewed-by: Fathi Boudra <fathi.boudra@linaro.org>	Benjamin Copeland	5 years ago
457	Whitelist: Update whitelist for ARM Whitelist: Update whitelist for ARM Change-Id: Id0d88667e8db2254aa2c6107c25ea14ee54837c8 Reviewed-on: https://review.linaro.org/28015 Reviewed-by: Julien Duraj <julien.duraj@linaro.org> Reviewed-by: Kelley Spoon <kelley.spoon@linaro.org> Reviewed-by: Benjamin Copeland <ben.copeland@linaro.org>	Benjamin Copeland	5 years ago