~philroche/simplestreams/bionic-i386-ova-not-expected : changes

~philroche/simplestreams/bionic-i386-ova-not-expected » Changes from revision 458

From Revision 437 to 418

Rev	Summary	Authors	Date
437	Create chksum for LXD metadata+root for squashfs (LP: #15779... Create chksum for LXD metadata+root for squashfs (LP: #1577922) This adds a 'combined_sqashfs_sha256' element to items that have both a LXD metadata tar and a squashfs root. Additionally, we have 'combined_rootxz_sha256' which wil have the same value as 'combined_sha256' and allow for sunsetting that ambiguous name.	Robert C Jennings	7 years ago
436	GlanceMirror: fix a couple of race-related problems GlanceMirror: fix a couple of race-related problems First, what can happen is that .sync() is interrupted for external reasons (service restarts, network issues,...) after first image has been uploaded to Glance when syncing multiple images. On re-run, the uploaded image is considered "synced" already, but since simplestreams index is written out only at the end of the sync, all the metadata is lost. We solve this by adding "simplestreams_metadata" as one of the properties of image in Glance where we put a JSON representation of all the metadata fields from the original simplestreams index entry. We then load them in load_products() if they are defined. This results in a slightly different auto.sync.json entry: "endpoint" and "region" are on the product entry, instead of on the disk1.img entry. Secondly, if sync is indeed interrupted after one or several images are completely uploaded (but not all of them), there's no index file at all, so any attempt to make use of them with juju would fail, even though it should be fully functional. We solve this by calling "insert_products()" at the end of the insert_item(). This means that the index is regenerated after every single image is uploaded. Along the way, I slightly improve insert_item() tests from the pre-req branch to add a test for minimal data required, rename the existing one using real world data to test_insert_item_full, and re-use that data for a test for a newly added call to insert_products().	Scott Moser	7 years ago
435	GlanceMirror: refactor insert_item for easier testing GlanceMirror: refactor insert_item for easier testing This change refactors GlanceMirror.insert_item() to allow for easier and more contained testing. I needed to do this to understand everything that was going on inside insert_item and other bits of code. There are now four distinct things happening in it: 1. Download image to a local file from a ContentSource 2. Construct extra properties to store in Glance along with image 3. Prepare arguments for GlanceClient.images.create() call 4. Adapt source simplestreams entry for an image for use in the target simplestreams index It should be fully backwards compatible, and test coverage for all the individual steps should be much better (I admit to it not being perfect, but it's a step in the right direction, imho at least).	Scott Moser	7 years ago
434	tools/toolutil.py: early yakkety images did not have ova fil... tools/toolutil.py: early yakkety images did not have ova files. this code will soon be obsolete but for now just ignore the ova files on older builds of yakkety.	Scott Moser	8 years ago
433	glance: ignore inactive images glance: ignore inactive images If connection to glance is broken during GlanceMirror.sync(), and image would be left over in "status": "saving". Glance itself tries to protect against that, however when it is restarted (eg. "service glance-api restart"), it does not clean the image up. This change makes load_product() ignore any images that do not have "status": "active". Images can only have deleted, active or saving status. Note that this isn't perfect. The image in "saving" state is never cleaned up, but at least the resulting cloud works. If we want to clean up broken images, it'd be hard to do on the simplestreams side (glance is better placed to do that: it knows if anything is going on with any of them, eg. is it being currently updated or not).	Scott Moser	8 years ago
432	tools/make-test-data: sort keys when writing json, sort prod... tools/make-test-data: sort keys when writing json, sort product names This uses the simplestreams/util 'dump_data' to dump json data in each place that was writing json in tools/make-test-data. Also, it sorts the 'products' data in the index stream.	Scott Moser	8 years ago
431	support stream output for yakkety builds support stream output for yakkety builds For yakkety builds: - yakkety build output renames the -disk1.img file to .img but stream data should maintain -disk1.img as the ftype and item key. - -root.tar.gz and -root.tar.xz files are not produced - squashfs images are available (and also were available partially through xenial) As a cleanup, we a helper to ubuntu_versions to compare versions more directly. String compare was less useful in some cases and also would end up breaking after 'z' because whatever release comes after z would string sort to < z. This also adds a simple tool for comparing two trees output of 'make exdata'. ./tools/compare-test-data ../trunk.dist ./ will output a 'diff' from old to new list of files indexed.	Scott Moser	8 years ago
430	Optimise read_signed Optimise read_signed Use join instead of iteratively "appending" to a string. In my use case, this reduces signature check time from over 400 seconds to almost a tenth of a second.	Robie Basak	8 years ago
429	set a default user-agent set a default user-agent Follow on to the custom user agent merge by adding a default user agent.	Scott Moser	8 years ago
428	glance mirror: do not require that hypervisor_config be pres... glance mirror: do not require that hypervisor_config be present This just allows for hypervisor_config to not be present.	Scott Moser	8 years ago
427	Add user-agent support Add user-agent support This adds user-agent support. The library user may want to provide a specific user agent when querying a remote mirror. Because of the code structure, there's some not-very-nice code in here, mainly the use of url_reader_factory function, but without knowing what parts of simplestreams API we want to keep backward compatible, I refrained from changing anything of the existing code too much. There are tests for all the new code, and for some of the old code that was untested.	Scott Moser	8 years ago
426	Switch to using virt key for hypervisor type in stream data. Switch to using virt key for hypervisor type in stream data. In order to be consistent with other stream data that uses the 'virt' tag name (such as aws) the glance mirror should translate the underlying hypervisor_type for the glance image into an appropriate 'virt' value for the cloud. Also, change the following virt types lxc -> lxd qemu -> kvm	Scott Moser	8 years ago
425	Set product aliases without modifying release data dictionar... Set product aliases without modifying release data dictionary In r424 we added support for aliases. With that code it is possible that the daily devel products will lack the intended 'devel' alias. Aliases for a release/arch are set from toolutil.REL2VER[release] and if the stream is not daily the 'devel' item in the list of aliases is removed, however the code from that commit accidentally removed 'devel' from the source dictionary rather than creating a local copy first. If the devel release is evaluated before the devel daily then the 'devel' alias will not be present in the dictionary. This patch will copy the aliases from the release dictionary and modify the copy for the product being indexed.	Ben Howard	8 years ago
424	[MERGE] Added alias' to support LXD [MERGE] Added alias' to support LXD	Ben Howard	8 years ago
423	Enable better support for multi-hypervisor regions in OpenSt... Enable better support for multi-hypervisor regions in OpenStack 1) Determine correct disk format. 2) Set hypervisor_type on images based on ftype field.	Scott Moser	8 years ago
422	dump_data: dump json data with sorted keys dump_data: dump json data with sorted keys make dump_data use sorted keys for dictionaries. The desire is to reduce differences in output that would otherwise be the same.	Scott Moser	8 years ago
421	tools/make-test-data: Fix missing lxc/lxd files for ppc64el ... tools/make-test-data: Fix missing lxc/lxd files for ppc64el builds. We had a whitelist on ppc64el ever since the first ppc64el builds. Removing that whitelist means we will now get correct file information for all ppc64el builds.	Scott Moser	8 years ago
420	Fix TestUrlContentSourcea to test with requests, add build-d... Fix TestUrlContentSourcea to test with requests, add build-depends Fix the test case to use Requests, then fix the build failure that actually testing it caused. Then, support skipping the requests tests if it is not present.	Scott Moser	8 years ago
419	sanitize size parameter to read on Readers sanitize size parameter to read on Readers This fixes the read() calls in Urllib2UrlReader, RequestsUrlReader, and the new FileReader so that they all support: read(-1) read(None) read() and pass the correct value down to the python below it. Essentially, python3 prefers read(size=None) while python2 prefers read(size-1) if the size is passed explicitly rather than using read().	Scott Moser	8 years ago
418	fix lintian warnings, and apply pull back distro package bui... fix lintian warnings, and apply pull back distro package build changes debian/control: - bump standards version - add build-depends on python-mock, python-yamnl, python3-mock - add build-depends on dh-python per sbuild output recommendation debian/copyright: fix license string agplv3 to AGPL-3 debian/rules: set SS_REQUIRE_DISTRO_INFO = 0 during build	Scott Moser	8 years ago