~smoser/simplestreams/trunk.moved-to-git : changes

~smoser/simplestreams/trunk.moved-to-git » Changes from revision 464

From Revision 444 to 425

Rev	Summary	Authors	Date
444	tools/run-flake8: run on all python programs in tools and fi... tools/run-flake8: run on all python programs in tools and fix errors now tools/run-flake8 will run on all python programs in tools by checking the first line for 'python'. Then, fix all the reported errors.	Scott Moser	7 years ago
443	tools/run-flake8: fix trusty tox environment tools/run-flake8: fix trusty tox environment syntax was just bad.	Scott Moser	7 years ago
442	Makefile: use 'any' flake8 Makefile: use 'any' flake8 this adds 'any' to tools/run-flake8. It then supports './tools/run-flake8 any' working * trusty: run 'flake8' - python3-flake8 Build-Depends gets flake8 command installed (python3-flake8 Depends on python-flake8 which gives /usr/bin/flake8) - python3 -m flake8 does not work. - there is no 'flake8' package * xenial, yakkety: - python3-flake8 does not get 'flake8' package - python3 -m flake8 works. * tox: we want to use the environment 'python' It was needed in order to run with the command ('flake8') on trusty	Scott Moser	7 years ago
441	Makefile: use 'any' flake8 Makefile: use 'any' flake8 this adds 'any' to tools/run-flake8. It then supports './tools/run-flake8 any' working * trusty: run 'flake8' - python3-flake8 Build-Depends gets flake8 command installed (python3-flake8 Depends on python-flake8 which gives /usr/bin/flake8) - python3 -m flake8 does not work. - there is no 'flake8' package * xenial, yakkety: - python3-flake8 does not get 'flake8' package - python3 -m flake8 works. * tox: we want to use the environment 'python' It was needed in order to run with the command ('flake8') on trusty	Scott Moser	7 years ago
440	add running of tox. add running of tox. This primarily moves a lot of things around to enable tox. Some things here: * add ./tools/install-deps this allows easily installing dependencies to build or run tox (the openstack libraries require some c python extentions so you end up needing python-dev and such in order to run tox). * add tools/create-gpgdir and tools/sign-examples this just moves that logic out of Makefile to these places. * make tools/js2signed only re-create .sjson and .json.gpg files if the .json is newer. It supports '--force' to disable this logic. * move to using flake8 instead of pyflakes and pep8 (flake8 uses pyflakes and pep8) * debian/control drop python-glanceclient and python3-keystoneclient these aren't available in trusty and thus can't build if they are there. Also adjusted tests to skip these if not present. * split signjson_file out of toolutil as toolutil required distroinfo which is only needed by make testdata. * add a trusty-flake8 tox environment for testing versions of flake8, pep8, and pyflakes that are on trusty. This allows us to catch what would be trusty specific build failures in tox.	Scott Moser	7 years ago
439	Update default LTS alias to point to Xenial (lp:1606606 ) Update default LTS alias to point to Xenial (lp:1606606 ) Following release of 16.04.1 update simplestream's default LTS alias to point to Xenial instead of Trusty. LXD exposes the values for image aliases.	Robert C Jennings	7 years ago
438	json2streams: Accept items with no size. json2streams: Accept items with no size. dict_to_item automatically converts size from a string to an int. This has the side effect of making size mandatory. Size should not be mandatory for image-ids.	Scott Moser	7 years ago
437	Create chksum for LXD metadata+root for squashfs (LP: #15779... Create chksum for LXD metadata+root for squashfs (LP: #1577922) This adds a 'combined_sqashfs_sha256' element to items that have both a LXD metadata tar and a squashfs root. Additionally, we have 'combined_rootxz_sha256' which wil have the same value as 'combined_sha256' and allow for sunsetting that ambiguous name.	Robert C Jennings	7 years ago
436	GlanceMirror: fix a couple of race-related problems GlanceMirror: fix a couple of race-related problems First, what can happen is that .sync() is interrupted for external reasons (service restarts, network issues,...) after first image has been uploaded to Glance when syncing multiple images. On re-run, the uploaded image is considered "synced" already, but since simplestreams index is written out only at the end of the sync, all the metadata is lost. We solve this by adding "simplestreams_metadata" as one of the properties of image in Glance where we put a JSON representation of all the metadata fields from the original simplestreams index entry. We then load them in load_products() if they are defined. This results in a slightly different auto.sync.json entry: "endpoint" and "region" are on the product entry, instead of on the disk1.img entry. Secondly, if sync is indeed interrupted after one or several images are completely uploaded (but not all of them), there's no index file at all, so any attempt to make use of them with juju would fail, even though it should be fully functional. We solve this by calling "insert_products()" at the end of the insert_item(). This means that the index is regenerated after every single image is uploaded. Along the way, I slightly improve insert_item() tests from the pre-req branch to add a test for minimal data required, rename the existing one using real world data to test_insert_item_full, and re-use that data for a test for a newly added call to insert_products().	Scott Moser	7 years ago
435	GlanceMirror: refactor insert_item for easier testing GlanceMirror: refactor insert_item for easier testing This change refactors GlanceMirror.insert_item() to allow for easier and more contained testing. I needed to do this to understand everything that was going on inside insert_item and other bits of code. There are now four distinct things happening in it: 1. Download image to a local file from a ContentSource 2. Construct extra properties to store in Glance along with image 3. Prepare arguments for GlanceClient.images.create() call 4. Adapt source simplestreams entry for an image for use in the target simplestreams index It should be fully backwards compatible, and test coverage for all the individual steps should be much better (I admit to it not being perfect, but it's a step in the right direction, imho at least).	Scott Moser	7 years ago
434	tools/toolutil.py: early yakkety images did not have ova fil... tools/toolutil.py: early yakkety images did not have ova files. this code will soon be obsolete but for now just ignore the ova files on older builds of yakkety.	Scott Moser	8 years ago
433	glance: ignore inactive images glance: ignore inactive images If connection to glance is broken during GlanceMirror.sync(), and image would be left over in "status": "saving". Glance itself tries to protect against that, however when it is restarted (eg. "service glance-api restart"), it does not clean the image up. This change makes load_product() ignore any images that do not have "status": "active". Images can only have deleted, active or saving status. Note that this isn't perfect. The image in "saving" state is never cleaned up, but at least the resulting cloud works. If we want to clean up broken images, it'd be hard to do on the simplestreams side (glance is better placed to do that: it knows if anything is going on with any of them, eg. is it being currently updated or not).	Scott Moser	8 years ago
432	tools/make-test-data: sort keys when writing json, sort prod... tools/make-test-data: sort keys when writing json, sort product names This uses the simplestreams/util 'dump_data' to dump json data in each place that was writing json in tools/make-test-data. Also, it sorts the 'products' data in the index stream.	Scott Moser	8 years ago
431	support stream output for yakkety builds support stream output for yakkety builds For yakkety builds: - yakkety build output renames the -disk1.img file to .img but stream data should maintain -disk1.img as the ftype and item key. - -root.tar.gz and -root.tar.xz files are not produced - squashfs images are available (and also were available partially through xenial) As a cleanup, we a helper to ubuntu_versions to compare versions more directly. String compare was less useful in some cases and also would end up breaking after 'z' because whatever release comes after z would string sort to < z. This also adds a simple tool for comparing two trees output of 'make exdata'. ./tools/compare-test-data ../trunk.dist ./ will output a 'diff' from old to new list of files indexed.	Scott Moser	8 years ago
430	Optimise read_signed Optimise read_signed Use join instead of iteratively "appending" to a string. In my use case, this reduces signature check time from over 400 seconds to almost a tenth of a second.	Robie Basak	8 years ago
429	set a default user-agent set a default user-agent Follow on to the custom user agent merge by adding a default user agent.	Scott Moser	8 years ago
428	glance mirror: do not require that hypervisor_config be pres... glance mirror: do not require that hypervisor_config be present This just allows for hypervisor_config to not be present.	Scott Moser	8 years ago
427	Add user-agent support Add user-agent support This adds user-agent support. The library user may want to provide a specific user agent when querying a remote mirror. Because of the code structure, there's some not-very-nice code in here, mainly the use of url_reader_factory function, but without knowing what parts of simplestreams API we want to keep backward compatible, I refrained from changing anything of the existing code too much. There are tests for all the new code, and for some of the old code that was untested.	Scott Moser	8 years ago
426	Switch to using virt key for hypervisor type in stream data. Switch to using virt key for hypervisor type in stream data. In order to be consistent with other stream data that uses the 'virt' tag name (such as aws) the glance mirror should translate the underlying hypervisor_type for the glance image into an appropriate 'virt' value for the cloud. Also, change the following virt types lxc -> lxd qemu -> kvm	Scott Moser	8 years ago
425	Set product aliases without modifying release data dictionar... Set product aliases without modifying release data dictionary In r424 we added support for aliases. With that code it is possible that the daily devel products will lack the intended 'devel' alias. Aliases for a release/arch are set from toolutil.REL2VER[release] and if the stream is not daily the 'devel' item in the list of aliases is removed, however the code from that commit accidentally removed 'devel' from the source dictionary rather than creating a local copy first. If the devel release is evaluated before the devel daily then the 'devel' alias will not be present in the dictionary. This patch will copy the aliases from the release dictionary and modify the copy for the product being indexed.	Ben Howard	8 years ago