~james-page/simplestreams/virt-openstack : changes

~james-page/simplestreams/virt-openstack » Changes from revision 429

From Revision 409 to 390

Rev	Summary	Authors	Date
409	Extract generate_simplestreams from tab2streams Extract generate_simplestreams from tab2streams Extract generate_simplestreams from tab2streams, so that it can be reused. Separating out FileNamer may seem strange, but Juju uses a distinct naming scheme for streams: 1. The proper index file is called index2.json (index.json is a cut-down version with only one content-id due to limitations of old jujus) 2. The content files are not named exactly after the content-ids. Instead, ':' is replaced with '-', because ':' is a special character on Windows.	Scott Moser	8 years ago
408	streams data: add 'support_eol' (end of life) streams data: add 'support_eol' (end of life) Each release will now have a 'support_eol' in the data.	Scott Moser	8 years ago
407	sstream-mirror: do not write status info to stdout sstream-mirror: do not write status info to stdout the dry run / report information was going to stdout. That seems wrong.	Scott Moser	8 years ago
406	Add JSON output format to sstream-query. Add JSON output format to sstream-query. adds a new JSON output format to sstream-query, as an alternative to --pretty or the default stringified dicts. Supporting JSON makes sense. It's well-known, expressive, and is also the native storage format of simplestreams. Juju QA are using JSON to generate streams, so --json lets us avoid a translation step.	Scott Moser	8 years ago
405	sstream-mirror: fix trace if .data is not present. sstream-mirror: fix trace if .data is not present. when reading from the mirror, don't bother reading the non '.data' path. As returning that as a product indicates that it has all been done.	Scott Moser	8 years ago
404	use gpgv if available use gpgv if available gpgv is much lighter than gpg, and as it is depended on by apt, its likely to be everywhere. We can use that specifically if we are given a keyring.	Scott Moser	8 years ago
403	tools: UEFI images are available for arm64. tools: UEFI images are available for arm64.	Scott Moser	8 years ago
402	tab2streams: skip lines starting with '#' tab2streams: skip lines starting with '#'	Scott Moser	8 years ago
401	export checksummer in simplestreams.util export checksummer in simplestreams.util When moving the checksumming things out of checksum_util, I added names for CHECKSUMS, item_checksums and ALGORITHMS, but failed to get checksummer. Simply export checksummer in util, pointing it to checksum_util.util	Scott Moser	8 years ago
400	provide insert_item with a contentsource that does checksumm... provide insert_item with a contentsource that does checksumming Previously, insert_item would receive a 'contentsource' (essentially a file like object to be read as a stream). The user of that object needed to calculate checksums and verify the data they read. Now, instead the contentsource will do checksumming as read() operations are done, and will raise a checksum error in any failure case. Thus to use this, the user now simply has to loop over reads and catch the exception. stream data is now expected to have valid checksums and size on all items with a path. If the user is using a stream source that does not have either size or checksum information, they have a few options: a.) [legacy/SRU only] set environment variable SS_MISSING_ITEM_CHECKSUM_BEHAVIOR can be set to silent: behave exactly as before. No checksumming is done, no warnings are emitted. The consumer of the contentsource must check checksums. warn: log messages at WARN level (same as default/unset) fail: the new behavior. raise an InvalidChecksum exception. b.) instantiate the BasicMirrorWriter with config checksumming_reader=False the default for that config setting is True, meaning that you will get a reader that checksums content as it goes and raises exception on bad data. c.) fix their source to have a sha256sum and a size The 'sstream-mirror' program now has a '--no-checksumming-reader' flag that does 'b' for this mirror.	Scott Moser	8 years ago
399	glance.py: fix KeyError due to typo glance.py: fix KeyError due to typo adding the os_version checked for existance of version in the data but then referenced 'os_version'.	Scott Moser	8 years ago
398	provide 'os_name' and 'os_version' to glance if available provide 'os_name' and 'os_version' to glance if available we recently added 'os' and 'version' strings to the stream data for Ubuntu cloud images. This takes those if they're there and adds them to the correctly named property fields in glance. See for more information: http://docs.openstack.org/cli-reference/content/ chapter_cli-glance-property.html	Scott Moser	8 years ago
397	Map ubuntu/dpkg arch strings to those that libvirt understan... Map ubuntu/dpkg arch strings to those that libvirt understands. Ensure all Ubuntu architectures get correctly mapped into ones that OpenStack and libvirt understand.	Scott Moser	8 years ago
396	make-test-data: support skipping unsupported releases make-test-data: support skipping unsupported releases This adds support for skipping the publication of data for releases that have reached end of life. We came to do this work when bad data on cloud-images.ubuntu.com (described in bug 1493395) meant that many referenced urls were not present. The general idea seems sane though, so we'll add this code. The behavior can be controlled by environment variable 'SS_SKIP_UNSUPPORTED' which defaults ot '0'. Setting to a value other than '0' will mean that the unsupported releases are simply skipped. Doing so means we must have distro-info (as it is how we get the supported flag for a release).	Scott Moser	8 years ago
395	require distro-info unless environment variable says otherwi... require distro-info unless environment variable says otherwise This is in a move to get 'supported' information consistently available. For development purposes you can override this setting with: SS_REQUIRE_DISTRO_INFO=0 The default is to require.	Scott Moser	8 years ago
394	tools/hook-check-downloads: fix typo in help output tools/hook-check-downloads: fix typo in help output	Scott Moser	8 years ago
393	add 'make-test-data' to run-pep8 and fix pep8 issues in it. add 'make-test-data' to run-pep8 and fix pep8 issues in it.	Scott Moser	8 years ago
392	make-test-data: add mechanism for skipping older builds make-test-data: add mechanism for skipping older builds since make-test-data crawls so much data from cloud-images.ubuntu.com it can be slow and painful on its first run. Once that data is obtained, its cached in exdata-query/FILE_DATA_CACHE, but its slow the first time. This lets you do something like: SS_DEBUG_MIN_SERIAL=20150101 make exdata and then any serial number < 20150101 will just be skipped.	Scott Moser	8 years ago
391	make-test-data: add 'version' back to download data make-test-data: add 'version' back to download data version was inadvertently dropped from download data in revision 387. It had stayed in the generated ec2 data. This adds 'new_ubuntu_product_dict' and uses it in both places to create the initial product data. That should ensure that we dont make download and ec2 different unintentionally in the future. This also will end up adding 'release_title' and 'release_codename' to the ec2 data. And also will add 'supported' for the case where python-distroinfo is available during the build (tools/ubuntu_versions.py only gets that data if using distro info so we are not expected to keep up to date information here).	Scott Moser	8 years ago
390	fix 'make pep8' on wily (1.6.2) fix 'make pep8' on wily (1.6.2) newer pep8 started complaining about these things.	Scott Moser	8 years ago