~ubuntu-core-dev/extract-changelogs/trunk : changes

~ubuntu-core-dev/extract-changelogs/trunk » Changes from revision 49

From Revision 49 to 30

Rev	Summary	Authors	Date
49	[cjwatson, r=gmason] Use archive.getPublishedSources(order_b... [cjwatson, r=gmason] Use archive.getPublishedSources(order_by_date=True) for a significant speedup. The query that extract-changelogs is currently relying on is very slow, and there are some subtle ways in which iterating over the collection can go wrong. For ddeb-retriever, we did a fair bit of work on this: https://bugs.launchpad.net/launchpad/+bug/1441729 https://code.launchpad.net/~cjwatson/launchpad/db-index-bpph-datecreated/+merge/255539 https://code.launchpad.net/~cjwatson/launchpad/getpublishedbinaries-sorting/+merge/255822 In the case of extract-changelogs, it should be sufficient to add order_by_date=True, which has the effect of joining fewer tables and using a reasonably well-indexed query to return a collection which is in decreasing ID order. If the collection changes during iteration (as long as you don't try to do any status filtering or similar, as explained in a comment here) then the worst case is that you get the same source package more than once, but extract-changelogs already handles this in LaunchpadChangelogsCrawler._unpack_changelogs_to_target. Please do test this! I have not done so. However, I hear that extract-changelogs times out when asked to work from a very old starting date, and this should make it behave a lot better. Review: moon127 ran this successfully. After discussion, it looks safe to merge this with IS superpowers.	Greg Mason	7 years ago
48	merge mvo's r41 which set a socket timeout to avoid hanging ... merge mvo's r41 which set a socket timeout to avoid hanging for days	Brian Murray	8 years ago
47	add date, process id to log file. explictly close the apt lo... add date, process id to log file. explictly close the apt lock.	Brian Murray	8 years ago
46	log how many symlinks we created log how many symlinks we created	Brian Murray	8 years ago
45	move symlink creation into its own function and try to creat... move symlink creation into its own function and try to create the symlink if even if the changelog has been extracted.	Brian Murray	8 years ago
44	Apply local changes from production Apply local changes from production	Stéphane Graber	10 years ago
43	Change the binary symlink logic to create symlinks under Change the binary symlink logic to create symlinks under binary/<poolhash>/<binary>/<version>	Stéphane Graber	10 years ago
42	Use urllib2 to download the files to avoid a weird urllib/la... Use urllib2 to download the files to avoid a weird urllib/launchpadlib interaction preventing me from downloading the files on my machine.	Stéphane Graber	10 years ago
41	Cleanup the existing code to pass pyflakes, pyflakes3 and pe... Cleanup the existing code to pass pyflakes, pyflakes3 and pep8	Stéphane Graber	10 years ago
40	lp-extract-changelogs.py: do not append "/" to dest lp-extract-changelogs.py: do not append "/" to dest	Michael Vogt	12 years ago
39	add debug option add debug option	Michael Vogt	12 years ago
38	add lockfile add lockfile	Michael Vogt	12 years ago
37	lp-extract-changelogs.py: deal with broken symlinks here lp-extract-changelogs.py: deal with broken symlinks here	Michael Vogt	12 years ago
36	lp-extract-changelogs.py: kill broken symlinks when unpackin... lp-extract-changelogs.py: kill broken symlinks when unpacking	Michael Vogt	12 years ago
35	lp-extract-changelogs.py: put back EDGE_SERVICE_ROOT as it a... lp-extract-changelogs.py: put back EDGE_SERVICE_ROOT as it appears older launchpadlibs need it	Michael Vogt	13 years ago
34	get-published-since.py: use anoymous api get-published-since.py: use anoymous api	Michael Vogt	13 years ago
33	lp-extract-changelogs.py: do not limit the query to "Publish... lp-extract-changelogs.py: do not limit the query to "Published" sources	Michael Vogt	13 years ago
32	lp-extract-changelogs.py: change substract of time to 2h ins... lp-extract-changelogs.py: change substract of time to 2h instead of 1h as it appears we are still missing packages (#685814, #372270)	Michael Vogt	13 years ago
31	get-published-since.py: minimal example for debugging get-published-since.py: minimal example for debugging	Michael Vogt	13 years ago
30	add script to initially populate the changelogs data add script to initially populate the changelogs data	Michael Vogt	13 years ago

Older »