~bac/charmworld/beaker-namespace : changes

~bac/charmworld/beaker-namespace » Changes from revision 513

From Revision 513 to 494

Rev	Summary	Authors	Date
513	Merge from trunk Merge from trunk	Brad Crittenden	9 years ago
512	Remove beaker:mongo stanza from setup.py Remove beaker:mongo stanza from setup.py	Brad Crittenden	9 years ago
511	Allow local charm ingestion. Allow local charm ingestion. To bin/ingest-queued the new option --local-repo has been added. Give the path to a parent directory and all subdirectories that are Bazaar branches will be ingested as charms. Fixes bug 1319140 where the --debug option wasn't declared in the correct place, causing bin/enqueue to die. While the work of that script is now a part of bin/ingest-queued it still has utility as a free-standing script and is kept. https://codereview.appspot.com/98280043/ R=jcsackett. Fixes: https://bugs.launchpad.net/bugs/1319140. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
510	Fixes a destructive test in Bug #1317567. Fixes: https://bug... Fixes a destructive test in Bug #1317567. Fixes: https://bugs.launchpad.net/bugs/1317567. Approved by Juju Gui Bot, Brad Crittenden.	Reed O'Brien	9 years ago
509	adds ngrams as multifield index to name. skips destructive t... adds ngrams as multifield index to name. skips destructive test -- see Bug #1317567. Approved by Juju Gui Bot, Richard Harding, j.c.sackett.	Reed O'Brien	9 years ago
508	Include annotations in bundle search results. Include annotations in bundle search results. The previous attempt at this problem was flawed in that API call for getting bundles directly pulls from Mongo while search?text= and search/interesting get results from ElasticSearch. QA: Load up some bundles with annotations: bin/ingest-queued --prefix="~charmers/charms/precise/ntp" bin/ingest-queued --prefix="~bac" Change juju-gui to point to locally running charmworld: diff --git a/app/config-debug.js b/app/config-debug.js index 39daca6..f2c86b6 100644 --- a/app/config-debug.js +++ b/app/config-debug.js @@ -32,7 +32,10 @@ var juju_config = { // These are the main application config items used and passed down into all // SubApps. consoleEnabled: true, + //charmworldURL: 'https://manage.jujucharms.com/', + charmworldURL: 'http://127.0.0.1:2464/', Run charmworld and juju-gui. In the GUI wait for the initial results to load. These are the Featured charms and bundles. Select the rails-scalable bundle. Note the layout in the inspector. Close the inspector and then drag the bundle (using the name not the icon) to the canvas. Note that the layout is the same as the preview. https://codereview.appspot.com/91140045/ R=jcsackett. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
507	Add in annotations from mongo when returning search results ... Add in annotations from mongo when returning search results from the API search. Approved by Juju Gui Bot, j.c.sackett.	Brad Crittenden	9 years ago
506	Handle old bundles in production. Handle old bundles in production. Some bundles in production do not have promulgated flags. Process them anyway. https://codereview.appspot.com/90420044/ R=bac. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
505	Fix migration wrt invalid constraints. Fix migration wrt invalid constraints. The migration 026 for rewriting contraints to be space-separated worked for staging. When run on production it had errors due to some of the bundles on production being older than the more restrictive bundle proofing requirements. Those bundles were in essence grandfathered in and never subject to the stricter rules. At least one bundle had an obsolete 'cpu' constraint and that caused the migration to fail since it was not checking for the thrown ValueError in this situation. The migration now checks for ValueError and will remove the affected bundle from the target database and proceed. The functions are now noted to throw ValueError. Test was added for this situation and logging checked. https://codereview.appspot.com/90420043/ R=jcsackett. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
504	Fix 026 exodus. Fix 026 exodus. A basket, the thing ingested from Launchpad, contains one or more bundles. If there are multiple bundles they may establish an inheritance relationship between the bundles in that basket. The inheritance is done before the basket is split into individual bundles and stored -- flattened in other words. The 'inherits' line is not removed, however. When the exodus happens, the re-use of 'store_bundles' attempts to flatten again, but the relationship to the other bundle is invalid as the 'basket' now just contains the one pre-processed bundle. The fix for this exodus is to simply manually delete the 'inherits' key from the bundle_data before proceeding. The exodus has been tested against a full ingest of all charms and bundles. The unit test has been modified to include an 'inherits' key in the data, which is then ignored. https://codereview.appspot.com/90300043/ R=jcsackett. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
503	Exodus to remove comma-sep constraints. Exodus to remove comma-sep constraints. Comma-separated constraints in bundles are deprecated. Currently newly processed bundles are converted to space-separated on ingestion but there are a number of old bundles in the db that need fixing. As this is an exodus, and only one can be present at a time, the 025 exodus is deleted. https://codereview.appspot.com/89770044/ R=jcsackett. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
502	Fix spurious error. Fix spurious error. All fixes are: Fix spurious test failure when version==0. Print bundle revid in debug message. Handle HTTPErrors from get_branch_tips and ValueErrors from parse_branch. Move implicit parse_branch test from test_lp to an explicit one in test_utils. Add Cache-Control to Launchpad API call when getting branch tips. Also use correct distribution name rather than alias. https://codereview.appspot.com/88980043/ R=sinzui. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
501	Do not index charm annotations in bundles. Do not index charm annotations in bundles. Charms in bundles have "annotations" which include information relating to the widget layout when placed in the GUI ('gui-x' and 'gui-y'). This information is ancilliary to the actual bundle and should not be searchable. The presence of such data has also caused operational problems. In production, some bundles are not being indexed due to parsing errors of the x and y coordinates. An example failure is shown here: 2014-04-11 19:12:30,096 DEBUG [pyelasticsearch][MainThread] got response {u'items': [{u'index': {u'_type': u'bundle', u'_id': u'~charmers/muletrain/wiki', u'error': u'MapperParsingException[failed to parse [data.data.services.mediawiki.annotations.gui-y]]; nested: NumberFormatException[For input string: "-117.5"]; ', u'_index': u'charms-78512'}}], u'took': 4} Unfortunately, this ingestion failure cannot be reproduced on staging or locally. Further, it appears pyelasticsearch is not returning an error in this situation but is instead silently failing to index the bundle. To QA the branch, in one window run 'make run'. In another, ingest some bundles: % bin/ingest-queued --prefix="~charmers/charms/bundles" Afterwards, go to 'localhost:2464' in your browser. Search for 'bundles' and see the ones that got ingested. Next search for 'gui-x' and see no results. https://codereview.appspot.com/87710043/ R=jcsackett. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
500	Bundle proof errors if juju-gui included. Bundle proof errors if juju-gui included. Updated error message from review but failed to land in previous commit. https://codereview.appspot.com/83270048/ R=hatch, juju-gui-bot. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
499	Bundle proof errors if juju-gui included. Bundle proof errors if juju-gui included. The juju-gui should not be a part of a bundle, though the juju-gui export mechanism has historically included it when doing the export. Because it has been in the exports, and most bundles start out as an export that then gets tuned, a lot of existing bundles have juju-gui in them. https://codereview.appspot.com/83270048/ R=hatch. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
498	Add a 'maintainer's field. Add a 'maintainer's field. It was decided on the mailing list that the 'maintainer' field should be renamed to 'maintainers' to make it clear that there can be more than one. Note the old 'maintainer' field was capable of being a single entity or a list just fine. So for backward compatability, 'maintainer' is preserved, 'maintainers' is added and the actual value used is the union of the two, with duplicates removed. On the charm page the heading has been changed to 'Maintainer(s)" and an unordered list is used to present the names. https://codereview.appspot.com/83620043/ R=rharding. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
497	Don't display 'Feature' for unpromulgated bundles. Don't display 'Feature' for unpromulgated bundles. https://codereview.appspot.com/78940044/ R=hatch. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
496	Add to GUI the ability to delete charms & bundles. Add to GUI the ability to delete charms & bundles. On the charm detail page, a 'Delete' link is added if the user has edit permission. On the bundle detail page, a new 'Actions' section is added that has 'Feature' and 'Delete' links. We talked about removing the charmworld/scripts/remove_charm.py file but that isn't done in this branch. QA: ingest some charms and bundles. While not logged in, visit the charm and bundle pages and notice you don't see the 'Delete' links. Log in. Now see the links. Click 'Delete' then cancel or submit without checking the box. Note the charm isn't deleted. Now delete it for real. Note that it is gone. Repeat for bundles. Try featuring a bundle. Late breaking thought: deleting a charm that a bundle depends upon may cause it to be undeployable. Refuse to delete a charm if a bundle depends on it or just leave it be? Note this is no worse than the remove_charm.py script, though it is easier to do. https://codereview.appspot.com/78220043/ R=benji, makyo, rharding. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
495	Improve pymongo connection management. Improve pymongo connection management. Despite code in MongoTestBase to try to intelligently manage mongo connections, sockets were being created and left open at a rate of about one per test. A default system has a limit of 1024 open connections. As our test suite exceeded that number failures were seen. Adding a call to 'conn.disconnect' in tearDown did not help. Research showed that we are using the deprecated call pymongo.Connection() to create our connections. The preferred connection class is now pymongo.MongoClient. This branch converts all connection instantiation to use the new method. By itself, this change was not very helpful. But, when the call to 'disconnect' was added, the result was better. Now the full suite uses about 250 open descriptors. It is unclear why are all of these sockets are still open. Some are shown to be ESTABLISHED while many are in CLOSE_WAIT. QA: Run the tests in one window. In another, find the pid of the test runner. % ps auxww \|grep nosetest Then, keep an eye on the number of open file descriptors with % watch 'lsof -p 39007 -Ff \| tail -10' Be careful as the results will be erased as soon as the tests finish. You should see something less than 300. https://codereview.appspot.com/77700043/ R=bac, rharding. Approved by Juju Gui Bot.	Brad Crittenden	9 years ago
494	Update scoring to be limited on ES side. Update scoring to be limited on ES side. - Drive-by to make sure we log the exception from ES for debugging purposes. Approved by Juju Gui Bot, Richard Harding, Benji York.	Rick Harding	9 years ago

Older »