~jcsackett/charmworld/bac-tag-constraints

« back to all changes in this revision

Viewing changes to charmworld/models.py

  • Committer: Tarmac
  • Author(s): Brad Crittenden
  • Date: 2014-04-14 20:07:28 UTC
  • mfrom: (500.1.9 es-missing-data)
  • Revision ID: tarmac-20140414200728-ew38mtkdrsuprcff
Do not index charm annotations in bundles.

Charms in bundles have "annotations" which include information relating to the
widget layout when placed in the GUI ('gui-x' and 'gui-y').  This information
is ancilliary to the actual bundle and should not be searchable.

The presence of such data has also caused operational problems.  In
production, some bundles are not being indexed due to parsing errors of the x
and y coordinates.

An example failure is shown here:
2014-04-11 19:12:30,096 DEBUG [pyelasticsearch][MainThread] got response {u'items': [{u'index': {u'_type': u'bundle', u'_id': u'~charmers/muletrain/wiki', u'error': u'MapperParsingException[failed to parse [data.data.services.mediawiki.annotations.gui-y]]; nested: NumberFormatException[For input string: "-117.5"]; ', u'_index': u'charms-78512'}}], u'took': 4}

Unfortunately, this ingestion failure cannot be reproduced on staging or
locally.  Further, it appears pyelasticsearch is not returning an error in
this situation but is instead silently failing to index the bundle.

To QA the branch, in one window run 'make run'.  In another, ingest some
bundles:

% bin/ingest-queued --prefix="~charmers/charms/bundles"

Afterwards, go to 'localhost:2464' in your browser.  Search for 'bundles' and
see the ones that got ingested.  Next search for 'gui-x' and see no results.

https://codereview.appspot.com/87710043/

R=jcsackett.

Approved by Juju Gui Bot.

Show diffs side-by-side

added added

removed removed

Lines of Context:
1854
1854
        and eligible to be featured?
1855
1855
    """
1856
1856
 
 
1857
    to_be_removed = ('annotations', )
 
1858
 
1857
1859
    if index_client is None:
1858
1860
        index_client = ElasticSearchClient.from_settings(settings)
1859
1861
 
1862
1864
    for bundle_name in deployer_config:
1863
1865
        # Set up indexing.
1864
1866
        data = get_flattened_deployment(deployer_config, bundle_name)
1865
 
        index_data[bundle_name] = make_bundle_doc(
 
1867
        doc = make_bundle_doc(
1866
1868
            data, owner, basket_id, bundle_name, first_change, last_change,
1867
1869
            changes, branch_spec, promulgated, services_info)
1868
1870
        if collection is not None:
1869
 
            collection.save(index_data[bundle_name])
 
1871
            collection.save(doc)
 
1872
 
 
1873
        # Remove annotations so that they aren't indexed.
 
1874
        if doc is not None:
 
1875
            data = doc.get('data') or {}
 
1876
            services = data.get('services', {})
 
1877
            for name in services:
 
1878
                for tbr in to_be_removed:
 
1879
                    if tbr in services[name]:
 
1880
                        del services[name][tbr]
 
1881
 
 
1882
        index_data[bundle_name] = doc
1870
1883
 
1871
1884
    index_client.index_bundles(index_data.values())
1872
1885