~jcsackett/charmworld/bac-tag-constraints

« back to all changes in this revision

Viewing changes to charmworld/tests/test_models.py

  • Committer: Tarmac
  • Author(s): Brad Crittenden
  • Date: 2014-04-14 20:07:28 UTC
  • mfrom: (500.1.9 es-missing-data)
  • Revision ID: tarmac-20140414200728-ew38mtkdrsuprcff
Do not index charm annotations in bundles.

Charms in bundles have "annotations" which include information relating to the
widget layout when placed in the GUI ('gui-x' and 'gui-y').  This information
is ancilliary to the actual bundle and should not be searchable.

The presence of such data has also caused operational problems.  In
production, some bundles are not being indexed due to parsing errors of the x
and y coordinates.

An example failure is shown here:
2014-04-11 19:12:30,096 DEBUG [pyelasticsearch][MainThread] got response {u'items': [{u'index': {u'_type': u'bundle', u'_id': u'~charmers/muletrain/wiki', u'error': u'MapperParsingException[failed to parse [data.data.services.mediawiki.annotations.gui-y]]; nested: NumberFormatException[For input string: "-117.5"]; ', u'_index': u'charms-78512'}}], u'took': 4}

Unfortunately, this ingestion failure cannot be reproduced on staging or
locally.  Further, it appears pyelasticsearch is not returning an error in
this situation but is instead silently failing to index the bundle.

To QA the branch, in one window run 'make run'.  In another, ingest some
bundles:

% bin/ingest-queued --prefix="~charmers/charms/bundles"

Afterwards, go to 'localhost:2464' in your browser.  Search for 'bundles' and
see the ones that got ingested.  Next search for 'gui-x' and see no results.

https://codereview.appspot.com/87710043/

R=jcsackett.

Approved by Juju Gui Bot.

Show diffs side-by-side

added added

removed removed

Lines of Context:
1756
1756
                None, None, None, None)
1757
1757
        self.assertItemsEqual(['wordpress-stage', 'wordpress-prod'], keys)
1758
1758
 
 
1759
    class FauxElasticSearchClient:
 
1760
        index_bundles_called = False
 
1761
        called_args = None
 
1762
 
 
1763
        @classmethod
 
1764
        def from_settings(cls, *args):
 
1765
            return cls
 
1766
 
 
1767
        @classmethod
 
1768
        def index_bundles(cls, *args):
 
1769
            cls.index_bundles_called = True
 
1770
            cls.called_args = args
 
1771
 
1759
1772
    def test_storing_a_bundle_includes_indexing_it(self):
1760
 
        class FauxElasticSearchClient:
1761
 
            index_bundles_called = False
1762
 
 
1763
 
            @classmethod
1764
 
            def from_settings(cls, *args):
1765
 
                return cls
1766
 
 
1767
 
            @classmethod
1768
 
            def index_bundles(cls, *args):
1769
 
                cls.index_bundles_called = True
1770
 
 
1771
 
        with patch(
1772
 
                'charmworld.models.ElasticSearchClient',
1773
 
                FauxElasticSearchClient):
 
1773
        with patch('charmworld.models.ElasticSearchClient',
 
1774
                    TestStoreBundles.FauxElasticSearchClient):
1774
1775
            store_bundles(
1775
1776
                self.db.bundles, {}, 'owner', 'wordpress-basket/5', None,
1776
1777
                None, None, None, None)
1777
1778
 
1778
 
        self.assertTrue(FauxElasticSearchClient.index_bundles_called)
 
1779
        self.assertTrue(
 
1780
            TestStoreBundles.FauxElasticSearchClient.index_bundles_called)
 
1781
 
 
1782
    def test_indexing_a_bundle_removes_some_annotations(self):
 
1783
        deployer_config = dedent("""\
 
1784
            wordpress-prod:
 
1785
                series: precise
 
1786
                services:
 
1787
                    blog:
 
1788
                        charm: cs:precise/wordpress
 
1789
                        constraints: mem=5
 
1790
                        annotations:
 
1791
                            'gui-x': 1
 
1792
                            'gui-y': 2
 
1793
                        options:
 
1794
                            tuning: optimized
 
1795
                            engine: apache
 
1796
            wordpress-stage:
 
1797
                series: precise
 
1798
                services:
 
1799
                    blog:
 
1800
                        charm: cs:precise/wordpress
 
1801
                        constraints: mem=2
 
1802
                        options:
 
1803
                            tuning: optimized
 
1804
                            engine: apache
 
1805
                        annotations:
 
1806
                            'gui-x': 1
 
1807
                            'gui-y': 2
 
1808
                            'color': 'purple'
 
1809
        """)
 
1810
 
 
1811
        parsed = yaml.safe_load(deployer_config)
 
1812
        with patch('charmworld.models.ElasticSearchClient',
 
1813
                    TestStoreBundles.FauxElasticSearchClient):
 
1814
            store_bundles(
 
1815
                self.db.bundles, parsed, 'owner',
 
1816
                'wordpress-basket/5', None, None, None, None, None)
 
1817
        self.assertTrue(
 
1818
            TestStoreBundles.FauxElasticSearchClient.index_bundles_called)
 
1819
 
 
1820
        called_args = sorted(
 
1821
            TestStoreBundles.FauxElasticSearchClient.called_args[0],
 
1822
            key=lambda x: x['_id'])
 
1823
        for arg in called_args:
 
1824
            self.assertFalse(
 
1825
                'annotations' in arg['data']['services']['blog'])
1779
1826
 
1780
1827
 
1781
1828
class BundleTestCase(MongoTestBase):