1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
|
QA Kit
#######
'qakit' is a tool that presents information that might be useful to UEQA engineers.
How Do I install qakit?
=======================
1) You already have the source code - well done!
2) Create a python 2 virtualenv (sadly the trello API is python 2 only) and activate it::
virtualenv ve
source ve/bin/activate
3) Install qakit::
python setup.py install
4) Run the 'qakit' command.
How Do I run the tests?
=======================
As above, but run `python setup.py test` instead of installing it.
How do I use qakit?
===================
Running `qakit` will give output like so::
$ qakit
usage: qakit [-h] {silo-backlog,silo-stats} ...
qakit: error: too few arguments
You must specify a command. Currently the only commands are 'silo-backlog' and 'silo-stats'. More will be added in the future. Running one of these commands produces output:
$ qakit silo-backlog
Silo Backlog:
+---------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
| Name | Lane |
+---------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
| ubuntu-rtm/landing-010 - indicator-bluetooth,indicator-location,indicator-messages,indicator-network,indicator-power,indicator-transfer : tedg, charles | Need QA Sign-off |
| ubuntu-rtm/landing-014 - account-polld : ralsina | Need QA Sign-off |
| ubuntu-rtm/landing-028 - ubuntu-app-launch : tedg | Need QA Sign-off |
| ubuntu-rtm/landing-020 - mir,platform-api,qtmir,qtmir-gles,unity-system-compositor : camako | Need QA Sign-off |
| ubuntu-rtm/landing-003 - dialer-app,history-service,messaging-app,telepathy-ofono,telephony-service : bfiller | Need QA Sign-off |
| ubuntu-rtm/landing-015 - unity-scope-click : alecu | Need QA Sign-off |
| ubuntu-rtm/landing-007 - qtbase-opensource-src,qtbase-opensource-src-gles : Mirv | Need QA Sign-off |
| ubuntu-rtm/landing-009 - ubuntu-touch-session : tedg | Need QA Sign-off |
+---------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
How do I process appstartup logs
================================
Environment configuration
-------------------------
Install pip by doing: "sudo apt-get install python-pip python-dev build-essential"
Download and Install eve (http://python-eve.org/install.html)
Run eve by doing python qakit/appstartup/eve/run.py (you can define the env variables, see in file settings.py)
Download and install mongodb (https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/)
Create a user for mongo and remember to update the eve settings file
To create the mongo user type: "mongo" and then
db.createUser(
{
user: "ubuntuUser",
pwd: "ubuntuPassword",
roles: [ { role: "readWrite", db: "test" } ]
}
)
Note: eve code is ready to run with python 2.7 and 3.X
Add to the config the following:
[PERF_GENERIC]
EVE_HOST = [dest_ip]
EVE_PORT = [dest_port]
JSON_REPORT = True
BUILDS_TO_PLOT = 10
[PERF_SPECIFIC]
NAMES = webbrowser, address_book, calculator, here, dialer, clock, messaging, ebay, camera, system_settings, music, gallery
TYPES = cold, hot
TITLES = Cold Start, Hot Start
COLORS = 3366FF, FF1A00
ECOLORS = FF1A00, 3366FF
Having a test-results.subunit file, run as following: python3 qakit/appstartup/orchestrator.py pathto/test-results.subunit
HTML and json reports are generated in the subunit directory. All the pictures will be stored in the appstartup_data dir.
Autodetection
=============
Crud
----
First step is to create autodetection settings in the database using the following commands.
list: Used to see all the settings created
python3 qakit/perf/autodetect/crud.py list
add: Used to create settings in the db
python3 qakit/perf/autodetect/crud.py add --id default_performance --email sergio.cazzolato@canonical.com --class performance --detect "{'missing':True, 'out_of_range':0.2, 'out_of_trend':True}"
delete: Used to delete settings from the db
python3 qakit/perf/autodetect/crud.py delete --id default_performance
Detect
------
Then the detection step is executed based on the settings already saved in the db.
python3 qakit/perf/autodetect/detection.py detect --id default_performance --input_report ./performance_report.json --output_report ./performance_errors.json
The input report is the path to the report generated by running the orchestration.
As result the json report will contain all the errors detected. This reports can be included in the dashboard to be part of the results.
Reporting
=============
CSV
---
It is possible to generate a csv report filtering data from the mongo db.
See an example:
python3 qakit/perf/reporting/reporting.py -c performance -t calls -n javascript -bf 351 -bt 355 -o test.csv
The available filters are:
[-c RCLASS]
[-t TYPE]
[-n NAME]
[-st TEST]
[-l LOAD]
[-d DEVICE]
[-ch CHANNEL]
[-bf BUILD_FROM]
[-bt BUILD_TO]
[-o OUTPUT]
Note: consider that a big query could affect the mongo db due to the number of records retrieved.
|