1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
|
-*- RST -*-
TODO
====
Reorganize this file.
Dependencies
============
* quicklisp
* cl-fad
* osicat
* lisp-unit
Some convention
===============
Definition of a `problem`:
A `problem` consists of the following components:
+ A *problem-code*, for internal use.
+ A human-readable *problem-name*.
+ An *input-dir* [in absolute or relative form?].
+ A *judge-program*.
+ A *time-limit* (default: 2.0 seconds).
+ A *mem-limit* (default: 2.0 megabytes)
Definition of a `solution`:
A `solution` consists of the following components:
+ The *problem-code* indicating which problem to test.
+ The *solution-file* in absolute path.
+ The *solution-output* in adsolute path.
+ The *language* which the solution uses (keyword).
+ The *solution-id*.
Definition of a `test-case`:
A `test-case` consists of the following components:
+ The *solution-id*.
+ The *problem-code*.
+ *from-test* and *to-test* numbers indicating the range of tests.
Other conventions:
* Each problem has a time limit for all tests.
* Each problem has a memory limit for all tests.
* Each problem doesn't need to know about its test cases. E.g. test
cases are stored in flat files and are automatically importing when
necessary.
* The test set of a problem is based on flat files. Hence, there's no
need to define a test information in the problem class.
* Ignore the unexisted test when testing.
* Each test set for a problem has to be packed into a .tar.bz2 file
and the tests has to be numbered consecutively.
- An input file has to be of the form <test-numeber>.in
- An output file has to be of the form <test-number>.out
- An answer file has to be of the form <test-number>.ans
* The `input_directory` passed to `run-test.sh` must end with a slash
``/``.
* The test output for each test must be of the form
`<solution-id>_<test-num>.out`.
Usage
=====
Initial setup:
+ Correct intepreter paths in `conf-interpreter.lisp`.
+ Correct problem database in `conf-problems.lisp`.
+ Correct the solution database in `conf-solutions.lisp`.
+ Custom the listening port in `conf-network.lisp`.
How it works?
=============
The testing process is described as follow:
* Initially, the judge system will read the configuration from 4
files:
- `conf-data-structures.lisp`
- `conf-interpreter.lisp`
- `conf-problems.lisp`
- `conf-solutions.lisp`
- `conf-network.lisp`
* It then setups the appropriate components for testing, including:
- Network: an IP and a listened port.
- Unix named pipe.
* Then, it initializes the `*solution-queue*` and `*result-queue*`
variables, set them to `nil`. Putting the system into an idle
state, ready for action.
* Whenever a problem is submitted, it will:
- Add the problem to `*problem-list*`.
- Add the problem to the database.
* Whenever a solution is submitted, it will:
- Be put into the `*solution-queue*`.
- For each solution in `*solution-queue*`, a process which does the
following things are created:
* Compile the solution, using the corresponding helper:
`*compile-<language>.sh*`
+ For compiled languages, the solution will be compiled as normal.
+ For interpreted languages, the solution wil be compiled into
bytecode with basic optimization options.
* Put the results of the compilation process into the appropriate
testing directory with the appropriate permission, default
`./jail/`.
* For each test, run the solution and immediately check for
+ *Memory limit*, halt the testing if a test failed.
+ *Time limit*, halt the testing if a test failed.
If no test failed for mem-limit of time-limit, the results of
every test case are combined into a list whose element is of the
form `(list test-number result)`. `result < 0` indicates that
there are some problems running the test:
+ `result = -1` time limit exceeded.
+ `result = -2` memory limit exceeded.
+ `result = 0` zero point.
If `result <= 0`, no more test case in the test set would be
performed.
* Report the result of the solution by putting it into the
`*result-queue*`. `*result-queue*` is then read via network of
named pipe, each time it is read, it is popped (proprosed data
structure: *multicast stream*)
TO-DO
=====
* Run each test case in a test set concurrently.
* Supporting problems which require more than
output-comparison-checking method.
* Proposed data structure for `*result-queue*` and `*solution-queue*`:
*broadcast-stream*.
* A user interface:
- Console (probably with cl-charms)
- GUI (probably with cl-gtk2)
|