-
Notifications
You must be signed in to change notification settings - Fork 81
[EN] 3. Package with a model solution
Generating output files without a model solution often turns out to be a rather difficult task. Fortunately, if we happen to own a program capable of generating model output, the SIO2 judge system can automatically generate output files based on your solution and inputs included in the package. It will also help you a lot if you need to modify your tests multiple times or alter the problem's conditions – you will be sure that your outputs are always complete and up-to-date.
This time we'll prepare a package for the problem Tree, to which we will naturally give tre as a short name. The user will be given a single natural number denoting a size of a Christmas tree. Their solution should output an ASCII representation of such a tree. The full statement can be found at tre1/doc/trezad.pdf. Note that if the tree's size given is 0, the user is asked to print "Too small to exist" rather than provide an empty output.
The complete package, which will be discussed in detail below, can be downloaded here.
The judge system treats whitespace characters in a very specific way. By default, any sequence of consecutive spaces and tabulations will be seen as no different than a single space. Thus, these two files:
1 1
1 1
will be judged equivalent. However, this file:
11
will be considered different, as the system will see a single number instead of two separate ones. The judge system will also skip all spaces and tabs at the end of each line and allow any number of whitespace characters at the end of the file.
Therefore, in order to force the user to adhere to the desired output format, instead of asking them to output some characters over white background (e.g. Christmas tree – asterisks, background – spaces), we will ask them to output the Christmas tree over a non-empty background. In our case we will combine hashes # (for the tree) and dots . (for the background). Of course, it IS possible to employ whitespace characters for printing a picture but then we wouldn't be able to compare the outputs properly. That is, not without using an output verifying program (a.k.a. output checker), which will be covered in the next tutorial.
In order to generate tests we will need a model solution. It can be implemented in any accepted programming language. Your model solution should be especially careful concerning whitespace characters (no line should end with a space, the output file should end with an empty line). Note that if it's done correctly, your contestants won't need to worry about any of this. For the sake of this tutorial we have prepared model solutions in C++ and Python. The model solution which will be used to generate the tests should be placed in the prog folder and named with the problem's short name (+ the file's extension, cf. the examples provided below). Additional model solutions (e.g. solving the problem differently or implemented in different programming languages) should have consecutive numbers follow the problem's short name (a common practice is to count from 2 onwards). Here's a model solution in C++ (the one used for output generation) tre/prog/tre.cpp, and here one in Python (an additional one) tre/prog/tre2.py.
Let's assume that we want to test the solutions in such a way that:
- those which do not consider the n=0 case will receive 60 points at most;
- those which always output Too small to exist will not receive any points whatsoever. With the knowledge we've got so far we are unable to do so (that is, not without e.g. asking the solver to output solutions for multiple queries at once, which is common on other judge systems, but not so much at SIO2 – you'll soon see why).
The SIO2 judge system comes with its own functionality that allows you to assign points for passing a test group rather than a single test. In other words, if you create tests tre1a.in and tre1b.in, the contestant's program will be awarded points only if it outputs a correct answer to each of them. In this manner, if the test tre1a.in contains the number 0, and the test tre1b.in contains the number 10, then of two programs: one that generally solves the problem correctly, and one that always outputs Too small to exist, neither will receive points for this group. Only a program which answers both inputs correctly will receive a positive score. From now on, we will refer to this mechanic as "test grouping". All tests belonging to the same group need to have the same number follow the problem's short name. They should be ordered with lowercase letters of the Latin alphabet. For example, if we want to have three tests belong to the second group, we shall name them tre2a, tre2b and tre2c (with appropriate extensions, that is .in or .out). If the number of tests in a group exceeds the number of letters in the alphabet, it is customary to name them like tre3aa, tre3ab etc.
The tests from the group 0 are called example tests. The results for these tests will be always shown to the participant, even if the full results for the problem are hidden. Additionally, those results will never affect the participant's score (a behavior specific for the 0 group). Usually some of them are provided as examples in the problem statement.
Another kind of non-scored tests are the ocen tests. They work similarly to the group 0 tests. A test will be considered an ocen test if its name adheres to the formula: the problem's short name + test number + ocen.in. Thus, if we create a test called tre1ocen.in, the user's program will be executed on it, the result will be shown, but no points will be awarded. NB: ocen tests are independent from test groups. They are only ordered by their numbers, however, they do not belong to any groups nor do they constitute a group of their own (unlike the tests from group 0).
Our package contains the test tre1ocen.in and two test groups (group 2 contains a test with 0 as an input).
Upon uploading a package, every solution code it contains will be automatically judged. SIO2 will summarise the results in a neat table, which we will also review in the end of this tutorial. This way we can easily verify that every kind of solution will receive an appropriate score. Wrong solutions are usually suffixed with b1.cpp, b2.py etc. Slow solutions are in turn suffixed with s1.py, s2.cpp etc. Of course, every solution's name ought to start with the problem's short name.
For our problem we've prepared the following additional solutions: treb1.py – always outputs Too small to exist treb2.py – draws a correct Christmas tree but doesn't consider the n=0 case.
Upon submitting our package to SIO2 (in the same way as in the previous part) we can see the results of all programs from the prog folder, as mentioned before. To access the view, go to Contest administration, then Problems. Finally, find the problem Tree that we've just uploaded and click on Model solutions next to its name. You can now see that the solution that always outputs Too small to exist receives 0 points and the solution that never considers the n=0 case loses 40 points, just like we wanted.