Frequently Asked Questions¶

What do the entries in the result tables mean?

For each test case, eight different performance measures are computed. These are the average absolute surface distance (AvgD) in mm and the average root mean square surface distance (RMSD) in mm, which are used to evaluate segmentation of femur and tibia. To evaluate femoral and tibial cartilage, we employ the volumetric overlap error (VOE) in percent and the relative volume difference (VD) in percent. All measures are translated to various scores in the range from 0 (lowest possible score) to 100 (perfect result), by comparing them with typical scores of an independent human observer. Finally, all scores are averaged to obtain one total score per test case. These scores are averaged to obtain the score for a system. The details of the error measures and the scoring system are explained in the ski10 paper that can be found here.

How often can I submit results?
In principle, you can upload as often as you want. Note however that all results you submit will appear on the website and every system should be substantially different from previous entries. The differences compared to other systems you have submitted must be evident from the submitted pdf file. In other words, you cannot submit different results using the same pdf file. We are committed to avoid 'training on the test set' and therefore do not want teams to send in a series of results that differ only in the settings of some parameters. For parameter tuning and related experiments, you should use the supplied training data.

What must be in the pdf document that is required for every submission?
Ideally, this document is a paper describing the system that has been used to generate the results in such detail that others can reimplement it, in other words, a standard scientific publication or technical report about your work. If you have published a paper describing your system, please upload that paper or, in case you are not allowed to have the paper in its original form downloadable from this site, upload a short description of it and a reference to the paper. If you have not yet written a detailed paper, or have submitted this for publication and do not want it to become publicly available, or if you have other reasons why you want to withhold detailed information about your method, please indicate the reasons for this in the pdf file you submit and describe the system only briefly.

Why do I have to provide a pdf document and/or a description with every result I submit?
We believe that it is not too interesting to report here the results of systems whose working is unknown. Therefore we require that a description of each system is provided.