Dear Organizers, I have a couple of questions regarding the usage of the challenge dataset: Q1. If I create manual labels for the validation dataset, am I allowed to use these labels to select the best model? Q2. Is it permissible to use the Task 1 dataset for Task 2 and vice versa? I would appreciate your guidance on these points. Best regards, Hyunwook Kim

Created by Hyunwook Kim wooks527
Dear Participant, Thank you for your questions; our answers are below. We would like to emphasize that using Task 1 data for Task 2 and vice versa is highly encouraged and recommended. Q1: Yes, you can utilize any approach to select the best model for your submission. If your process for model selection includes creating your own labels for the validation dataset, you are welcome to do this; however, note that our experts have already assigned labels to the Task 1 validation set, but the labels are not available to participants. For the validation phase, we only release image data and we will never reveal the ground truth labels for this validation set. Further, these same experts have already predetermined the labels for the test set, of which no data will be publically released. In the validation phase, we evaluate your csv file submission against our pre-determined validation labels to provide your team with a validation score. The ground truth may differ from your manual labels and therefore selecting a model based on your manual labels may skew the outcome to be unfavorable during test set evaluation as test set data were again labeled by the same experts that labeled the validation data. We recommend selecting the model that scores best during the validation phase. Q2: Yes, you can use data interchangeably between tasks. The task 1 dataset may be utilized for task 2 training and vice versa. However, at test time, the input will be ONLY the data provided for the specific task. In other words, we expect your MLCube submission for Task 2 to utilize only Task 2 data to make a segmentation prediciton. We hope this makes things a bit clearer. Please let us know if we can clarify any of our points further. Thank you, LISA 2024 Challenge Orgnaizers

Request for Guidance on Dataset Usage page is loading…