Would it be possible to have more environment variables in the next phase?
e.g.
EXPRESS_LANE={1,0}
CHALLENGE_MODE={preprocessing, training, scoring_sc1, scoring_sc2}
This would be particularly helpful for the Express Lane (so we can use the same script for both for troubleshooting).
Thanks.
Created by David Gutman davecg @davecg All the submissions have now access to the environment variable `WALLTIME_MINUTES`. On the Express Lane, the value of this variable is set to 30 as we have increased the number of Express Lane machines and decided to increase the wall time from 20 to 30 minutes. Hopefully, you can use this information in combination with a timer to estimate if you have the time to process another subject before returning with the status code 0 (success).
Thanks! Thanks for getting back to me Thomas.
I know I could run the same script for both Model Training and the Model Training Express Lane, but even with the smaller express dataset most of my scripts do not finish in 20 minutes.
So, for example, I would want to change the number of training epochs on the Express Lane vs full Model Training so I could run through my complete script. (Depending on timing I might have to choose a smaller subset of the data for the express lane too.)
Hi David,
I'm not sure to understand why you would like the environment variables EXPRESS_LANE and CHALLENGE_MODE. The current design allows you to run the same container with the same scripts (preprocess.sh, train.sh, sc1_infer.sh, etc.) on both the Express Lane and the Challenge Data. The idea is that you would first run your container on the Express Lane to check that it runs correctly, then submit this "validated" container to run on the Challenge Data. One approach could be for us to run systematically your container on the express lane before running it on the Challenge Data. However, all the issues can not be detected, e.g. 200 GB of RAM could be enough to process the Express Lane data but a method could experience an Out of Memory exception at some point on the Challenge Data if the method has not been correctly implemented. Dear David,
Thanks for bringing up this suggestion. I will pass this along to other challenge organizers.
Best,
Thomas I know it's possible to do something hacky to guess (e.g. Look at permissions and existence of various directories to guess type of image running), but think it would be better to make it explicit in case some of the things we assume change in the future. The express lane has only about 1% of the training data in the tsv files, so maybe thats a quick solution for you