Hi there I am trying to test out a docker image containing a rather large stacked model which I believe is breaking in the fastlane because it isnt getting past the importing of the model stage and might require more than 16Gb of RAM, despite being able to run it locally with
```
docker run --cpus ="4.0" --memory="16g"
```
parameters. Anyways I noticed that the GPU submission allows for 61GB of RAM and was wondering if there was a was large memory queue or GPU fastlane that I could test out on.
submission ID 9724760
Thanks,
Michael
Created by Michael Jochum MADscientist314 Hi @MADscientist314 ,
Thank you for the thorough debugging and follow-up! I looked into your situation and it was indeed a RAM issue. For any RAM-"intensive" models, we recommend using the GPU queues for evaluation.
Glad you were able to get a submission in! @vchung Just a heads up before I had posted this in the discussion forum I had actually submitted the docker image to the Task1 GPU queue.
It looks like It sat overnight in the queue but just now received notification that it proceeded and was successfully scored.
I am at a complete loss as to how to explain how same docker image failed in 1 queue yet passed in another.
My only guess is that it was a RAM issue due to the following:
- The logs from the fast lane stop right after attempting to loading the model,
- I am unable to reproduce the issue locally,
- I have been troubleshooting the code left and right and am unable to get it to pass through the fast lane or the regular Task 1 queue.
Either way I got it through submission, here is hoping this helps someone else reading this in the competition!
submission (ID: 9724845)
Thanks for your help,
Michael
Hi @MADscientist314 ,
For submission ID 9724760, we received the following error:
```r
[1] "LOG: PTB predictions Started: 2022-09-01 20:44:03"
Error in `$<-.data.frame`(`*tmp*`, "call_text", value = c("... %>% select(specimen, pred, prob)", :
replacement has 34 rows, data has 33
Calls: %>% ... probFunction -> -> predict -> predict.randomForest
Execution halted
```
I am not sure why the Docker log did not catch this.... apologies for that!