The process of using a container for pre-processing seems serial. Is it possible to save a jpg/png image inside the /preprocessedData/ path on the fly while training? My issue is that I am using luajit where I can't use a dicom package to load images. So I wish to use python to create a few images into the preprocessedData directory and then load them into luajit as I train. Is this possible? As in will we have write access into this directory while executing a program on the main container? Also, I think it would really great to get some log file/notification if the training has begun successfully. Is that possible to implement?
Created by Amrit Krishnan amrit110 Hi Amrit,
> Also, I think it would really great to get some log file/notification if the training has begun successfully. Is that possible to implement?
Emails are already sent when the pre-processing and training start. > The process of using a container for pre-processing seems serial.
We developed a system where you can save up to 10 TB of pre-processed data to avoid you spending your time quota on repeating the same pre-processing tasks over and over.
> My issue is that I am using luajit where I can't use a dicom package to load images. So I wish to use python to create a few images into the preprocessedData directory and then load them into luajit as I train. Is this possible? As in will we have write access into this directory while executing a program on the main container?
The directory _/preprocessedData_ is available in read-write mode in the pre-processing phase and then only in read mode for the training phase (see [Submitting Models](https://www.synapse.org/#!Synapse:syn4224222/wiki/401759)). We do that to increase the modularity and reproducibility of the pipeline. Please consider developing a method that avoid redoing unnecessarily the same pre-processing of the data for multiple runs.