Hi.
My submissions are stuck in 'Validated' status for a few days (eg. submission ID: 8452548). Could you please check?
The end of round 3 is approaching and we cannot train our models.
Also, could you please email me(swkim@lunit.io) the log file for submission ID:8452710 (inference for sub-challenge 2)?
I have a bug that is not reproducible in local and express environment and it's really hard to debug with only the last few lines of error.
Thank you.
Created by Seung Wook Kim swkim > I have a bug that is not reproducible in local and express environment and it's really hard to debug with only the last few lines of error.
A longer tail of the log is below for SC-2 inference submissions 8452710. Hope this helps.
```
...
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Invalid PNG header, data size 0
[[Node: DecodePng = DecodePng[channels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"](ReadFile)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Invalid PNG header, data size 0
[[Node: DecodePng = DecodePng[channels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"](ReadFile)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Invalid PNG header, data size 0
[[Node: DecodePng = DecodePng[channels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"](ReadFile)]]
W tensorflow/core/framework/op_kernel.cc:975] Invalid argument: Invalid PNG header, data size 0
[[Node: DecodePng = DecodePng[channels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"](ReadFile)]]
hello
0
['subjectId', 'examIndex', 'imageIndex', 'view', 'laterality', 'filename']
['3', '1', '1', 'CC', 'R', '000024.dcm']
Headers--------------------------------
['subjectId', 'examIndex', 'daysSincePreviousExam', 'cancerL', 'cancerR', 'invL', 'invR', 'age', 'implantEver', 'implantNow', 'bcHistory', 'yearsSincePreviousBc', 'previousBcLaterality', 'reduxHistory', 'reduxLaterality', 'hrt', 'antiestrogen', 'firstDegreeWithBc', 'firstDegreeWithBc50', 'bmi', 'race']
restoring weights for test set from: ./saved_model/20-sc2-model.ckpt-epoch12@
Number of test cases: 17106
0 3-2
Traceback (most recent call last):
File "run.py", line 135, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "run.py", line 132, in main
run_helper.execute()
File "/dm-resnet/run_helper.py", line 431, in execute
step_llogits, step_rlogits = sess.run([tst_llogits,tst_rlogits], feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid PNG header, data size 0
[[Node: DecodePng = DecodePng[channels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"](ReadFile)]]
[[Node: resblock0_0/BatchNorm_1/moving_mean/read/_111 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_117_resblock0_0/BatchNorm_1/moving_mean/read", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Caused by op u'DecodePng', defined at:
File "run.py", line 135, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "run.py", line 132, in main
run_helper.execute()
File "/dm-resnet/run_helper.py", line 384, in execute
inputs, pls = batchiter.multimodal_preprocessed_inputs(for_training=False)
File "/dm-resnet/batchiter.py", line 169, in multimodal_preprocessed_inputs
imgs,llabel,rlabel, pls = multimodal_read_data()
File "/dm-resnet/batchiter.py", line 31, in multimodal_read_data
img_rcc = tf.image.decode_png(tf.read_file(pl_img_rcc),1) # returns a tensor with shape [h,w,c]
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_image_ops.py", line 345, in decode_png
channels=channels, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Invalid PNG header, data size 0
[[Node: DecodePng = DecodePng[channels=1, dtype=DT_UINT8, _device="/job:localhost/replica:0/task:0/cpu:0"](ReadFile)]]
[[Node: resblock0_0/BatchNorm_1/moving_mean/read/_111 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_117_resblock0_0/BatchNorm_1/moving_mean/read", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Traceback (most recent call last):
File "ensemble_probs.py", line 10, in
open_files.append(open(file_name, 'rb').readlines())
IOError: [Errno 2] No such file or directory: '/output/model0.csv'
``` Submission 8452548 is running now.