In the guide for submitting models via the synapseclient (https://www.synapse.org/#!Synapse:syn1768504/wiki/56101) it refers to an ID for the evaluation. What is this ID for this challenge?
Thanks.
Created by John Reid Epimetheus Sure. We'll release the integrated scoring scripts and activate them on the leaderboards by early next week. then can you please display the leaderboard according to what you will display on the final test, Costello?
see, i already made a bottom-ranking submission with real name to activate the leaderboard, why can't you display the integrated leaderboard?
---
otherwise, to me it looks like you can make any team win and any team not win... metrics has to be finalizeded at least 2 weeks prior to deadline, right?
---
(you see, previously when i asked this question, the answer is mostly 'we assure you that we do not select winners based on who we like'. then even though they prove so in the end, it is not very convincing. can we hear something different this time, please?) Hi Epimetheus, for the leaderboards, we just selected AUPRC for display purposes, you can rerank by clicking on any of the metrics. Thanks for your help. That looks great, except for the ordering but I guess that is someone else's code doing that. > If you invalidate my last submission (id: 7150070) then I will try again.
I just did so.
> And FWIW I was following the advice at the synapse wiki when naming the arguments. You might want to change that.
Yes. I have opened an issue in our issue tracking system to update the doc':
https://sagebionetworks.jira.com/browse/SYNPY-385
i found it is exclusively ranked by AUPRC. that is the sin of having multiple metrics, eventually someone smart will play with the system.
and then make it a non-scientific problem but a play of statistics. they need a statistic consultant, really... Also I have question regarding the ranking of the leaderboard. My submission appears to score 2/3 on 2 of the criteria and 3/3 on the other. Should it not be in second place therefore? I did try again as suggested and the same thing happened again. I see you have edited your answer to provide different details. If you invalidate my last submission (id: 7150070) then I will try again. And FWIW I was following the advice at the synapse wiki when naming the arguments. You might want to change that. Thanks for your help. I marked submission 7123889 as "INVALID" which will allow you to submit a new one in its place.
I think I see the problem: In your script you have:
```
submission = syn.submit(evaluation, upload_file, name='Submission', teamName='')
```
while in the doc's
http://docs.synapse.org/python/Client.html#synapseclient.Synapse.submit
it says
```
submission = syn.submit(evaluation, entity, name='Our Final Answer', team='Blue Team')
```
i.e. the argument is 'team', not 'teamName'. Could you modify your script and try again?
and yes that is the submission I referred to Yes I do mean that script. I used 'Saturn' as the team name. Hi, Epimetheus,
I am on the support team and here to help. I see a submission here, id= 7123889, under your user name:
https://www.synapse.org/#!Synapse:syn6131484/wiki/402499
Is this the problematic submission?
You ask:
> It submitted my predictions as an individual (Epimetheus) when the rest of my predictions have been submitted under my team (Saturn). Does anyone know how I can fix the script?
By 'script' are you referring to the Python script you posted in this discussion thread? If so, exactly what string did you use for ?
Thank you.
Hi Epimetheus, We are having the Synapse engineering team look into this. Stay tuned for updates. see, epim? that's one of the biggest benefit of not working in team, so you don't need to ask them to remove accidental individual submissions which block all your future submissions... Actually the script has a problem. It submitted my predictions as an individual (Epimetheus) when the rest of my predictions have been submitted under my team (Saturn). Does anyone know how I can fix the script? The wiki isn't that detailed. Also can anyone move my recent GABPA submission from an individual submission to a team submission?
Thanks. Just to also say that the wiki (https://www.synapse.org/#!Synapse:syn1768504/wiki/56101) seems to be wrong. There is no `joinEvaluation` method on the `syn` object and it does not seem to be necessary. I include my batch submission script below for anyone that finds it useful. You will need the docopt python module installed.
```
#!/usr/bin/env python
#
# Submit files to challenge
#
doc = """Usage: make-submissions SUBMISSIONFILES..."""
from docopt import docopt
import synapseclient
from synapseclient import File
#
# Parse options
#
opts = docopt(doc)
submission_files = opts['SUBMISSIONFILES']
#
# Log in
#
print('Logging in.')
syn = synapseclient.Synapse()
syn.login()
#
# Our project
#
print('Getting project.')
project = syn.get('')
#
# Get the object representing the evaluation queue
#
print('Getting evaluation queue.')
evaluation = syn.getEvaluation(7071644)
for filename in submission_files:
#
# Upload our file
#
print('Uploading: {}'.format(filename))
upload_file = File(filename, description='Submission', parentId = project.id)
upload_file = syn.store(upload_file)
#
# Submit the file to the evaluation queue
#
print('Submitting: {}'.format(filename))
submission = syn.submit(evaluation, upload_file, name='Submission', team='')
```
Hi, sorry that id is not easy to find. It is 7071644. We have also updated the Submitting Models page to include this id for others.
Drop files to upload
Synapse id of the evaluation for the challenge page is loading…