**[Full text of the proposal](https://www.synapse.org/Portal/filehandle?ownerId=syn5659209&ownerType=ENTITY&xsrfToken=1EA1466FCA55F7EAE33833333900F1BC&fileName=Idea3.pdf&preview=false&wikiId=414654)** ### Anonymous Review 1 and Authors Response _**Impact:** I think this an important problem, but there have been other large scale studies to predict drug target affinities._ **Response:** This is true, and we are also aware of many computational studies for predicting drug target affinities. However, the existing prediction models are often constructed and evaluated under overly-simplified data and setups that do not really reflect the real-life problems in actual drug discovery or drug repurposing applications. Therefore, even though these theoretical prediction studies and models may have a great potential for guiding the experimental bioactivity mapping, their practical benefits for the real-world applications remain largely unknown, as most modelling works lack direct experimental evaluation of their predictions on a larger scale. This is why we and also others (see e.g. Reviewer 2 and 3 overall evaluations) think that the field is now matured enough for organizing a DREAM challenge, in which the computational teams can apply their favorite prediction models they believe are theoretically ready for drug-target affinity prediction. Then, based on a large and standardized experimental dataset collected, as well as new data generated in this Challenge, the organizers can systematically evaluate the model predictions so that the community will finally know which approaches are best for producing most accurate and robust predictions, as well as exciting biomedical discoveries, for instance, in the form of novel drug repurposing opportunities.   _**Feasibility:** It is feasible, but there are some details that are missing as discussed in the detailed review. In particular, it is important to have a strategy to determine which specific targets and drugs will be tested experimentally._ **Response:** We at FIMM are specifically interested in kinase inhibitors as kinases represent an important class of targets for anticancer treatments, as well as in many other diseases, e.g., Alzheimer’s diseases, and those are the types of drugs we are testing at FIMM using our high-throughput compound testing setup (https://vimeo.com/74608507). However, we think it will be important to carefully discuss and plan the set of drugs and targets tested together with the experimental collaborators that are supposed to generate the new experimental dataset(s), in close collaboration with us and the DREAM Challenge organizers, so that all the parties will be maximally committed to the Challenge. From the modeling and prediction point of view, kinase inhibitor target activity selectivity prediction poses a relatively challenging problem, due to the high structural and sequence similarity shared by protein kinase domains, hence leading to profound target promiscuity and polypharmacological effects. However, we are also interested in many other target classes, including GPCRs, ion channels or nuclear receptors, whichever are exciting for the data generators. As biologics today represent the major type of newly approved as well as revenue-making oncology drugs, these could provide an additional interesting drug class for the Challenge, even if target selectivity typically is much clearer with these agents.   _**Overall evaluation:** The goal of this proposal is to systematically create a comprehensive dataset of drug-target interaction affinities. To this end the investigators propose to (1) use a crowd-sourcing effort to create a single data resource that would systematically curate and store drug-target datasets using a standardized format, and, (2) to collect experimental data of drug and target interactions for about 500 drugs._ _Strengths:_ _1. The proposal is well written and is aiming to address an important problem._ _2. The created datasets would serve as a rich community resource for predicting potential targets of a drug, or predicting drugs of a potential target protein._ _3. The investigators have already developed an online web platform called Drug Target Commons which can be used for crowd-sourcing the manual curation of existing drug-target interaction datasets. The availability of this web-based resource could significantly aid in the creation of curated training and test sets._ _4. Preliminary evidence that the predictions from the KronRLS model is able to identify some valid off-targets that is supported with experimental data._ _Weaknesses:_ _1. The budget mentions number of compounds but not the number of targets. It would be helpful to know this. Also the budget does not including any aspect of the DTC. Perhaps including some incentive for contributing to the DTC could help improve the crowd-sourcing effort._ **Response:** the number and type of targets depends on the number and type of drug compounds tested (and the budget available). For instance, in case we want to focus on kinase inhibitors (e.g., those ca. 500 compounds we are currently profiling at FIMM), then the target space would cover all the human protein kinases (ca. 500 protein targets). For the other compound and target classes, these numbers differ respectively. Please see Figure 2 for some examples of target numbers and promiscuity for various target classes (e.g., GPCRs, ion channels, nuclear receptors, kinases and other enzymes). However, to systematically evaluate the computational models, it will be important to collect/generate as large and standardized target selectivity data matrix as possible, which spans over multiple compound and target families to investigate also the potential of the models to predict off-target class relationships (e.g., some kinase inhibitors may also target GPCRs, and vice versa). Such cross-class-reactivity may lead to toxic side effects, but also to novel drug repurposing opportunities. As for the DrugTargetCommons (DTC) budget, we agree that an incentive for contributing to the DTC would indeed greatly help the crowd-sourced data standardization effort. The incentive could be, for instance, authorship in the planned paper as part of the DREAM Challenge Consortium. So far, DTC has been a FIMM in-house effort (involving around 25 PhD students and postdocs), but for the Challenge, we will definitely need a larger community of domain experts to collect and standardize large enough and fully-curated experimental dataset. We have already made contacts to potential collaborators who have expressed their interest in joining such as crowd-sourcing effort, including several research institutes in Europe and in the US, as well as pharma companies that are nowadays more open for open-data research collaborations. Pharma companies are in a unique position in such an effort as they have often comprehensively profiled their own compounds for on/off-target activities, resulting in large target selectivity data matrices spanning many approved drugs currently in use.   _2. The proposed model (Kroneker regularized least square) predictive model is not described in any detail and there is no linked paper that one could read to understand exactly how the model is working. The authors also mention random forests but don't report results using these models._ **Response:** We could not unfortunately detail the models in the 5-page proposal. The Kronecker regularized least squares (KronRLS) model has been described in detail in the pilot paper we cited in the proposal (please see Suppl. Methods in AddSuppFiles-2 of Pahikkala et al. Toward more realistic drug-target interaction predictions. Brief Bioinform. 2015. doi: 10.1093/bib/bbu010). However, we note that KronRLS is just one machine learning method that we have used for such prediction problem, but the computational part of the Challenge would allow the community and teams using various models to make even better predictions. We propose that we could use KronRLS as a baseline model against which all the submissions from the DREAM Challenge participants will be benchmarked. Although we believe our preliminary results are promising, it is unlikely that the quite simple KronRLS will give the best result. Random Forests (RF) were used in the pilot study merely to show that similar trends in the results under the different, practical evaluation setups were obtained using also other models than KronRLS, to avoid reporting any model-specific artifacts (the RF results were shown in Suppl. Tables in AddSuppFiles-3 of Pahikkala et al. 2015). However, even though the two models behaved rather similarly in the practical setups S1-S4, there were many differences in the actual drug-target predictions between KronRLS and RF models, and therefore it will be interesting and important to evaluate the prediction accuracy of a wide range of computational models, both statistical and machine learning methods, in this Challenge.   _3. Oftentimes a lot of auxilliary features of the drug as well as the target is used and has been shown to be useful, including some references mentioned by the investigators. There is no mention of how these features might be used._ **Response:** This is exactly the idea of the Challenge; to make a crowd-sourced assessment of the methods and features that are most predictive of drug-target affinities. We and other have used specific structural and molecular features in the past models, and we do have further ideas that might improve the results, which can be shared and provided during the Challenge start-up phase; but most likely the community will figure out even more clever features that originate from various sources of auxiliary side information, not only those provided by us, and also ways how to use these for this prediction problem.   _4. There are many regression modeling approaches and it is not clear why the KronRLS approach should be used without showing that other regression models don't perform as well._ **Response:** Like explained above, we propose that the KronRLS model could be used as a reference baseline model against which all the submissions and models from the DREAM Challenge participants will be benchmarked. We feel our preliminary results are promising, and that it is good to have such a benchmark model available also in this Challenge, but it would be very surprising if the simple KronRLS would give the best prediction results. Like the Reviewer pointed out, there are many other modelling approaches, including more advanced regression models, which will likely lead to improved results.   _5. It is not clear how the predictions generated from the KronRLS (or other model) will be used for determining the experimental dataset creation. Currently  the experimental part of the proposal and the computational part are not well integrated._ **Response:** We apologize for not being clear enough in the proposal wording: the computational models are not meant to be used for determining the experimental dataset creation in the first place; instead, the existing experimental data will be initially collected and standardized through DTC for systematic training and initial assessment of the models submitted. The new set of experimental data will be generated at the same time to objectively test the computational models in an independent compound-target data matrix not available at the moment to avoid model over-fitting. However, if possible in this Challenge, budget or time-wise, it would be exciting to have also a follow-up, post-competition phase, where the top models from the first phase will be next applied to guiding new experimental drug-target mapping efforts, e.g., in terms of prioritizing the most potent, novel off-target interactions for further experimental evaluation. This would enable iterative, model-guided data generation process, toward really demonstrating the practical utility of the model predictions in real-life application cases, both for phenotype- and target-based drug discovery applications.