**[Full text of the proposal](https://www.synapse.org/Portal/filehandle?ownerId=syn5659209&ownerType=ENTITY&xsrfToken=1EA1466FCA55F7EAE33833333900F1BC&fileName=Idea2.pdf&preview=false&wikiId=414654)** ###Anonymous Review 1 and Authors Response _ **Impact: ** The problem of completeness of a gold standard for gene regulation is very important. Predictions cannot be made unless having a complete and well described gold standard. Still, when considering its impact, the main handicap is the species for which the gold standard is proposed. The study for mammals would boost the impact of the proposal. However, the key problems for mammals, among others, are the remodeling of chromatin, RNA post-processing isoforms (i.e. by splicing), or distant interactions of chromosomes. Preliminary results are obtained for yeast, but then the impact of the gold standard is lower (although I think it is still high and relevant for the development of methods of network-inference and construction)._ **Response:** We thank the reviewer for their positive remarks. We completely agree with the reviewer that the impact of this work would be greatest if the gold standard were generated for a mammalian system and in fact this is what we propose. Although most of our preliminary work is in yeast, we are interested to generate a gold standard for a mammalian system. In Figure 3 of our proposal we had begun to evaluate our network inference approach in the mouse ES system, where we show that there is a significant discrepancy in what can be predicted when considering a ChIP-chip/seq based gold standard versus a knockout/knockdown based gold standard. This was one of our primary motivations to propose creating a high quality gold standard for a mammalian system. We appreciate the reviewer?s concern about the complexity of mammalian regulation. While for this challenge we did not propose to explicitly model splicing, we do want to include post-translational modifications via the inclusion of phospho-proteomics data. We are also, in parallel, developing tools to examine long-range gene regulation that can be integrated into the network inference approach.   _**Feasibility: ** If the gold standard is constructed for yeast the project may be feasible. Still, the work is very hard and requires a large amount of data collection on the interaction network involved in different types of signaling and in gene regulation. It may also help additional regulation data produced by synthetic transcription factors, perturbation of the network with CRISP (knocking and/or activating genes), TFs binding test by means of PBMs (which can also be used for network inference). Studying the implication of large complexes of transcription, modeling the structure of the complex or including information on protein-protein interactions can improve the reliability of the gold-standard regulatory network._ **Response:** We agree that the study would be more feasible in yeast, however, we feel that as a community we should try to tackle a mammalian system. We agree that additional perturbation data including CRISPR or synthetic TFs could help to establish the functional aspect of the network. If budget permits and we can find wet-lab collaborators who can do the CRISPR experiments, we would be happy to include these. We can also include modeling of TF complexes (by integrating protein-protein interaction networks) at the promoter of a gene as part of the network inference approach. We plan to include available PBM data for mouse or human as motif priors in our model.   _**Overall evaluation:** Such a gold-standard is very important to validate many of the computational models dedicated to this problem. However, we are still far from its application to more complex systems, like for mammals that require more information on the genome structure._ **Response:** We thank the reviewer for their overall enthusiasm for our project. We hope that with the help of the DREAM IDEA community we will be able to make some advances to this difficult but important problem.

Created by Chloé-Agathe Azencott caz
Thank you for your comments. About 1, yes we completely agree that we need to carefully define what we mean by an edge. We don't think ChIP-chip alone or knockdown/knockout alone can provide these edges as each has its own limitations. We propose to test predictions with ChIP-chip (physical) and knockdown (functional). We think that by asking for support from both these types of measurements would bolster our support of what we believe as a true regulatory connection. About 2, yes, this was implicit in the proposal but we agree that we absolutely need to have an iterative approach. Our idea was to prioritize edges for validation using the model that we build. The model itself should be able to predict expression that we would like to compare in an experimental setting. Then we would use the results of the validation experiments to update the model. We can repeat this procedure several times to enable iterative model building and refinement.
I would like to start by thanking the authors for submitting this proposal. Generation of a good gold standard for network inference is very important and is likely to advance the field significantly. Network inference also happens to be a personal interest of mine. I see several problems with the current proposal and would like to bring forward the main ones in the hope that we as a community could reshape this proposal into a feasible project. 1. What do you mean by a link in the network you want to infer? In the proposal perturbations with measurement of expression changes as well as use of ChIP-chip is suggested. Perturbation of genes and observation of resulting expression changes in all these genes provide information about influence links. These influences captures the information flow of the gene regulation, i.e. the intra cellular control system. As stressed by several researchers, influences are not in general the same thing as physical interactions, i.e. binding of molecules. Two molecules can bind without any information flow and influence on the expression of any gene and an influence can exist without any binding event. Consider for example binding of a transcription factor that does not change the rate of transcription or a concentration change that modifies the probability of a reaction. Inferred influences should therefore not be evaluated based on physical interactions such as transcription factor binding data from ChIP-chips. Actually, influences can in general only be observed indirectly through their influence on the gene expression, so to understand the control system of cells we need to map out influences. To understand the physical interactions that I personally believe to act as a back-bone of the information flow ChIP-chip is helpful. 2. The main problem with existing data sets that have been used for network inference is that they are not informative enough for network inference. The system is by evolution designed to amplify some signals and attenuate other a property that has been called Interampatteness. In particular the attenuation is problematic when inferring networks, because the weakest signal is the one that essentially determines the quality of the model in inverse problems. The weakest signal is typically hidden in noise, which means that the network models become poor. The only way to overcome this problem is to iteratively design perturbation experiments until the signal to noise ratio is sufficient in all directions spanned by the states of the system. I see no mentioning of an iterative design and experiment strategy in the proposal and it is therefore unlikely to succeed in generating a good golden standard.
### Anonymous Review 4 and Authors Response _**Impact:** Learning the structure of regulatory networks in molecular biology is a widely-studied problem, yet the ground truths usually used to evaluate said structures are flawed. This proposal is to generate new gold standard data that is more complete than what has been used before. I believe the results would be used widely by many groups studying networking biology._ _**Feasibility:** In general, the study is feasible and clearly written. One thing that was unclear is the coverage to be obtained in the experiments that involve sequencing. Applicant quotes TF ChIP-seq profiling at $150 per sample. This seems incredibly low. In general I wonder whether applicant will be able to supply all of the deliverables here at the listed cost, but whatever fraction they get would surely be helpful._ _A plan for dealing with the impact of fatal changes in the proposed perturbations would also be useful. Applicants will not be able to collect detailed genomic profiling data when a knockdown kills the cell._ _I believe this study is likely to succeed in achieving some of its goals._ **Response:** We agree that our proposal is ambitious in terms of the datasets we propose to collect. We have some wet-lab collaborators, however, we are also expecting to collaborate with the IDEA challenge organizers to help us find collaborators who can assist with the data collection. The reviewer makes an excellent point about scenarios where the experiment might have fatal consequences on a particular cell type. We will work with our experimentalist collaborators to titrate the extent of perturbation based on siRNA knockdown experiments and consider other types of perturbation (for example overexpression of the regulator) to avoid completely lethal perturbations.   _**Overall evaluation:** The most important part of the proposal is to generate comprehensive data to provide a high-quality benchmark for regulatory network learning methods. Applicant proposes to generate several kinds of genome-scale data (RNA-seq, ChIP-seq, ATAC-seq, and proteomics), sometimes after siRNA knockdown. This will be very useful data for many groups._ _Applicants also plan to use these data to perform validation of several probabilistic graphical model methods, including MERLIN, Inferelator, TIGRESS, GENIE3, and CLR. This sort of benchmarking will be very useful in driving further progress in the field of regulatory network learning._ _The principal investigator is a highly-regarded expert in computational epigenomics and has access to an excellent research environment._ **Response:** We thank the reviewer for their enthusiastic support of our proposal and comments.
### Anonymous Review 3 and Authors Response _**Impact:** The proposal is to generate a "golden" test set for programs that predict  gene regulatory network. The submitter proposed to test 20TF and 20 signaling  proteins._ _**Feasibility:** I think this could very useful to have such a dataset (perhaps that should not be called  gold standard, but it would provide a reasonable reference)  but I have several concerns:_ _1. The authors do not provide any estimate of how many TF with known antibodies have not been used for Chip-Seq? If  this is to be a benchmark, it has to be independent of data that is currently in the databases._ ** Response:** Response: We thank the reviewer for this comment. We do not want to replicate all experiments for which high quality ChIP-seq datasets already exist for the cell type of interest. We will refer to the Biocompare website (www.biocompare.com) to look for ChIP-validated antibodies in mouse and that have not already been profiled in the cell type of interest (for example by the ENCODE or the BLUEPRINT consortiums).   _2. For siRNA for human/mouse cell there is a serious issue of off-target effect. If this is to be a high confidence set, the authors should consider using different siRNA for the same target. Replicas are good, but different siRNA would be even more important._ **Response: ** We thank the reviewer for this comment. We will use different siRNAs for each regulator.   _3. How exactly the 40 regulators will be selected?_ **Response:** We will use different regulator prioritization techniques using topological and functional properties of the network (See also response to Reviewer 2?s first comment). Based on this reviewer?s advice we will additionally avoid trying to recreate datasets that already exist.   _**Overall evaluation:** Overall, if successful,  the project will generate a very useful dataset.  My main concern is lack of any discussion of  how to generate such set so that it would be orthogonal to existing data and not biased, say, towards collecting data on regulators that are more studied and thus easier to predict.  Also the submitter haven't flashed out how to use single cell data. Unlike other data considered in this proposal, for which it is obvious how it should be applied, single cell data is noisy data and its applicability has to be supported by describing the algorithm  that will be applied to the data. The proposal has not much  to say about it. Granted, it will be interesting data to have, but to convince that this particular  dataset is important  for building of gold standard set, additional information of how  it will be used is needed._ **Response:** To avoid overlap with datasets that already exist (for example ChIP-seq experiments for well-studied regulators), we will use regulatory network-based prioritization techniques. While these techniques can pull out known and well-studied regulators, we expect to identify several novel regulators as well. We believe that the orthogonality of the dataset also depends upon the specific cell type we are studying. We will take these factors into consideration while nominating regulators for validation experiments. Regarding the single cell data, we admit that the proposal did not describe in detail why such a dataset is useful in the context of regulatory network inference. Our proposal is to examine whether the cell-to-cell variation measured in single cell RNA-seq experiments can be informative for network inference. Our plan is to apply standard network inference algorithms to study whether we can infer a network from this data, and if so, whether it is better or worse than the networks we get from bulk data. We realize that there are numerous statistical challenges that arise due to the overabundance of zeros in single cell RNA-seq data. We plan to use recent diffusion-based smoothing based techniques to interpolate zeros. In addition, we have already downloaded several single cell RNA-seq datasets on which we plan to carry out a pilot study of computational network inference. The newly generated single cell RNA-seq datasets can be used to supplement the published dataset or can be used for validation after regulator perturbation.
### Anonymous Review 2 and Authors Response _ **Impact:** The proposal aims to create a small gold standard data set for regulatory interactions for benchmarking regulatory network inference models. 20 transcription factors and 20 kinases involved in human or mouse embryonic stem cells are planned to be characterized physically and functionally using sequencing and chromatin data as well as proteomic profiling for both wild type and perturbed cell lines. Previously developed regulatory inference models will be used in combination with sequence motifs on promoters and network structure information trained a priori._ _ **Feasibility:** The study is likely to succeed in generating data on a small set of proteins and verify a subset (if not all) of the regulatory interactions inferred from the data. See below for my detailed comments._ _ **Overall evaluation:** It is not entirely clear how the regulators to be tested are going to be prioritized. Which data sets are planned to be used? Is it possible to reduce costs by selecting proteins with potentially higher number of overlapping downstream proteins or proteins for which publicly available sequencing data exist in GEO._ **Response:** We thank the reviewer for the suggestion of trying to prioritize regulators based on the number of downstream proteins. Different graph centrality based measures can capture some of these properties (e.g., we believe the reviewer is referring to prioritizing nodes based on a graph reachability analysis). We will use topological measures (e.g. degree centrality, eigenvector centrality, betweenness centrality), as well as functional measures such as those described in Chasman et al 2016 (Ref 16) which uses the parameters of the network to determine regulator importance. These measures will be computed on the inferred regulatory network, which in turn will be first learned from publicly available human/mouse ES gene expression data (or any other mammalian cell type of interest to wet-lab collaborators) and also static interaction data derived from sequence-specific motifs. We will also consider the inclusion of published protein-protein interaction datasets (to address Reviewer 1?s suggestion).   _I do not follow how the single-cell sequencing data is going to be used towards achieving the objective of the proposal. Is it going to be used to identify natural variants? It is also unclear how the cell level heterogeneity will be taken into account for regulatory network inference._ **Response:** The single-cell RNA-seq data was proposed to examine the question if this type of data could enable us to infer a different and potentially more correct regulatory network. Our hypothesis is that the cell-to-cell heterogeneity is a useful type of variation that could be useful for network inference. The network inferred from single cell data would be examined using similar regulator prioritization techniques to propose regulators for validation. As preliminary work we have already downloaded several single cell RNA-seq datasets for mouse ESCs. We will use these for network inference, which can be further supplemented with new single-cell RNA-seq datasets that collaborators can generate.   _ "We observe that prior-base methods perform significantly better than expression-based methods in recovering the structure of the gold standard." From the figure, this seems to be true only for the case of natural variants._ **Response:** We apologize that this point is not clear. We reported PR curves in Fig 1A for the natural variation data, however the area in the precision recall curves (AUPRs) are similarly high for the other datasets as well (See Ref 5). In addition to AUPRs, we also used the number of predictable TFs as an additional metric (Fig 1B), which is the number of TFs for which we can reliably predict their ChIP-chip targets. We find that the number of predictable TFs is higher for the methods that use priors (PGG+Prior, MERLIN+Prior), compared to their counterparts that do not use priors (PGG, MERLIN) on the natural variants dataset (Nat. Var.) as well as in the Knockout and Stress Resp. datasets.   _ Potentially due to the space limitations, sections 3-4 lack methodological details that makes it tricky to corroborate the conclusions based on the provided evidence. _ **Response:** Several of our results are published and described in Siahpirani & Roy (Ref 5) and Chasman et al (Ref 16).   _ How can the author make sure that the proposed experiments will provide enough data that will tackle the main conclusion derived from the preliminary results? That is will the generated data improve substantially over current gold standard in terms of providing a meaningful/complete benchmarking data set?_ **Response:** The reviewer makes a very good point. We have proposed experiments that measure both the physical and functional part of the network that will be conducted by the same wet-lab collaborators. Current gold standards either rely on ChIP-chip/seq (physical) or knockout/knockdown (functional) data alone, and there is little overlap of such datasets being collected at the same time. Furthermore, we will be measuring the role of signaling proteins as potential regulators in controlling transcriptional output, which to our knowledge would be a novel aspect of our dataset. Hence, by collecting different types of measurements for a set of prioritized regulators for the same system under study, we expect to improve on the current gold standards.   _ Minor comments:_ _ - Ref 26 is incorrect repetition of ref 1_ **Response:** We apologize for this and will correct this.   _ - "Singe-cell RNA-seq data, which ..., is and as yet unexplored type of perturbation for network inference" Long and unclear sentence. I dont see how the sequencing data can be a "perturbation type"._ **Response:** We have attempted to explain this point in the comments above. But briefly, we hypothesize that the variation between individual cells could be harnessed to infer regulatory networks. It is currently unknown whether this type of variation is useful for network inference or not. The perturbation is the variation between cells.   _ - "the number of available bulk transciptome .. comparable to the ESC state": Sentence unclear._ _ - "structural priors" could use some explanation in the sentence where it is first mentioned._ _ - missing parentheses where refs 18-20 and 21,21 are referred._ _ - eigen\*vector\* centrality_ **Response:** Thanks, we will correct / clarify this.

Idea 2: Towards building a better gold standard for computational network inference approaches page is loading…