Is the the parameter of interest the sample average treatment effect, i.e. the true average cf difference in the sample or is it for say a million person population as generated by the data generating system? Please clarify!
Created by Jonathan Levy jlstiles Yeah, good point. One could see the SATE as $$\( \sum_{i=1}^n E[Y(1)-Y(0) \mid W_i] \)$$ which is what I was driving at.
I agree that using math as our common language would be helpful. Although in this case we also need a common language around causality. So from a statistical causal perspective, the equation you have written is just the difference in mean outcomes between treated and untreated conditional on covariates (assuming A denotes binary treatment and W covariates and that n is the sample size). This is not a causal estimand. The causal estimand SATE (in my world) is defined as $$\(\frac{1}{n}{\sum^n_{i=1} Y(1) - Y(0)}\)$$. Your equation and mine might be equal in expectation under the assumption of strong ignorability which requires $$\(Y(1), Y(0) \perp A \mid W\)$$ and $$\(0 \lt Pr(A \mid W) \lt 1\)$$. My read of what the organizers have told us is the former condition has been satisfied in this competition. They have not ensured the latter condition (common support) and in fact it is pretty clear from the training data that it does not always hold. All of this highlights how hard it is to talk across fields and even sub-fields in precise ways! Thanks, it really ought to be defined mathematically as follows where the expectations are taken over the truth: $$\(\frac{1}{n}\sum_{i=1}^{n}(E[Y|A=1,W_i]-E[Y|A=0,W_i])\)$$. This kind of connotes a population effect since the expectations are taken wrt a true underlying distribution but it also depends on the sample so calling this a population effect is unclear and even more difficult to clear up via English, as we see on this thread! $$\(\)$$ It's easy for these terms to be confusing. Thanks for the clarification -- super helpful!! First, thanks for clarifying your point, excuse me for my misunderstanding.
Second, that being the case, the estimand evaluated is indeed SATE.
[Counterfactual files, like the ones provided, are used to determine the true effect. Each counterfactual outcome file contains the same individuals as in it's corresponding observed file (rather than containing the values for the entire 100k pool of subjects). Therefore, the true effect for a particular ufid is truly a sample average effect]. Sorry, no, this doesn't clarify. Are you calculating the true average treatment effect by taking the average difference between the counterfactuals
across everyone in the full population of 100,000 included in the covariate file? Or, are you taking the average different between the counterfactuals for
everyone in the sample file corresponding to a particular ufid? The first would be (a version of ) PATE; the second would be SATE. They are not
(in general) equal. It is true that if the sample is drawn randomly from the population then an unbiased estimate of SATE should also be an unbiased
estimate of PATE; but that is not the same as SATE being equal to PATE. Moreover, there are still implications for uncertainty quantification since
estimation of PATE includes an additional source of variation. In other targeting SATE versus PATE would really affect our confidence intervals. Please
clarify which of the above estimands is the benchmark. Thank you. The challenge is based on a benchmarking code that uses the counterfactual files to calculate the true effect ([for example](https://github.com/IBM-HRL-MLHLS/IBM-Causal-Inference-Benchmarking-Framework/blob/5e6d30e8ec970d8675dd260fa4fa94732660673b/causalbenchmark/evaluate.py#L86)).
Since the entire population is bounded by the the size of the original LBID dataset (we don't have a million person population, the simulations are based on a real covariates matrix) and since the sampling of individuals for each simulation file is unbiased, the SATE is the same as the PATE.
The term "population" describes here an aggregated measure so to differ it from the "individual effect" track.
Sorry for any confusion, I hope this clears it. Hi Ehud. Thanks for the reply but this is confusing since the submission instructions over and over explicitly call for "population effect estimation".
Can you confirm we are supposed to be performing inference for SATE and not PATE?? Sample average treatment effect.
Sorry for the late response!