Friday, March 14, 2014

Kepler False Positives: Separating the True from the False

The Problem: Electronic False Positives in Kepler Results

As described in my last post, the Kepler mission’s data pipeline has a flaw that generates electronic false positives – signals that look like a planet but correspond to nothing in the real world at all. My purpose here is to characterize that flaw in order to better determine which of the Threshold Crossing Events (TCEs) detected by Kepler are real.

One of the challenges for Kepler’s transiting method is determining when temporary dimming events in a star’s brightness are caused by a real planet transiting its star and when they are due to random fluctuations, called noise. This determination begins with the calculation of a TCE's signal-to-noise ratio (SNR), the strength of the dimming signals divided by the size of the variation that is typical in measurements of that star’s brightness. When SNR is high, the likelihood increases that the observation was of a real event, and not noise.

Observations take place over time. For a planet orbiting a sunlike star in an earthlike, the signal – a real transit – is up to about 12 hours long, whereas the noise is calculated over a period of up to 38 days. A problem with this approach is that changes in the performance of the electronics in Kepler’s instrument cause noise to vary significantly on time scales less than 38 days, so the actual noise at the time of a dimming event can be significantly higher than the average over 38 days. This means that noise is underestimated, and therefore SNR is overestimated. That is the root cause of an excess of electronic false positives: A modest discrepancy in SNR corresponds to a large increase in the probability of spurious detections. For example, a 20% overestimate in SNR, given a Gaussian distribution of occurrence and a detection threshold of SNR>7.1, increases the false positive rate by a factor of 10,800 (from 6.2•10-13 to 6.7•10-9).

The Kepler data processing pipeline is sophisticated, and the factors cited above are generally known to the mission team. Nevertheless, they are not fully accounted for. What I offer here is not a complete, analytical correction of the flaw in the pipeline, but a demonstration that unaccounted-for variations in noise produce the overwhelming majority of false positives in the TCE pipeline, and a starting point for accounting for that flaw.

Seasonal Noise

Noise varies widely for different Kepler targets, and for each target it varies over time. One way of measuring noise in Kepler light measurements, called Combined Differential Photometric Precision (CDPP), have been published for each Kepler target, for each quarter that it was observed. A formula that predicts the rate of Annual TCE false positives is as follows:

For a Kepler target, t, observed in three quarters x, x+4, and x+8, with the lowest three quarters of CDPP for t occurring in quarters a, b, and c, we define the seasonal noise, NS(t, x) as:

NS(t, x) = (CDPP(t, x) + CDPP(t, x+4) + CDPP(t, x+8))/(CDPP(t, a)  + CDPP(t, b)  + CDPP(t, c))

For example, for a Kepler target, t, with CDPP = 40 in quarters 1, 5, and 9, and CDPP = 20 in all other quarters, the
NS(t, 1) = 120/60 = 2, and
NS(t, 2) = NS(t, 3) = NS(t, 4) = 60/60 = 1.

This formula is motivated by two inferences: That the minimum noise over three quarters defines a useful baseline for a target's noise and that quarterly excess in comparison to that baseline will correlate with the potential excess noise at the time of a false transit in comparison to the ~38-day time scales over which the Kepler pipeline calculates noise. These inferences are shown to be justified by the empirical results given below, showing that the formula is excellent at predicting the circumstances in which false positives occur.

Results: Noise and False Positives, All Kepler Targets

113,860 Kepler targets were observed each quarter during Q1-Q12. Each such Kepler target was therefore observed during four seasons, where a season consists of three quarters, {x, x+4, x+8}. Each target-season combination has an associated value of seasonal noise, according to the definition given above.

For each Kepler target in this sample, we can rank its four seasons in terms of NS. If NS does not correlate with the detection of Annual TCEs, then we would expect its four seasons to have, on average, the same number of detected Annual TCEs. However, the set of all targets' lowest-NS seasons yielded only 8 Annual TCEs, whereas the set of highest-NS seasons yielded 866 Annual TCEs, 108 times higher than the lowest-NS seasons, and 88.4% of the total detected in all four seasons. We also see a strong effect of NS on the detection of multiple Annual TCEs. There were 168 targets with multiple Annual TCEs observed in the same season, and for 157 of those (93.5%), it was in the target's season of maximum NS.

The number of real planets transiting stars is unrelated to variations in Kepler's performance, so the excess seen in noisy quarters is wholly due to electronic false positives, and make up the great majority of all Annual TCEs. These results make it clear that seasonal noise is overwhelmingly the governing factor in the generation of false positives in the Kepler pipeline.

Results: Noise and False Positives, Sunlike Stars

To focus on finding earthlike planets around sunlike stars, we narrow the focus to stars that are about as hot as the Sun (classes K, G, and F) and about the same size (logg, a measure of the star's density, between 4.0–4.9), and we only consider Annual TCEs with periods that are earthlike (300-425 days). We also loosen one constraint from the previous analysis by including stars that were observed for as few as six quarters out of Q1-Q12.

It is easier to comprehend the data if we put the count of Annual TCEs in terms of the number of planets per star implied if all the detections are real. To do this, we need to estimate the completeness of the data: For every such planet that actually exists, how many could Kepler possibly detect? For the aforementioned earthlike/sunlike conditions, only 0.47% of such planets will have orbits aligned to allow a transit, and only 23.3% of those planets will happen to have had three transits recorded while Kepler was making observations, so only about 0.11% of all such planets that exist could possibly have been detected. It's actually worse than that if we take into account the fact that small planets can be lost in a star's noise, whereas for large planets like Jupiter and Saturn, that's rarely a concern. Here, we'll ignore that and note that any results we get could be considerably higher if we took into account planet size. We will call the number of planets in earthlike orbits per sunlike star ƒE, equalling the count of Annual TCEs per Kepler target divided by 0.11%, and remember that our measure of ƒE ignores the undercount of small planets.


Recalling the observation that certain areas of the Kepler instrument’s detector are responsible for an excess of Annual TCE detections, we call those areas, shaded pink at right, Hot Zones, and the remainder of the detector surface Cool Zones. In the graph below, we show ƒE as calculated for the Cool Zones in blue, and for the Hot Zones in red. In addition, we show the incidence of systems with multiple Annual TCEs, scaled in the same way, as dashed lines, also separated into Cool and Hot zones using the same color coding.



We see, as in the previous section, the significant impact of NS on the detection of Annual TCEs. For higher levels of NS, even in the Cool Zone, the detection is approximately 30 times that of the lowest values of NS. Hot Zones show higher rates of detection than Cool Zones, which indicates that NS alone does not account for all of the factors governing false positive generation, and I would suggest that this is because quarterly CDPP correlates with noise fluctuation over the shorter ~38-day time scale of noise calculation highly, but imperfectly. We also see that the proportion of targets showing multiple Annual TCEs is low, but not zero, in the lowest-NS condition, but nearly equal to the number of single-detection cases for the noisiest cases at right. The NS metric identifies the leading cause of electronic false positives, and further refinement that takes into account shorter time scales might be able to clean up the results considerably.

Real planets in earthlike orbits


It is intriguing to consider the cases of least noise (the solid blue line, furthest at left). This is the condition with the smallest number of electronic false positives. (Besides electronic false positives, there are also astrophysical false positives, which are larger bodies that only seem to be the size that Kepler seems to show, which is another story.) The analysis here doesn’t show that the rate is zero for the lowest values of NS, but it is lower than elsewhere. In my next post, I’ll examine the 87 TCEs that resemble earthlike planets orbiting sunlike stars and use NS as a filter to begin examination of the ones most likely to be real.

No comments :

Post a Comment