Tag Archives: best

The posts that aren’t all that bad…

Studies show reading this essay will make you smarter

Recently, there was an interesting article in BusinessWeek about the flip-flop of studies on the efficacy of echinacea to cure the common cold. The article focused on the possibility of incorrectly performed studies. But, there may have been nothing wrong with any of the studies, even though they differed in their results. The statistical nature of clinical studies means there is always a small possibility that false effects will be seen. However, biases inherent to statistical research may result in a surprisingly large percentage of published studies being wrong. In fact, it has been suggested that the majority of such studies are.

First, I’ll have to briefly explain something about how statistically-based studies are done. When people do such trials, they consider as “significant” any result that would only happen by chance 1 in 20 times. In the language of statistics, they design the study so that the “null” hypothesis (e.g. that echinacea has no effect on a cold) would only be rejected falsely at most 5% of the time based on the normal random variability expected in their study. In other words, they accept that 5% of the time (at most) they will erroneously see an effect where there truly isn’t any. This 5% chance of a mistake arises from unavoidable randomness, such as the normal variation in disease duration and severity; in the case of the echinacea studies you might just happen to test your drug on a group of people who happened to get lucky and got colds which were abnormally weak.

In summary, to say a drug study is conducted at the 5% significance level, you are saying that you designed the study so that you would falsely conclude a positive effect when there were none only 5% of the time. In practice, scientists usually publish the p-value, which is the lowest significance (which you can only compute after the fact) that would have still allowed you to conclude an effect. The main point, however, is that any study that is at least significant at the 5% level is generally considered significant enough to publish.

So, being wrong at most 1 in 20 times is pretty good, right? The cost of putting a study out there that is wrong pales in comparison to the good of the 19 that actually help, right? Does it really matter if, of the 1000s of studies telling us what we should and shouldn’t do, dozens of them are wrong? In theory, there will always be an order of magnitude more studies that are truly helpful.

The problem is, this conclusion assumes a lot. Just because the average study may have a p-value of, say 2%, it doesn’t mean only 2% of the studies out there are wrong. We have no idea how many studies are performed and not published. Another way of looking at the significance level of an experiment is “How many times does this experiment have to be repeated before I have a high probability of being able to publish the result I want?” This may sound cynical, but I’m not suggesting any dishonesty. This kind of specious statistics occurs innocently all the time due to unknown repeated efforts in the community, an effect called publication bias. Scientists rarely publish null findings, and even if they do, such results are unlikely to get much attention.

Taking 5% as the accepted norm for statistical significance, this means only 14 groups need to have independently looked at the same question, in the entire history of medicine, before it’s probable that one of them will find a falsely significant result. Perhaps more problematically, consider that many studies actually look at multitudes of variables, and it becomes clear that if you just ask enough questions on a survey, you’re virtually guaranteed to have plenty of statistically significant “effects” to publish. Perhaps this is why companies find funding statistical studies so much more gratifying than funding the physical sciences.

None of what I have said so far is likely to be considered novel to anybody involved in clinical research. However, I think there is potentially another, more insidious source of bias that I don’t believe has been mentioned before. The medical research community is basically a big hypothesis generating machine, and the weirder, the better. There is fame to be found in overturning existing belief and finding counterintuitive effects, so people are biased towards attempting studies where the null hypothesis represents existing belief. However, assuming that there is some correlation between our current state of knowledge and the truth, this implies a bias towards studies where the null hypothesis is actually correct. In classical statistics, the null hypothesis can only be refuted, not confirmed. Thus, by focusing on studies that seek to overturn existing belief, there may be an inherent bias in the medical profession to find false results. If so, it’s possible that a significant percentage of published studies are wrong, far in excess of that suggested by the published significance level of the studies. One might call this predisposition toward finding counterintuitive results “fame bias”. It may be how we get such ludicrous results as “eating McDonald’s french fries decreases the risk of breast cancer,” an actual published result from Harvard.

Statistical studies are certainly appropriate when attempting to confirm a scientific theory grounded in logic and understanding of the underlying mechanism. A random question, however, is not a theory, and using statistics to blindly fish for novel correlations will always produce false results at a rate proportional to the effort applied. Furthermore, as mentioned above, this may be further exacerbated by the bias towards disproving existing knowledge as opposed to confirming it. The quality expert W. Edwards Deming (1975) once suggested that the reason students have problems understanding hypothesis tests is that they “may be trying to think.” Using statistics as a primary scientific investigative tool, as opposed to merely a confirmative one, is a recipe for the production of junk science.

How modern art can be so horrid

Every day I walk by Gehry’s Stata center, which overall I have to admit is one of the more interesting and visually appealing modern buildings. The “centerpiece” of the building, however, is this bit of architectural self-abuse:

Stata Center

The building cost almost half a billion dollars to make, and over 15 million dollars went to the architect, Frank Gehry. It was intended to be a masterpiece on the vanguard of modern architecture, representing MIT’s engineering audacity. Upon its completion the head of MIT’s campus development proudly boasted of the genius of the building, breathlessly noting the way the snorkel of the central section playfully echoes and mocks the radar dish at the top of the neighboring Green building. The head of the computer science department waxed poetic about the way the light interacts with the angles and the “spaces.” This kind of guileless, sychophantic adoration by intellectuals in the academic community is revealing of the culture in which contemporary art manages to flourish despite its near general popular rejection (in NYC, the attendance at the Met is five times that of MOMA). [Update: As pointed out by a commenter, this may be a specious argument to use.] What’s most telling is the self-conscious way the praise must always be justified in (pseudo) intellectual terms, as they try to hitch their ego to the train of the artist.

Continue reading

Digital camera buying tip from an engineer

Last week my digital camera (Canon A610) died, after only two years of light service. (It turns out that a large batch of cameras made by Canon late 2005 had a bad CCD connector which tends to die after a year or two.) While I was obviously frustrated, part of me was also secretly happy since it meant I could have the fun of shopping for another camera.

After a fair amount of research (but not enough, as it turns out) I ended up buying an 8 million pixel Olympus camera (SP-560UZ) to replace the 5 million dead ones on my old camera. Progress, right? Not really. I was surprised to find that the 8 MP camera, made over two years after my dead camera, produced images of lesser quality. Sure, they were higher resolution, but that was about it. How is that possible?

I looked a little more into the sensor elements used (called CCDs, for charge coupled devices) and it seems that the digital camera companies generally increase pixel counts without actually making the sensors any bigger. In fact, the CCD on the Olympus is actually about half the area of the Canon! [Note: as pointed out by Leonid, below, they don’t have to much leeway to do otherwise in the megazoom cameras.] So the pixels get much smaller, and since each pixel requires a certain amount of circuitry (that can’t shrink any further) the sensor actually becomes even less sensitive as a whole. Furthermore, the amount of noise experienced by each pixel doesn’t shrink as quickly as the pixel size (for reasons that are a bit complicated), so an 8 MP sensor experiences significantly more noise than a 5 MP sensor. They make up for this with somewhat better CCD technology and clever image processing, but there is only so much that can be done.

Another, perhaps even worse, result of smaller pixel size is that the maximum number of photoelectrons that can be stored in each pixel is lowered. A pixel on a CCD acts like a bucket for electrons. (It’s sometimes incorrectly stated that CCD pixels store photons.) A photon of light hitting the CCD pixel has a certain probability of causing an electron to be “freed” from the silicon and dropped in the bucket. While this is just a metaphor, electrons do actually follow a lot of the same rules as water filling a bucket do. Once the bucket is full, the electrons spill out, often into a neighboring pixel that isn’t yet full. Furthermore, if the bucket (pixel) is shrunk, it can’t hold as many electrons.

The increased noise plus the smaller capacity to hold electrons means that each pixel can’t handle a very large difference between light and dark in a scene. In other words, as you increase the exposure, the pixels “fill up” much quicker than if the pixels were larger. The ability to measure large variatation between light and dark in a scene is called dynamic range. A lack of dynamic range shows up as washed out highlights and lack of detail in shadows. An example of this is shown in the following zoom from a picture taken with the Olympus:

Washed out sky

It looks like a cloudy day, but this picture was actually taken at 3 pm on a nice, sunny day. The blue sky was not bright at all to the eye, but it was enough to cause the pixels which saw the sky to all max out and overflow. The fact that they maxed out is indicated by the pure white that resulted, and the overflow of photoelectrons into neighboring pixels is evidenced by the “bleeding” of the white into the tree branches. Admittedly, this is not strictly proof of anything, as I would have to provide a picture of the same scene taken with a better camera for you to be able to truly verify my claims, so you’re just going to have to trust me that this day wasn’t. Here is the full picture, to show that the rest of the picture was not overexposed:

The Olympus is a great camera in most every way, but it appears that they have pushed the pixel count so high that picture quality has suffered. All manufacturers of cameras in this class appear to do the same thing, and not one of them seems to have the guts to say “enough.” From an engineering standpoint, this pixel race makes absolutely no sense. At some point, adding pixels is counterproductive and actually lowers the effective resolution for most situations, due to the effects of increased noise and loss of detail due to dynamic range reduction. In my opinion, this point was reached on the small sensors used in compact point-and-shoot cameras at about 4-5 MP. Unfortunately, engineers don’t run companies, marketing types do. And the marketing lemmings invariably decide that putting a sticker that says “8 MP” on the side of the camera is more important that the quality of the images it produces. Most likely, they don’t even understand the trade-offs involved in doing so, and only hear the first three words when the engineering manager says “Yes, we can do that, but…”

People buy into this because they, understandably, assume that companies couldn’t possibly be so crass and cynical as to intentially fool people into paying more for an inferior product. There was probably a time when that was a fair assumption, but those days are long gone in the Persian bazaar that is the consumer electronics industry.

The counterintuitive upshot is that you can actually get a better quality image from a $200 low-end camera than from the higher-end $400 model from the same brand. If you’re buying a digital camera, consider intentionally buying a 5 or 6 MP model (if you can still find them) even if you can afford the 8 or 10 MP version. Check the specifications and buy the one with the largest CCD you can find. This means avoiding the cute pocket cameras, if you care at all about image quality. If you need to print poster-sized enlargments that require more than 5 MP, you just need to bite the bullet and splurge for a digital SLR; they use much larger sensors that operate on a fundamentally different read-out principle, and as a result they can produce 10 MP images with incredibly low noise. The idea of a small consumer level point-and-shoot camera with 8 MP is a bit crazy, if you ask me, and a terrible engineering choice.

The net result of all my research is the realization that I had a great thing in that little Canon A610, which makes its loss even worse. It was one of the last models where Canon used a relatively large 1/1.8 inch CCD, and to my eye it struck the right balance between resolution and image quality. After all this, I’m just going to try to find a used A610.