How good are the new cameras, and how was it determined?
With every new camera we get the same set of posts on Internet forums (and even by reviewers who should know better): "A is noisier than B." "B is noisier than A." Then come the quicky reviews that annoint a new camera a "winner" or "loser" in the noise battle (and other image quality abilities, as well).
Really? Based on what criteria?
Well, there's the rub, noise is almost always based on one of two criteria: (1) what people see at 100% pixel view; or (2) a Standard Deviation value based upon a plain gray test target taken at the camera's defaults. Neither of these actually tell you how "noisy" a camera is or isn't. One criteria is subjective, the other objective but not really meaningful.
In the world of sensor and camera design, the engineers look at a number of very specific factors, most notably well capacity, shot noise, and read noise. They actually count electrons and figure out what the verifiable signal-to-noise (SN) ratio is. Ah, there's a magic word: signal-to-noise. Ever see a review that tried to make sure that their test setup was optimized to the full signal-to-noise ratio a camera was capable of? None that I know of (I'll even admit that I don't usually do that, though I do usually test to see what the maximum SN ratio is [on a D300, somewhere just above 11 stops at the base ISO]).
Funny thing is, we've been through this scenario before the parade of DSLRs began. I remember it most with audio equipment, but it's happened with other electronics, as well. First, you get the casual fan-type reviews. Then you get the stately "we've got lots of test equipment and here are the numbers" reviews. Eventually, you get something like the high-end magazine Stereophile produces: a combination of complete and diligent number testing combined with very well trained and thorough experts testing in actual use and taking long enough to form a real opinion about what the product can and can't do. As I write this, we're still in the fan-type reviews (both pro-fan [fanboy] and con-fan [troll], by the way; sometimes it's difficult to tell one from the other).
So we're already seeing people post "it's better than a D200" (or 40D or whatever) and others posting "it's no better than the D200" (or 40D or whatever). I've even seen such posts from less than a day's worth of use of the camera. We've also seen a few number results (including my 11 stop SN ratio number above). Can you conclude anything useful from these quick judgements? Not really. The quantum efficiency of the D300 sensor does seem to be clearly 50%+ better than that of the D200 sensor, for instance, but that doesn't really tell us anything absolutely useful from a photographic sense.
Why not? Because quite a few other factors enter into critical image quality assessments, including some subjective ones. First, we have the JPEG versus NEF differences. JPEGs are processed in-camera and have lossy compression in them. NEFs are processed out-of-camera with a variety of possible converters and you don't (usually) have compression losses (unless you convert to JPEG ;~).
For in-camera JPEGs, the camera settings come into play. Push sharpening up, for example, and you'll see more noise as any neighbor value changes are interpreted as edges and increased in contrast. Push noise reduction up and you mask the noise at the expense of detail. Even the color controls come into play, as in incandescent light, for example, you don't have as intense blue wavelengths as red, so you end up with a "noisier" blue channel than red; select any control that boosts blues and you might suddenly see more noise. Oh, and all those folks testing in sRGB: you're not seeing the full capability of what the camera can do, as the reduced Color Space also comes into play (which brings up a question that Nikon hasn't answered: what Color Space is being used in internal processing? Is it the user-selected Color Space or is it something like ProPhotoRGB and only reduced to sRGB or AdobeRGB at the last moment, just like the JPEG compression?).
But even when we go to lossless 14-bit NEFs in AdobeRGB Color Space we still end up with a lot of issues to deal with. Basically, interpretation of raw conversions becomes a "how good is the person doing the conversion" game, followed by a "how well can the person see and interpret differences" game. I trust my own processing and eyesight, but can I trust Joe Internet?
Moreover, how's the image going to be used? People who reduce to 1/4 size for use on the Web have a different assessment of image quality than those printing on top-quality devices at 24x36". Even amongst a single group--let's pick on wedding photographers today ;~)--you'll find differences in opinion. Some wedding photographs believe that you only have to go to 8x10" and "just good enough" while others think they need to produce works of art that will pass muster when printed to a wall-sized murals examined at closer than a foot. (Okay, I exaggerate, but there really is a large range of what wedding photographers consider acceptable in terms of image quality.)
Quite a few people are waiting for my judgement on the D300 and the D3. All I can say so far is that the D300 appears to be somewhat better than the D200 when the camera is set optimally, and when conversions are done optimally. How much better I can't say and won't until I've had a chance to shoot with the D300 for some time and do more exhaustive side by side tests and conversions. On the other hand, the D3 is clearly better than any previous Nikon DSLR, though again how much better is open to question and requires a lot more use and testing to determine. These are very important cameras for Nikon, so I'm not going to rush to judgement. Just the opposite, in fact. I intend to shoot with both, shoot many varied subject types with both, and convert images from both for some time before trying to write any critical analysis of their image quality, especially noise (and yes, that involves my "basketball test").
Don't be a rusher to judgment. Your world isn't going to collapse if you don't know just how good the D300 and D3 are in the next two hours.