How to Interpret Thom's Lens Ratings
This article applies to all lens reviews starting 2/8/10 that get posted to this site. Earlier reviews may not yet be updated to reflect these statements. Check the "Last Updated" date to see if it is post 2/8/10. If it is, the review uses these guidelines. If not, it uses older ratings that are more lax and will eventually be updated to the new ratings.
In doing more lens reviews and getting them deployed to the site plus getting ready for a move to a new design, I decided that it was time to get a bit more formal with my rating system for lenses. This should help make them more consistent and usable. I now rate nine things: overall value, plus features, focus speed, optical quality (previously called performance), and build quality rated on both an absolute scale and a value scale. I rate all of these items on a scale of 1 (lowest/worst) to 5 (highest/best). It's also possible that you may sometimes see a value of 0, which means that I haven't gotten around to rating that item yet.
Here's what the ratings mean in each category:
- 5 (Exceptional). Means that the lens has all the goodies that one could expect from a lens, perhaps even some additions you might not expect. For telephoto lenses, for example, VR, a solid but rotating tripod mount, manual focus override, a state-of-the-art maximum aperture, a quality lens hood, the ability to use filters, plus perhaps other things like the ability to restrain focus to smaller ranges, weather sealing, focus hold buttons, and so on, would get the lens a 5 on the absolute scale. A macro lens needs to get to 1:1 to obtain this rating, and should have enough working distance to adequately light a subject at minimum focus distance. Wide angle lenses need front element protection, the ability to use filters, a good lens hood, and an accurate and usable distance scale to obtain this rating. All zooms would require VR to get this rating, and they need to have a reasonable and necessary focal length range. Prime lenses are the toughest call, as by definition they tend to be simple in design definition. However, a prime would need a near state-of-the-art maximum aperture, Nano coating, AF-S, and perhaps more to obtain this rating. On a value scale, things change. At high prices, you expect a high-priced lens to have most of the things I've outlined here, so to achieve a five star rating on the value scale a lens has to have many features you don't expect for the price.
- 4 (Above Expected). The lens has somewhat more than the expected feature set, meaning that it has something well beyond the basic focal length expectations. That could be a faster-than-normal maximum aperture, addition of some useful features like VR, DC, close focus, or a focal range that extends beyond what is normal for the class of lenses to which it is normally judged (e.g. a 24-105mm lens versus a 28-105mm lens). On a value scale, to achieve this rating a lens needs to have at least one feature you wouldn't expect for the price.
- 3 (Expected). Most lower-cost consumer lenses automatically fit into this category, as they have an expected focal range, a modest maximum aperture specification, and few if any additional features. Barebones primes like the 50mm f/1.8D also tend to live in this category. The defining element is "100% rudimentary feature set." Most lower-cost consumer lenses are likely to fall into this category; indeed, they are normally designed to hit this category, both in absolute and value terms. In value terms, this rating is basically "you get what you pay for."
- 2 (Below Expected). The lens is missing one or more things that are really necessary for it to be fully useful, or has less focal length range for an equivalent lens than is commonly accepted now. This is a tricky category, as there are lenses that have some extras but are missing others. On a value scale, I wouldn't rate something as a 2 unless the missing feature(s) clearly detract from the usefulness of this lens. For example, the old 300mm f/4 would probably be a 2 compared to the newer 300mm f/4 AF-S being a 3. Both are missing VR, which for a telephoto lens is a bit of an issue. Both have a basic tripod mount, though not a solid enough one to be considered beyond the expectation. But the AF of the older model is sluggish (even on high-end cameras) and you really need good AF performance on a telephoto of this length. Thus, the new version manages to fall in the "expected" category and gets a 3, the older version fails to quite live up to purpose, so gets a 2.
- 1 (Fails completely). I don't expect many lenses to fall this low in their features rating, but if they do, it means that I just don't feel that the feature set has anything in it that would compel you to buy it. On occasion, a flaw in feature definition might doom a lens to this rating. For example, an 85mm f/3.5 without VR or AF-S and with a close focus distance of 12 feet would almost certainly live here, because the combination of long focus, 85mm, and f/3.5 simply doesn't add up to anything usable (you'd never be able to use this lens for portraits, for example, because you couldn't get close enough or throw the background out of focus enough, and the slow aperture and lack of VR/AF-S probably dooms its use for other things). Lens designers rarely make mistakes that would put a lens in this category, though. If they do, my suggestion is that they find another job. On a value scale, it's possible that a lens falls into this category because it is high priced and has less than the expected feature set.
While this will be controversial, I won't take into account whether a lens has an aperture ring or not in determining this rating. It seems clear to me that Nikon has moved to no aperture rings. While some will bemoan the loss, it also means fewer points of failure and a better ability to seal the lens mount.
My expectation is that most lenses will end up getting 3 to 5 ratings for features on the absolute scale, 3 or 4 on the value scale. None of these ratings should be construed as a lens you avoid: the value rating is most useful comparing two similar lenses you're considering.
- 5. State of the art, period. Snaps to accurate focus with no hunting.
- 4. Fast, responsive focus performance that's accurate; rare hunting.
- 3. Needs to clearly exceed that of most compact cameras, but isn't always snap-to fast, and may tend to hunt sometimes.
- 2. Slower than expected focus performance, barely above compact camera focus speeds.
- 1. Slow and/or imprecise focus performance.
A handful of exotic lenses will manage a 5 in this category. Most AF-S lenses should achieve a 4 on the absolute scale. A rating of 3 on the absolute scale should cause you to investigate whether the lens has the focus performance on your camera that you require, as in this category a 3 is decidedly average at best. Lenses with ratings of 1 and 2 should be considered manual focus lenses; in other words, if you buy one, don't expect the autofocus system to give you acceptable results, you're better off tweaking focus by hand. In general, value ratings tend to be a bit below absolute ratings in this category. It's a rare low-priced lens that has absolute focus performance.
- 5. No flaws for which you'd immediately invoke post processing. State of the art performance in the corners. State of the art performance at maximum aperture. Very high MTF results, generally performing at maximum resolution of the cameras on which it can be used.
- 4. Lens may have a visual flaw you might want to correct, like linear distortion, or a modest visible issue that you'd invoke a tool to correct completely (minor chromatic aberration). Should have excellent performance at optimal aperture even into the corners, but may have one or more modest issues at maximum aperture. Very high MTF results at optimal aperture, but perhaps some modestly lower numbers at maximum aperture.
- 3. You'll likely find several things that you want to correct in post processing, or aperture and focal length combinations that you need to avoid to achieve good visual results with this lens. It may not perform at high levels of quality at maximum aperture, for example. Of, if it's a zoom with a long focal range, the extreme focal length(s) may be weaker than the rest of the range. Linear distortion is either extreme, or complex, and thus harder to remove. Chromatic aberrations may be highly visible and harder to remove. But overall, the lens is, with the right settings and post processing, capable of producing visual results that match the cameras on which it can be used. Good but not state-of-the-art MTF numbers, usually with lower numbers in the corners and/or at maximum aperture.
- 2. The lens has a significant optical flaw that cannot be corrected or applies to the entire range. Mis-centering of lens elements can cause this, but so can a lens that simply was designed for optical standards less than the current cameras can capture. Lower-than-the-test-camera MTF numbers, even at optimal aperture. Often edge-to-edge differences.
- 1. Lens simply failed to fall in the category I'd call "usable" even at optimal aperture. Often has low MTF numbers and/or very visible optical flaws in pictures.
Most modern lenses should be a 3 to 5 on the absolute scale. It's a rare lens that I would give a 5 to on the value scale here. Consumer and lower cost lenses tend to be near 3 in this category for value almost by definition: it's expensive to produce lenses that produce very high MTF numbers and have only modest flaws, which is what it takes to get a 4 in my system. Many of Nikon's recent consumer lenses actually do manage to get to 4, which shows that Nikon has both the design and production issues of producing really good lenses at moderate prices down pat. But, in general, the more you pay, the higher the number here tends to be, in both absolute and value terms.
You may have noted the "maximum resolution of the cameras on which it can be used" phrase. This is my attempt to finesse some issues that come up with DX versus FX. I test DX issues on DX bodies and base my ratings on this. Some DX lenses can be used on FX bodies at some focal lengths, but have less than stellar performance in the FX corners. I wouldn't penalize their rating for that, though.
- 5 (pro). The build quality of a pro-caliber lens has a lot of very minor things that start to come into play. For example, focal length extension doesn't have side-to-side play in the barrel. The lens tends to stay the same size and nothing rotates. The lens should be weather sealed. The rings are smooth in rotation, easily identified, and easy to grip. The lens barrel can sustain minor to moderate bumps and hits without impairing performance. Buttons are easily found and identified, and have clear positions they snap to. Tripod mounts are solid and show no play at all. Glass and/or high quality elments are used throughout. Expensive front elements are protected by clear filter supplied with the lens. Everything is clearly marked.
- 4 (better than consumer). The reason a lens would fall from a 5 to a 4 rating is that some attention to the detail mentioned above is missing, but the lens overall is still of very high quality.
- 3 (consumer). Narrow rings that are perhaps not smooth. Lens barrels that have potential side-to-side shift when extended. Lens barrel might sustain a minor bump without impairing performance, but confidence level in that result is not high. Materials used are less than highest grade or have "feel" issues. Buttons may be vulnerable to damage if hit, or don't move securely from position to position.
- 2 (worse than consumer). Clear vulnerability to rough handling. May have Inferior materials that show cosmetic changes/wear with use. Often have Sticky, jerky rings. Materials feel significantly cheap. May have wide use of plastic lenses, especially front element. May come with clear poor workmanship.
- 1 (inadequate). A lens that you'd immediately return to the seller.
Most modern lenses should be a 3 to 5 on the absolute scale, a 3 or 4 on the value scale. Consumer and lower cost lenses tend to be near 3 for value in this category almost by definition: they often use materials and construction techniques that place them here. A few of Nikon's recent consumer lenses actually do manage to get to 4 in Build Quality value, but in general, the more you pay, the higher the number here tends to be, both in absolute and value terms.
- 5. The lens is clearly underpriced for what it does and how it does it. It needs to outperform higher priced lenses to get this rating, and usually be significant margin. I would also expect such lenses to hold their value well if well maintained.
- 4. The lens is arguably a better value than competitive offerings, but doesn't clearly excel on all counts compared to those others. It may have only one or two features that give it more value than the competitors.
- 3. Reasonably priced for what it does and how it's built. Basically, you'll be getting what you expected, nothing more, and nothiing less.
- 2. The lens is behind its competitive offerings in one or more points, yet the price isn't enough lower to justify those drawbacks.
- 1. The lens isn't usually worth purchasing. Yes, that's harsh, but the only reason you'd purchase a lens in this category is that the optical aspects aren't obtainable in any other way. Otherwise, the competitive products would be the clear choice.
Again, most modern lenses should be a 3 or a 4 in this category. You might ask how some portion of "most" could get to 4 in this category when the definition is that 3 is basically the expectation. Chalk it up to progress. Each new generation of lenses seems to push features and performance up a notch while keeping price level or pushing it down, so a new lens can often garner a 4 in this category if it is top notch. Sometimes competitive pricing also pushes a new lens in at a price that would undercut the current leader. As with several of the categories, it's really only the 1 and 2 ratings that should cause you any pause prior to purchasing.
Overall, you'll note that "average" or "expected" or "normal" should all garner a lens about a 3 in my rating system. A 3 is not a death warrant in this rating system--I own and use several lenses that are clear 3's in many categories. What I've tried to do is build a system that will more clearly differentiate a lens that stands out (5) from one that is competent, but unremarkable (3), plus will also clearly tell you when you need to be very sure that the lens does what you need it to before purchasing it (1 or 2 ratings). Also, by splitting the absolute and value ratings for the four primary categories, you can tell if a lens is basically as good as it gets (5 in absolute terms) but perhaps a little pricey (4 in value in the same category) or even clearly overpriced (a 3 in value for the same category as it gets a 5 in for absolute performance).