Forum Twitter Facebook Google + RSS

Author Topic: -.-  (Read 4655 times)

Offline SiliconVoid

  • Newbie
  • *
  • Posts: 9
« on: March 08, 2014, 10:09:09 pm »
This a comment for this page on the website
Once again we hear a DxO proclamation that this or that provides the best image quality over this or that.. yawn..
Yet little has changed over the years, photographers continue to purchase, in greater numbers, the cameras that are no where near the top of the grand list.. Why you ask? Because the testing methods DxO uses do not mean anything to the majority of the public, they do not take into account the range of operation, the range of output, the range of use of said product.
To give an example, the current publicly ranked 'king' D800e, can only produce its 'advertised' results at ~100-200 ISO (plenty of studio work in that range) BUT only in the form of a ~75% downsampled ~8mp image. While that may be fine for a percentage of D800/e users, the reality (and reason for its corporate revenue killing sales) is that the majority of users are wanting to output full resolution 36mp images in some form. Everything from full rez prints (landscape, macro, sports) to the 'croppers' (wildlife, sports, daily, peepers, journalists, etc) where the IQ performance at 1:1 is what matters. It is in this area DxO is apparently under the impression that no one uses a digital camera for anything but 100iso 8mp 8x10 output, so there is no need to evaluate that area of performance in our ranking... and even that thinking would be fine if said camera produced its ranked results through its operating range.
**It is here we find the crux of the story**
It does not take much effort to click a few options in the comparison tool to see that the D800/e are the worst performing Nikon FF camera when using native resolution in any form, and at base ISO! Even if one were to speculate that the performance curve from native to 75% downsample was in its favor, one would be wrong. It also does not take much effort to see that of the cameras that outperform the D800/e (D4, D4s, D600, D610, D3s, D700) their performance at their native resolutions surpasses the D800/e even when downsampled to those resolutions. Meaning that even when downsampled to 24mp, the D800/e cannot match the output of the D610 at its native 24mp... it cannot match the 16mp performance of the D4 even when downsampled to 16mp.

Understand this is not a bash on Nikon, or the D800/e, merely the most identifiable examples of the two primary questions DxO fails to satisfactorily comment on for unknown reasons.
1: What possible reason could there be that DxO does not 'rank' sensors based on their entire operating range? We all understand that a certain area of photography world rarely has to go above 100iso, but these tools are not designed solely for 100iso. In actuality the rest of the photography world shoots considerably higher than base ISO. Certainly that would mean that the sensors testing, measurement, and ranking should reflect its performance across its ISO range and not just one setting...
2: What possible reason could there be that DxO does not 'rank' sensors based on their entire output range? We all understand that there are certain areas of photography that operate in the 8x10 world, but there are exponentially more images printed in magazines and other forms of media that are much smaller. Therefore allowing all equipment being tested to benefit accurately from the same percentage of downsampling. So instead of focusing on what one camera can achieve at some arbitrary output setting, DxO could certainly test, measure, and rank the performance of the device across all possible output ranges.
Then rank the device based on an averaged scope of its performance.

It has been mentioned by DxO that the ~8mp base ISO output fills some esoteric demand in the photography industry, and therefore represents an established criteria for measurement. However that niche was established back in the film days where an 8x10 or 8x12 was essentially the max output of 35mm film - not to mention the largest size the average person could reliably get printed. One could print larger, but if limiting the print to the highest amount of detail, sharpness, and color whether scrutinized by eye or loupe, that was around 8x11. Today we have magazines that are A3/B3 in size, so focusing on the size of the print is not really the optimal measurement criteria. We are also in a time where digital media is larger than print media by a quantum measure, so an 8mp image sample becomes even more arbitrary when the typical 1080p monitor only needs 2mp of data to fill it. We understand that there is a threshold, a point of diminishing return, in downsampling so why not decide on a percentage as opposed to output size. If a digital image of something like a D800/e no longer returns appreciable benefits beyond say a 75% downsample, then pick that as the ratio that all images are measured at. For a 16mp sensor a 75% downsample is not going to result in a 240-300dpi 8x10, but it would still produce an image size used significantly across other printed media - but more importantly it represents the beneficial result of a specific downsample that any sensor cam achieve and be fairly measured on. The higher the mp of the sensor the more knowledge and assurance that there is no more to be gained and most importantly nothing lost, so no brand or model is disadvantaged due to mp count versus downsample ratio.

The most significant discrepancy however is in the sensitivity range of the equipment being tested, as it clearly does include any calculation of the poor performance curve in some equipment in the 'scoring'.. Here again we have a situation where DxO has commented that there would be no clear range in which to measure and rank equally all devices given the advancements of technology. That is simply a load of self-absorbed empirically asserted horse manure. Just pick a range, any range, and go with it.. If the majority of digital imaging devices today are designed to operate between 100iso to 6400iso, then pick that range, or 100iso-3200iso. For older devices that cannot achieve the chosen ISO range well they have already been surpassed by modern devices in every way, so they would continue to be ranked below newer devices regardless. As newer devices are released they should improve on that performance range as mere consequence of trying to reach for higher ISO ranges. So If XX camera today achieves an average ISO score of 2900 for a designed 100-6400iso operating range, and two years from now YY camera improves the operating range to 100-51200iso without improving, or even depreciating, their previous performance then they are not working in the right direction and would be deserved of a lower score - otherwise their score within the same range should increase, regardless of any higher operating ranges designed, and ranking would continue to reflect that. See, it isn't that hard to really 'measure' and 'rank' the equipment without giving any advantage to one or artificial disadvantage to another.

More than anything else it would accurately test, measure, and 'rank' the devices on what they were designed to do, for any user, under all design conditions, instead of some esoteric criteria. The measurement and ranking methods used by DxO are certainly unique to DxO, and perhaps that is why you feel so validated in your empirical proclamations. To give some perspective of the anomaly though, if your testing, measurement, and ranking methods were applied to anything else we would have the best performing automobile based on the speed it can achieve while coasting downhill during a tail-wind with two of its four windows rolled down and tires under inflated by 15psi...
Of course one could make those 'settings' on every product tested, and therefore feel quantified in the results, but there are many more variables of the products design and operation range that determine its performance that any badge of accomplishment not based/averaged on overall operation is meaningless.

Offline zodiacfml

  • Newbie
  • *
  • Posts: 7
Re: -.-
« Reply #1 on: March 09, 2014, 06:16:14 am »
I understand your point but DXO simplified many things so that ordinary consumers can make a sound judgement in terms of which is better, albeit, in a small but still relevant typical usage range.
High ISO performing cameras can easily be achieved by tweaking and sacrificing low ISO performance. This seems what is Canon doing now which is why the performance of their sensors at High ISOs remain competitive versus Nikon despite using way older sensors and technology.

In my opinion and personal preference, low/base ISO performance is still relevant and top priority (unless or until smaller format cameras can compete with FF/MF cameras in low ISO of the same generation).

On topic, this post of DXO is the poorest of them all.  Why include and hype a video camera in a sea of digital cameras?

As an authority in sensor benchmarks, I would like them to investigate the image quality differences between CCD and the new CMOS implementation in MF cameras. 

Offline leongqijin

  • Newbie
  • *
  • Posts: 1
Re: -.-
« Reply #2 on: March 10, 2014, 02:45:26 pm »
You seem grossly misinformed about the image size issue. DxO has to choose a standard image size to compare all the image from in a fair way. Yes, DxO can use a larger image size, but that only change all of the scores of the cameras altogether. Their scores are still the same compared to each other, relatively, their scores are still the same. Comparing a normalized 8MP picture and a normalized 16MP picture won't be very different in terms of relative performance.

And take note, 1 or 2 points in DxOMark scores are well within their margin of error. I often see people comparing a sensor with another which have about 100 ISO difference, which is just 1/10th of a stop for larger sensors.

Yes, DxO can review the entire operating range of a camera, but that would really be unfair to cameras which have a greater operating range but as we all know, image quality normally suffer at that range. And, in fact, they do review the entire operating range of a camera - just look at specific scores. For fair comparison, you can't account for those into the overall score. The overall score is just a simplification of performance according to their set standards. For more detailed information about your use case, why not delve deeper into those specific scores? With so many variables, it's not possible to include all of it into the final score. The only way is to find a set of items that allows effective comparison between sensors.

Anyway, my point is, DxO isn't any wrong in their way of testing the sensors and their method of scoring sensors. But for anyone other than beginners or a very general use case, I suppose you would go deeper and compare between sensors in a more precise way. That is what DxOMark is for.

Offline randybenter

  • Newbie
  • *
  • Posts: 33
Re: -.-
« Reply #3 on: March 10, 2014, 10:27:01 pm »
That is a very long post. You could have saved a bunch of time by just posting this instead: "I have no clue why DXO equalizes images in order to make comparisons fair, nor do I know how to read their charts showing the test results at all ISOs".

Maybe if you post questions next time, instead of a rant, then you could learn how to interpret DXO test results.

Offline shawnington

  • Newbie
  • *
  • Posts: 1
Re: -.-
« Reply #4 on: March 12, 2014, 10:40:03 pm »
Yes, the results are a bit biased, as some who shoots with both a D4, and a D800, there is no comparison. The d800 absolutely slaughters the D4 in image quality in most situations. Only real advantages the d4 has are when the light gets very low, and framerate is important.

Offline martinmartinov

  • Newbie
  • *
  • Posts: 4
Re: -.-
« Reply #5 on: March 23, 2014, 02:10:02 am »
Let's take S/N. As I get this, DxO uses statistical parameters to determine the sums or averages of discrepancy/deviation of all deviations on a pixel level of all 8 million pixels. So when you downsample the image, you get better results of course. If the original file is better, then the downscaled file will better as well. So there's your downscaling answer. Btw D800 is excellent, plus what's the point of purchasing D800? Studio work (ISO 100 mostly). If you don't like the overall scoring then look up the measurements, they apply to all ISOs at all parameters. At first I was doubtful about the scores, but in reality there is a difference between Nikon and Canon, just as indicated by DxO. I've shot with 5DMk3 (the newest 5D) and an APSC Nikon 16MPix Sony sensor (D5100). Believe it or not the Nikon clearly showed better performance in the DR showing considerably less noise when working in a RAW converter (I didn't use DxO, but other). So, I realized that regardless of the format the Nikons really show better performance in DR. The detail, of course was a bit better albeit slightly on the Canon, a parameter DxO doesn't value at in sensor ratings, but in lens ratings which is clearly understandable because the lens determines the IQ ;)