Creating Raw Histograms From Digital Camera Raw Files

Contents
- Click here to see a list of all pages related to DSLR camera testing

Click Here, the Mystery of the Canon DSLR Combing Artifacts in Raw Histograms is Solved!

The Raw Histogram Creator Program

I wrote a quick-and-dirty program to create raw histograms from Canon (and others?) digital camera raw files. It is a command line (console) Windows application - no installation is required. The source code is available here if you are interested.

Usage:
rawhistogram.exe [-?][-help] file
	-?, -help	Gives a help message
	file		PGM input file created with dcraw.

The input file must be a raw file processed into 16 bit raw PGM by using dcraw. For example if you have a Canon raw file called IMG_0005.CR2 first use dcraw:


	dcraw -D -4 -t 0 -o 0 -v IMG_0005.CR2

this will create IMG_0005.pgm. Note that the options are critical to get the right (16 bit raw) conversion - you can omit "-o 0 -v" if you want but the rest are required. Now run:


	rawhistogram IMG_0005.pgm

The result will be a (text) file called IMG_0005.csv that contains the raw histogram data. This file can easily be opened in a spreadsheet program (e.g. Excel) to plot the RGGB histograms.

To try the raw histogram creator (v.0.93 updated to handle 14 bit cameras and create statistics) on your own raw files you can download this file:
findcombedraws.zip
it contains rawhistorgam.exe, dcraw.exe and a batch file - findcombedraws.bat - that helps in processing many raw files automatically and searching for combed ones. Please refer to the included README.TXT to find out how to use the batch file.

The version of dcraw I used is compiled from this version of the source and Little cms version 1.15.

Raw Histograms From Canon 30D and Other DSLR Cameras

Using the above program I created raw histograms of some photographs shot with a Canon 30D DSLR. Below you can find the small/fine in camera JPEG files, the CSV files created by rawhistogram.exe and plots made from them. The interesting thing to note are the combing artifacts in the histograms of some of the 30D and 5D photos which should not be there if the data was "pure unmodified raw" as it came from the A/D converters of the camera. For a discussion of the phenomenon see this thread in the DPReview forums. Also I would like to thank the people at the Open Photography Forums who helped a lot in trying to figure out what causes this behaviour. There are some theories as to what is causing the combing in the conclusion.

In the list below the combed photos are marked with a bold "Lens" field.

Canon EOS 30D Photos

Click here for a deteiled test of the 30D.

On the 30D at ISO 100 the pixels saturate before the A/D converters; the saturation level is 3398 raw counts out of the possible 4096. See IMG_1829 for an example, see also the Canon EOS 1D Mark III below. Another interesting fact is that the intermediate 1/3 stop ISO settings of the camera (i.e. ISO 125, 160, 250, 320, 500, 640, 1000 and 1250) are implemented in the firmware by setting the sensor amplifiers for the closest full stop and (mathematically) scaling the raw data up or down. Therefore all 1/3 stop ISOs are combed and this is not a mystery.

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram file
IMG_0005	EF-S18-55mm f/3.5-5.6	ISO 1600, EC 0, 1/200 f/4.5, flash, FEC 0	IMG_0005.png	IMG_0005.csv
IMG_0008	EF-S18-55mm f/3.5-5.6	ISO 100, EC 0, 1/13 f/22	IMG_0008.png	IMG_0008.csv
IMG_0010	EF-S18-55mm f/3.5-5.6	ISO 1600, EC 0, 1/200 f/29	IMG_0010.png	IMG_0010.csv
IMG_0011	EF-S18-55mm f/3.5-5.6	ISO 100, EC 0, 1/13 f/29	IMG_0011.png	IMG_0011.csv
IMG_0541	EF100mm f/2 USM	ISO 100, EC 0, 1/400 f/2	IMG_0541.png	IMG_0541.csv
IMG_0843	EF100mm f/2 USM	ISO 100, EC +1/3, 1/500 f/2	IMG_0843.png	IMG_0843.csv
IMG_0903	EF50mm f/1.8	ISO 800, EC +1/3, 1/250 f/1.8, flash, FEC 0	IMG_0903.png	IMG_0903.csv
IMG_0947	EF100mm f/2 USM	ISO 800, EC +2, 1/40 f/2	IMG_0947.png	IMG_0947.csv
IMG_0963	EF50mm f/1.8	ISO 800, EC +2/3, 1/125 f/1.8	IMG_0963.png	IMG_0963.csv
IMG_1753	EF50mm f/1.8	ISO 100, EC 0, 1/80 f/22, 2006-07-22 15:35:14	IMG_1753.png, IMG_1753_zoom.png	IMG_1753.csv
IMG_1754	EF50mm f/1.8	ISO 125, EC 0, 1/125 f/22, 2006-07-22 15:35:20	IMG_1754.png, IMG_1754_zoom.png	IMG_1754.csv
IMG_1755	EF50mm f/1.8	ISO 160, EC 0, 1/125 f/22, 2006-07-22 15:35:24	IMG_1755.png, IMG_1755_zoom.png	IMG_1755.csv
IMG_1829	EF50mm f/1.8	ISO 100, EC +1/3, 1/30 f/8, 2006-07-23 14:33:38	IMG_1829.png	IMG_1829.csv
IMG_1830	EF50mm f/1.8	ISO 100, EC +1/3, 1/640 f/1.8, 2006-07-23 14:34:26	IMG_1830.png	IMG_1830.csv

Canon EOS 20D Photos

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram file	Source of the photo
IMG_0119	16.0 - 35.0mm	ISO 100, EC 0, 1/125 f/5.6	IMG_0119.png	IMG_0119.csv	20D review at DPReview??
IMG_9505	24.0 - 70.0mm	ISO 100, EC 0, 1/400 f/7.1	IMG_9505.png	IMG_9505.csv

Canon EOS 300D (EOS DIGITAL REBEL) Photos

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram file
CRW_7394	24.0 - 70.0mm	ISO 100, EC 0, 1/317.9 f/8	CRW_7394.png	CRW_7394.csv

Canon EOS D60 Photos

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram file	Source of the photo
CRW_3057	28.0 - 75.0mm	ISO 100, EC 0, 1/201.7 f/5	CRW_3057.png	CRW_3057.csv	http://www.aim-dtp.net/aim/digicam/dcraw/

Canon EOS 350D (EOS DIGITAL REBEL XT) Photos

On the 350D the black point is calibrated at raw level 256 unlike the 300D, D60, 20D, 30D and 5D where it is at 128.

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram file	Source of the photo
XTI_6755	18.0 - 55.0mm	ISO 100, 1/25 f/8, flash, FEC 0	XTI_6755.png	XTI_6755.csv	A very good friend
XTI_6784	18.0 - 55.0mm	ISO 100, 1/8 f/16	XTI_6784.png	XTI_6784.csv	A very good friend
XTI_6787	18.0 - 55.0mm	ISO 100, 1/8 f/16	XTI_6787.png	XTI_6787.csv	A very good friend
XTI_6833	18.0 - 55.0mm	ISO 100, 1/8 f/36	XTI_6833.png	XTI_6833.csv	A very good friend

Canon EOS 5D Photos

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram f/ile	Source of the photo
5DI_0187	EF50mm f/1.4 USM	ISO 1600, EC 0, 1/100 f/1.4	5DI_0187.png	5DI_0187.csv	Mike Robbins
5DI_1285	50.0mm	ISO 1600, EC 0, 1/20 f/1.8	5DI_1285.png	5DI_1285.csv	Mike Robbins
5DI_2259	EF50mm f/1.4 USM	ISO 1600, EC 0, 1/30 f/1.8	5DI_2259.png	5DI_2259.csv	Mike Robbins
5DI_2272	EF50mm f/1.4 USM	ISO 1600, EC 0, 1/30 f/1.8	5DI_2272.png	5DI_2272.csv	Mike Robbins
5DI_2278	EF50mm f/1.4 USM	ISO 1600, EC 0, 1/30 f/1.8	5DI_2278.png	5DI_2278.csv	Mike Robbins
5DI_2303	EF50mm f/1.4 USM	ISO 1600, EC 0, 1/30 f/1.8	5DI_2303.png	5DI_2303.csv	Mike Robbins

Canon EOS 1D Mark III Photos

Click here for a deteiled test of the Mark III.

This camera has 14bit analog to digital converters - notice the scale on the histograms! The raw data has a potential range from 0 to 16383 compared to the 0-4095 range on all previous DSLRs with 12bit A/D converters. In addition the black point appears to be calibrated at 1024, while on the 300D, D60, 20D, 30D and 5D it is at 128 and on the 350D it is at 256. Like some previous models, at ISO 100 the pixels saturate before the A/D converters; the saturation level is 15280 counts out of the possible 16383 (see also IMG_1829 from the 30D above).

Photo	Lens	Shooting conditions	Raw histogram plot	Raw Histogram file	Source of the photo
Y0B0B0327	50mm	ISO 100, 1/200 f/5.6	Y0B0B0327.png	Y0B0B0327.csv	Imaging Resource
Y0B0B0430	50mm	ISO 100, 1/200 f/6.4	Y0B0B0430.png	Y0B0B0430.csv	Imaging Resource
Y0B0B0494	50mm	ISO 100, 1/200 f/6.4	Y0B0B0494.png	Y0B0B0494.csv	Imaging Resource

Note: You can see the full EXIF information from some of the JPEG files.
Note: There are two green lines in the histograms - G1 and G2 - this is because there are two green filters in each 2x2 set of pixels in the bayer matrix. The two lines overlap almost perfectly most of the time of course.
Note: The photographs were not chosen for their artistic merit :-)
Note: Only the 30D photos are mine the rest are collected from either people I know or the Web.

Results From Detailed Analysis of a Big Collection of Canon 30D Raw Files

The mystery of the combing artifacts is solved. This section is kept only for "historic" reference"

To try to understand what is causing the combing I analyzed my whole collection of raw files. Here are the results.

Lens	Photos total	Combed photos	%Combed	Non Comed ISOs	Combed ISOs
EF50mm f/1.8	419	128	30.55%	100, 200, 400, 800, 1600	100, 200, 400, 800, 1600
EF100mm f/2 USM	452	127	28.10%	100, 200, 400, 800, 1600	100, 200, 400, 800
EF24-85mm f/3.5-4.5 USM	3487	11	0.32%	100, 200, 400, 800, 1600	500
EF-S18-55mm f/3.5-5.6	396		0.00%	100, 200, 400, 800, 1600
Sigma 170-500mm	110		0.00%	100, 200, 400, 800
Tokina 12-24mm	7		0.00%	200, 400, 800
Tamron 200-500mm	46		0.00%	100, 200
No lens (pinhole body cap or LensBaby)	43		0.00%
Total	4541	138	3.04%

The combed files from the EF24-85mm f/3.5-4.5 USM are all at ISO 500 and that is always combed on the 30D, so they are not a mystery.

Lens	EF50mm f/1.8			EF100mm f/2 USM			EF24-85mm f/3.5-4.5 USM			EF-S18-55mm f/3.5-5.6			Sigma 170-500mm			Tokina 12-24mm			Tamron 200-500mm
f/	Total	Combed	%	Total	Combed	%	Total	Combed	%	Total	Combed	%	Total	Combed	%	Total	Combed	%	Total	Combed	%
Total	419	128	30.55%	452	127	28.10%	3487	11	0.32%	396			110			7			46
1.8	128	115	89.84%
2	60	13	21.67%	163	127	77.91%
2.8	22			47
3.2				1
3.5	2			4			306	1	0.33%	1
4	118			2			196	1	0.51%	1						3
4.5	2						647	2	0.31%	5
5							71			13			1
5.6	18			3			169			154			63						1
6.3				2			106			9			9						9
7.1							88			1
8	37			156			791			7			23			3			20
9	2						143			17									1
10							132			67			2						2
11				34			536	7	1.31%	88			9						13
13							122			9			2			1
14				4			86			2			1
16	4			36			74			19
22	26						12			1
29										2
32							8

Conclusion Regarding the Combing Artifacts