PixInsight versus Maxim: Subframe Calibration Comparison

Introduction

While trying to process some images taken with my SBig STF-8300M in PixInsight I was encountering problems; PI would not
correctly read images from my SBig STF-8300M, either the original image collected by Maxim or the calibrated image. This seems to have to do with whether the images are saved in 16-bit signed or unsigned format; there is a lot of discussion in the PI forum about how this should be done, Maxim using non-standard FITS files, etc. In the end it appears the “fix” is to manually reformat all of the calibrated light subs in Maxim into 32 bit floating point format.

During the forum discussion people made statements like

      It’s a really bad idea to let maxim touch your files at all… the less maxim does, the better.
      Stop letting maxim calibrate images and do it all in PI. the results are superior anyway.
      Avoid MaximDL like the plague once it has captured the image.

Interesting. Until now, I have been happy with Maxim calibrating the images. It is easy, happens automatically under ACP/Maxim, and the only problems encountered were caused by Some Idiot using bad flat or dark subs. Googling around, I do not find any studies showing the relative performance of Maxim versus PI in the calibration of images. I don’t know the source of these statements indicating the superior performance of PI in calibration.

On the other hand, my earlier tests on image alignment showed that PI was superior to Maxim and Registar. Maybe PI has better performance in calibration? Would this offset the extra manual work required to prepare and calibrate the images for PI
calibration (compared to almost no effort in Maxim calibration)?

So, here goes my test. Does PixInsight calibrate light subs better than Maxim?

Test data

I assembled a test data set of bias, dark, flat, and light frames. I use the same frames for calibrating under both PI and Maxim.

  • 1 light frame (an OIII light sub, 30 minute exposure). This is the image to be calibrated. The camera is trying to get to -10C, but the ambient is high enough that the actual temperature is -4C. I just grabbed a sub from last night’s imaging run. Maxim calibrates using the original file. Since PI cannot open this file correctly, it uses a copy of the same file saved as 32 bit floating point.
  • 20 Dark frames at -10C, 30 minute subs each. Collected in Maxim.
  • 25 dusk sky flats with OIII filter collected in Maxim under ACP’s control. These flats are between -5 and -10 degrees C; at dusk the ambient temperature is still around 100 Fahrenheit so the camera can’t quite cool to -10C.
  • 40 Bias frames at -10C collected in Maxim.

In PI, I set the probe to display values in 16 bit integer mode. Hopefully this allows comparison of “ADU” units with the
corresponding Maxim values.

The Maxim calibration involved using “Set Calibration” to load the various calibration individual subs into Maxim and clicking
the button that creates Masters for the groups of files (Bias, Dark, and Flat). I then opened the raw file and clicked “Calibrate”.

Maxim Calibration results showing the nebula (upper left) and the 3 master files. Click for full size.

Maxim Calibration results showing the nebula (upper left) and the 3 master files. Click for full size.

In the fight to get PI to calibrate the images, I tried several different cases as listed below. The various cases are variations on using floating point versions of the various calibration files for PI.

Note that only Case 4 adequately calibrated the images.

Case 1 – All files float

+ all files (bias, dark, flat, light) are exported in floating format from Maxim
+ FITS configured to rescale with 0-65535

FITS file config used to access floating point files.

FITS file config used to access floating point files.

Result: the PI conversion runs, but is not correct.

All individual subs (light, dark, flat, bias) exported from Maxim as 32 bit float. PI builds its own masters. Click to enlarge.

All individual subs (light, dark, flat, bias) exported from Maxim as 32 bit float. PI builds its own masters. Click for full size.

  • Calibrated image is too dark, zeros in background, no nebulosity.
  • flat is wrong; it looks a lot like the bias.
  • values in the PI created calibration master files are different from Maxim.
  • Bias ranges 40000..15000..40000 across frame.
    Maxim => 19500..1900..1950
  • Flat ranges 0..6000..0 across frame
    Maxim => 15500..18000..15500
  • Dark values around 1000 across frame
    Maxim => around 300 across frame

Case 2 – Fits configuration at 32767

This was the same as Case 1, except that the FITS configuration is set to 0..32767 instead of 65535. This was tried in an attempt to address image values being off by a factor of 2.

Result: PI Conversion fails to run.

After PI calibrates the individual flats the integration process tries to stack the flats to make the master. This process fails because one of the calibrated flats is “open in another process” and cannot be opened. On different runs, it will be a different file that is open in another process.

Case 3 – Use Maxim calibration masters

It turns out the Maxim calibration masters are already in float format. This case uses those masters directly PI, rather than converting the individual subs to float for PI to create its own masters.

FITS is configured as 65535 again (for the light frame).

  • Result: Calibration runs, but it looks like the flats are not correctly applied.
  • Dust motes are visible.
  • Darker center of image. Looks like flats are too strong?
  • Values (flat, dark, bias) of the masters when opened in PI match those reported by Maxim.

Case 4 – Use original individual 16 bit Maxim calibration files

Another weirdness – it turns out the individual calibration subs created by Maxim are readable by PI, even though they are in the same 16-bit format which doesn’t work for the light frames. For unknown reasons the calibration files can be read.

So, in this case allow PI to use the original Maxim individual subs to build it’s masters. The only thing different between PI and Maxim processing in this case is PI starts with the float version of the light sub.

+ Calibration files (bias, dark, flat) are left in “16 bit” state as created by Maxim. No conversion performed.
+ Light file is exported in 32 bit float format. Can tell this is necessary because PI does not open this 16 bit file correctly.
+ FITS configured to rescale with 0..65536

Result: As seen below, the calibration looks pretty good, very similar to the Maxim result.

PI calibration using Maxim individual calibration subs directly to create its own masters.

PI calibration using Maxim individual calibration subs directly to create its own masters. Click for full size.

  • Dark master displays around 2000 “ADU”, versus Maxim’s 300.
  • Bias master is slightly lower than Maxim: 1870..1800..1870
  • Flat master is similar to Maxim, 15800..18000..15800
  • Bias has much larger values than dark. I don’t understand this. However, the same thing is true in Maxim, so OK.

Comparison of PI versus Maxim calibrated light sub

So, how does the case 4 calibration of the light frame differ between the Maxim process and the PI process? The image below shows the two images, where the left image is the PI result and the right image is the Maxim result (converted to float).

PI and Maxim calibration results. Click to enlarge.

PI (left) and Maxim (right) calibration results. Click for full size.

Visual results:

  • The Maxim file is smaller, 2Mb versus 8Mb for Float form. The PI result is also an 8 Mb xisf file. I don’t know how Maxim got the file down to 2Mb, maybe that is a clue why PI can’t open it?
  • PI result has more hot pixels.
  • PI result has more noise (much grainier to my eye).
  • Stars look the same (roundness, brightness) even when zoomed in significantly.
  • Maxim result may have more dark pixels; a little hard to tell because of the increased noise in the PI image.

Tool Measurements

I used several PI tools to measure the noise and SNR for the converted images. Measurements for the Maxim result used the Maxim calibrated image converted to 32 bit float.

Statistics tool results:

Maxim K    PI K
count (%)   99.75290    99.95980
count (px)  2116573    2120963
mean        0.0072616    0.0075407
median      0.0071210    0.0072508
stdDev      0.0068079    0.0073814
avgDev      0.0011528    0.0013096
MAD         0.0009753    0.0009751
minimum     0.0000006    0.0000476
maximum     0.9739980    0.9766412

  • PI has more pixels?
  • PI has slightly higher background (median)
  • PI has larger deviations

Noise comparison script:

The NoiseEvaluation tool was used after performing a linear fit on the PI image to match it to the Maxim result.

PI    = 8.6e-4
Maxim = 9.34e-4

  • Maxim has slightly more noise? This does not jive with what I see visually.

SNR Calculation:

Calculated the SNR ratio as SNR ratio = MAD / Noise
PI = 1.13   Maxim = 1.044

  • Again, the PI result seems to have a slightly better SNR ratio by 0.09, although visually I wouldn’t have thought so.

ContrastBackgroundNoiseRatio tool:

Parameters: Background percentile 100th  cycle-spins 8
Maxim    PI
Median        466.48    475.15
Contrast        61.31    68.67
Background Mean    466.41    481.45
Background Noise    66.63    77.95
CBNR        0.92    0.88

  • This tool indicates the Maxim result has a better contrast (0.92 vs 0.88) and less noise (66.63 vs 77.95).

Conclusions:

Visually it seems clear the Maxim result is better. The various PI tools indicate that the two calibrations are fairly close; some tools indicate the Maxim image is better, some indicate the PI image is better. One might suspect I am not running the tools in the correct way:)

Case 5 – Case 4 plus Superbias and optimized dark frames

I turned on the optimize dark frames parameter (0.004) in the Darks frame screen. I also created 2 Superbias frames, one based on the Case 1 Bias master (from float format bias frames) and one based on the Case 4 bias master (based on master created from original Maxim bias subs).

Adding Superbias and dark frame optimization. Click to enlarge.

Adding Superbias and dark frame optimization. Click for full size.

This did not work. The background is over-reduced (lots of zero values). Somehow the superbias is not working correctly. Both superbias files had the same result.

I also tried only setting the Optimize Darks parameter = 0.004. This is the only difference between this frame and the Case4 result. It is fairly indistinguishable from the Case4 result; optimizing the darks didn’t seem to help any.

Thus, it seems that

  • the superbias makes things worse, and
  • optimizing dark frames does not help appreciably.

Final Conclusions

  1. The Maxim calibration appears to be at least as good as the PI calibration; I think it is better.
  2. There may be further fine tuning possible in PI to improve the results.
  3. Using PI for calibration requires a lot of manual operations compared to the almost automatic operation of Maxim. I see no reason to change to PI for calibration.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s