Flat-Field Correction (FFC) (imaWorx CXP-12 Quad)

The Flat-Field Correction (FFC) feature allows you to remove non-uniformities in the image caused by differing light sensitivities of the sensor pixels or by inhomogeneous illumination.

A correction of non-uniformities in the image caused by inhomogeneous illumination is called "shading correction" or "vignetting correction" and can be compensated with a flat field correction.

Non-uniformities in the image caused by differing light sensitivities of the sensor pixels is caused by differences of the amplifiers and ADCs of the sensor. For older CMOS sensor, this can be seen in a strong column noise.

This has three different effects:

• Dark signal non-uniformity (DSNU):
  • A possible offset for each individual pixel
  • Is independent of the illumination
  • Depends on the temperature of the sensor
  • Has a stronger effect for long exposure times
• Photo response non-uniformity (PRNU):
  • Is caused by different gains of each pixel
• Different responses in sensor areas
  • Some sensors have different responses in specific sensor areas
  • Reasons are the sensor design itself or temperature differences in specific areas
  • For color cameras, this can cause color differences in some areas of the images

All effects can be corrected with a flat field correction. However, it should be noted that a pixel-wise correction isn't possible with a block flat field correction.

The correction itself compensates these inhomogeneities. This is done by a subtraction of the offset values followed by a gain to correct lighting response.

The Flat-Field Correction (FFC) feature is delivered as plugin with the file name EnhAppletsFFC.dll for pylon. You find information for installing and using this plugin in section Using the Flat-Field Correction for Frame Grabbers in pylon.

How It Works

The block flat field correction of the EnhancedApplets follow the following process:

Determine Offset and Gain Coefficients

Before calculation of the correction values some preparation is required.

• Your illumination should be as homogeneous as possible
• Lens aperture and focus should be correctly set
• The exposure time needs to be configured as well
• All pre-processing features in the camera such as analog gain, etc. need to be configured.
• The camera should run at normal operating temperature

Offset

Calculating the offset coefficients for the correction follows a standard way with some individual deviations which is explained in this section.

To determine the offset coefficients to correct the DSNU or any sensor-related offsets, you must acquire a camera image where the sensor doesn't get any light e.g. with lens cap closed. In theory, the sensor should output all zero values, however, due to the DSNU, the pixel values can have an individual offset. Thus, the acquired dark image represents the offset field, i.e. the offset coefficients.

As the DSNU varies with the camera settings and the temperature, you should acquire the images under real conditions, especially regarding exposure and analog gain settings. Moreover, the offset values should be derived from the mean value of multiple images to eliminate the noise.

• D(x, y): average dark pixel value over several images at position x,y
• Offset(x, y) = D(x, y)

Gain

The acquired objects in your application have different brightness values. You must identify the brightest parts in your image, e.g white or reflective parts. The exposure, aperture and lighting should be set so that these parts are close but not in saturation. (Unless you want to, because you are interested in dark parts only).

For example, you want to investigate bright objects on a dark background. In this case put the object in front of the lens under real exhibition conditions. The brightest spot of the investgigated object should be close, but not in saturation. Lets say for 8 Bit at pixel value 240. Next, you need to illuminate the sensor homogenously with an object close to the same brightness value. For example, this could be a white paper without any structures. Again, none of the pixel should be in saturation. Record some of these reference images and calculate an average image P(x,y)

Next, several methods exist to derive the reference pixel value for the gain correction.

1. Mean value in reference image: If the brightest parts in P(x,y) are equal to the brightest parts of the acquired objects. Thus, you can use the mean pixel value of P(x,y) as a reference. PMean = Mean(P(x,y)) The gain coefficients will then be:

   

2. User define reference value or brightest spot: If the mean brightness in your reference image is not equal to the desired maximum pixel values, you can define a reference value by yourself. This could be the value of the brightest spot in your image or the difference between your reference object and reference bright image. The gain values will then be calculated so that the pixels of the homogenous images are corrected to this reference value Pref.

   

In both formulas, the previously determined offset is subtracted prior to the division. This must not be done, if P(x,y) is already corrected by the offset in a two step procedure.

FFC for Bayer Pixel Format

For the Bayer pixel format, the FFC is processed on the RAW Bayer images before converting them to RGB or any other color space. This gives a maximum precision. Calculating the correction coefficients is done individually for each color component. As a side effect, the FFC also does the color calibration as the reference image is corrected to a perfect white image.

Definition of Block Flat Field Correction

The block-based flat field correction is used when there isn't enough memory to store a correction value for each individual pixel. In that case, the image is split into blocks whose sizes you can define. Each block has its own starting offset and gain coefficient at the corners.

For all pixels in-between, the coefficients are determined using linear interpolation inside the frame grabber.

This results in a very precise correction, but it can't correct individual pixels especially for DSNU and PRNU caused by individual pixel variations.

Manual Calculation of the Block Size and Applet Parameter Values

To use the block FFC in the applet, you must perform two steps after the correction values have been calculated:

1. Configure the block size.
2. Load the correction coefficients onto the frame grabber.

Configuring the Block Size

As you can connect any camera with different resolutions to the frame grabber, the applet implementation is made to adapt to different resolutions.

All blocks have the same size, but they don't need to be squares, i.e. height and width might differ. The frame grabber offers a total of 16384 blocks for Offset and Gain correction values. Thus, you must adapt the block so that the blocks can be distributed to the image size.

Example: The boost v boA5120-230cm has a resolution of 5120 x 4096 pixels. If you want to use the full resolution, you must distribute the available 16384 blocks to all these blocks.

The blocks don't necessarily need to be square-shaped. Maybe your shading is stronger in X- than in Y-direction, so you should have a smaller block size in X and a larger block size in Y.

The following image shows non-squared blocks, where the resolution in X-direction is higher than in Y-direction:

To equally distribute the 16384 blocks as square blocks over the whole image, use the following formulas.

• Image width: W
• Image height: H
• Total number of blocks: N

The total area of the image is: Image Area = W x H

Each block is square with a side length s. The area of one block is: Block Area = s²

The number of blocks is:

To find s:

For the example above: W = 5120, H = 4096, N = 16384:

Not every block side length can be configured in the applets. For example, the block size in X-direction needs to be a multiple of 16.

For the given example, the block size is rounded down to B_x= 32.

To calculate the block height, you need to consider non-squared blocks to fully use the available blocks:

s² = block size in X x block size in Y = B_x x B_y

The block height can then be determined as:

For the example above, this will be:

The actual used number of blocks will then be:

You can now set the applet parameters:

• FlatFieldCorrectionBlockWidth = 32
• FlatFieldCorrectionBlockHeight = 40

![Screenshot FFC Block Width]

Loading the Correction Coefficients Offset and Gain

Each block requires a set of four offset and gain coefficient values Q₁₁,Q₁₂,Q₂₁,Q₂₂. Those values are the support-positions at the edges of the block.

The end position of a block is the starting position of the next block. So Q₁₁,Q₁₂ of a current block is equal to Q₂₁,Q₂₂ of the next horizontal block.

The following example shows a block size of 4x3 at an image of size 8x6. It's obvious that the end-positions overlap with the start-positions of the next block. Moreover, the block end support positions at the image edge are outside the image boundaries. If the block size isn't an integer divisor of the image size, the blocks at the image edge will be even further outside of the image.

• Block 0
  • Q₁₁ at x = 0, y = 0
  • Q₁₂ at x = 4, y = 0
  • Q₂₁ at x = 0, y = 3
  • Q₂₂ at x = 4, y = 3
• Block 1
  • Q₁₁ at x = 4, y = 0
  • Q₁₂ at x = 8, y = 0
  • Q₂₁ at x = 4, y = 3
  • Q₂₂ at x = 6, y = 3
• Block 2
  • Q₁₁ at x = 0, y = 3
  • Q₁₂ at x = 4, y = 3
  • Q₂₁ at x = 0, y = 6
  • Q₂₂ at x = 4, y = 6
• Block 1
  • Q₁₁ at x = 4, y = 3
  • Q₁₂ at x = 8, y = 3
  • Q₂₁ at x = 4, y = 6
  • Q₂₂ at x = 6, y = 6

The values for Offset and Gain are represented as fixed point values with a specific range and resolution. The applet has some info parameters to check the range.

For the given example, the Gain values have the following range:

In other words, the applet allows gain corrections up to 3,999. For example, a pixel value 50 could be corrected at maximum to 50 x 3.999 = 199.

To convert a float value into a fixed point value, multiply it with the fractional bits and limit it to the ranges [0,2^GainBits - 1], where GainBits = GainInteger + GainFractional.

The fixed- point values Q₁₁,Q₁₂,Q₂₁,Q₂₂ are appended to a large integer value for each block.

Or as a bit shift operation:

For example, float values:

are converted to fixed-point using the resolution above to

The applet value becomes

Hence the applet parameter value becomes

In hexadecimal representation:

Using the Flat-Field Correction for Frame Grabbers in pylon

The Flat-Field Correction (FFC) feature is delivered as plugin with the file name EnhAppletsFFC.dll for pylon. To install the pylon FFC plugin, copy the file EnhAppletsFFC.dll to the pylon installation folder Applications\x64\lib\pylonviewer\plugins.

Before using the plugin, you should do some preparations:

• To understand the underlying theory, see the sections How It Works.
• To understand what the plugin is doing to determine the FFC correction values, see Determine Offset and Gain Coefficients.

Before calculating the correction values, you must ensure the following:

• Your illumination should be as homogeneous as possible.
• Lens aperture and focus should be correctly set.
• The exposure time needs to be configured as well.
• All pre-processing features in the camera such as analog gain, etc. must be configured.
• The correct EnhancedApplet is loaded.
• Set the correct camera pixel format in camera and applet.
• The camera should run at normal operating temperature.
• Don't set any pre-processing in the applet. You must obtain an image at full ROI.

Basler recommends to do this in microDisplay X first, so you can use the features Histogram and Line Profile for calibration.

The screenshot shows the acquisition of an image in pylon Viewer. As you can see from the line profile, there is a strong shading towards the corners of the image. Furthermore, some pixels are in saturation. This is on purpose since the darker parts of the object are the ones of interest. This is also a realistic example for machine vision applications.

Now, remove the object and provide a homogeneously illuminated background, which you see also in the example.

Then you can start the plugin:

1. A camera must be selected.
2. Stop the acquisition.
3. In the pylon Viewer, select Tools > EnhancedApplets > Flat Field Correction.

Once the plugin is started, you see information of your settings, whether the applet can do a block FFC and a live image:

Click Next to get to the next step. You can now calculate the offset correction values by acquiring a sequence of dark images. Click Calculate Offset. The plugin displays a scaled image and some statistics so that you can see the maximum and average offset values:

Click Next to get to the next step. On the next page, you can acquire a sequence of bright reference images to obtain the gain correction value. Place a homogenous bright object under the lens and start the calculation. Again, the displayed image visualizes the gain. You can select whether you want to see the acquired image or corrected image instead.

You can see in the screenshot above that the bright reference image isn't close to the saturation. So you might want to enter a reference value, which is higher than the automatically derived value from the image average. Set a value to e.g. 230or even 300.

Click Next. On the next page, set the block size or do an automatic calculation. For the given example, an image size of 5120 x5 120 px is used, so for example a block size of 48 x 34 px results in a total of 16157 blocks, which is less than the available total of 16384 blocks. Set edge handling to half block and set Linear extension.

Store the correction values to disc so that the FFC in the applet can load and apply them. Click Save to Folder… and select an empty folder. Once the calculation is successfully processed, files are generated into this folder.

Click Next. On the next page, you can apply the determined coefficients directly to the loaded applet and you can see the results.

The Line Profile now shows a homogenous image and a high-quality image can be acquired. Use the FFC mode parameter to toggle the FFC on or off.

Using Correction Values in SDK Applications

Each time you restart your computer or open the frame grabber application, the FFC settings must be re-loaded to the frame grabber. A frame grabber has a volatile memory, so no data can be stored as opposed to the camera user sets on the cameras.

Thus, in your SDK application (pylon API, GenApi, GenTL consumer or Framegrabber SDK), you must configure the FFC:

1. Set the block width and height.
2. Load the offset and gain correction files which you have stored using the plyon Viewer FFC plugin.
3. While saving the files, the plugin also generates a configuration text file for your reference. pylon Viewer offers the possibility to obtain a sample C++ code for the pylon C++ API.

Using the Hierarchical Box in VisualApplets

The functional blocks can be provided as VisualApplets User Libraries or hierarchical boxes in a design. This is done depending on your project and project environment. You can get the hierarchical box for the enhanced applets from your local Basler Sales representative or field application engineer.

The use of the hierarchical box in VisualApplets is similar to the use of the Enhanced Applets. The process is as follows:

1. Do the calculation of the coefficients with the pylon Viewer FFC plugin and Enhanced Applets.
2. Write the correction coefficients and configuration to files using the FFC plugin.
3. Open the flat-field correction hierarchical box in a design of VisualApplets.

4. Configure the static parameters of the FFC hierarchical block in VisualApplets.

   The hierarchical box automatically adapts to the pixel bit width, parallelism and image format. First, you must parameterize the input link. It's important to do this before setting the static parameters.

   Allowed formats are:

   • Bit width: 8 to 16
   • Artithmetic: unsigned
   • Parallelism: 1 to 64
   • Kernel Columns, Kernel Rows: 1
   • Color Format: VAF_GRAY
   • Color Flavor: FL_NONE
   • Max. Image Width: any
   • Max. Image Height: any

   The static parameters define the precision and FPGA resource usage of the block-based FFC:

   • OffsetBits: Defines the number of bits used for offset correction. This is the sum of the integer and fractional bits.
   • OffsetFractionalBits: Defines the number of fractional bits. For fixed point arithmetic explanation see section Loading of the Correction Coefficients Offset and Gain.
   • GainBits: Defines the number of bits used for gain correction. This is the sum of the integer and fractional bits.
   • GainFractionalBits: Defines the number of fractional bits. For fixed point arithmetic explanation see section Loading of the Correction Coefficients Offset and Gain.
   • MemoryType: Select either UltraRAM, Block RAM, or LUT RAM. UltraRAM is recommended.
   • MaxNumbrOfBlocks: Select the maximum number of blocks used for the block FFC. If you had selected URAM as MemoryType, only 16384 blocks can be used.
   • MaxNumberOfBlocksInX: Define the maximum number of blocks in X-direction. Usually this should be Max. Image Width / Parallelism.
   • MaxNumberOfBlocksInY: Define the maximum number of blocks in Y-direction.
   • MaxBlockSizeInX: Define the maximum block size in X-direction. Choose a suitable value for your application. For a general purpose use, you can set the full image width.
   • MaxBlockSizeInY: Define the maximum block size in Y-direction. Choose a suitable value for your application. For a general purpose use, you can set the full image height.

   The dynamic parameters can be changed during runtime to the application requirements:

   • CoefficientsXLength: This value needs to match to the calculated and uploaded coefficient files, e.g. the ROI size of the Enhanced Applet which you have used in the pylon Viewer and the FFC plugin.
   • CoefficientsYLength: This value needs to match to the calculated and uploaded coefficient files, e.g. the ROI size of the Enhanced Applet which you have used in the pylon Viewer and the FFC plugin.
   • BlockSizeMode: Select Manual or Auto. In auto mode, the available blocks will be distributed over the image. For manual mode, the following parameters can be used to set the block size. If you used the FFC plugin in the pylon Viewer, copy the corresponding values:
     • BlockSizeInX: Define the block size if BlockSizeMode is set to Manual. If you used the FFC plugin in the pylon Viewer, copy the corresponding values to this parameter.
     • BlockSizeInY: Define the block size if BlockSizeMode is set to Manual. If you used the FFC plugin in pylonViewer, copy the corresponding values to this parameter.
     • EnoughBlocksForImageSize: This read-only parameter indicates whether the defined maximum number of blocks are enough for the given block size and ROI setting.
     • OffsetCoefficients: Is a field parameter for the offset coefficients. Upload the text file from the FFC pylon plugin. You find a description of the format in section Loading of the Correction Coefficients Offset and Gain.
     • GainCoefficients: Is a field parameter for the offset coefficients. Upload the text file from the FFC pylon plugin. You find a description of the format in section Loading of the Correction Coefficients Offset and Gain.
     • FlatFieldCorrectionMode: Select, if you want the FFC Offset and Gain on or off. This can be done during image acquisition.

5. Configure the dynamic parameters of the FFC hierarchical block either during design time in VisualApplets or during runtime with your SDK application (pylon API, GenApi, GenTL consumer or Framegrabber SDK).

The following screenshot shows the use of the FFC hierarchical box in VisualApplets together with the parameters.