Image Processing for Jellybean Quality Control - Software Development

Test Image Preparation

We will develop image processing software for use by a confectionary manufacturer for automated quality control. Our software will examine groups of jellybeans on the production line before prior to them either being funneled into a packet and sealed or rejected.  

To simulate the area of the assembly line that contains candidate jellybeans for packaging or rejection we will use a sheet of A4 paper. Empty your packet onto the sheet of paper, taking care not to ingest any. Add your calibration chip.

Next, take a picture using your mobile phone. The picture should include only the sheet of paper and jellybeans/calibration chip, not the table or floor. 

Try to take the picture in relatively diffused lighting so that there aren't too many pronounced specular reflections. Don't be tempted to move the jellybeans around manually - it is fine if they are close to one another, although they should not be piled on top of each other (on the production line this is ensured by shaking the conveyer belt to promote separation).

Tasks 1. Display normalised RGB histogram(s) for your original jellybean image either on a single axis, or on three separate axes. Perform histogram stretch on the image and show the RGB histogram(s) again. Calculate and display the RMS contrast before and after the stretch operation.

2. Threshold your histogram-stretched image from question 1 to produce a set of binary images, one for each jellybean colour you have. You can use any combination of low pass, high pass, band-pass and band-reject thresholds on one or more channels, as necessary, to identify pixels belonging to each colour from the background and other unwanted colours. Each binary image should contain a 1 where that pixel is in the current bean colour, and 0 otherwise. Don’t worry if the binary images are slightly noisy to start with (i.e., there might be holes in some jellybeans, or there may be pixels allocated to a particular jellybean colour that shouldn’t really be there). However, do your best to select your threshold values very carefully. Also do this for the calibration chip you have used.

3. Show the union of the set of binary images you produced in question 2, above. Next show the images produced when you perform entry-wise multiplication of each of your binary images, and the union of the binary images, by the original RGB image (i.e., use your binary images as masks to select the original pixels). If any of the images look particularly poor, go back and revise your threshold values and threshold methods.

4. Next, we will try to tidy up the binary images we produced in question 3, above. Experiment to see which of the following methods enable you to more accurately separate pixels belonging to each jellybean colour from the remaining unwanted pixels: (i) non-linear majority filter, (ii) morphological erode, dilate, open and close. For the morphological operators, you will also need to experiment with different structuring elements. Rev. Date: 08-10-19 Copyright © 2019 Ian van der Linde Page 2

5. Your binary images should have been tidied up by step

4 (i.e., 1s only for the beans of interest, with fewer or no holes), so repeat step 3 to demonstrate this and report on how many pixels are in each jellybean colour pixel group before and after your selected best-performing method from step 4.

6. Using the calibration chip, estimate the area of each bean in mm2 . We are doing this to ensure that there aren’t any excessively small, large, or malformed beans on the production line. If there are, we will reject this batch.

7. Calculate the Euclidean and city block distance of the centre of each bean to the image centre. We are doing this because the packaging robotic apparatus requires that the jellybeans are relatively well clustered towards the middle of the conveyer to prevent jams.

8. Count how many beans of each colour there are in the image. We are doing this because if the production line robot determines that there are too many beans of the same colour, the batch will be rejected rather than packaged. Similarly, if there are too many or two few beans on the conveyor belt, then they are unsuitable for packaging.

9. Extract each individual jellybean (i.e., obtain a sub-matrix bounding box around it) and rotate these sub-images so that the longest dimension (from tip-to-tip) of the bean runs in a vertical direction. Place the bean images side by side to create a mosaic of jellybean images for occasional manual inspection by the factory supervisor. Calculate the length, in mm, of each bean from tip-to-tip using the calibration chip.