Difference between revisions of "S16: Helios"

Revision as of 04:55, 23 May 2016

Automated Assembly Line

This project is an automated assembly line capable to defect detection, rejection and sorting with the help of wireless communication and image processing.

Abstract

The project is an automated assembly line capable of object defect detection, object rejection and sorting with the help of image processing algorithms and wireless communication. The entire system comprises of three micro controllers (2 SJOne boards and one Raspberry pi 2). A wired interface exists between raspberry pi 2 and SJOne transmitter board, while the two SJOne boards communicate via nordic wireless. Raspberry pi 2 is equipped with a linux operating system with Open CV support to run image processing algorithms for color and shape detection. All three micro controllers are mounted to a miniature assembly line with object moving on it. The SJOne transmitter board is capable of object rejection using servo motor and IR object detector module, while the SJOne receiver board is capable of simple sorting using servo motor based on the result of object color and shape detection algorithms running on raspberry pi 2.

Objectives & Introduction

• To design a automated assembly line with motor drive.
• To implement nordic wireless communication on SJOne board for communication of control signals.
• To design and implement color and shape detection algorithms using Open CV.
• To use servo motors with mechanical parts to perform object rejection and sorting.
• To be able to process images of objects on the belt and decide the acceptance of object for future processing on the assembly line.

There is a need for Assembly lines to incorporate intelligence to identify and reject defective products. Image processing is one of the foremost technologies responsible for achieving this intelligence. An assembly line with wireless and object detection, sorting capabilities is the prime goal of this project. The need for a wireless assembly line arises due to large wiring overhead in wired systems.

A camera captures the pictures of items that move on an Assembly line. The raspberry pi 2 board processes these pictures and assesses color and shape information and communicates to boards further down the aseembly line to take appropriate action. The SJOne transmitter board receives the information and rejects defective items from the assembly line while retaining fit items for the future stages of the assembly line. The SJOne receiver board uses servo motor to bifurcate the products into two separate belts based on color for future actions.

Team Members & Responsibilities

Kartick Parameswaran

• OpenCV installation, Rasberry Pi environment setup, Hardware support

Anush Shankar

• Open CV installation support, Nordic wireless communication, Stepper and servo control, Conveyor and Hardware

Sindhuja Gopalakrishnan Elango

• Open CV algorithm design, Nordic wireless communication, Sensor implmentation, Hardware support

Akshay Dabholkar

• Open CV algorithm design,implementation and verification, Hardware support

Schedule

This section of the report provides the team schedule for the Assembly line project, indicating the milestones to be achieved during the course of the project.

Parts List & Cost

System Block Diagram

System Block Diagram

Design & Implementation

This section of the report provide the Hardware and Software design of the project. Details of all design considerations and implementation are provided in this section.

Hardware Design

This section describes the various hardware design parameters involved in the project. It involves conveyor system design, object rejection hardware, object sorting hardware,object detection hardware and conveyor drive hardware.

SJOne Board

SJOne board

The SJOne board is one of the development boards used for the project. Some of the features offered by SJOne board are as follows:

• Built-in Nordic Wireless (Board-to-Board communication).
• Software Stack for Mesh Network.
• 4 Switches and 4 LEDs (both hard-wired).
• Many GPIOs with two SPI, Multiple UARTs, and I2C availability.
• Power from USB or External Power.
• PWM support.

The SJOne board offers whole lot of features out of which wireless communication (Nordic), GPIO, I2C and PWM timers are utilized for the project. The I2C feature was used to communicate with the Raspberry pi, GPIO's used to drive stepper motors, PWM's to drive servo motors, Nordic for wireless communicaton between SJOne boards.

Raspberry Pi

Raspberry Pi is a linux based embedded computer created by the Raspberry Pi Foundation. The Foundation aims at providing cheap and affordable computing solutions.

Raspberry Pi features

Microcontroller: The Raspberry Pi contains a microcontroller manufactured by Broadcom - BCM2836. The processor within this chip is a quad-core Cortex-A7. Operating System: Raspberry pi has many compatible operating systems available. We chose the Raspbian Jessie operating system as it popular. The underlying core for this operating system is a debian linux kernel. Peripherals: The various hardware available on board can be found here.

Installing OpenCV libraries on Raspberry Pi

OpenCV is a collection of libraries that provide various C, C++ and Python functions that can be directly used for Image processing. The advantage is that it provides data structures to hold RGB images and complex image processing algorithms, such as edge detection, is already present and just have to be called within our program. OpenCV installation can be done using various methods:

1. Cross compiling on a general purpose computer: This is a common method for compiling OpenCV. This method involves compiling OpenCV on a general purpose computer which has a cross compiler installed intended for your target platform, such as the arm-gcc. The advantage of cross compiling is that the compiled libraries can be now used on both the target and host computers. It would now be possible to compile program by using the arm-gcc on your host platform and then copying the generated executable into the target platform and directly run it.
2. Installing by building a new kernel image for your target platform: Another popular method is to use tools such as Buildroot which compile a kernel image for your target platform. This is useful if you are installing a new OS and want the OpenCV libraries installed along with this new OS. This tool automatically searches and compiles any dependencies that would be needed to install OpenCV, which is the reason this is popular.
3. Installing directly on the target platform: This is a less preferred method because of the large amount of time it can take and because it needs an . Compiling libraries is a resource intensive process and thus takes a large amount of time (more than 2-3 hours) on a slower processor.

For our project we chose method three for the following reasons:

1. Raspberry Pi 2 with its quad-core processor is capable of handling these loads.
2. It was easier to compile on the native system than cross-compiling.
3. With an internet connectivity adding dependencies was easy using “sudo apt-get install”

After searching the internet for methods to compile the libraries we came across this script to install the libraries and its dependencies.

Conveyor Belt

Conveyor belt is a mechanical system with which materials can be transferred from one location to another. Conveyor systems are used in many industrial applications for safely transferring materials where several processing steps happen on the material when it travels on the belt. As the conveyor system is fully automated, it can be ensured to a have a uniform throughput of the deliverable. Industries thus use the conveyors for increasing their production rate. Conveyor systems are very popular in material handling, packaging, automotive, food processing and many other industries.

Basic conveyor structure

Webcam

USB webcam

The USB webcam is an integral part of the Image processing system running as part of the automated assembly line. The camera acts as eyes of the system and is used to capture images, that may be processed later using open CV algorithms. Choice of camera is crucial to the success of the project. A camera with high definition image quality and wide angle view provides better image processing control and lower pixel corruption due to external light related noise. A microsoft HD lifecam 3000 was chosen due to the various high end features it offers and comparatively low cost. Some of the features of this camera are as follows:

• Widescreen - 16:9 format offers cinematic video recording.
• TrueColor technology - TrueColor technology automatically delivers bright and colorful video, in virtually all lighting conditions.
• Universal attachment base - Works on your desktop, laptop, or notebook.
• Small size and ease of mounting.
• 2.0 USB connectivity.

Nordic wireless

Nordic communication

SJone board has a nordic wireless chip nRF24L01+ manufactured by the Nordic Semiconductors interfaced to it. Through a Serial peripheral Interface it is possible to operate and configure the Nordic chip. The air data rate, operating modes and RF channel frequency need to be set by configuring the registers in the nRF24L01+.

Air data rate is the modulating signalling rate that is used by nRF24L01+ for transmitting and receiving data. It can be 250 Kbps, 1 Mbps and 2 Mbps. Lower air data rate gives better receiver sensitivity while higher data rate gives a reduced current consumption. The RF channel frequency determines the centre of the channel frequency in which nRF24L01+ operates. Typical operating frequencies are from 2.400GHz to 2.525GHz

For communication to happen,

• Both nordic transmitter and receiver must be set at the same air data rate. We have set the air data rate at 250 Kbps.
• Both nordic transmitter and receiver must be operated at the same RF channel frequency.We have set the channel number to 2.499 GHz.

IR Detector

IR Detector

IR Detector Calibration

IR Detector

IR Detector OSEPP IRDET-01 detects obstacles when they are in the calibrated range. The module has 3 pins which are the ground(G), VCC(V) and Signal(S). The signal wire line is digital. It can be either high or low. High if the detected range is further than the calibrated distance, low if the detected range is less than the calibrated distance. The pins are connected to the GND, VCC and GPIO pin of the SJone board, The pin direction is set as input. When no object is detected the pin will read 0 and when object is detected the pin will read 1.

Calibration

IR detector module device has two potentiometers that can be used to calibrate the device. One for frequency (F) and one for distance (D). Much like an IR Remote, the IR light is oscillated at a defined frequency and detected at that same frequency; this enables the sensor to work in noisy environments. The frequency of the detector module can be adjusted to avoid interference with other detectors or other IR sources.

Turning the frequency potentiometer clockwise increases the frequency and turning it anti clockwise decreases the frequency.The distance at which the sensor is triggered is determined by adjusting the Distance potentiometer. When the sensor is triggered the LED will light up. Turning this pot clockwise increases the triggering distance, turning it counter-clockwise decreases the distance.

Servo Motor

SG90 is a tiny and lightweight servo motor with high output power. Servo can rotate approximately 180 degrees (90 in each direction), and is driven by a PWM wave. It fits in small places and offers good torque. Using servo motors provides precise control of angular movement using relevant duty cycle of a constant frequency PWM wave. Servos consist of an internal motor with a feedback system for position.

SG90 Servo Motor

Servo PWM Waves

The wiring information for SG90 is as follows:

Some of the specifications if SG90 servo are as follows:

• Weight: 9 g.

• Dimension: 22.2 x 11.8 x 31 mm approx.

• Stall torque: 1.8 kgf·cm.

• Operating speed: 0.1 s/60 degree.

• Operating voltage: 4.8 V (~5V).

A PWM wave of 50 Hz or 20 ms is needed to drive the servo. The servo works with duty cycle between 5-10%. Provided a wave with duty cycle 5% the motor rotates 90 degree to the left from neutral position (extreme left position), with 10% it rotates 90 degree to the right from neutral position(extreme right position) and with 7.5% duty cycle moves to neutral position.

Stepper Motor

Stepper motor working

A stepper motor is a DC motor which moves in discreet steps. The motor has poles which when energizedFor driving the conveyer system 28BYJ-48 stepper motors were used with ULN2003 darlington array module. The ULN2003 is a high voltage, high current darlington array with 7 NPN darlington pairs for providing high current required by the stepper motor.

28BYJ-48 Stepper motor

Some of the features of the stepper are :

• Voltage: 5V.
• Diameter: 28mm.
• Step angle: 5.625 x 1 / 64.
• Reduction ratio: 1 / 64
• 5 Line 4 phase can be driven by ordinary ULN2003 chip.
• Driven using PWM or GPIO.
• Simple connections with easy to plug connectors and switch for power.
• A, B, C, D four-phase LED indicates the status of the stepper motor work.

Hardware Interface

The hardware interface is divided into two parts. Components that were interfaced with SJOne Master board and components that were interfaced with the SJOne Slave board. Both the SJone boards had to be mounted one on each end of the conveyor system. Decsions made by the SJone master board will be implemented by the SJOne slave board.

Building our own conveyor system

Initially it was decided to buy or rent conveyor systems and interface components to it. First option was to buy the conveyor system. On enquiring with many dealers who supplied industrial mini conveyor systems to small scale and large scale industries , it turned out that a simple system was around 1200 US dollars. Second option was to rent the conveyor belt. Renting it for a day would cost around 200 dollars. But the requirement for the project was definitely more than a day. There were development and testing phases of each component that had to be mounted with the conveyor belt and tested frequently. So the options of buying and renting were eliminated. Having an objective of building an economical automated conveyor system, the decision to self build the conveyor system was made by the team.
Nevertheless, it wasn't that easy to build a conveyor system. The conveyor system required the following

• A rigid frame on both sides to hold the conveyor belt
  
• Rotating cylinders that had to be mounted to either ends of the frame.
  
• A conveyor belt that could slide on the cylinders bordered by the rigid frame
  

Interface with SJOne Master Board

Interface with SJOne Slave Board

Software Design

Show your software design. For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the code. For example, do not show exact code, but you may show psuedocode and fragments of code. Keep in mind that you are showing DESIGN of your software, not the inner workings of it.

Assembly line Mechanism

Master SJOne board Flow Sequence

Slave SJOne board Flow Sequence

Object Detection and Analysis with OpenCv and image processing

Image processing is the process of manipulating image data in order to make it suitable for computer vision applications. Computer vision helps to obtain relevant information from images and make decisions based on that information. In our project, we have used OpenCV with C++ to develop our computer vision application. OpenCV is an open source C++ library for image processing and computer vision, originally developed by Intel and now supported by Willow Garage. OpenCV has many inbuilt functions for image processing and computer vision that are mainly aimed at real-time processing.

Digital Image and pixels

A digital image is a collection of pixels. A pixel is the smallest element in a digital image. In an image, each pixel can have a particular value to represent that image. For example, consider a grayscale image where each pixel can be represented by 8-bit value. So, each pixel can have a range of 0 to 255 where 0 is the complete black and 255 is a completely white pixel.

Matrix_representation_of_GRAYSCALE

A color image consists of 3 planes Red, Green, and Blue. Any color can be formed with the combination these three planes. These planes are called channels of the image. So, an RGB i.e. a color image has 3 channels. Each pixel in RGB has a combination of three values. (255, 0, 0) represent pure red. (0, 255, 0) represent pure green. (255, 0, 255) represents pure violate. Thus, each pixel is represented by 24-bits in an RGB.

RGB_Planes

RGB_Planes

Color Detection

Object detection and segmentation is the most important task of computer vision. The easiest way to detect and segment the object is through color detection.

The method used for color detection was to first convert the RGB image into HSV and then by thresholding the HSV image, the binary version (Black and White) of the HSV can be obtained. In this binary image, the white space would represent the color which wants to detect. This binary image can then be used to detect the object we are looking using color detection and also to find the shape of the detected object using the available OpenCv library functions.

Need to convert from RGB to HSV

RGB color space describes colors in terms of the amount of red, green, and blue present. HSV color space describes colors in terms of the Hue, Saturation, and Value. The HSV image separates luma (intensity component), from chroma (color component). So that we can now perform histological equalization of the image by adjusting the minimum and maximum values of H, S, and V to such levels that only the pixel with the desired color will be illuminated (white) and the rest of the pixels will be dark.

void inRange(InputArray src, InputArray lowerb, InputArray upperb, OutputArray dst) library function was used to threshold the HSV image. 

Src is the source image. Dst is the thresholded image. The upperb and lowerb boundary parameters are scalars (minimum and maximum values of HSV) which can be determined using trackbars.

HSV__and_Original_Image

Thresholded__and_Original_Image

Trackbars

Trackbar is a very useful tool which allows us to change various parameters while our OpenCv application is running. In our project, we made extensive use of trackbars to determine upper and lower boundary parameters for the particular color which we want to detect. Show the image of trackbars and thresholded image. As shown in the fig. we can change the minimum values of H, S, and V in such a manner that only the desired color will remain as a whitespace and rest of the image should be dark. Once the minimum and maximum values are determined we can lock down these values with inRange() function to get the thresholded image.

  int createTrackbar(const string& trackbar name, const string& winname, int* value, int count, TrackbarCallback)  is the library function which was used to create the trackbars.

RGB_Planes

Trackbarname is the name of the trackbar. Winname is the window name to which trackbar is attached. Integer, pointed by this pointer, holds the value associated with the position of the trackbar. count - The maximum value of the trackbar. The minimum value is always zero.

findContours library function

Using find contours we can retrieve the contours from the binary image that we obtained through thresholding the HSV image. void findContours(InputOutputArray image, OutputArrayOfArrays contours, OutputArray hierarchy, int mode, int method, Point offset=Point()) We can pass appropriate modes to retrieve the different contour information from the image. The contour information is retrieved in the contour is a vector of points. From the contours information, we can find the area of the whitespaces using moments method which is actually the target color and can detect the color.

Algorithm for Color Detection
Step1: Initialize the camera.
Step2: Capture the image frame.
Step3: Convert the image from RGB to HSV.
Step4: Threshold the image for the required color.
Step5: Apply morphological operations to eliminate noise.
Step6: Find contours of the binary image.
Step7: Find the number of whitespace regions using the contour information.
Step8: If no whitespaces found go to step 2, else proceed to step9.
Step9: Check if the area of whitespace is greater than the minimum area. If area is less than the minimum area set, it may be a noise then go to step 2, else proceed to step 10.
Step10: Set object found flag to true. Continue scanning go to step2.

Flowchart for Color Detection

Color_Detection_Flowchart

Morphological Operations

Filtering is an important part of computer vision. In most of the applications, filtering should be applied to the images before processing on them. In our project, we applied morphological operations to remove noise and to do proper segmentation of the images. The two morphological operations we performed were: Dilation and Erosion.

Dilation: Dilation increases the white pixels along the boundaries of the foreground object. Consider the following example.

Image_Dilation

As we can see that dilation increases white pixels along the boundary of the image.

Erosion: Erosion is opposite to dilation. It increases the dark pixels along the boundaries of the foreground object. The example of the erosion is as shown in the figure.

Image_Erosion

As we can clearly see the erosion operation increased the black pixels along the boundary of the image. In our project, we applied erosion followed by dilation because the erosion sinks our object, so we dilate it.

Implementation

This section includes implementation, but again, not the details, just the high level. For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.

Testing & Technical Challenges

Describe the challenges of your project. What advise would you give yourself or someone else if your project can be started from scratch again? Make a smooth transition to testing section and described what it took to test your project.

Include sub-sections that list out a problem and solution, such as:

My Issue #1

Installing OpenCV on ARM 11

Using a pre-built conveyor system

Lighting Problem

Lighting was the most critical problem faced during image-processing. In our project, we converted the RGB matrix frames to HSV for the thresholding. The HSV frame extracts the lighting information ‘Luminosity’ and ‘Chroma’ from the RGB planes, so any slight change in the light source angles would alter the thresholding values for HSV frames. Also, we faced a lot of problems due to distortions arising from the light source. We tried to handle the problem by applying morphological operations and implemented the color detection Algorithm. But, still due to slight distortions still present we faced problem in implementing our shape detection Algorithm where even slight distortions would cause problems in detecting the vertices of the detected object.

Conclusion

Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?

Project Video

YouTube

Project Source Code

• Sourceforge Source Code Link

References

Acknowledgement

Any acknowledgement that you may wish to provide can be included here.

References Used

List any references used in project.

Appendix

You can list the references you used.