S16: Helios
Contents
Automated Assembly Line
This project is an automated assembly line capable to defect detection, rejection and sorting with the help of wireless communication and image processing.
Abstract
The project is an automated assembly line capable of object defect detection, object rejection and sorting with the help of image processing algorithms and wireless communication. The entire system comprises of three micro controllers (2 SJOne boards and one Raspberry pi 2). A wired interface exists between raspberry pi 2 and SJOne transmitter board, while the two SJOne boards communicate via nordic wireless. Raspberry pi 2 is equipped with a linux operating system with Open CV support to run image processing algorithms for color and shape detection. All three micro controllers are mounted to a miniature assembly line with object moving on it. The SJOne transmitter board is capable of object rejection using servo motor and IR object detector module, while the SJOne receiver board is capable of simple sorting using servo motor based on the result of object color and shape detection algorithms running on raspberry pi 2.
Objectives & Introduction
- To design a automated assembly line with motor drive.
- To implement nordic wireless communication on SJOne board for communication of control signals.
- To design and implement color and shape detection algorithms using Open CV.
- To use servo motors with mechanical parts to perform object rejection and sorting.
- To be able to process images of objects on the belt and decide the acceptance of object for future processing on the assembly line.
There is a need for Assembly lines to incorporate intelligence to identify and reject defective products. Image processing is one of the foremost technologies responsible for achieving this intelligence. An assembly line with wireless and object detection, sorting capabilities is the prime goal of this project. The need for a wireless assembly line arises due to large wiring overhead in wired systems.
A camera captures the pictures of items that move on an Assembly line. The raspberry pi 2 board processes these pictures and assesses color and shape information and communicates to boards further down the aseembly line to take appropriate action. The SJOne transmitter board receives the information and rejects defective items from the assembly line while retaining fit items for the future stages of the assembly line. The SJOne receiver board uses servo motor to bifurcate the products into two separate belts based on color for future actions.
Team Members & Responsibilities
- OpenCV installation, Rasberry Pi environment setup, Hardware support
- Open CV installation support, Nordic wireless communication, Stepper and servo control, Conveyor and Hardware
Sindhuja Gopalakrishnan Elango
- Open CV algorithm design, Nordic wireless communication, Sensor implmentation, Hardware support
- Open CV algorithm design,implementation and verification, Hardware support
Schedule
This section of the report provides the team schedule for the Assembly line project, indicating the milestones to be achieved during the course of the project.
| SI No. | Start Date | End Date | Task | Status | Actual Completion Date |
|---|---|---|---|---|---|
| 1 | 03/20/2016 | 03/29/2016 |
|
Completed | 03/29/2016 |
| 2 | 03/30/2016 | 04/05/2016 |
|
Completed | 04/05/2016 |
| 3 | 04/06/2016 | 04/12/2016 |
|
Completed | 04/12/2016 |
| 4 | 04/13/2016 | 04/19/2016 |
|
Completed | 04/25/2016 |
| 5 | 04/20/2016 | 04/26/2016 |
|
Completed | 05/05/2016 |
| 6 | 04/27/2016 | 05/03/2016 |
|
Completed | 05/12/2016 |
| 7 | 05/04/2016 | 05/10/2016 |
|
Completed | 05/15/2016 |
| 8 | 05/11/2016 | 05/17/2016 |
|
Completed | 05/21/2016 |
Parts List & Cost
| Item# | Part Description | Vendor | Qty | Cost |
|---|---|---|---|---|
| 1 | Conveyer system | Amazon | 1 | $34.62 |
| 2 | IR Object detector | Fry's | 2 | $21.78 |
| 3 | Antenna - Wireless communication | Amazon | 2 | $10.9 |
| 4 | 5V Stepper Motor with driver break out board | Amazon | 2 | $11.98 |
| 5 | Microsoft USB HD camera | Amazon | 1 | $24.24 |
| 7 | Wire Cutter | Amazon | 1 | $2.99 |
| 8 | Hardware Items | Home Depot | 1 | $15.49 |
| 9 | SJOne Board | Purchased from Preet | 3 | $240 |
| 10 | Raspberry Pi 2 Board | 1 | $30 | |
| Total Cost | $383.87 |
System Block Diagram
Design & Implementation
This section of the report provide the Hardware and Software design of the project. Details of all design considerations and implementation are provided in this section.
Hardware Design
This section describes the various hardware design parameters involved in the project. It involves conveyor system design, object rejection hardware, object sorting hardware,object detection hardware and conveyor drive hardware.
SJOne Board
The SJOne board is one of the development boards used for the project. Some of the features offered by SJOne board are as follows:
- Built-in Nordic Wireless (Board-to-Board communication).
- Software Stack for Mesh Network.
- 4 Switches and 4 LEDs (both hard-wired).
- Many GPIOs with two SPI, Multiple UARTs, and I2C availability.
- Power from USB or External Power.
- PWM support.
The SJOne board offers whole lot of features out of which wireless communication (Nordic), GPIO, I2C and PWM timers are utilized for the project. The I2C feature was used to communicate with the Raspberry pi, GPIO's used to drive stepper motors, PWM's to drive servo motors, Nordic for wireless communicaton between SJOne boards.
Raspberry Pi
Raspberry Pi is a linux based embedded computer created by the Raspberry Pi Foundation. The Foundation aims at providing cheap and affordable computing solutions.
Raspberry Pi features
Microcontroller: The Raspberry Pi contains a microcontroller manufactured by Broadcom - BCM2836. The processor within this chip is a quad-core Cortex-A7. Operating System: Raspberry pi has many compatible operating systems available. We chose the Raspbian Jessie operating system as it popular. The underlying core for this operating system is a debian linux kernel. Peripherals: The various hardware available on board can be found here.
Installing OpenCV libraries on Raspberry Pi
OpenCV is a collection of libraries that provide various C, C++ and Python functions that can be directly used for Image processing. The advantage is that it provides data structures to hold RGB images and complex image processing algorithms, such as edge detection, is already present and just have to be called within our program. OpenCV installation can be done using various methods:
- Cross compiling on a general purpose computer: This is a common method for compiling OpenCV. This method involves compiling OpenCV on a general purpose computer which has a cross compiler installed intended for your target platform, such as the arm-gcc. The advantage of cross compiling is that the compiled libraries can be now used on both the target and host computers. It would now be possible to compile program by using the arm-gcc on your host platform and then copying the generated executable into the target platform and directly run it.
- Installing by building a new kernel image for your target platform: Another popular method is to use tools such as Buildroot which compile a kernel image for your target platform. This is useful if you are installing a new OS and want the OpenCV libraries installed along with this new OS. This tool automatically searches and compiles any dependencies that would be needed to install OpenCV, which is the reason this is popular.
- Installing directly on the target platform: This is a less preferred method because of the large amount of time it can take and because it needs an . Compiling libraries is a resource intensive process and thus takes a large amount of time (more than 2-3 hours) on a slower processor.
For our project we chose method three for the following reasons:
- Raspberry Pi 2 with its quad-core processor is capable of handling these loads.
- It was easier to compile on the native system than cross-compiling.
- With an internet connectivity adding dependencies was easy using “sudo apt-get install”
After searching the internet for methods to compile the libraries we came across this script to install the libraries and its dependencies.
Conveyor Belt
Conveyor belt is a mechanical system with which materials can be transferred from one location to another. Conveyor systems are used in many industrial applications for safely transferring materials where several processing steps happen on the material when it travels on the belt. As the conveyor system is fully automated, it can be ensured to a have a uniform throughput of the deliverable. Industries thus use the conveyors for increasing their production rate. Conveyor systems are very popular in material handling, packaging, automotive, food processing and many other industries.
Webcam
The USB webcam is an integral part of the Image processing system running as part of the automated assembly line. The camera acts as eyes of the system and is used to capture images, that may be processed later using open CV algorithms. Choice of camera is crucial to the success of the project. A camera with high definition image quality and wide angle view provides better image processing control and lower pixel corruption due to external light related noise. A microsoft HD lifecam 3000 was chosen due to the various high end features it offers and comparatively low cost. Some of the features of this camera are as follows:
- Widescreen - 16:9 format offers cinematic video recording.
- TrueColor technology - TrueColor technology automatically delivers bright and colorful video, in virtually all lighting conditions.
- Universal attachment base - Works on your desktop, laptop, or notebook.
- Small size and ease of mounting.
- 2.0 USB connectivity.
Nordic wireless
SJone board has a nordic wireless chip nRF24L01+ manufactured by the Nordic Semiconductors interfaced to it. Through a Serial peripheral Interface it is possible to operate and configure the Nordic chip. The air data rate, operating modes and RF channel frequency need to be set by configuring the registers in the nRF24L01+.
Air data rate is the modulating signalling rate that is used by nRF24L01+ for transmitting and receiving data. It can be 250 Kbps, 1 Mbps and 2 Mbps. Lower air data rate gives better receiver sensitivity while higher data rate gives a reduced current consumption. The RF channel frequency determines the centre of the channel frequency in which nRF24L01+ operates. Typical operating frequencies are from 2.400GHz to 2.525GHz
For communication to happen,
- Both nordic transmitter and receiver must be set at the same air data rate. We have set the air data rate at 250 Kbps.
- Both nordic transmitter and receiver must be operated at the same RF channel frequency.We have set the channel number to 2.499 GHz.
IR Detector
IR Detector OSEPP IRDET-01 detects obstacles when they are in the calibrated range. The module has 3 pins which are the ground(G), VCC(V) and Signal(S). The signal wire line is digital. It can be either high or low. High if the detected range is further than the calibrated distance, low if the detected range is less than the calibrated distance. The pins are connected to the GND, VCC and GPIO pin of the SJone board, The pin direction is set as input. When no object is detected the pin will read 0 and when object is detected the pin will read 1.
Calibration
IR detector module device has two potentiometers that can be used to calibrate the device. One for frequency (F) and one for distance (D). Much like an IR Remote, the IR light is oscillated at a defined frequency and detected at that same frequency; this enables the sensor to work in noisy environments. The frequency of the detector module can be adjusted to avoid interference with other detectors or other IR sources.
Turning the frequency potentiometer clockwise increases the frequency and turning it anti clockwise decreases the frequency.The distance at which the sensor is triggered is determined by adjusting the Distance potentiometer. When the sensor is triggered the LED will light up. Turning this pot clockwise increases the triggering distance, turning it counter-clockwise decreases the distance.
Servo Motor
SG90 is a tiny and lightweight servo motor with high output power. Servo can rotate approximately 180 degrees (90 in each direction), and is driven by a PWM wave. It fits in small places and offers good torque. Using servo motors provides precise control of angular movement using relevant duty cycle of a constant frequency PWM wave. Servos consist of an internal motor with a feedback system for position.
The wiring information for SG90 is as follows:
| Sl. No | Servo Pin Color | Purpose |
|---|---|---|
| 1 | Brown | Ground |
| 2 | Red | VCC (max 4.8 Volts) |
| 3 | Yellow | PWM input to motor |
Some of the specifications if SG90 servo are as follows:
- Weight: 9 g.
- Dimension: 22.2 x 11.8 x 31 mm approx.
- Stall torque: 1.8 kgf·cm.
- Operating speed: 0.1 s/60 degree.
- Operating voltage: 4.8 V (~5V).
A PWM wave of 50 Hz or 20 ms is needed to drive the servo. The servo works with duty cycle between 5-10%. Provided a wave with duty cycle 5% the motor rotates 90 degree to the left from neutral position (extreme left position), with 10% it rotates 90 degree to the right from neutral position(extreme right position) and with 7.5% duty cycle moves to neutral position.
Stepper Motor
A stepper motor is a DC motor which moves in discreet steps. The motor has poles which when energizedFor driving the conveyer system 28BYJ-48 stepper motors were used with ULN2003 darlington array module. The ULN2003 is a high voltage, high current darlington array with 7 NPN darlington pairs for providing high current required by the stepper motor.
Some of the features of the stepper are :
- Voltage: 5V.
- Diameter: 28mm.
- Step angle: 5.625 x 1 / 64.
- Reduction ratio: 1 / 64
- 5 Line 4 phase can be driven by ordinary ULN2003 chip.
- Driven using PWM or GPIO.
- Simple connections with easy to plug connectors and switch for power.
- A, B, C, D four-phase LED indicates the status of the stepper motor work.
Hardware Interface
The hardware interface is divided into two parts. Components that were interfaced with SJOne Master board and components that were interfaced with the SJOne Slave board. Both the SJone boards had to be mounted one on each end of the conveyor system. Decsions made by the SJone master board will be implemented by the SJOne slave board.
Building our own conveyor system
Initially it was decided to buy or rent conveyor systems and interface components to it. First option was to buy the conveyor system. On enquiring with many dealers who supplied industrial mini conveyor systems to small scale and large scale industries , it turned out that a simple system was around 1200 US dollars. Second option was to rent the conveyor belt. Renting it for a day would cost around 200 dollars. But the requirement for the project was definitely more than a day. There were development and testing phases of each component that had to be mounted with the conveyor belt and tested frequently. So the options of buying and renting were eliminated. Having an objective of building an economical automated conveyor system, the decision to self build the conveyor system was made by the team.
Nevertheless, it wasn't that easy to build a conveyor system. The conveyor system required the following
- A rigid frame on both sides to hold the conveyor belt
- Rotating cylinders that had to be mounted to either ends of the frame.
- A conveyor belt that could slide on the cylinders bordered by the rigid frame
Interface with SJOne Master Board
The SJOne master board had the following components to it:
- I2C connection with Raspberry Pi: Raspberry pi passes information about the object detected to the SJOne Master board through I2C. The SJOne board is configured as a slave so that it can receive the information of the color of the object.
- Nordic transmitter:The information received through I2C is relayed to the slave SJOne board for further sorting of the objects.
- IR sensor: The IR sensor detects the presence of an object. This is needed because the camera can detect only the valid colors, but there is no way for the camera to know whether there is an object that is of the wrong color or if there are no objects.
- Servo motor for rejection: The servo motor requires has three pins: VCC, GND and PWM signal. The PWM signal is provided at port 2 pin 1. This servo motor is responsible for rejecting any object that is not blue or green in color. The decision to swing the servo motor arm is based on the information received on the I2C interface and the IR sensor input.
Interface with SJOne Slave Board
The SJOne slave board has three interfaces:
- Nordic wireless receiver:The wireless receiver receives information of the object on the conveyor belt and will take the decision to sort it based on the color of the object
- IR sensor to detect object: The IR sensor detects the presence of an object. This is needed because the camera can detect only the valid colors, but there is no way for the camera to know whether there is an object that is of the wrong color or if there are no objects.
- Servo motor for sorting: The servo motor connected to the slave is responsible for sorting the green object, but allow the blue object to pass through. The decision to swing the servo motor arm is based on the information received on the Nordic wireless interface and the IR sensor input.
Software Design
Show your software design. For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the code. For example, do not show exact code, but you may show psuedocode and fragments of code. Keep in mind that you are showing DESIGN of your software, not the inner workings of it.
Assembly line Mechanism
Object Detection and Analysis with OpenCv and image processing
Image processing is the process of manipulating image data in order to make it suitable for computer vision applications. Computer vision helps to obtain relevant information from images and make decisions based on that information. In our project, we have used OpenCV with C++ to develop our computer vision application. OpenCV is an open source C++ library for image processing and computer vision, originally developed by Intel and now supported by Willow Garage. OpenCV has many inbuilt functions for image processing and computer vision that are mainly aimed at real-time processing.
Digital Image and pixels
A digital image is a collection of pixels. A pixel is the smallest element in a digital image. In an image, each pixel can have a particular value to represent that image. For example, consider a grayscale image where each pixel can be represented by 8-bit value. So, each pixel can have a range of 0 to 255 where 0 is the complete black and 255 is a completely white pixel.
A color image consists of 3 planes Red, Green, and Blue. Any color can be formed with the combination these three planes. These planes are called channels of the image. So, an RGB i.e. a color image has 3 channels. Each pixel in RGB has a combination of three values. (255, 0, 0) represent pure red. (0, 255, 0) represent pure green. (255, 0, 255) represents pure violate. Thus, each pixel is represented by 24-bits in an RGB.
Color Detection
Object detection and segmentation is the most important task of computer vision. The easiest way to detect and segment the object is through color detection.
The method used for color detection was to first convert the RGB image into HSV and then by thresholding the HSV image, the binary version (Black and White) of the HSV can be obtained. In this binary image, the white space would represent the color which wants to detect. This binary image can then be used to detect the object we are looking using color detection and also to find the shape of the detected object using the available OpenCv library functions.
Need to convert from RGB to HSV
RGB color space describes colors in terms of the amount of red, green, and blue present. HSV color space describes colors in terms of the Hue, Saturation, and Value. The HSV image separates luma (intensity component), from chroma (color component). So that we can now perform histological equalization of the image by adjusting the minimum and maximum values of H, S, and V to such levels that only the pixel with the desired color will be illuminated (white) and the rest of the pixels will be dark.
void inRange(InputArray src, InputArray lowerb, InputArray upperb, OutputArray dst) library function was used to threshold the HSV image.
Src is the source image. Dst is the thresholded image. The upperb and lowerb boundary parameters are scalars (minimum and maximum values of HSV) which can be determined using trackbars.
Trackbars
Trackbar is a very useful tool which allows us to change various parameters while our OpenCv application is running. In our project, we made extensive use of trackbars to determine upper and lower boundary parameters for the particular color which we want to detect. Show the image of trackbars and thresholded image. As shown in the fig. we can change the minimum values of H, S, and V in such a manner that only the desired color will remain as a whitespace and rest of the image should be dark. Once the minimum and maximum values are determined we can lock down these values with inRange() function to get the thresholded image.
int createTrackbar(const string& trackbar name, const string& winname, int* value, int count, TrackbarCallback) is the library function which was used to create the trackbars.
Trackbarname is the name of the trackbar.
Winname is the window name to which trackbar is attached. Integer, pointed by this pointer, holds the value associated with the position of the trackbar.
count - The maximum value of the trackbar. The minimum value is always zero.
findContours library function
Using find contours we can retrieve the contours from the binary image that we obtained through thresholding the HSV image. void findContours(InputOutputArray image, OutputArrayOfArrays contours, OutputArray hierarchy, int mode, int method, Point offset=Point()) We can pass appropriate modes to retrieve the different contour information from the image. The contour information is retrieved in the contour is a vector of points. From the contours information, we can find the area of the whitespaces using moments method which is actually the target color and can detect the color.
Algorithm for Color Detection
Step1: Initialize the camera.
Step2: Capture the image frame.
Step3: Convert the image from RGB to HSV.
Step4: Threshold the image for the required color.
Step5: Apply morphological operations to eliminate noise.
Step6: Find contours of the binary image.
Step7: Find the number of whitespace regions using the contour information.
Step8: If no whitespaces found go to step 2, else proceed to step9.
Step9: Check if the area of whitespace is greater than the minimum area. If area is less than the minimum area set, it may be a noise then go to step 2, else proceed to step 10.
Step10: Set object found flag to true. Continue scanning go to step2.
Flowchart for Color Detection
Morphological Operations
Filtering is an important part of computer vision. In most of the applications, filtering should be applied to the images before processing on them. In our project, we applied morphological operations to remove noise and to do proper segmentation of the images. The two morphological operations we performed were: Dilation and Erosion.
Dilation: Dilation increases the white pixels along the boundaries of the foreground object. Consider the following example.
As we can see that dilation increases white pixels along the boundary of the image.
Erosion: Erosion is opposite to dilation. It increases the dark pixels along the boundaries of the foreground object. The example of the erosion is as shown in the figure.
As we can clearly see the erosion operation increased the black pixels along the boundary of the image. In our project, we applied erosion followed by dilation because the erosion sinks our object, so we dilate it.
Implementation
The implementation of the object color identification was done through image-processing. We used Raspberry pi and OpenCV to perform the image-processing. The image frames were captured from a USB camera connected to the Raspberry pi. Each frame was processed with our OpenCV algorithm to detect the desired object. The information about the detected object was then passed to SJOne board interfaced to the Raspberry pi through I2C. The values for the object present in front of the camera could have three values:
- Blue
- Green
- Invalid
The value invalid is for both when there is no object in front of the camera or an object that does not have the desired color. Once this information is passed to the SJOne Master board the IR sensor connected to this board identifies whether the invalid classification was for an invalid colored object or for an absence of any object. If the decision made computes as an invalid object, the servo motor activates and pushes the object out of the conveyor system else the servo motor remains in its idle position. A similar logic is employed at the slave SjOne board. The information for the color of the object is relayed to this slave board via the Nordic wireless interface. If the object is green, then the sorting servo pushes it away and if it is blue, it lets it pass through.
The flow chart below explains this process graphically.
Testing & Technical Challenges
Testing
Testing for this project was carried out in phases. This was a conscious decision on part of all the team members as previous experiences have taught is that testing everything all at once can lead to problems that could otherwise have been identified earlier. The testing involved the following phases:
Phase 1
Phase once consisted of unit testing each part. The image processing algorithm was first tested on a desktop computer. This was then ported to the Raspberry Pi which resulted in an instant success on the Raspberry Pi. Parallel to this the motor systems, I2C, IR sensor calibration and Nordic wireless messaging were tested and the code was fixed.
Phase 2
Phase 2 testing was to combine the smaller parts of the system. The code for image processing and I2C was merged and tested. At the same time the IR and servo motor was tested so that they worked in conjunction with the messages received over I2C(for the SJOne Master board) and Nordic wireless(for the SJOne slave board). At this point we also ensured the conveyor system was in place and working smoothly.
Final testing
Final testing consisted of combining all the parts and securing them in the housing and testing the rejection system first and then the rejection system was tested. Phased testing enabled us to quickly identify any changes that would be needed when the entire system was in place. This resulted in reducing the time spent on integrating the system.
Technical Challenges
This project was a challenging one due to the varied components involved: from choosing the correct board for image processing to building our own conveyor system. There were a lot of difficult areas in this project. The most significant being the entire housing for all the components and the conveyor system and a lot of the ideas we had at the start had to be changed in order to have the desired result. The most significant challenges are listed below:
Installing OpenCV on ARM 11 (FriendlyARM Tiny6410 SDK)
The Raspberry Pi was not our first choice for a board that can perform image processing, rather it was another board: The FriendlyARM Tiny6410 SDK, it was readily available with us. This board has a Samsung ARM 11 based microcontroller with good memory and decent speed. The board comes with a preinstalled Linux image. The problem was, the documentation for this board is not very comprehensive and the support community is not very active. This meant a lot of trial and error. We tried to install OpenCV but were not successful. The cross compilation kept failing for this board, and eventually we had to give up the idea of using this board due to time constraints in favor of the highly popular Raspberry Pi.
Using a pre-built conveyor system
At the outset, we tried to minimize the amount of custom build required for this project. This meant needed a readymade conveyor system. We found and ordered a conveyor system that could be manually cranked. We thought we could attach motors and drive the conveyor leading to a hassle free conveyor system. The result, however, was the opposite. The conveyor could not be dismantled and so the correct area for the motor drive could not be accessed. We managed to attach a motor at one end, but, by this time the structural integrity of the rubber conveyor belt was compromised and so we could not get the motor to drive the conveyor successfully. Eventually we had to build our own conveyor system and housing for all the components. This was a great build exercise as we decided to choose recyclable material for this and made the assembly using cartons and paper.
Lighting Problem
Lighting was the most critical problem faced during image-processing. In our project, we converted the RGB matrix frames to HSV for the thresholding. The HSV frame extracts the lighting information ‘Luminosity’ and ‘Chroma’ from the RGB planes, so any slight change in the light source angles would alter the thresholding values for HSV frames. Also, we faced a lot of problems due to distortions arising from the light source. We tried to handle the problem by applying morphological operations and implemented the color detection Algorithm. But, still due to slight distortions still present we faced problem in implementing our shape detection Algorithm where even slight distortions would cause problems in detecting the vertices of the detected object.
Conclusion
The learning experience our team wanted to have was to learn about things we had not done before. This is reflected in the major components in the project, such as the use of image processing. This further led to the realization that SJOne cannot be used for image processing and hence we had to to look for other resources. The use of an embedded linux computer enhanced our understanding of an RTOS. The use of two different Operating Systems: FreeRTOS and Raspbian Jessie helped us in understanding the different use cases for them and how the software complements the hardware. Another takeaway from this project is the build. Building the conveyor system was challenging and needed a lot of original thinking. Cross compiling of OpenCV was another major part of this project and it helped us understand how libraries and compilers work together and help in writing programs easily. This aspect of programming is often neglected in application programming and this project gave us a deep insight into this.In conclusion, it can be said that the project was a success and it has been successful in making us understand Embedded Software better.
Project Video
Project Source Code
References
Acknowledgement
We would like to thank our advisor Preetpal Kang for encouraging us to take on a challenging project.
