S15: Tree Node using Google Protocol Buffers
Contents
- 1 Grading Criteria
- 2 Tree Node: using Google Protocol Buffers
- 3 Problem Statement
- 4 Our Solution
- 5 Abstract
- 6 Objectives & Introduction
- 7 Project Schedule
- 8 Parts List & Cost
- 9 Protocol Buffers
- 10 Nordic Wireless
- 11 Design & Implementation
- 12 Testing & Technical Challenges
- 13 Conclusion
- 14 References
Grading Criteria
- How well is Software & Hardware Design described?
- How well can this report be used to reproduce this project?
- Code Quality
- Overall Report Quality:
- Software Block Diagrams
- Hardware Block Diagrams
- Schematic Quality
- Quality of technical challenges and solutions adopted.
Tree Node: using Google Protocol Buffers
Problem Statement
With the race to harness the value of big data in full swing, no one is happy about storing multiple versions of data. Its expensive, delays data availability, and just seems like a waste of time. Machines cannot afford to send and receive large chunk of data among each other. Sending large amount of data usually increases the transmission delays in the communication system and as a result the processing time of over all system increases. So, there is a burning need of faster, lighter and smarter way to process and transmit the data in the world of internet of things.
Basic Questions Asked:
- What kind of protocol to use to transmit serial data ?
- What is an efficient method for storing and exchanging the data?
Our Solution
Data Serialization
Historically, handling huge data has been a challenge for developers, mostly due to dependencies across multiple platforms and data access layers. Developers face situations where we want to deploy new code, but some other client relies on the old code. It can stop a deployment dead in its tracks. As models and needs mature more towards data as a service, developers expect that multiple clients would run different versions of a software platforms and even use different languages. In these cases, data serialization become quite important and serves as the best possible option.
So basically, serialization is used when large amounts of data have to be stored in flat files and retrieved at a later stage or to transport over the network. To achieve big data handling without serialization, it becomes too tedious, error-prone and complicated as the data structure is complex. So, we needed a serialization technique that is fast, compact format, easy to define, extensible and platform independent. That's where Google Protocol Buffers come into picture.
Google protocol buffers (GPB) is the best available way for data serialization between servers. By implementing lighter and micro controller compatible version of GPB - nanoPB - we created a faster and better machine to machine communication network. Google Protocol buffers provides neutral and efficient way of serializing structured data for the use in communication protocols.
Abstract
This project is to implement google protocol buffers for serialization of the data. In this main role of protocol buffers is to encode the data before sending it out via nordic wireless communication and decoding the received data on the receiver side.
- This project mainly focuses on Following topics:
- Studying different types of Protocol Buffers (GPB)
Google Protocol Buffers
NanoPB
Protobuf-c
Flat Buffers
- Implementing Nordic wireless communication between different boards by using these implemented google protocol buffers.
- Studying network architecture - Tree Achitecture
Objectives & Introduction
Show list of your objectives. This section includes the high level details of your project. You can write about the various sensors or peripherals you used to get your project completed.
Team Members & Responsibilities
Project Schedule
Following table gives the complete description of in-advance planning and deliverable for the project. This project schedule help us in scoping out the project and keep oneself in check with the progress.
| Week No. | Start Date | Planned End Date | Task | Status | Actual End Date |
|---|---|---|---|---|---|
| 1 | 03/31/2015 | 04/07/2015 |
* Decision on all required modules and placing the order online. * Research on Google Protocol Buffers |
|
04/07/2015 |
| 2 | 04/06/2015 | 04/13/2015 |
* Environment setup of Google Protocol Buffer on windows and accomplish an example task such as hello world. |
we are able to compile the basic program successfully.(AI: to figure out run part of the compiled code).
|
04/29/2015 |
| 3 | 04/13/2015 | 04/20/2015 |
* Nordic Wireless: Ping Test between two SJOne boards. |
|
04/29/2015 |
| 4 | 04/20/2015 | 04/27/2015 |
* Nordic wireless all development and testing for 2-3 SJOne Boards. * Successful implementation of nanoPB ( GPB variant) |
|
04/30/2015 |
| 5 | 04/27/2015 | 05/04/2015 |
* Start code development for LCD module. * Testing Code via Google Protocol Buffers. |
|
--/--/---- |
| 6 | 05/04/2015 | 05/11/2015 |
* Integration of all modules with parallel testing. |
|
05/20/2015 |
| 7 | 05/11/2015 | 05/18/2015 |
* Integration of the code with Google Protocol Buffers. |
|
05/20/2015 |
| 8 | 05/18/2015 | 05/25/2015 |
* Final Testing and integration of additional features if time permits. |
|
--/--/---- |
| 9 | 05/25/2015 | 05/25/2015 |
* Final Demo Day |
|
--/--/---- |
Parts List & Cost
Following are the components and modules required for this project.
Parts List & Cost
| Item# | Part Desciption | Vendor | Part Number | Qty | Cost |
|---|---|---|---|---|---|
| 1 | SJOne Board | SCU Room | Revision 2 | 3 | $240 |
| 2 | Antenna's | SCU room | CMPE 295 | 3 | $12 |
| 3 | LCD Module | New Haven | NHD-0216B3Z-FL-GBW-V3-ND | 1 | $27 |
| 4 | Reserved for extra modules | NA | NA | NA | NA |
Protocol Buffers
In the below sections we discuss the different types of Protocol Buffers we explored and how we concluded that Nanopb was suitable for our project.
Google Protocol Buffers
Protocol Buffers was developed by Google for serializing data.
So, what is Serialization? According to Wikipedia, "serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and reconstructed later in the same or another computer environment". Google Protocol Buffers is one of the methods used for serializing. The other famous method is XML. But, the main advantage of using Google Protocol Buffer over XML is that it is light weight, which makes it fast. It also has an easier implementation.
Currently, GPB has extended support for many programming languages, but C++, Java and Python are widely used.
Getting Started
The first step for us was to install the package available in https://github.com/google/protobuf/ and follow the instructions given in it. We first tried it in Windows Operating System. Because we were not able to successfully install in Windows, we moved to Linux.
Installation in Windows(Unix terminal)
The SJSU development package consists of a folder MSYS inside MINGW. The proto compiler is the most important thing and it was downloaded from https://developers.google.com/protocol-buffers/docs/downloads. It isused to compile .proto files which produces pb.h and pb.cc files. The protoc binary executable file was then placed inside the bin folder and its path defined in the environment variables. Msys is a unix like terminal used for configuring and building of applications which depend on Unix tools. The below steps for installing GPB were followed -
- Double click on ./autogen.sh. It will run in command prompt. This will generate a gtest folder. It consists of configure files, source code and libraries.
- Now in another window, open msys.bat file to start the unix like terminal.
- Navigate to the gtest folder and run ./configure
- This takes time and checks for gcc compatibility. Also, make sure that your antivirus is temporarily off. If not, the configure file may not be executed successfully and it will give errors like gcc configured
- Once configure runs successfully, type 'make' and execute it.
- After executing make, type 'make install' and execute it. Here is where, we faced issues.
The error screenshot is below -
After posting on various google groups, we found out that, support has not yet been released for installing GPB with MinGW in Windows. Although, in Windows Visual Studio could have been used, we tried to use eclipse plugins to compile and run.
Installation in Windows(Eclipse) The plugin to download was available in https://code.google.com/p/protobuf-dt/wiki/Installing. The steps required to follow in eclipse are -
- Xtext and protobuf-dt is installed from http://download.eclipse.org/modeling/tmf/xtext/updates/composite/releases/. This is a descriptor i.e. highlights the keywords. To install the given link in Eclipse -
- Inside the Eclipse CDT, click on 'Help' -> 'Install New Software'. A new dialog box opens where you will have to copy paste the link and click on 'Add'. The software finds the required plugins and you can complete the installation.
- This can be similarly done for 'protobuf-dt' plugin.
- After installing the plugin, we check if the option 'Protocol Buffers' is present in Windows -> Preferences. As seen in the below figure, we will use protoc in PATH variable and descriptor.proto location is given. Under the Options tab , Gen-C++ is checked so that after compiling, C++ src-gen (Source) is generated.
Installation in Linux
Installing GPB in Linux is simple. Follow the methods similar to Windows Installation in Unix terminal and use it in Terminal in Linux. The installation will be successful because all make files are supported for linux.
Sample Schema
Defining a sample proto file
Below is the definition of the proto file and it is saved with a .proto extension. All the proto files are defined through keyword message. The keyword required is a modifier. Other modifiers are optional and repeated. When required is defined, then that particular variable has to be initialized in the main or as mentioned in the below code with default value. For example the below code is defined as hello.proto
message hello
{
required string text= 1 [default="Hello World!"];
}
Compiling the proto file and defining source code
Once the proto file is defined, it is then compiled in the terminal. The command protoc --cpp_out = hello.proto will compile the file and generate .pb.cc and .pb.h files. These files contain the API's for encoding and decoding the file. Once these are generated, a source file is created called hello.cpp. The following sample code shows its demonstration.
int main()
{
hello demo; // instance "demo" of the "hello" class defined in proto
const std::string& test = demo.data(); // data is the field where the text was stored
cout << test << "\n";
demo.set_data("How do you do?"); // changing the data
cout << demo.data() << "\n";
return 0;
}
Eclipse
In Eclipse once, we define the proto file we can build it, since we have defined the path for protoc compiler and installed the protobuf plugin. This generates a src-gen folder which consists of hello.pb.h and hello.pb.cc files. Further issues were faced when we tried to integrate it with Preet's folder structure. That will be discussed in Testing and Technical challenges section.
NANOPB
Although Google has provided an official protobuffer implementation for C/C++, we were unable to utilize the official version as the embedded system's C library is not fully POSIX compliant, preventing the official implementation from cross-compiling. Protobuffer is also not memory efficient, limiting its implementation on memory constrained embedded systems.
So we turned to nanopb. It is a variant of protocol buffers, specifically designed for embedded systems. Unlike google protocol buffers which generate C/C++ typedef structures, nanopb generates associated C structures which has tiny footprint. Nanopb comes with minimal requirements for RAM and code space and it is primarily suited for 32-bit microcontrollers like ours.
Features
- No requirement for dynamic memory allocations.
- Pure C runtime.
- Memory efficient: small RAM and code size
- Backward compatible.
- Callback mechanism for handling messages larger than can fit in available RAM.
- Most of the protobuffer features by sacrificing some speed.
Structure
Protobuffer compiler compiles .proto file by linking them to the library functions.
Protobuffer compiler compiles .proto file to .pb file which is not human readable. This architecture is built over the nanopb generator. nanopb_generator.py is python tool script which call upon a number of nanopb library functions and any other user defined application or library functions to generate protobuffer header and source files (.pb.h and .pb.c). Both of these runtime generated files purely consists of C library functions, that’s why low RAM and code space usage.
For the runtime program, you always need pb.h for type declarations. Depending on whether you want to encode, decode, or both, you also need pb_encode.h/c or pb_decode.h/c. Application must declare pb.h, encoding/decoding/both header files and pb.h generated by nanopb_generator tool script at the runtime.
Application layer is composed of main function along with nanopb header files and protocol buffer messages.
Installation
Linux
Nanopb is primarily made for linux flavored environments. So a lot of support and literature is available for linux environment. Linux scripts made available with the nanopb installation files and out of the box python support make it very easy to compile and generate protobuffer files. So I would not like to go into these details but they can be found here [1]
Windows
As said earlier nanopb is primarily buil for linux environment, so windows installation is bit complicated and is built over the linux environment. Main issue with the lack of out of the support for python in windows environment. So combination of Dynamic-link library (DLL) files and executables need to be handled during installation.
Steps to be followed:
- Download nanopb-0.x.x-windows-x86.zip (our version is 0.3.3) from [2]. Extract downloaded zip into a folder.
- Install python 2.7 (must use version 2.7 version and not 3.4) from [3] and python protobuf from [4]
- Now go to extracted folder -> generator-bin. Copy python27.dll from python in the nanopb folder -> generator-bin. Make sure you have python27.dll at this location, if not copy it from C:/Windows/System32
- Copy pb.h, pb_common.c/h, pb_encode.c/h and pb_decode.c/h in the current directory.
- Use this as working directory to generate protobuffer source and header files along with file_name.options file in which you can define micro controller related constraints if not done earlier in the .proto file.
Eclipse environment setup
- Though this step is optional but highly recommended. It will simply ignore the protobuffer files if included unknowingly in the working directory. Eclipse has protobuffer plugin for which you will need egit plugin. Go to eclipse -> Help -> Eclipse Marketplace and search for egit and cdt-protobuf.
- Copy pb.h, pb_common.c/h, pb_encode.c/h and pb_decode.c/h in the Eclipse working directory.
Nordic Wireless
For the wireless communication between two or more SJone controllers, we have used the Nordic wireless communication. SJone board itself have onboard Nordic wireless chip nrf24L01 configured with Serial Peripheral Interface(SPI) which also features Enhanced ShockBurst™. This helps in automatic packet handling and timing. During transmit, ShockBurst™ assembles the packet and clocks the bits in the data packet for transmission. During receive, ShockBurst™ constantly searches for a valid address in the demodulated signal.
The nrf24L01 + Full three level FIFO for both TX and RX direction helps in achieving maximum 0-10Mbps with 4 wire SPI interface. The following Diagram shows internal working of nordic wireless.
Hardware Design
Discuss your hardware design here. Show detailed schematics, and the interface here.
Hardware Interface
In this section, you can describe how your hardware communicates, such as which BUSes used. You can discuss your driver implementation here, such that the Software Design section is isolated to talk about high level workings rather than inner working of your project.
Software Design
Show your software design. For example, if you are designing an MP3 Player, show the tasks that you are using, and what they are doing at a high level. Do not show the details of the code. For example, do not show exact code, but you may show psuedocode and fragments of code. Keep in mind that you are showing DESIGN of your software, not the inner workings of it.
Insert here
Design & Implementation
Implementation
This section includes implementation, but again, not the details, just the high level. For example, you can list the steps it takes to communicate over a sensor, or the steps needed to write a page of memory onto SPI Flash. You can include sub-sections for each of your component implementation.
Nanopb:
- Create sensor.proto file in the nanopb/generator_bin directory or the directory in which nanopb_generator is set up earlier.
- Open cmd -> protoc -osensor.pb sensor.proto. This will generate sensor.options and sensor.pb
protoc -osensor.proto
- Cmd -> nanopb_generator sensor.pb. It will create header and source files, sensor.pb.c and sensor.pb.h.
nanopb_generator osensor.pb
- Include sensor.pb.c/h files in the eclipse working directory either by including the path to these files or by copying them into the project folder.
- As discussed earlier it is very important to include pb.h, encode.h, decode.h along with sensor.pb.h at the run time.
Wireless
- Include wireless task with the highest priority.
Testing & Technical Challenges
Describe the challenges of your project. What advise would you give yourself or someone else if your project can be started from scratch again? Make a smooth transition to testing section and described what it took to test your project.
Include sub-sections that list out a problem and solution, such as:
My Issue #1
The first issue that we faced was during installation of GPB in Windows environment. As described in the GOOGLE PROTOCOL BUFFER section, while installing in Windows, we received error. We then tried to compile it in Eclipse. The compilation was successful but, after the generated src-gen folder, we faced a deadlock. We could not build it after defining the main program. Then we decided to import all the libraries required for protobuf and when we tried to build it we faced many errors such as closedir was not declared in this scope.
Example-
After giving it a lot of thought, we concluded that we were never supposed to use closedir and all other functionality. We then switched to Nanopb as its tiny footprint is the most suitable for 32 bit micro-controller.
1) Out of scope GPB functions displaying errors like closedir was not declared in this scope.
Solution:
· Switched to Nanopb as its tiny footprint is suitable for 32 bit micro-controller.
- ===Nanopb installation issues===
2) LCD blinking an error message and not showing the GUI.
Solution:
· Check whether the IDE has copied the project files in the microSD card. Generally it copies 3 files.
· Silly thing, but check whether the microSD is inserted behind the LCD.
· And one last thing, check if the microSD card is formatted in FAT file system.
· We had first used a SD card which wasn’t supported by the LCD, so we changed the SD card and used of some other manufacturer and it worked perfectly.
Conclusion
Conclude your project here. You can recap your testing and problems. You should address the "so what" part here to indicate what you ultimately learnt from this project. How has this project increased your knowledge?
Project Video
Upload a video of your project and post the link here.
Project Source Code
References
Acknowledgement
Any acknowledgement that you may wish to provide can be included here.
References Used
List any references used in project.
- Google's Nanopb web: http://koti.kapsi.fi/jpa/nanopb/docs/concepts.html
- Nanopb API references: http://koti.kapsi.fi/jpa/nanopb/docs/reference.html
Appendix
You can list the references you used.
