Google protobuf at run-time deserialization example in C++

Lately I have been looking at Google’s format for interchanging data between multiple processes or Protocol Buffers. As you would expect for an inter-process data format Protocol Buffers are supported on different platforms and implementations are available for most popular programming languages. An interesting example of such a platform is the stm32l1 micro controller, which is supported by a third party protobuf implementation named nano pb.

One of the interesting feats of Google’s C++ protobuf implementation is that it supports deserializing arbitrary protobuf objects at run-time. As long as the proto definition is known at run time, the library is capable of parsing any message back into the individual fields from the proto definition. This is especially useful in situations where a program has to be able to handle proto definitions that are unknown at compile time. In this case the standard approach of using the proto compiler (protoc) to construct C++ class definitions, writing (de)serialization code and linking the resulting program with these definitions is obviously not possible.

This post addresses the lack of examples that demonstrate this ‘at run-time deserialization’ feature of Google’s C++ protobuf implementation. The example below was constructed by combining the protobuf unit tests with this example provided by . It has been tested with the v3.0.0-beta-3 from Google’s github repo. Use at your own discretion. The example should be self explanatory.

/* At run-time deserialization of a protobuf buffer to a C++ object example
 * Based on https://cxwangyi.wordpress.com/2010/06/29/google-protocol-buffers-online-parsing-of-proto-file-and-related-data-files/
 * @author: Floris Van den Abeele <floris@vdna.be>
 *
 * Starting from a protobuf definition, main() does the following:
 * 1) Translate protobuf definition to FileDescriptorProto object using the
 * Parser from protoc. FileDescriptorProto seems to be nothing more than an
 * in-memory representation of the proto definition.
 * 2) Use a DescriptorPool to construct a FileDescriptor. FileDescriptor
 * seems to contain all necessary meta data to describe all the members of a
 * message that adheres to the proto definition. DescriptorPool can be used to
 * 'resolve' any other proto types that might be used by our proto definition.
 * 3) Print the parsed proto definition.
 * 4) Use DynamicMessageFactory and ParseFromArray to deserialize a binary
 * buffer to a Message that follows the proto definition
 * 5) Use Reflection to print the data fields present in the deserialized
 * object
 *
 * Note that this example code does not look at error handling.
 */
#include <iostream>

#include <google/protobuf/descriptor.h>
#include <google/protobuf/dynamic_message.h>
#include <google/protobuf/io/zero_copy_stream_impl.h>
#include <google/protobuf/io/tokenizer.h>

#include <google/protobuf/compiler/parser.h>

// Proto definition (can come from any IO source: disk, network, ...):
char text[] = "syntax = \"proto2\";\n"
  "message APIPort3 {"
  "required uint32 AppLedStateOn = 1;"
  "required uint32 PotiPercentage = 2;"
  "required uint32 VDD = 3;"
  "}";
std::string message_type("APIPort3");

int main() {
  using namespace google::protobuf;
  using namespace google::protobuf::io;
  using namespace google::protobuf::compiler;

  ArrayInputStream raw_input(text, strlen(text));
  Tokenizer input(&raw_input, NULL);

  // Proto definition to a representation as used by the protobuf lib:
  /* FileDescriptorProto documentation:
   * A valid .proto file can be translated directly to a FileDescriptorProto
   * without any other information (e.g. without reading its imports).
   * */
  FileDescriptorProto file_desc_proto;
  Parser parser;
  if (!parser.Parse(&input, &file_desc_proto)) {
    std::cerr << "Failed to parse .proto definition:" << text;
    return -1;
  }

  // Set the name in file_desc_proto as Parser::Parse does not do this:
  if (!file_desc_proto.has_name()) {
    file_desc_proto.set_name(message_type);
  }

  // Construct our own FileDescriptor for the proto file:
  /* FileDescriptor documentation:
   * Describes a whole .proto file.  To get the FileDescriptor for a compiled-in
   * file, get the descriptor for something defined in that file and call
   * descriptor->file().  Use DescriptorPool to construct your own descriptors.
   * */
  google::protobuf::DescriptorPool pool;
  const google::protobuf::FileDescriptor* file_desc =
    pool.BuildFile(file_desc_proto);
  if (file_desc == NULL) {
    std::cerr << "Cannot get file descriptor from file descriptor proto"
      << file_desc_proto.DebugString();
    return -1;
  }

  // As a .proto definition can contain more than one message Type,
  // select the message type that we are interested in
  const google::protobuf::Descriptor* message_desc =
    file_desc->FindMessageTypeByName(message_type);
  if (message_desc == NULL) {
    std::cerr << "Cannot get message descriptor of message: " << message_type
      << ", DebugString(): " << file_desc->DebugString();
    return -2;
  }

  // Print fields that are defined in the proto definition:
  // Note that proto defintion field numbers start from 1, but we assume that
  // the proto definition uses increasing field numbering without any gaps
  for (uint8_t i = 1; i <= message_desc->field_count(); i++) {
     const FieldDescriptor* field = message_desc->FindFieldByNumber(i);
     if (field)
       std::cout << field->name() << ": " << field->type_name() << " ("
         << field->label() << ")" << std::endl;
     else
       std::cerr << "Error fieldDescriptor object is NULL, field_count() = "
         << message_desc->field_count() << std::endl;
  }

  // Create an empty Message object that will hold the result of deserializing
  // a byte array for the proto definition:
  google::protobuf::DynamicMessageFactory factory;
  const google::protobuf::Message* prototype_msg =
    factory.GetPrototype(message_desc); // prototype_msg is immutable
  if (prototype_msg == NULL) {
    std::cerr << "Cannot create prototype message from message descriptor";
    return -3;
  }

  google::protobuf::Message* mutable_msg = prototype_msg->New();
  if (mutable_msg == NULL) {
    std::cerr << "Failed in prototype_msg->New(); to create mutable message";
    return -4;
  }

  // Deserialize a binary buffer that contains a message that is described by
  // the proto definition:
  uint8_t buffer[] = {0x08, 0x00, 0x10, 0x64, 0x18, 0xF5, 0x2D};
  if (!mutable_msg->ParseFromArray(buffer, 7)) {
    std::cerr << "Failed to parse value in buffer";
  }

  //std::cout << mutable_msg->DebugString();

  // Use the reflection interface to examine the contents.
  const Reflection* reflection = mutable_msg->GetReflection();
  std::vector<const FieldDescriptor*> fields;
  reflection->ListFields(*mutable_msg, &fields);
  // Loop over the fields that are present in the deserialized object:
  for (auto field_it = fields.begin(); field_it != fields.end(); field_it++) {
    const FieldDescriptor* field = *field_it;
     if (field) {
       // For our APIPort3 proto definition all types are UInt32, so
       // we don't have to inspect field->type_name() here:
       uint32 value = reflection->GetUInt32(*mutable_msg, field);
       std::cout << field->name() << " -> " << value << std::endl;
     } else
       std::cerr << "Error fieldDescriptor object is NULL" << std::endl;
  }

  return 0;
}

Note to self, compile with:

g++ -g -I~/GIT/protobuf/src -c -std=c++11 protobuf-example.cc
g++ protobuf-example.o -lprotoc -lprotobuf

Cross compiling C projects with external dependencies: Click modular router and the Raspberry PI

Introduction

At IBCN – my research group – we rely on Click router for implementing and evaluating various Internet communication protocols. Click is a modular software router in C++ that was developed at the end of the last century by Eddie Kohler and others at MIT (the original research paper is available here). Apart from the IP routing functionality that ships with Click, we’ve added a CoAP implementation that has been widely tested at ETSI Plugtest interoperability events. We also have a working 6LoWPAN implementation that was tested in  at last year’s summer IETF in Berlin. Lately, we’ve also added DTLS integration via CyaSSL. Continue reading

Bibliography available

In the interest of computer science research (ahum) I have put online my bibliography with PDF downloads of my own papers (i.e. where I am the first author).

I would urge any researcher to do the same as there is little to gain by isolating your work behind a pay wall. Apart from pay walls, hosting your own bibliography also makes accepted but unpublished work accessible to others sooner!

Cfr. http://cr.yp.to/bib/online.html

Hello world!

This post marks the beginning of my blog where I plan to write on various topics related to my professional activities as well as my personal interests.

Professionally, I’m currently engaged in a PhD focused on the Internet of Things. I’m employed at the Mobile research group at the department of Information Technology (INTEC) at Ghent University under supervision of prof. Ingrid Moerman. My `academic career’ can be tracked via Google scholar and/or my university’s digital library profile page. As the Internet of Things is a rather broad domain, let me be more specific. Currently I’m focusing on the impact of standardization on the Internet of Things.  I’ll write about the need for standardization in a separate post, as it would lead us to far for the purpose of this introductory post. In the future, you can also expect content about more generic topics related to computer science than just standardization though.

Apart from my academic ponderings, I also plan to write about some of my personal interests. Some of these are quite geeky, while others are more mundane. On the geek-side of the spectrum interests include communication networks/technology, Linux systems (and administration), web development and an occasional computer game. I’ll write about my home network setup in a later post, but for now I’ll say that I’m running a (non-native) dual stack and that this blog is hosted on a Raspberry PI with me at home. One consequence is that this blog is also reachable over IPv6 (try it)! More common interests include cycling (like most Belgians), (in-door) football, the classical piano, (fantasy) novels and hiking. I’ve included a list of my five latest bike rides on Strava on the right-hand side. As far as the piano goes, I’m currently enrolled in my second year at a local music school and am steadily improving (although I haven’t played for some time due to summer break).

Voilà, be sure to check back regularly for more content!