MongoDBInputFormat

Introduction

MongoDBInputFormat is used for reading data from MongoDB in Husky. It is requested to configure the MongoDB as Sharding in the cluster. Reading from standalone version is not considered since it is very easy by calling the API of MongoDB driver. In details, the reading process is divided in to two periods: the first is that Husky Master will ask the config server for the information of chunks; each Husky Worker will ask Husky Master for an available and unread chunk to read, until all the chunks are delivered.

Usage

The code can be found at io/input/mongodb_inputformat.{c,h}pp. Here is an example.

husky::io::MongoDBInputFormat infmt;
infmt.set_server("MONGODB_ROUTE_ADDRESS");
infmt.set_ns("DB_NAME", "COLLECTION_NAME");
infmt.set_auth("USERNAME", "PASSWORD");  // If it needs to auth at first.
infmt.set_query("{id: $lte 10000}");