postface.txt

.output postface.wd
++ Postface

+++ Tales from Out There

I asked some of the contributors to the Guide to tell us what they were doing with 0MQ. Here are their stories.

++++ Rob Gagnon's Story

"We use 0MQ to assist in aggregating thousands of events occurring every minute across our global network of telecommunications servers so that we can accurately report and monitor for situations that require our attention. 0MQ made the development of the system not only easier, but faster to develop, and more robust and fault-tolerant than we had originally planned in our original design.

"We're able to easily add and remove clients from the network without the loss of any message. If we need to enhance the server portion of our system, we can stop and restart it as well, without having to worry about stopping all of the clients first. The built-in buffering of 0MQ makes this all possible."

++++ Tom van Leeuwen's Story

"I was looking for creating some kind of service bus connecting all kinds of services together. There were already some products that implemented a broker, but they did not have the functionality I wanted/needed. By accident I stumbled upon 0MQ which is awesome. It's very lightweight, lean, simple and easy to follow since the zguide is very complete and reads very well. I've actually implemented the Titanic pattern and the Majordomo broker with some additions (client/worker authentication and workers sending a catalog explaining what they provide and how they should be addressed).

"The beautiful thing about 0MQ is the fact that it is a library and not an application. You can mold it however you like and it simply puts boring things like queuing, reconnecting, tcp sockets and such to the background, making sure you can concentrate on what is important for you. I've implemented all kinds of workers/clients and the broker in Ruby, because that is the main language we use for development, but also some php clients to connect to the bus from existing php webapps. We use this service bus for cloud services connecting all kinds of platform devices to a service bus exposing functionality for automation.

"0MQ is very easy to understand and if you spend a day in the zguide, you'll have good knowledge of how it works. I'm a network engineer, not a software developer, but managed to create a very nice solution for our automation needs! 0MQ: Thank you very much!"

++++ Michael Jakl's Story

"We use 0MQ for distributing millions of documents per day in our distributed processing pipeline. We started out with big message queuing brokers that had their own respective issues and problems. In the quest of simplifying our architecture, we chose 0MQ to do the wiring. So far it had a huge impact in how our architecture scales and how easy it is to change/move the components. The plethora of language bindings lets us choose the right tool for the job without sacrificing interoperability in our system. We don't use a lot of sockets (less than 10 in our whole application), but that's all we needed to split a huge monolithic application into small independent parts.

"All in all, 0MQ lets me keep my sanity and helps my customers to stay within budget."

++++ Vadim Shalts's Story

"I am team leader in the company ActForex, which develops software for financial markets. Due to the nature of our domain, we need to process large volumes of prices quickly. In addition, it's extremely critical to minimize latency in processing orders and prices. Achieve a high throughput is not enough. Everything must be handled in a soft real-time with a predictable ultra low latency per price. The system consists of multiple components which exchanging messages. Each price can take a lot of processing stages, each of which increases total latency. As a consequence, low and predictable latency of messaging between components becomes a key factor of our architecture.

"We investigated different solutions to find suitable for our needs. We tried different message brokers (RabbitMQ, ActiveMQ Apollo, Kafka), but failed to reach a low and predictable latency with any of them. In the end, we have chosen ZeroMQ used in conjunction with ZooKeeper for service discovery. Complex coordination with ZeroMQ requires a relatively large effort and a good understanding, as a result of the natural complexity of multi-threading. We found that external agent like ZooKeeper is better choice for service discovery and coordination while ZeroMQ can be used primary for simple messaging. ZeroMQ perfectly fit into our architecture. It allowed us to achieve the desired latency using minimal efforts. It saved us from a bottleneck in the processing of messages and made processing time very stable and predictable.

"I can decidedly recommend ZeroMQ for solutions where low latency is important."

+++ How the Guide Happened

When I set out to write the Guide, we were still debating the pros and cons of forks and pull requests in the 0MQ community. Today, for what it's worth, this argument seems settled: the "liberal" policy which we adopted for libzmq in early 2012 broke our dependency on a single prime author, and opened the floor to dozens of new contributors. More profoundly, it allowed us to move to a gently organic evolutionary model, very different from the older forced-march model.

The reason I was confident this would work was that our work on the Guide had, for a year or more, shown the way. True, the text is my own work, which is perhaps as it should be. Writing is not programming. When we write, we tell a story and one doesn't want different voices telling one tale, it feels strange.

For me the real long-term value of the Guide is the repository of examples: about 65,000 lines of code in 24 different languages. It's partly about making 0MQ accessible to more people. People already refer to the Python and PHP example repositories -- two of the most complete -- when they want to tell others how to learn 0MQ. But it's also about learning programming languages.

Here's a loop of code in Tcl:

[[code language="Tcl"]]
while {1} {
    # Process all parts of the message
    zmq message message
    frontend recv_msg message
    set more [frontend getsockopt RCVMORE]
    backend send_msg message [expr {$more?"SNDMORE":""}]
    message close
    if {!$more} {
        break ; # Last message part
    }
}
[[/code]]

And the same loop in Lua:

[[code language="Lua"]]
while true do
    --  Process all parts of the message
    local msg = frontend:recv()
    if (frontend:getopt(zmq.RCVMORE) == 1) then
        backend:send(msg, zmq.SNDMORE)
    else
        backend:send(msg, 0)
        break;      --  Last message part
    end
end
[[/code]]

And this particular example (rrbroker) is in C#, C++, CL, Clojure, Erlang, F#, Go, Haskell, Haxe, Java, Lua, Node.js, Perl, PHP, Python, Ruby, Scala, Tcl, and of course C. This code base, all licensed as open source under the MIT/X11, may form the basis for other books, or projects.

But what this collection of translations says most profoundly is this: the language you choose is a detail, even a distraction. The power of 0MQ lies in the patterns it gives you and lets you build, and these transcend the comings and goings of languages. My goal as a software and social architect is to build structures that can last generations. There seems no point in aiming for mere decades.

+++ Removing Friction

I'll explain the technical tool chain we used in terms of the friction we removed. With the Guide we're telling a story and the goal is to reach as many people as possible, as cheaply and smoothly as we can.

The core idea was to host the Guide on github and make it easy for anyone to contribute. It turned out to be more complex than that, however.

Let's start with the division of labor. I'm a good writer and can produce endless amounts of decent text quickly. But what was impossible for me was to provide the examples in other languages. Since the core 0MQ API is in C, it seemed logical to write the original examples in C. Also, C is a neutral choice; it's perhaps the only language that doesn't create strong emotions.

How to encourage people to make translations of the examples? We tried a few approaches and finally what worked best was to offer a "choose your language" link on every single example, in the text, which took people either to the translation, or to a page explaining how they could contribute. The way it usually works is that as people learn 0MQ in their preferred language, they contribute a handful of translations, or fixes to the existing ones.

At the same time I noticed a few people quite determinedly translating //every single// example. This was mainly binding authors who realized that the examples were a great way to encourage people to use their bindings. For their efforts, I extended the scripts to produce language-specific versions of the Guide. Instead of including the C code, we'd include the Python, or PHP code. Lua and Haxe also got their dedicated Guides.

Once we have an idea of who works on what, we know how to structure the work itself. It's clear that to write and test an example, what you want to work on is //source code//. So we import this source code when we build the Guide, and that's how we make language-specific versions.

I like to write in a plain text format. It's fast and works well with source control systems like git. Since the main platform for our websites is Wikidot, I write using Wikidot's very readable markup format.

At least in the first chapters, it was important to draw pictures to explain the flow of messages between peers. I found Ditaa, a lovely tool that chews up line-drawings and spits out elegant graphics. Having the graphics in the text, as text, makes it remarkably easy to work.

By now you'll realize that the toolchain is highly customized, though it uses a lot of external tools. All are available on Ubuntu, which is a small mercy, and the whole toolchain is in the Guide repository in the bin subdirectory.

Let's walk through the editing and publishing process. Here is how we produce the online version:

[[code]]
bin/mkguide
[[/code]]

Which works as follows:

* The original text sits in a series of text files (one per chapter).
* The examples sit in the examples subdirectory, classified per language.
* We take the text, and processes this into a set of Wikidot-ready files. It does this for each of the languages that get their own Guide version.
* We extract the graphics and calls Ditaa on each one to produce image files, which it stores in the images subdirectory.
* We extract inline listings (which are not translated) and stores these in the listings subdirectory.
* We use pygmentize on each example and listing to create a marked-up page in Wikidot format.
* We upload all changed files to the Guide website on Wikidot.

Doing this from scratch takes a while. So we store the SHA1 signatures of every image, listing, example, and text file, and only process and upload changes, and that makes it easy to publish a new version of the Guide when people make new contributions.

To produce the PDF and Epub formats we do this:

[[code]]
bin/mkpdfs
[[/code]]

Which works as follows:

* We use the mkbook script on all the input files to produce a DocBook output.
* We push the DocBook format through docbook2ps and ps2pdf to create clean PDFs, in each language.
* We push the DocBook format though db2epub to create Epub books, and in each language.

It's important, when you create a community project, to lower the "change latency", i.e. the time it takes for people to see their work live. Or at least, to see that you've accepted their pull request. If that is more than a day or two, you've often lost your contributor.

+++ Licensing

I want people to reuse the Guide in their own work: in presentations, articles, and even other books. However, the deal is that if they remix my work, others can remix theirs. I'd like credit, and have no argument against others making money from their remixes. Thus, the Guide is licensed under cc-by-sa.

For the examples, we started with GPL, but it rapidly became clear this wasn't workable. The point of examples is to give people reusable code fragments so they will use 0MQ more widely, and if these are GPL that won't happen. We switched to MIT/X11, even for the larger and more complex examples that conceivably would work as LGPL.

However when we started turning the examples into stand-alone projects (as with Majordomo), we used the LGPL. Again, remixability trumps dissemination. Licenses are tools, use them with intent, not ideology.

+++ Contributors

Thanks to Bill Desmarais, Brian Dorsey, Daniel Lin, Eric Desgranges, Gonzalo Diethelm, Guido Goldstein, Hunter Ford, Kamil Shakirov, Martin Sustrik, Mike Castleman, Naveen Chawla, Nicola Peduzzi, Oliver Smith, Olivier Chamoux, Peter Alexander, Pierre Rouleau, Randy Dryburgh, John Unwin, Alex Thomas, Mihail Minkov, Jeremy Avnet, Michael Compton, Kamil Kisiel, Mark Kharitonov, Guillaume Aubert, Ian Barber, Mike Sheridan, Faruk Akgul, Oleg Sidorov, Lev Givon, Allister MacLeod, Alexander D'Archangel, Andreas Hoelzlwimmer, Han Holl, Robert G. Jakabosky, Felipe Cruz, Marcus McCurdy, Mikhail Kulemin, Dr. Gergő Érdi, Pavel Zhukov, Alexander Else, Giovanni Ruggiero, Rick "Technoweenie", Daniel Lundin, Dave Hoover, Simon Jefford, Benjamin Peterson, Justin Case, Devon Weller, Richard Smith, Alexander Morland, Wadim Grasza, Michael Jakl, Uwe Dauernheim, Sebastian Nowicki, Simone Deponti, Aaron Raddon, Dan Colish, Markus Schirp, Benoit Larroque, Jonathan Palardy, Isaiah Peng, Arkadiusz Orzechowski, Umut Aydin, Matthew Horsfall, Jeremy W. Sherman, Eric Pugh, Tyler Sellon, John E. Vincent, Pavel Mitin, Min RK, Igor Wiedler, Olof Åkesson, Patrick Lucas, Heow Goodman, Senthil Palanisami, John Gallagher, Tomas Roos, Stephen McQuay, Erik Allik, Arnaud Cogoluègnes, Rob Gagnon, Dan Williams, Edward Smith, James Tucker, Kristian Kristensen, Vadim Shalts, Martin Trojer, Tom van Leeuwen, Pandya Hiten, Harm Aarts, Marc Harter, Iskren Ivov Chernev, Jay Han, Sonia Hamilton, Nathan Stocks, Naveen Palli, and Zed Shaw for their contributions.

Thanks to Stathis Sideris for [http://www.ditaa.org Ditaa].