Remove Queues and Replace With Disruptor

[sirinath]

Hi,

Lest remove queues / lists and replace with disruptor where possible.

Also try not to use strings for messaging other than the class is for string messaging.

S

[akaigoro]

Strings are used in tests only, and the classes using strings are not part of the core.
As for Disruptor, for most applications its usefulness is questionable. First, in most cases, nodes in a pipe work at different speed, so fastest node typically would consume an event and then sleep, so the executor's working thread switches to another task, which means access to an ordinary queue in multithreaded context - very time consuming operation. So first disruptor-based executor should be developed. Then, its remain to be seen if disruptor can be used with the same Sink and Source interfaces. If not, then the whole pipeline-core project should be reimplemented, or better say, another project should be created, because most applications can live without complexities of disruptor and disruptor is less convenient (programmer has to point out the size of the ring buffer).

[sirinath]

Disruptor makes sense because of this. Have a look at this: https://code.google.com/p/disruptor/wiki/DisruptorWizard

Problem with this kind of thing is that queueing locking and switching has over head. This is exactly what I am trying to avoid.

See https://code.google.com/p/disruptor/wiki/DisruptorWizard
https://code.google.com/p/disruptor/wiki/PerformanceResults
https://code.google.com/p/disruptor/wiki/PresentationsAndInterviews

[akaigoro]

Premature optimization is the root of all evil (c) D. Knuth.
To be convinced, I have to look at your workflow. The best way to avoid
queueing overhead is not to use queues at all, doing work in a single task.
The second is to make tasks larger, and access to queues less frequent.

thanks,
Alexei

On Wed, Sep 4, 2013 at 2:08 AM, Suminda Dharmasena <[email protected]

wrote:

Disruptor makes sense because of this. Have a look at this:
https://code.google.com/p/disruptor/wiki/DisruptorWizard

Problem with this kind of thing is that queueing locking and switching has
over head. This is exactly what I am trying to avoid.

See https://code.google.com/p/disruptor/wiki/DisruptorWizard
https://code.google.com/p/disruptor/wiki/PerformanceResults
https://code.google.com/p/disruptor/wiki/PresentationsAndInterviews

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-23738631
.

[akaigoro]

An article "When Disruptor is not a good fit":

http://blog.parwy.com/2012/03/when-disruptor-is-not-good-fit.html

I don't want to say that I reject using of Disruptor, but want to point out that Disruptor cannot replace classic design and should be considered as option.

[sirinath]

I am looking to see if you can do about 30m transactions per second on a single machine.So the design should be about speed speed and speed alone.

http://reactor.github.io/reactor/ these guys have made it upto 100m non network transactions.

[akaigoro]

I did not find where they say about 100 millions. LMAX reports of 50
million tps on advanced hardware, and 6-8 millions on stock hardware.

Anyway, I cannot do more than 1 million tps. My goal is not super speed,
but easy-to-use interface. I do it for fun, and in hope for wide
acceptance. Super speed makes interfaces ugly, and is interested to narrow
user community.

thanks,
Alexei

On Wed, Sep 4, 2013 at 11:06 PM, Suminda Dharmasena <
[email protected]> wrote:

I am looking to see if you can do about 30m transactions per second on a
single machine.So the design should be about speed speed and speed alone.

http://reactor.github.io/reactor/ these guys have made it upto 100m non
network transactions.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-23801750
.

[sirinath]

https://groups.google.com/forum/#!searchin/reactor-framework/benchmark/reactor-framework/e3q9IqOKHdk/Et9zuw896JEJ

[akaigoro]

They say about events/sec, not transactions/sec. Transaction execution
includes several event transfers, some task switches, and business
calculations. So 50 M tps is really a great result, and 30 M is significant
achievement, unreachable without tedious painstaking work.

thanks,
Alexei

On Thu, Sep 5, 2013 at 12:44 AM, Suminda Dharmasena <
[email protected]> wrote:

https://groups.google.com/forum/#!searchin/reactor-framework/benchmark/reactor-framework/e3q9IqOKHdk/Et9zuw896JEJ

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-23809204
.