Persistent Dataflow

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Persistent Dataflow

Ming Fang
Is anybody working on extending Dataflow to make channels persistent?
I'm looking for something that functions like Kestrel but using  Gpars stead.

Sent from my iPad

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Persistent Dataflow

Russel Winder-3
On Wed, 2012-11-28 at 15:35 -0500, Ming Fang wrote:
> Is anybody working on extending Dataflow to make channels persistent?
> I'm looking for something that functions like Kestrel but using  Gpars stead.

I am wondering what you mean by persistent here. Are you thinking of
realizing channels with operating system sockets rather than in memory
data structures?

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Persistent Dataflow

Ming Fang
Not sockets, but files.

Sent from my iPad

On Nov 29, 2012, at 2:42 AM, Russel Winder <[hidden email]> wrote:

> On Wed, 2012-11-28 at 15:35 -0500, Ming Fang wrote:
>> Is anybody working on extending Dataflow to make channels persistent?
>> I'm looking for something that functions like Kestrel but using  Gpars stead.
>
> I am wondering what you mean by persistent here. Are you thinking of
> realizing channels with operating system sockets rather than in memory
> data structures?
>
> --
> Russel.
> =============================================================================
> Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Persistent Dataflow

Russel Winder-3
On Thu, 2012-11-29 at 07:17 -0500, Ming Fang wrote:
> Not sockets, but files.

I am not aware of any such implementation of channels in GPars, though I
can envisage an implementation on Linux using inotify. What's the use
case that motivates such an approach, it would be quite slow?

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Persistent Dataflow

Ming Fang
We use persistence for several reasons.
Main ones are 1- the incoming message rate can burst and overflow the available memory and 2- the saved journal files can be use for replay for troubleshooting and regression tests.

In terms of speed, it's slower than memory but it's fast enough for most applications.
The journal implementation we use is Java-Chronicles and it can handle millions of messages a second.
https://github.com/peter-lawrey/Java-Chronicle

Here are two other projects that leverages files for queueing
http://incubator.apache.org/kafka/performance.html
https://github.com/robey/kestrel

I'm looking for a way to integrate GPars with one of the projects above.

On Nov 29, 2012, at 7:46 AM, Russel Winder <[hidden email]> wrote:

> On Thu, 2012-11-29 at 07:17 -0500, Ming Fang wrote:
>> Not sockets, but files.
>
> I am not aware of any such implementation of channels in GPars, though I
> can envisage an implementation on Linux using inotify. What's the use
> case that motivates such an approach, it would be quite slow?
>
> --
> Russel.
> =============================================================================
> Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
> 41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email