does gpars use persistent collections?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

does gpars use persistent collections?

Jochen Theodorou
Hi,

first of all, I really wish I could spend some real time on gpars, but
groovy-core is a never ending source of work ;)

My newest "task" is a new MOP and in that I intend to use persistent
collections where possible. All the mutable data in those global
structures did cost so many problems in the past. Anyway. since I don't
want to reinvent the wheel and since I thought gpars is the most likely
place in the groovy world for something like this to be used these days
I wanted to ask if gpars does use persistent collections and if yes,
what library and why.

The only one I came up with so far, that really seems to meet my needs
is pcollections, which had not much work on it for a while, but still
looks promising.

I need to use those things from Java, so I am not sure the Clojure stuff
would be a good choice for me for example. Tim (which I asked directly
prior to this mail) suggested also to look at functional Java.

So any ideas?

bye blackdrag

--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: does gpars use persistent collections?

Russel Winder-3
On Mon, 2013-01-14 at 16:16 +0100, Jochen Theodorou wrote:
> Hi,
>
> first of all, I really wish I could spend some real time on gpars, but
> groovy-core is a never ending source of work ;)

I know the feeling, I have some stuff I want to get into GPars 1.1 and I
haven't even started yet.

> My newest "task" is a new MOP and in that I intend to use persistent
> collections where possible. All the mutable data in those global
> structures did cost so many problems in the past. Anyway. since I don't
> want to reinvent the wheel and since I thought gpars is the most likely
> place in the groovy world for something like this to be used these days
> I wanted to ask if gpars does use persistent collections and if yes,
> what library and why.

I am not sure what you would want persistent collections for a runtime
system. Could you outline the "use case"?

GPars doesn't have persistent collections, it is a runtime only toolkit.

> The only one I came up with so far, that really seems to meet my needs
> is pcollections, which had not much work on it for a while, but still
> looks promising.

PCollections will have to be checked for consistency with the new Java 8
Collections stuff, but should be a small lightweight thing.  I assume
the persistence is by serializing out to disc, which is horrendously
slow compared to in-memory data structured.

> I need to use those things from Java, so I am not sure the Clojure stuff
> would be a good choice for me for example. Tim (which I asked directly
> prior to this mail) suggested also to look at functional Java.

I suspect the Clojure stuff has some impedance mismatch with the Java
Collections, but I haven't had time to do much with it.

Functional Java has it's own data structures including mutable ones, but
I am not aware it has persistent ones.

I'd be worrying about more dependencies, isn't it just easier to use
things like ConcurrentHashMap to store stuff?

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Floor versions [Re: [gpars-user] does gpars use persistent collections?]

Russel Winder-3
In reply to this post by Jochen Theodorou
Jochen,

Will Groovy 3 require Java 7? I do hope so.

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Floor versions [Re: [gpars-user] does gpars use persistent collections?]

Guillaume Laforge-2
It's likely going to be the case, although we've not made the final decision.
Groovy 3 will definitely build on top of invoke dynamic, thus requiring Java 7.
But we might see how we can still support JDK < 7 for users who have not switched yet -- with a compatibility layer or module, an agent, or something like that.
So it's not a black or white answer :-)


On Mon, Jan 14, 2013 at 4:42 PM, Russel Winder <[hidden email]> wrote:
Jochen,

Will Groovy 3 require Java 7? I do hope so.

--
Russel.
=============================================================================
Dr Russel Winder      t: <a href="tel:%2B44%2020%207585%202200" value="+442075852200">+44 20 7585 2200   voip: [hidden email]
41 Buckmaster Road    m: <a href="tel:%2B44%207770%20465%20077" value="+447770465077">+44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



--
Guillaume Laforge
Groovy Project Manager
SpringSource, a division of VMware

Reply | Threaded
Open this post in threaded view
|

Re: does gpars use persistent collections?

Jochen Theodorou
In reply to this post by Russel Winder-3
Am 14.01.2013 16:41, schrieb Russel Winder:
> On Mon, 2013-01-14 at 16:16 +0100, Jochen Theodorou wrote:
[...]

>> My newest "task" is a new MOP and in that I intend to use persistent
>> collections where possible. All the mutable data in those global
>> structures did cost so many problems in the past. Anyway. since I don't
>> want to reinvent the wheel and since I thought gpars is the most likely
>> place in the groovy world for something like this to be used these days
>> I wanted to ask if gpars does use persistent collections and if yes,
>> what library and why.
>
> I am not sure what you would want persistent collections for a runtime
> system. Could you outline the "use case"?
>
> GPars doesn't have persistent collections, it is a runtime only toolkit.

I guess there is a misunderstanding. I did mean
http://en.wikipedia.org/wiki/Persistent_data_structure The term
"persistent collections" is nowadays mainly used for immutable and
thread safe data structures, that consists themselves only of these as
well. I can for example not simple mutate a list to add an element, I
can though make a new list consisting of the old list and the new
element (cons anyone?). The new MOP is supposed to get along largely
without any synchronization mechanisms at the expense of not having the
current meta class visible everywhere at the same time. If I want to
avoid synchronization but have something in theory visible to another
thread, then this something better consists only as something according
tho Groovy's @Immutable, meaning immutables only, final fields only and
so on. And that is why persistent collections are of use to me in my task.

>> The only one I came up with so far, that really seems to meet my needs
>> is pcollections, which had not much work on it for a while, but still
>> looks promising.
>
> PCollections will have to be checked for consistency with the new Java 8
> Collections stuff, but should be a small lightweight thing.

good thinking... I completely forgot about that.

>> I need to use those things from Java, so I am not sure the Clojure stuff
>> would be a good choice for me for example. Tim (which I asked directly
>> prior to this mail) suggested also to look at functional Java.
>
> I suspect the Clojure stuff has some impedance mismatch with the Java
> Collections, but I haven't had time to do much with it.

In the meantime I checked a bit more and the classes used by Clojure are
quite wired to the runtime. This is mostly interfaces as far as I can
see, but still I would have to pull in a lot of classes and the license
is not 100% fitting as well.

> Functional Java has it's own data structures including mutable ones, but
> I am not aware it has persistent ones.

I have to check... functional languages should have them a lot

> I'd be worrying about more dependencies, isn't it just easier to use
> things like ConcurrentHashMap to store stuff?

ConcurrentHashMap is not a bad one, yes, and even if it uses only a
volatile to check for modifications, that is still bad for your
many-CPU-environment.

bye blackdrag
--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: does gpars use persistent collections?

Russel Winder-3
On Mon, 2013-01-14 at 17:13 +0100, Jochen Theodorou wrote:
[…]

> I guess there is a misunderstanding. I did mean
> http://en.wikipedia.org/wiki/Persistent_data_structure The term
> "persistent collections" is nowadays mainly used for immutable and
> thread safe data structures, that consists themselves only of these as
> well. I can for example not simple mutate a list to add an element, I
> can though make a new list consisting of the old list and the new
> element (cons anyone?). The new MOP is supposed to get along largely
> without any synchronization mechanisms at the expense of not having the
> current meta class visible everywhere at the same time. If I want to
> avoid synchronization but have something in theory visible to another
> thread, then this something better consists only as something according
> tho Groovy's @Immutable, meaning immutables only, final fields only and
> so on. And that is why persistent collections are of use to me in my task.
For me persistent means stored on the filestore in some way. The
confusion is entirely my fault. We are now though on the same
wavelength. Sorry about that. I should have read the PCollections
example :-)

It is being reported that with the G1 garbage collector object creation
and deletion is not the problem it used to be with the CMS or other
collectors. However I think the benchmarks are for small objects, there
is an assumption that big data structures will be handled by agents.

> > PCollections will have to be checked for consistency with the new Java 8
> > Collections stuff, but should be a small lightweight thing.
>
> good thinking... I completely forgot about that.

I think there will not have to be any actual changes to PCollections
since all the lambda expression stuff will be handled in the interfaces
via defender methods.

> In the meantime I checked a bit more and the classes used by Clojure are
> quite wired to the runtime. This is mostly interfaces as far as I can
> see, but still I would have to pull in a lot of classes and the license
> is not 100% fitting as well.

Certainly the cons lists will be suited only for recursive algorithms. I
suspect the array-like classes are not sufficuiently useful to pull in
the whole Clojure system — unless you are using Clojure.

> > Functional Java has it's own data structures including mutable ones, but
> > I am not aware it has persistent ones.
>
> I have to check... functional languages should have them a lot

My fault, wrong use of term persistent. Functional Java does indeed have
the sort of data structures you are looking for, but I am not sure You
get enough "bang for your buck" unless you are going to write the whole
Groovy system in a very functional Java style.

They also use the wrong lambda function syntax compared to Java 8.

> > I'd be worrying about more dependencies, isn't it just easier to use
> > things like ConcurrentHashMap to store stuff?
>
> ConcurrentHashMap is not a bad one, yes, and even if it uses only a
> volatile to check for modifications, that is still bad for your
> many-CPU-environment.

Sorry this last comment confuses me. ConcurrentHashMap is about as
parallel a thread-safe map that you can get. If there is an example of
failure then a test case needs shipping to Doug Lea.

--
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:[hidden email]
41 Buckmaster Road    m: +44 7770 465 077   xmpp: [hidden email]
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: does gpars use persistent collections?

Jochen Theodorou
Am 14.01.2013 19:38, schrieb Russel Winder:
> On Mon, 2013-01-14 at 17:13 +0100, Jochen Theodorou wrote:
> […]
>> ConcurrentHashMap is not a bad one, yes, and even if it uses only a
>> volatile to check for modifications, that is still bad for your
>> many-CPU-environment.
>
> Sorry this last comment confuses me. ConcurrentHashMap is about as
> parallel a thread-safe map that you can get. If there is an example of
> failure then a test case needs shipping to Doug Lea.

there are two "models" on working on data structures in parallel.

In model T1 a change made in Thread A must become visible in Thread B as
soon as possible. To get this job done Java has synchronized, locks,
volatile and all that. ConcurrentHashMap is of that model. Meaning, if
you request a key in one Thread and another Thread happens to have just
added the key, you will get the assigned value, while if the request is
handled before the key is assigned it will not work (return null) of course

In model T2 a change made in Thread A must not become visible in Thread
B as soon as possible. Instead you are ok to wait for a memory
synchronization that happens anyway. In Java there is no direct
construct (afaik) for this, though the happens-before rules of the java
memory model allow you to piggyback on another synchronization. That
means you use for example no volatile fields, do not synchronize or lock
yourself, instead you let other code do that, and "simply" make your
code being able to accept this discontinuity. Groovy 1.8 is partially
using this already and I intend to extend that in Groovy 3.

Let us say Thread A adds a method to Integer.metaClass, then Thread A
will see that addition right away of course, but Thread B may or may not
see it.

Anyway, the structures you share in both models need to be threadsafe of
course, but what you share in T2 be better also immutable or else you
may face the problem of seeing a partially initialized instance (java
memory model hitting here again). Let us for example assume you have a
class C and two non-final fields F1 and F2 and each stores an immutable.
Now you mutate the C instance by setting new values for F1 and F2.
Another thread seeing that instance can now see the new F1, the new F2
or both new, in any order. If F1 and F2 are final, then the thread will
either see the new F1 and F2 or in both the old, depending on if the
Thread will see the new instance or the old.

bye blackdrag

--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: does gpars use persistent collections?

Alex Tkachman
In reply to this post by Jochen Theodorou
Jochen,

if I correctly imagine what you need persistent collections (in fact I believe you need only map and never vector) for in Groovy Core (meta class tables etc.) for performance reasons I would recommend to implement it manually for your own need (multi key case etc.) it is not complex and I did it few times.


On Mon, Jan 14, 2013 at 5:16 PM, Jochen Theodorou <[hidden email]> wrote:
Hi,

first of all, I really wish I could spend some real time on gpars, but groovy-core is a never ending source of work ;)

My newest "task" is a new MOP and in that I intend to use persistent collections where possible. All the mutable data in those global structures did cost so many problems in the past. Anyway. since I don't want to reinvent the wheel and since I thought gpars is the most likely place in the groovy world for something like this to be used these days I wanted to ask if gpars does use persistent collections and if yes, what library and why.

The only one I came up with so far, that really seems to meet my needs is pcollections, which had not much work on it for a while, but still looks promising.

I need to use those things from Java, so I am not sure the Clojure stuff would be a good choice for me for example. Tim (which I asked directly prior to this mail) suggested also to look at functional Java.

So any ideas?

bye blackdrag

--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





--
Best regards,
Alex

Best way to call / chat with me
http://lucy.me/alex
Reply | Threaded
Open this post in threaded view
|

Re: does gpars use persistent collections?

Jochen Theodorou
Am 21.01.2013 08:05, schrieb Alex Tkachman:
> Jochen,
>
> if I correctly imagine what you need persistent collections (in fact I
> believe you need only map and never vector) for in Groovy Core (meta
> class tables etc.) for performance reasons I would recommend to
> implement it manually for your own need (multi key case etc.) it is not
> complex and I did it few times.

I decided to make an HAMT based version for my needs

bye blackdrag


--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email