Using a counter inside a Parallelized method

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Using a counter inside a Parallelized method

Martin Jones-2
Hi,

I am new to concurrent programming, so I want to make sure I do this
the right way.  I want to use eachParallel(), but also have a counter
that keeps track of the total number of elements processed.  Since
this counter will be accessed by multiple threads, is the best way to
do this with an AtomicInteger? Something like

def counter = new AtomicInteger()
Parallelizer.doParallel {
     myLargeArray.eachParallel {
          counter.incrementAndGet()
          doProcessing(it)
     }
}

Assuming that my doProcessing() method does not have any side effects.

Thanks,
Martin

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Using a counter inside a Parallelized method

Dierk König
Hi Martin,

welcome to GPars!

Yes, you are doing it right.

However, one rule of thumb: using references that are not
local to you closure (anything but "it" in this case) is often a sign
that some other concurrency concept may be more appropriate
or that you need an additional one (DataFlow, Safe,  or maybe even Actor).

cheers
Dierk


Am 14.04.2010 um 10:45 schrieb Martin Jones:

> Hi,
>
> I am new to concurrent programming, so I want to make sure I do this
> the right way.  I want to use eachParallel(), but also have a counter
> that keeps track of the total number of elements processed.  Since
> this counter will be accessed by multiple threads, is the best way to
> do this with an AtomicInteger? Something like
>
> def counter = new AtomicInteger()
> Parallelizer.doParallel {
>     myLargeArray.eachParallel {
>          counter.incrementAndGet()
>          doProcessing(it)
>     }
> }
>
> Assuming that my doProcessing() method does not have any side effects.
>
> Thanks,
> Martin
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Using a counter inside a Parallelized method

Martin Jones-6
Hi Dierk,

Thanks for the reply.  I am actually using Actors to do the job I
mentioned below - I have modified the Load Balancer example from the
GPars demo.  But then I was browsing the Parallelizer documentation
and it occurred to me that I could do the same job with much less code
using eachParallel (or collectParallel). It seems to behave like a
simple load balancer. I will read up on DataFlow and Safe.

I am very excited to have found the GPars project - I am working with
problems in bioinformatics, in which there is a lot of interest in
parallel processing and GPars is fantastic for trying out concurrency
ideas.  I am going to show my code to my colleagues to try to convince
them to give GPars a go.

Martin



On 14 April 2010 11:46, Dierk König <[hidden email]> wrote:

> Hi Martin,
>
> welcome to GPars!
>
> Yes, you are doing it right.
>
> However, one rule of thumb: using references that are not
> local to you closure (anything but "it" in this case) is often a sign
> that some other concurrency concept may be more appropriate
> or that you need an additional one (DataFlow, Safe,  or maybe even Actor).
>
> cheers
> Dierk
>
>
> Am 14.04.2010 um 10:45 schrieb Martin Jones:
>
>> Hi,
>>
>> I am new to concurrent programming, so I want to make sure I do this
>> the right way.  I want to use eachParallel(), but also have a counter
>> that keeps track of the total number of elements processed.  Since
>> this counter will be accessed by multiple threads, is the best way to
>> do this with an AtomicInteger? Something like
>>
>> def counter = new AtomicInteger()
>> Parallelizer.doParallel {
>>     myLargeArray.eachParallel {
>>          counter.incrementAndGet()
>>          doProcessing(it)
>>     }
>> }
>>
>> Assuming that my doProcessing() method does not have any side effects.
>>
>> Thanks,
>> Martin
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Using a counter inside a Parallelized method

Dierk König
Cool!

If you have an updated demo that you would like to share, then this
is more than welcome!
Demos are particularly instructive when they show how to achieve
the same goal with different means.

The best way to share is using a github pull request but posting the
code here would also work for a short self-contained demo.

thanks for participating
Dierk

Am 14.04.2010 um 12:55 schrieb Martin Jones:

> Hi Dierk,
>
> Thanks for the reply.  I am actually using Actors to do the job I
> mentioned below - I have modified the Load Balancer example from the
> GPars demo.  But then I was browsing the Parallelizer documentation
> and it occurred to me that I could do the same job with much less code
> using eachParallel (or collectParallel). It seems to behave like a
> simple load balancer. I will read up on DataFlow and Safe.
>
> I am very excited to have found the GPars project - I am working with
> problems in bioinformatics, in which there is a lot of interest in
> parallel processing and GPars is fantastic for trying out concurrency
> ideas.  I am going to show my code to my colleagues to try to convince
> them to give GPars a go.
>
> Martin
>
>
>
> On 14 April 2010 11:46, Dierk König <[hidden email]> wrote:
>> Hi Martin,
>>
>> welcome to GPars!
>>
>> Yes, you are doing it right.
>>
>> However, one rule of thumb: using references that are not
>> local to you closure (anything but "it" in this case) is often a sign
>> that some other concurrency concept may be more appropriate
>> or that you need an additional one (DataFlow, Safe,  or maybe even Actor).
>>
>> cheers
>> Dierk
>>
>>
>> Am 14.04.2010 um 10:45 schrieb Martin Jones:
>>
>>> Hi,
>>>
>>> I am new to concurrent programming, so I want to make sure I do this
>>> the right way.  I want to use eachParallel(), but also have a counter
>>> that keeps track of the total number of elements processed.  Since
>>> this counter will be accessed by multiple threads, is the best way to
>>> do this with an AtomicInteger? Something like
>>>
>>> def counter = new AtomicInteger()
>>> Parallelizer.doParallel {
>>>     myLargeArray.eachParallel {
>>>          counter.incrementAndGet()
>>>          doProcessing(it)
>>>     }
>>> }
>>>
>>> Assuming that my doProcessing() method does not have any side effects.
>>>
>>> Thanks,
>>> Martin
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this list, please visit:
>>>
>>>    http://xircles.codehaus.org/manage_email
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Using a counter inside a Parallelized method

Martin Jones-6
Hi,

I'd love to contribute something to the GPars project if it would be
useful.  At the moment my load balancer code doesn't really add
anything to the existing demo, but I will definitely make a post if I
come up with anything more creative!

I have a further question about Parallelizer, but I will put it in a
separate thread to make it easier for future searching.

Martin



On 14 April 2010 12:19, Dierk König <[hidden email]> wrote:

> Cool!
>
> If you have an updated demo that you would like to share, then this
> is more than welcome!
> Demos are particularly instructive when they show how to achieve
> the same goal with different means.
>
> The best way to share is using a github pull request but posting the
> code here would also work for a short self-contained demo.
>
> thanks for participating
> Dierk
>
> Am 14.04.2010 um 12:55 schrieb Martin Jones:
>
>> Hi Dierk,
>>
>> Thanks for the reply.  I am actually using Actors to do the job I
>> mentioned below - I have modified the Load Balancer example from the
>> GPars demo.  But then I was browsing the Parallelizer documentation
>> and it occurred to me that I could do the same job with much less code
>> using eachParallel (or collectParallel). It seems to behave like a
>> simple load balancer. I will read up on DataFlow and Safe.
>>
>> I am very excited to have found the GPars project - I am working with
>> problems in bioinformatics, in which there is a lot of interest in
>> parallel processing and GPars is fantastic for trying out concurrency
>> ideas.  I am going to show my code to my colleagues to try to convince
>> them to give GPars a go.
>>
>> Martin
>>
>>
>>
>> On 14 April 2010 11:46, Dierk König <[hidden email]> wrote:
>>> Hi Martin,
>>>
>>> welcome to GPars!
>>>
>>> Yes, you are doing it right.
>>>
>>> However, one rule of thumb: using references that are not
>>> local to you closure (anything but "it" in this case) is often a sign
>>> that some other concurrency concept may be more appropriate
>>> or that you need an additional one (DataFlow, Safe,  or maybe even Actor).
>>>
>>> cheers
>>> Dierk
>>>
>>>
>>> Am 14.04.2010 um 10:45 schrieb Martin Jones:
>>>
>>>> Hi,
>>>>
>>>> I am new to concurrent programming, so I want to make sure I do this
>>>> the right way.  I want to use eachParallel(), but also have a counter
>>>> that keeps track of the total number of elements processed.  Since
>>>> this counter will be accessed by multiple threads, is the best way to
>>>> do this with an AtomicInteger? Something like
>>>>
>>>> def counter = new AtomicInteger()
>>>> Parallelizer.doParallel {
>>>>     myLargeArray.eachParallel {
>>>>          counter.incrementAndGet()
>>>>          doProcessing(it)
>>>>     }
>>>> }
>>>>
>>>> Assuming that my doProcessing() method does not have any side effects.
>>>>
>>>> Thanks,
>>>> Martin
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe from this list, please visit:
>>>>
>>>>    http://xircles.codehaus.org/manage_email
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this list, please visit:
>>>
>>>    http://xircles.codehaus.org/manage_email
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Using a counter inside a Parallelized method

Vaclav
Administrator
As a first-shot tip for a shared counter I would suggest using an agent aka Safe, which nicely wraps shared mutable state into an isolated entity accepting update commands from the outside world.
Direct use of an AtomicInteger, however, will certainly perform slightly better even after our recent performance improvements for 0.10. As for the Safe concept, I recommend you checked-out the updated snapshot guide at http://gpars.org/SNAPSHOT/guide/index.html

Vaclav


On Wed, Apr 14, 2010 at 1:38 PM, Martin Jones <[hidden email]> wrote:
Hi,

I'd love to contribute something to the GPars project if it would be
useful.  At the moment my load balancer code doesn't really add
anything to the existing demo, but I will definitely make a post if I
come up with anything more creative!

I have a further question about Parallelizer, but I will put it in a
separate thread to make it easier for future searching.

Martin



On 14 April 2010 12:19, Dierk König <[hidden email]> wrote:
> Cool!
>
> If you have an updated demo that you would like to share, then this
> is more than welcome!
> Demos are particularly instructive when they show how to achieve
> the same goal with different means.
>
> The best way to share is using a github pull request but posting the
> code here would also work for a short self-contained demo.
>
> thanks for participating
> Dierk
>
> Am 14.04.2010 um 12:55 schrieb Martin Jones:
>
>> Hi Dierk,
>>
>> Thanks for the reply.  I am actually using Actors to do the job I
>> mentioned below - I have modified the Load Balancer example from the
>> GPars demo.  But then I was browsing the Parallelizer documentation
>> and it occurred to me that I could do the same job with much less code
>> using eachParallel (or collectParallel). It seems to behave like a
>> simple load balancer. I will read up on DataFlow and Safe.
>>
>> I am very excited to have found the GPars project - I am working with
>> problems in bioinformatics, in which there is a lot of interest in
>> parallel processing and GPars is fantastic for trying out concurrency
>> ideas.  I am going to show my code to my colleagues to try to convince
>> them to give GPars a go.
>>
>> Martin
>>
>>
>>
>> On 14 April 2010 11:46, Dierk König <[hidden email]> wrote:
>>> Hi Martin,
>>>
>>> welcome to GPars!
>>>
>>> Yes, you are doing it right.
>>>
>>> However, one rule of thumb: using references that are not
>>> local to you closure (anything but "it" in this case) is often a sign
>>> that some other concurrency concept may be more appropriate
>>> or that you need an additional one (DataFlow, Safe,  or maybe even Actor).
>>>
>>> cheers
>>> Dierk
>>>
>>>
>>> Am 14.04.2010 um 10:45 schrieb Martin Jones:
>>>
>>>> Hi,
>>>>
>>>> I am new to concurrent programming, so I want to make sure I do this
>>>> the right way.  I want to use eachParallel(), but also have a counter
>>>> that keeps track of the total number of elements processed.  Since
>>>> this counter will be accessed by multiple threads, is the best way to
>>>> do this with an AtomicInteger? Something like
>>>>
>>>> def counter = new AtomicInteger()
>>>> Parallelizer.doParallel {
>>>>     myLargeArray.eachParallel {
>>>>          counter.incrementAndGet()
>>>>          doProcessing(it)
>>>>     }
>>>> }
>>>>
>>>> Assuming that my doProcessing() method does not have any side effects.
>>>>
>>>> Thanks,
>>>> Martin
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe from this list, please visit:
>>>>
>>>>    http://xircles.codehaus.org/manage_email
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this list, please visit:
>>>
>>>    http://xircles.codehaus.org/manage_email
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





--
E-mail: [hidden email]
Blog: http://www.jroller.com/vaclav
Linkedin page: http://www.linkedin.com/in/vaclavpech
Reply | Threaded
Open this post in threaded view
|

Re: Using a counter inside a Parallelized method

Martin Jones-6
Thanks, I'll check that out.

Martin


On 15 April 2010 10:06, Vaclav Pech <[hidden email]> wrote:

> As a first-shot tip for a shared counter I would suggest using an agent aka
> Safe, which nicely wraps shared mutable state into an isolated entity
> accepting update commands from the outside world.
> Direct use of an AtomicInteger, however, will certainly perform slightly
> better even after our recent performance improvements for 0.10. As for the
> Safe concept, I recommend you checked-out the updated snapshot guide at
> http://gpars.org/SNAPSHOT/guide/index.html
>
> Vaclav
>
>
> On Wed, Apr 14, 2010 at 1:38 PM, Martin Jones <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> I'd love to contribute something to the GPars project if it would be
>> useful.  At the moment my load balancer code doesn't really add
>> anything to the existing demo, but I will definitely make a post if I
>> come up with anything more creative!
>>
>> I have a further question about Parallelizer, but I will put it in a
>> separate thread to make it easier for future searching.
>>
>> Martin
>>
>>
>>
>> On 14 April 2010 12:19, Dierk König <[hidden email]> wrote:
>> > Cool!
>> >
>> > If you have an updated demo that you would like to share, then this
>> > is more than welcome!
>> > Demos are particularly instructive when they show how to achieve
>> > the same goal with different means.
>> >
>> > The best way to share is using a github pull request but posting the
>> > code here would also work for a short self-contained demo.
>> >
>> > thanks for participating
>> > Dierk
>> >
>> > Am 14.04.2010 um 12:55 schrieb Martin Jones:
>> >
>> >> Hi Dierk,
>> >>
>> >> Thanks for the reply.  I am actually using Actors to do the job I
>> >> mentioned below - I have modified the Load Balancer example from the
>> >> GPars demo.  But then I was browsing the Parallelizer documentation
>> >> and it occurred to me that I could do the same job with much less code
>> >> using eachParallel (or collectParallel). It seems to behave like a
>> >> simple load balancer. I will read up on DataFlow and Safe.
>> >>
>> >> I am very excited to have found the GPars project - I am working with
>> >> problems in bioinformatics, in which there is a lot of interest in
>> >> parallel processing and GPars is fantastic for trying out concurrency
>> >> ideas.  I am going to show my code to my colleagues to try to convince
>> >> them to give GPars a go.
>> >>
>> >> Martin
>> >>
>> >>
>> >>
>> >> On 14 April 2010 11:46, Dierk König <[hidden email]> wrote:
>> >>> Hi Martin,
>> >>>
>> >>> welcome to GPars!
>> >>>
>> >>> Yes, you are doing it right.
>> >>>
>> >>> However, one rule of thumb: using references that are not
>> >>> local to you closure (anything but "it" in this case) is often a sign
>> >>> that some other concurrency concept may be more appropriate
>> >>> or that you need an additional one (DataFlow, Safe,  or maybe even
>> >>> Actor).
>> >>>
>> >>> cheers
>> >>> Dierk
>> >>>
>> >>>
>> >>> Am 14.04.2010 um 10:45 schrieb Martin Jones:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I am new to concurrent programming, so I want to make sure I do this
>> >>>> the right way.  I want to use eachParallel(), but also have a counter
>> >>>> that keeps track of the total number of elements processed.  Since
>> >>>> this counter will be accessed by multiple threads, is the best way to
>> >>>> do this with an AtomicInteger? Something like
>> >>>>
>> >>>> def counter = new AtomicInteger()
>> >>>> Parallelizer.doParallel {
>> >>>>     myLargeArray.eachParallel {
>> >>>>          counter.incrementAndGet()
>> >>>>          doProcessing(it)
>> >>>>     }
>> >>>> }
>> >>>>
>> >>>> Assuming that my doProcessing() method does not have any side
>> >>>> effects.
>> >>>>
>> >>>> Thanks,
>> >>>> Martin
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe from this list, please visit:
>> >>>>
>> >>>>    http://xircles.codehaus.org/manage_email
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe from this list, please visit:
>> >>>
>> >>>    http://xircles.codehaus.org/manage_email
>> >>>
>> >>>
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe from this list, please visit:
>> >>
>> >>    http://xircles.codehaus.org/manage_email
>> >>
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe from this list, please visit:
>> >
>> >    http://xircles.codehaus.org/manage_email
>> >
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>
>
> --
> E-mail: [hidden email]
> Blog: http://www.jroller.com/vaclav
> Linkedin page: http://www.linkedin.com/in/vaclavpech
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email