Using a counter inside a Parallelized method
Locked
Using a counter inside a Parallelized method
 
|
Hi,
I am new to concurrent programming, so I want to make sure I do this the right way. I want to use eachParallel(), but also have a counter that keeps track of the total number of elements processed. Since this counter will be accessed by multiple threads, is the best way to do this with an AtomicInteger? Something like def counter = new AtomicInteger() Parallelizer.doParallel { myLargeArray.eachParallel { counter.incrementAndGet() doProcessing(it) } } Assuming that my doProcessing() method does not have any side effects. Thanks, Martin --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
Re: Using a counter inside a Parallelized method
|
|
Hi Martin,
welcome to GPars! Yes, you are doing it right. However, one rule of thumb: using references that are not local to you closure (anything but "it" in this case) is often a sign that some other concurrency concept may be more appropriate or that you need an additional one (DataFlow, Safe, or maybe even Actor). cheers Dierk Am 14.04.2010 um 10:45 schrieb Martin Jones: > Hi, > > I am new to concurrent programming, so I want to make sure I do this > the right way. I want to use eachParallel(), but also have a counter > that keeps track of the total number of elements processed. Since > this counter will be accessed by multiple threads, is the best way to > do this with an AtomicInteger? Something like > > def counter = new AtomicInteger() > Parallelizer.doParallel { > myLargeArray.eachParallel { > counter.incrementAndGet() > doProcessing(it) > } > } > > Assuming that my doProcessing() method does not have any side effects. > > Thanks, > Martin > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
Re: Using a counter inside a Parallelized method
|
|
Hi Dierk,
Thanks for the reply. I am actually using Actors to do the job I mentioned below - I have modified the Load Balancer example from the GPars demo. But then I was browsing the Parallelizer documentation and it occurred to me that I could do the same job with much less code using eachParallel (or collectParallel). It seems to behave like a simple load balancer. I will read up on DataFlow and Safe. I am very excited to have found the GPars project - I am working with problems in bioinformatics, in which there is a lot of interest in parallel processing and GPars is fantastic for trying out concurrency ideas. I am going to show my code to my colleagues to try to convince them to give GPars a go. Martin On 14 April 2010 11:46, Dierk König <[hidden email]> wrote: > Hi Martin, > > welcome to GPars! > > Yes, you are doing it right. > > However, one rule of thumb: using references that are not > local to you closure (anything but "it" in this case) is often a sign > that some other concurrency concept may be more appropriate > or that you need an additional one (DataFlow, Safe, or maybe even Actor). > > cheers > Dierk > > > Am 14.04.2010 um 10:45 schrieb Martin Jones: > >> Hi, >> >> I am new to concurrent programming, so I want to make sure I do this >> the right way. I want to use eachParallel(), but also have a counter >> that keeps track of the total number of elements processed. Since >> this counter will be accessed by multiple threads, is the best way to >> do this with an AtomicInteger? Something like >> >> def counter = new AtomicInteger() >> Parallelizer.doParallel { >> myLargeArray.eachParallel { >> counter.incrementAndGet() >> doProcessing(it) >> } >> } >> >> Assuming that my doProcessing() method does not have any side effects. >> >> Thanks, >> Martin >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
Re: Using a counter inside a Parallelized method
|
|
Cool!
If you have an updated demo that you would like to share, then this is more than welcome! Demos are particularly instructive when they show how to achieve the same goal with different means. The best way to share is using a github pull request but posting the code here would also work for a short self-contained demo. thanks for participating Dierk Am 14.04.2010 um 12:55 schrieb Martin Jones: > Hi Dierk, > > Thanks for the reply. I am actually using Actors to do the job I > mentioned below - I have modified the Load Balancer example from the > GPars demo. But then I was browsing the Parallelizer documentation > and it occurred to me that I could do the same job with much less code > using eachParallel (or collectParallel). It seems to behave like a > simple load balancer. I will read up on DataFlow and Safe. > > I am very excited to have found the GPars project - I am working with > problems in bioinformatics, in which there is a lot of interest in > parallel processing and GPars is fantastic for trying out concurrency > ideas. I am going to show my code to my colleagues to try to convince > them to give GPars a go. > > Martin > > > > On 14 April 2010 11:46, Dierk König <[hidden email]> wrote: >> Hi Martin, >> >> welcome to GPars! >> >> Yes, you are doing it right. >> >> However, one rule of thumb: using references that are not >> local to you closure (anything but "it" in this case) is often a sign >> that some other concurrency concept may be more appropriate >> or that you need an additional one (DataFlow, Safe, or maybe even Actor). >> >> cheers >> Dierk >> >> >> Am 14.04.2010 um 10:45 schrieb Martin Jones: >> >>> Hi, >>> >>> I am new to concurrent programming, so I want to make sure I do this >>> the right way. I want to use eachParallel(), but also have a counter >>> that keeps track of the total number of elements processed. Since >>> this counter will be accessed by multiple threads, is the best way to >>> do this with an AtomicInteger? Something like >>> >>> def counter = new AtomicInteger() >>> Parallelizer.doParallel { >>> myLargeArray.eachParallel { >>> counter.incrementAndGet() >>> doProcessing(it) >>> } >>> } >>> >>> Assuming that my doProcessing() method does not have any side effects. >>> >>> Thanks, >>> Martin >>> >>> --------------------------------------------------------------------- >>> To unsubscribe from this list, please visit: >>> >>> http://xircles.codehaus.org/manage_email >>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> >> > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
Re: Using a counter inside a Parallelized method
|
|
Hi,
I'd love to contribute something to the GPars project if it would be useful. At the moment my load balancer code doesn't really add anything to the existing demo, but I will definitely make a post if I come up with anything more creative! I have a further question about Parallelizer, but I will put it in a separate thread to make it easier for future searching. Martin On 14 April 2010 12:19, Dierk König <[hidden email]> wrote: > Cool! > > If you have an updated demo that you would like to share, then this > is more than welcome! > Demos are particularly instructive when they show how to achieve > the same goal with different means. > > The best way to share is using a github pull request but posting the > code here would also work for a short self-contained demo. > > thanks for participating > Dierk > > Am 14.04.2010 um 12:55 schrieb Martin Jones: > >> Hi Dierk, >> >> Thanks for the reply. I am actually using Actors to do the job I >> mentioned below - I have modified the Load Balancer example from the >> GPars demo. But then I was browsing the Parallelizer documentation >> and it occurred to me that I could do the same job with much less code >> using eachParallel (or collectParallel). It seems to behave like a >> simple load balancer. I will read up on DataFlow and Safe. >> >> I am very excited to have found the GPars project - I am working with >> problems in bioinformatics, in which there is a lot of interest in >> parallel processing and GPars is fantastic for trying out concurrency >> ideas. I am going to show my code to my colleagues to try to convince >> them to give GPars a go. >> >> Martin >> >> >> >> On 14 April 2010 11:46, Dierk König <[hidden email]> wrote: >>> Hi Martin, >>> >>> welcome to GPars! >>> >>> Yes, you are doing it right. >>> >>> However, one rule of thumb: using references that are not >>> local to you closure (anything but "it" in this case) is often a sign >>> that some other concurrency concept may be more appropriate >>> or that you need an additional one (DataFlow, Safe, or maybe even Actor). >>> >>> cheers >>> Dierk >>> >>> >>> Am 14.04.2010 um 10:45 schrieb Martin Jones: >>> >>>> Hi, >>>> >>>> I am new to concurrent programming, so I want to make sure I do this >>>> the right way. I want to use eachParallel(), but also have a counter >>>> that keeps track of the total number of elements processed. Since >>>> this counter will be accessed by multiple threads, is the best way to >>>> do this with an AtomicInteger? Something like >>>> >>>> def counter = new AtomicInteger() >>>> Parallelizer.doParallel { >>>> myLargeArray.eachParallel { >>>> counter.incrementAndGet() >>>> doProcessing(it) >>>> } >>>> } >>>> >>>> Assuming that my doProcessing() method does not have any side effects. >>>> >>>> Thanks, >>>> Martin >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe from this list, please visit: >>>> >>>> http://xircles.codehaus.org/manage_email >>>> >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe from this list, please visit: >>> >>> http://xircles.codehaus.org/manage_email >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
|
Administrator
|
As a first-shot tip for a shared counter I would suggest using an agent aka Safe, which nicely wraps shared mutable state into an isolated entity accepting update commands from the outside world.
Direct use of an AtomicInteger, however, will certainly perform slightly better even after our recent performance improvements for 0.10. As for the Safe concept, I recommend you checked-out the updated snapshot guide at http://gpars.org/SNAPSHOT/guide/index.html Vaclav On Wed, Apr 14, 2010 at 1:38 PM, Martin Jones <[hidden email]> wrote: Hi, -- E-mail: [hidden email] Blog: http://www.jroller.com/vaclav Linkedin page: http://www.linkedin.com/in/vaclavpech |
Re: Using a counter inside a Parallelized method
|
|
Thanks, I'll check that out.
Martin On 15 April 2010 10:06, Vaclav Pech <[hidden email]> wrote: > As a first-shot tip for a shared counter I would suggest using an agent aka > Safe, which nicely wraps shared mutable state into an isolated entity > accepting update commands from the outside world. > Direct use of an AtomicInteger, however, will certainly perform slightly > better even after our recent performance improvements for 0.10. As for the > Safe concept, I recommend you checked-out the updated snapshot guide at > http://gpars.org/SNAPSHOT/guide/index.html > > Vaclav > > > On Wed, Apr 14, 2010 at 1:38 PM, Martin Jones <[hidden email]> > wrote: >> >> Hi, >> >> I'd love to contribute something to the GPars project if it would be >> useful. At the moment my load balancer code doesn't really add >> anything to the existing demo, but I will definitely make a post if I >> come up with anything more creative! >> >> I have a further question about Parallelizer, but I will put it in a >> separate thread to make it easier for future searching. >> >> Martin >> >> >> >> On 14 April 2010 12:19, Dierk König <[hidden email]> wrote: >> > Cool! >> > >> > If you have an updated demo that you would like to share, then this >> > is more than welcome! >> > Demos are particularly instructive when they show how to achieve >> > the same goal with different means. >> > >> > The best way to share is using a github pull request but posting the >> > code here would also work for a short self-contained demo. >> > >> > thanks for participating >> > Dierk >> > >> > Am 14.04.2010 um 12:55 schrieb Martin Jones: >> > >> >> Hi Dierk, >> >> >> >> Thanks for the reply. I am actually using Actors to do the job I >> >> mentioned below - I have modified the Load Balancer example from the >> >> GPars demo. But then I was browsing the Parallelizer documentation >> >> and it occurred to me that I could do the same job with much less code >> >> using eachParallel (or collectParallel). It seems to behave like a >> >> simple load balancer. I will read up on DataFlow and Safe. >> >> >> >> I am very excited to have found the GPars project - I am working with >> >> problems in bioinformatics, in which there is a lot of interest in >> >> parallel processing and GPars is fantastic for trying out concurrency >> >> ideas. I am going to show my code to my colleagues to try to convince >> >> them to give GPars a go. >> >> >> >> Martin >> >> >> >> >> >> >> >> On 14 April 2010 11:46, Dierk König <[hidden email]> wrote: >> >>> Hi Martin, >> >>> >> >>> welcome to GPars! >> >>> >> >>> Yes, you are doing it right. >> >>> >> >>> However, one rule of thumb: using references that are not >> >>> local to you closure (anything but "it" in this case) is often a sign >> >>> that some other concurrency concept may be more appropriate >> >>> or that you need an additional one (DataFlow, Safe, or maybe even >> >>> Actor). >> >>> >> >>> cheers >> >>> Dierk >> >>> >> >>> >> >>> Am 14.04.2010 um 10:45 schrieb Martin Jones: >> >>> >> >>>> Hi, >> >>>> >> >>>> I am new to concurrent programming, so I want to make sure I do this >> >>>> the right way. I want to use eachParallel(), but also have a counter >> >>>> that keeps track of the total number of elements processed. Since >> >>>> this counter will be accessed by multiple threads, is the best way to >> >>>> do this with an AtomicInteger? Something like >> >>>> >> >>>> def counter = new AtomicInteger() >> >>>> Parallelizer.doParallel { >> >>>> myLargeArray.eachParallel { >> >>>> counter.incrementAndGet() >> >>>> doProcessing(it) >> >>>> } >> >>>> } >> >>>> >> >>>> Assuming that my doProcessing() method does not have any side >> >>>> effects. >> >>>> >> >>>> Thanks, >> >>>> Martin >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe from this list, please visit: >> >>>> >> >>>> http://xircles.codehaus.org/manage_email >> >>>> >> >>>> >> >>> >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe from this list, please visit: >> >>> >> >>> http://xircles.codehaus.org/manage_email >> >>> >> >>> >> >>> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe from this list, please visit: >> >> >> >> http://xircles.codehaus.org/manage_email >> >> >> >> >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe from this list, please visit: >> > >> > http://xircles.codehaus.org/manage_email >> > >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > > > -- > E-mail: [hidden email] > Blog: http://www.jroller.com/vaclav > Linkedin page: http://www.linkedin.com/in/vaclavpech > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email |
«
Return to GPars - user mailing list
|
1 view|%1 views
| Free forum by Nabble | Edit this page |