This post offers simple example and installation tips for “doSMP” the new Parallel Processing backend package for R under windows.
* * *
Update:
The required packages are not yet now available on CRAN, but until they will get online, you can download them from here:
REvolution foreach windows bundle
(Simply unzip the folders inside your R library folder)
* * *
Recently, REvolution blog announced the release of “doSMP”, an R package which offers support for symmetric multicore processing (SMP) on Windows.
This means you can now speed up loops in R code running iterations in parallel on a multi-core or multi-processor machine, thus offering windows users what was until recently available for only Linux/Mac users through the doMC package.
Installation
For now, doSMP is not available on CRAN, so in order to get it you will need to download the REvolution R distribution “R Community 3.2” (they will ask you to supply your e-mail, but I trust REvolution won’t do anything too bad with it…)
If you already have R installed, and want to keep using it (and not the REvolution distribution, as was the case with me), you can navigate to the library folder inside the REvolution distribution it, and copy all the folders (package folders) from there to the library folder in your own R installation.
If you are using R 2.11.0, you will also need to download (and install) the revoIPC package from here:
revoIPC package – download link (required for running doSMP on windows)
(Thanks to Tao Shi for making this available!)
Usage
Once you got the folders in place, you can then load the packages and do something like this:
require(doSMP)
workers <- startWorkers(2) # My computer has 2 cores
registerDoSMP(workers)
# create a function to run in each itteration of the loop
check <-function(n) {
for(i in 1:1000)
{
sme <- matrix(rnorm(100), 10,10)
solve(sme)
}
}
times <- 10 # times to run the loop
# comparing the running time for each loop
system.time(x <- foreach(j=1:times ) %dopar% check(j)) # 2.56 seconds (notice that the first run would be slower, because of R's lazy loading)
system.time(for(j in 1:times ) x <- check(j)) # 4.82 seconds
# stop workers
stopWorkers(workers)
Points to notice:
- You will only benefit from the parallelism if the body of the loop is performing time-consuming operations. Otherwise, R serial loops will be faster
- Notice that on the first run, the foreach loop could be slow because of R's lazy loading of functions.
- I am using startWorkers(2) because my computer has two cores, if your computer has more (for example 4) use more.
- Lastly - if you want more examples on usage, look at the "ParallelR Lite User's Guide", included with REvolution R Community 3.2 installation in the "doc" folder
Updates
(15.5.10) :
The new R version (2.11.0) doesn't work with doSMP, and will return you with the following error:
Loading required package: revoIPC
Error: package 'revoIPC' was built for i386-pc-intel32
So far, a solution is not found, except using REvolution R distribution, or using R 2.10
A thread on the subject was started recently to report the problem. Updates will be given in case someone would come up with better solutions.
Thanks to Tao Shi, there is now a solution to the problem. You'll need to download the revoIPC package from here:
revoIPC package - download link (required for running doSMP on windows)
Install the package on your R distribution, and follow all of the other steps detailed earlier in this post. It will now work fine on R 2.11.0
Update 2: Notice that I added, in the beginning of the post, a download link to all the packages required for running parallel foreach with R 2.11.0 on windows. (That is until they will be uploaded to CRAN)
Update 3 (04.03.2011): doSMP is now officially on CRAN!
Hi. This works on a 32 bit distribution of R but not the 64 bit distribution. Any fixes?
I am sorry to say I don’t know of any fixes (or of the problem). If you come across a solution, please come back to share.
Are there any new updates for 64 bit doSMP. I am new to parallel computing but I could make it to work on 32 bit. I installed on 64 bit without errors but the problem is when I start running, its not responding (even for stopWorkers function).
My machine is Windows 7 with 4 cores and 8 logical processors (intel(R) Core(TM) i7 CPU).
Great post – I’m very excited about parallel computing. So far I’m doing things on a single PC, but maybe someday I will be able to set up a cloud (as soon as sky falls down and my faculty starts using linux). It would be also nice if something similar would be done for Windows (sensu the SETI project), with a single application that would turn host’s PC into a node that would share resources for R over the internet or local network.
Thank you for the kind words Romunov.
I hear your pain regarding the cloud in your local system.
My University uses Condor, which I still haven’t fully figured out how to use for my needs (I did once, but never got the motivation to systematically do it again, and more scalable for my work).
In the next few months I’ll probably write a bit more on options to do this using Amazon cloud.
But only after I have learned it a bit more myself 🙂
Cheers,
Tal
multicore package has Windows support (albeit in testing). o/
CRAN doesn’t have it yet, but it’s already on R-forge (http://www.rforge.net/multicore/).
Thanks Roman. I will have a look at it sometime later, and hope to try to reproduce all that I have done in this post with that package.
has anyone had any luck with this? parallel processing in windows 7 with a 64 bit build would make r much more powerful for me..
I am using the trick described in the post on Win 7 64bit.
Regarding the multicore, I haven’t yet – if you will find it successful, I’d be glad to know about it.
which post? link?
I am saying that in this post (the one we are having the discussion on), there is all the code needed on how to use parallel computing in R.
I tried it on my code but Rterm crashed (for each spawned child). :/
Any ideas where to drop my logs for the developer to see?
Since I don’t see any other option, I would guess you might wish to try here:
[email protected]
See:
https://cran.r-project.org/web/packages/multicore/index.html
That’s what I thought as well. I’ve already emailed mr. Urbanek.
Has anyone bothered to find out how to “kill” all SMP processes? I was sloppy and didn’t clean after myself before powering down R and now I’m without the primary object I used to registerDoSMP. Is there a way to “reboot” all the cores?
This post helped me much Tal. Thanks and all the best.
My pleasure, thanks for sharing it in the comments 🙂
Hi Tal,
Thanks for the helpful instructions and example code. It seems that doSMP is now available from R-Forge (https://r-forge.r-project.org/projects/dosmp/), and can be installed on native R with the install.packages() command.
install.packages(“doSMP”, repos=”http://R-Forge.R-project.org”)
It works fine on R-2.12.0 on Windows 7 64bit.
Failed with error: ‘package ‘revoIPC’ is not installed for ‘arch=i386’’
win7 64 bit
It still can not work for win7 64 bit system. Is that right?
doSMP is now removed from CRAN – and I need it for my disseration analysis! what do i DO?
Thank you for the update.
I read your parallel multicore processing in R post and I’d like to try it out. Â Unfortunately, I haven’t been able to install the doSMP package. Â It isn’t listed on CRAN (I found a .tar.gz file in the archives, but I’m not sure how to install it).
I’d appreciate any help that you can give me. Â Thanks!
Hi Jon,
It is (sadly) the case that doSMP was removed from CRAN due to bugs in the package (when used with the newest R). Â You will either have to wait for a newer version, use an older version of R (not suggested), or try one of the other options (for example, the {cluster} package in R).
With regards,
Tal
I am glad to know that 🙂
I am glad to know 🙂
The doSMP package is no longer actively supported in R 2.15 – it is now a deprecated package.
 The doParallel package has taken over the functionality of doSMP.
Stephen Weller
Revolution Analytics Technical Support Engineer
Hy,
So if I understood correctly then I Should use doParallel with R Revolution yeas?