Distributed processing
Jeremy Udkoff
Hello,
I was wondering if I could get some help with distributed processing.
I have been building models on my laptop and started using distributed processing on a 6 core desktop --I'm on a gigabit network. However, even during 10k+10k Monte Carlo/Microsim trials (which take hours), the desktop doesn't go above 25% CPU usage in TA and is only using 400mb of ram. Is there anyway to increase the usage on that computer? Do I need to specify 6 cores?
Also, if I stop a process during computation from the master computer, I need to restart the computer for distributed processing to work again. Anyway around this? (restarting TA doesn't help)
Thank you very much!
Comments
What is your cycle length and roughly how many cycles do you need to complete on average to complete 1 trial? I find that the thing that consumes most time is a large average number of cycles required per trial.
A common scenario is that you need a short cycle length at the beginning to capture the upfront nuance of the disease process but then people move into a long-term sate where they don't do much. That's where the short cycle-length kills you.
Since you're running samples+trials, you are essentially running a large number of patient level simulations. If the description above sounds like your situation, you can make your model much more efficient by using dynamic cycle length - i.e. cyclelength becomes a variable with local definitions associated with each state. I have more on this if this interests you.
Another option is to use the threading option in simulations so that you can be sure that the work is divided among cores. Also consider increasing the Java Heap space.
A really good option is to run TreeAge on a souped up virtual computer on AWS. I have detailed instructions for that I can share with you.
Ultimately, you could consider reformatting your model as a DES which is very fast.
Email me at david.naimark@sunnybrook.ca if you want to follow up
David
Thank you! I sent you an email.
Note that non-fixed cycle length is tricky. Many of the built-in functions like _stage and discounting will be of no use to you. Instead, you would need to track time yourself for each patient in a tracker.
I'm not sure I would recommend this, but David is very skilled with TreeAge Pro and may have some good advice for you.
Please sign in to leave a comment.