Research:ChaNGa
From Astronomy Facility Wiki
ChaNGa (Charm N-body GrAvity solver) is a code to perform collisionless N-body simulations. It can perform cosmological simulations with periodic boundary conditions in comoving coordinates or simulations of isolated stellar systems. It uses a Barnes-Hut tree to calculate gravity, with hexadecapole expansion of nodes and Ewald summation for periodic forces. Timestepping is done with a leapfrog integrator with individual timesteps for each particle.
ChaNGa's novel feature is to use the dynamic load balancing scheme of the CHARM runtime system in order to obtain good performance on massively parallel systems. See our Supercomputing '06 poster for scaling results up to 20,000 processors.
ChaNGa uses the NChilada File Format or the Tipsy file format.
Contents |
Obtaining ChaNGa
First you need to install Charm. The standard Charm distribution will not work. You will need to get either the nightly tarball or check out from the CVS archive.
The most recent version of ChaNGa itself can be obtained from the CVS archive at the PACS software server. The ChaNGa module in the CVS repository will get the directories you need, which includes changa and structures:
setenv CVSROOT hpcc.astro.washington.edu:/net/CVS/nchilada cvs checkout ChaNGa
A particular release can also be downloaded from the software server. Please register before downloading ChaNGa so that we can keep statistics on its use.
License
The distribution of ChaNGa is licensed under the GPL.
Building ChaNGa
Now go into the ChaNGa/changa subdirectory, set the environment variable CHARM_DIR to point at the root of your charm distribution and run configure. (Note that if the copy of charm is put into the "ChaNGa" directory, or the directory above, then the CHARM_DIR variable need not be set.) This will configure the Makefile in both the changa and structures directory. make should then produce the charmrun and ChaNGa executables.
Machine specific instructions
For instructions on particular parallel architectures see Research:ChaNGaBuildInstructions.
Running ChanNGa
ChaNGa accepts Tipsy files as initial conditions. The running of the program is controlled by either a parameter file or command line switches, in the style of PKDGRAV. See the testcosmo or the teststep subdirectories for example parameter files. ChaNGa --help will give a list of all available options. Their meaning is described in Research:ChaNGaOptions. ChaNGa can be run in parallel or in serial. Generally (depending on the architecture) to run in parallel requires starting ChaNGa with the charmrun program. For
example
charmrun +p4 ./ChaNGa cube300.param
will start ChaNGa on four processors using the cube300.param parameter file. Note that on MPI architectures, charmrun is just a shell script wrapper around whatever command is used to start MPI jobs (e.g poe on IBM, mpirun on mpich.) See the CHARM installation and usage manual for more information on parallel execution.
Outputs are also in TIPSY format and are in files that end with the timestep. For example to visualize the final output of the testcosmo simulation, fire up tipsy, and type
openbinary cube300.000128 loadstandard 1.0 zall
This should display the clustering of galaxies on a 300 Mpc scale.
Restarts
A simulation can be restarted from a checkpoint using the syntax:
charmrun +p4 ./ChaNGa +restart cube300.chk0
All parameters will be restored from the checkpoint directory.
Getting Help
An email list has been set up at changa-users at u.washington.edu. Please subscribe to the list before posting to it.
Bugs and feature requests can be submitted to the NChilada product of our bugzilla server. Because of Spammers, a password is needed; ask for it on the mailing list.
Also check out our list of Research:ChaNGa Issues for common errors when running ChaNGa.
Documentation
While there is no comprehensive body of documentation detailing the ChaNGa code, the recent refactoring efforts are outlined and discussed here. The refactoring process unearthed the answers to some nuances of the existing code as well, so one would do well to look through these articles.
Acknowledgements
The development of ChaNGa was supported by a National Science Foundation ITR grant PHY-0205413 to the University of Washington, and NSF ITR grant NSF-0205611 to the University of Illinois. Contributors to the program include Graeme Lufkin, Tom Quinn, Filippo Gioachin, Sayantan Chakravorty, Amit Sharma, Celso Mendes, and Joachim Stadel.
