Research:ChaNGa
From Astronomy Facility Wiki
ChaNGa (Charm N-body GrAvity solver) is a code to perform collisionless N-body simulations. It can perform cosmological simulations with periodic boundary conditions in comoving coordinates or simulations of isolated stellar systems. It uses a Barnes-Hut tree to calculate gravity, with hexadecapole expansion of nodes and Ewald summation for periodic forces. Timestepping is done with a leapfrog integrator with individual timesteps for each particle.
ChaNGa's novel feature is to use the dynamic load balancing scheme of the CHARM runtime system in order to obtain good performance on massively parallel systems. See our Supercomputing '06 poster for scaling results up to 20,000 processors.
ChaNGa uses the NChilada File Format or the Tipsy file format.
Contents |
Obtaining ChaNGa
First you need to install Charm. The standard Charm distribution will not work. You will need to get either the nightly tarball or check out from the CVS archive.
The most recent version of ChaNGa itself can be obtained from the CVS archive at the PACS software server. The ChaNGa module in the CVS repository will get the directories you need, which includes changa and structures:
setenv CVSROOT hpcc-cvs.astro.washington.edu:/local/CVS/nchilada cvs checkout ChaNGa
A particular release can also be downloaded from the software server. Please register before downloading ChaNGa so that we can keep statistics on its use.
License
The distribution of ChaNGa is licensed under the GPL.
Building ChaNGa
Charm needs to be built first. If it is not build it with the build script in the top level of the charm distribution. Typically, a command like build charm++ net-linux will build the charm++ compiler. For more recent versions of ChaNGa the cache module also needs to be built with build cache net-linux.
Now go into the ChaNGa/changa subdirectory, set the environment variable CHARM_DIR to point at the root of your charm distribution and run configure. (Note that if the copy of charm is put into the "ChaNGa" directory, or the directory above, then the CHARM_DIR variable need not be set.) This will configure the Makefile in both the changa and structures directory. make should then produce the charmrun and ChaNGa executables.
Machine specific instructions
For instructions on particular parallel architectures see Research:ChaNGaBuildInstructions.
Running ChanNGa
ChaNGa accepts Tipsy files as initial conditions. The running of the program is controlled by either a parameter file or command line switches, in the style of PKDGRAV. See the testcosmo or the teststep subdirectories for example parameter files. ChaNGa --help will give a list of all available options. Their meaning is described in Research:ChaNGaOptions. ChaNGa can be run in parallel or in serial. Generally (depending on the architecture) to run in parallel requires starting ChaNGa with the charmrun program. For
example
charmrun +p4 ./ChaNGa cube300.param
will start ChaNGa on four processors using the cube300.param parameter file.
net-linux Architectures
The "net" version of charm starts multiple processes by invoking ssh; therefore an ssh server needs to be installed on the target machine. For example, on Redhat/Fedora machines the openssh-server package needs to be installed. yum install openssh-server will accomplish this. Ssh needs to be installed even if you are running multiple cores on a single node. Also by default, ssh requires you to enter your password. This can be avoided by setting up your ssh keys correctly. See the SSH with keys HOWTO for information on how to do this.
MPI Architectures
On MPI architectures, charmrun is just a shell script wrapper around whatever command is used to start MPI jobs (e.g poe on IBM, mpirun on mpich.)
See the CHARM installation and usage manual for more information on parallel execution. Also see Research:ChaNGaPerformanceAnalysis to evaluate how these options affect the parallel performance.
Outputs are also in TIPSY format and are in files that end with the timestep. For example to visualize the final output of the testcosmo simulation, fire up tipsy, and type
openbinary cube300.000128 loadstandard 1.0 zall
This should display the clustering of galaxies on a 300 Mpc scale.
Restarts
A simulation can be restarted from a checkpoint using the syntax:
charmrun +p4 ./ChaNGa +restart cube300.chk0
All parameters will be restored from the checkpoint directory.
Visualization
ChaNGa now (as of 7/2009) has on demand visualization capabilities via the liveViz module of CHARM++. To use it, set bLiveViz = 1 in the parameter file, and start ChaNGa with
charmrun +p4 ++server ++server-port NNNNN ./ChaNGa run.param
where NNNNN is an unused TCP port number. Images of the running simulation can be optained by using the liveViz java client from the CHARM++ distribution in java/bin/liveViz. The syntax is liveViz hostname NNNNN where hostname is the machine on which charmrun is running, and NNNNN is the port number given above. A window will pop up with an image that will continually be refreshed from the running program. The image view is controlled by the .director file. See Research:ChaNGaOptions#Movie_Making_options.
Getting Help
An email list has been set up at changa-users at u.washington.edu. Please subscribe to the list before posting to it.
Bugs and feature requests can be submitted to the NChilada product of our bugzilla server. Because of Spammers, a password is needed; ask for it on the mailing list.
Also check out our list of Research:ChaNGa Issues for common errors when running ChaNGa.
Documentation
While there is no comprehensive body of documentation detailing the ChaNGa code, the recent refactoring efforts are outlined and discussed here. The refactoring process unearthed the answers to some nuances of the existing code as well, so one would do well to look through these articles.
Acknowledgements
The development of ChaNGa was supported by a National Science Foundation ITR grant PHY-0205413 to the University of Washington, and NSF ITR grant NSF-0205611 to the University of Illinois. Contributors to the program include Graeme Lufkin, Tom Quinn, Rok Roskar, Filippo Gioachin, Sayantan Chakravorty, Amit Sharma, Pritish Jetley, Celso Mendes, and Joachim Stadel.
