Chapter 2: Simple Programs and Basic Chares¶

Charm++ programs are written in standard C++. To support parallel features, Charm++ requires a small amount of additional information about which objects are parallel and which of their methods can be invoked remotely. This information is provided in an interface file with the extension .ci.

Every Charm++ program therefore involves (at minimum) two source files:

A .ci file declaring the parallel structure
A .C (or .cpp) file containing the C++ implementation

The Charm++ compiler, charmc, processes the .ci file to generate two header files (<module>.decl.h and <module>.def.h) that are included in the .C file.

Chares: Parallel C++ Objects¶

A chare is a C++ object with three special properties:

It inherits from a system-generated base class (CBase_ClassName).
It can be created on any processor — the creator does not choose which one.
Some of its methods, called entry methods, can be invoked asynchronously from any other processor.

Everything else about a chare is ordinary C++: it has data members, private methods, constructors, and so on. The parallel machinery is layered on top of normal C++ objects, not in place of them.

Hello World¶

The simplest Charm++ program has a single chare — the main chare — whose constructor is the program entry point, analogous to main() in sequential C.

The Interface File¶

// hello.ci
mainmodule hello {
  mainchare main {
    entry main(CkArgMsg *m);
  };
};

Several things to note:

Every Charm++ program is organized into modules. Each module has one .ci file. Exactly one module is the mainmodule.
The mainchare keyword designates the class whose constructor starts execution.
The constructor is declared as an entry method. Any method that can be invoked remotely — including all constructors of chare classes — must be declared as entry in the .ci file.
CkArgMsg is a system class carrying argc and argv. All main chares receive it.

The Implementation File¶

// hello.C
#include "hello.decl.h"      // generated by charmc from hello.ci

class main : public CBase_main {
public:
  main(CkMigrateMessage *m) {}   // required for migratability
  main(CkArgMsg *m) {
    CkPrintf("Hello World\n");
    delete m;
    CkExit();
  }
};

#include "hello.def.h"       // generated by charmc from hello.ci

Key points:

CBase_main is generated by charmc from the .ci declaration. Inheriting from it makes main a proper chare class.
hello.decl.h is included at the top of the file (before the class definition). hello.def.h is included at the bottom (after all class definitions).
CkPrintf is the Charm++ replacement for printf. It buffers output so that lines from different processors do not interleave. ckout provides the same guarantee for iostream-style output.
CkExit() — not exit() — shuts down all processors cleanly. Calling exit() directly would leave other processors hanging.

Build and Run¶

charmc hello.ci                         # generates hello.decl.h and hello.def.h
charmc -c hello.C                       # compiles
charmc -language charm++ -o hello hello.o  # links

./hello +p1                             # run on 1 PE (no charmrun needed for single process)
./charmrun ++local ./hello +p4          # run on 4 PEs on local machine

The +pN flag tells Charm++ to use N processing elements (PEs). On a single machine, ++local avoids the need for ssh.

Creating Multiple Chares¶

A single chare uses only one PE. To use multiple processors, the program must create multiple chares so the runtime can distribute them across PEs.

The main chare always creates the first wave of work. Those chares may create further chares, and so on.

To create a new chare, use the ckNew static method of its proxy class:

// In the .ci file:
chare Interim {
  entry Interim(int x);
};

// In the .C file:
CProxy_Interim::ckNew(42);   // create an Interim chare with argument 42

ckNew is non-blocking. It tells the runtime to create the chare on some processor of its choice, then immediately returns. The actual construction happens asynchronously.

For every chare class Foo declared in the .ci file, charmc generates a proxy class CProxy_Foo. Proxies are location-oblivious handles to chares — they work correctly regardless of which processor the chare lives on, and even if the chare migrates.

Proxies and Entry Methods¶

Creating a chare with ckNew returns a proxy to it:

CProxy_Compute sim = CProxy_Compute::ckNew(pi);

You can then invoke entry methods on the chare through its proxy:

sim.findArea(5.0);    // non-blocking: enqueues the call on sim's processor

This looks like a regular method call but behaves very differently: - The parameters are packed into a message by the runtime. - The message is sent to whatever processor currently holds sim. - Control returns immediately to the caller. - At some future time, sim's scheduler picks up the message and invokes findArea.

Entry methods must be declared in the .ci file:

chare Compute {
  entry Compute(double pi);
  entry void findArea(double radius, bool done);
};

Every chare also has thisProxy — a proxy to itself, inherited from CBase_*. This is how a chare passes a reference to itself to others:

CProxy_Compute::ckNew(pi, thisProxy);   // pass self-reference to new chare

Out-of-Order Execution¶

Charm++ does not guarantee that entry method invocations are delivered in the order they were sent. This surprises programmers coming from sequential or MPI backgrounds.

Consider:

for (int i = 1; i <= 10; i++)
  sim.findArea(i);
sim.findArea(10, /*done=*/true);   // wrong: this may arrive first

The done=true invocation may reach sim before some earlier ones, causing premature termination.

There are two reasons Charm++ does not enforce ordering: 1. Guaranteeing order has a cost (sequence numbers, buffering). The base model avoids paying that cost unless necessary. 2. The scheduler may deliberately reorder messages — for example, to execute high-priority messages first.

The correct approach is to think asynchronously and design programs that work regardless of delivery order. A standard pattern: count callbacks.

// Main chare sends 10 findArea requests and waits for 10 responses:
class Main : public CBase_Main {
  int count;
public:
  Main(CkArgMsg *m) {
    CProxy_Compute sim = CProxy_Compute::ckNew(pi, thisProxy);
    for (int i = 1; i <= 10; i++) sim.findArea(i);
    count = 10;
  }
  void doneArea() {          // entry method called by Compute for each result
    if (--count == 0) CkExit();
  }
};

Now termination is correct regardless of delivery order: CkExit fires only after all 10 responses arrive.

Note: a chare's own private methods (not declared entry) are called directly and execute synchronously, like ordinary C++. An entry method called via thisProxy.foo() is enqueued in the scheduler and executes later; the same method called as foo() or this->foo() executes immediately.

Passing Arrays to Entry Methods¶

Entry methods can receive arrays. The size parameter must appear before the array in the .ci declaration:

// .ci file
entry void process(int n, double data[n]);

// .C file — receiving side
void process(int n, double *data) {
  for (int i = 0; i < n; i++) { ... }
}

// sending side
double buf[100];
proxy.process(100, buf);    // runtime packs the array into the message

The runtime handles packing and unpacking. The receiving pointer is valid only for the duration of the entry method.

Grainsize¶

A key question in any Charm++ program is: how many chares, and how much work per chare? This is the grainsize question.

Define: - g — computation time of one entry method invocation (the grain) - t_o — per-grain overhead (scheduling, message packing/unpacking): typically 1–5 µs

On one processor, total execution time is:

T₁ = T + t_o · (T/g)

where T is the sequential execution time. The overhead term is small when g >> t_o. A practical rule: keep g at least 10–20× t_o.

On P processors (assuming good load balance):

Tp ≈ max(g, T₁/P)

The max term captures Amdahl's law: you cannot run faster than the largest grain.

Measured Example: Primality Testing¶

Testing primality of 1 million random integers, varying the number of integers per chare (batch size M), on a single PE:

M (numbers/chare)	Chares	Time (s)
1	1,000,000	2.29
10	100,000	1.29
20	50,000	1.23
50	20,000	1.21
100	10,000	1.19
1,000,000	1 (sequential)	1.19

Per-chare overhead measured at ~1.1 µs. With ~1.2 µs of computation per number, the grain needs to be only ~20 numbers (M ≈ 20) to bring overhead below 5% of sequential time. Beyond that, the curve is essentially flat — a wide range of grainsizes all perform near-optimally.

Key insight: the flat region is wide. There is no need to find a precise optimal grainsize. Choose a grain large enough to amortize overhead, and the runtime does the rest.

Note on measurement: these timings were collected by running the program directly (without charmrun) on a single PE. When using charmrun with SMP mode, inter-process communication machinery adds per-message overhead even on a single logical PE, artificially inflating the measured overhead and introducing noise. For grainsize experiments, run directly.

Grainsize and Parallelism¶

On multiple PEs, the optimal grainsize shifts. With 4 PEs, a finer decomposition (more chares) allows the runtime more flexibility to overlap work and communication. In the primality example, the 4-PE optimum was around M = 10,000 — coarser than strictly necessary for overhead, but fine enough to give the runtime 100 chares to schedule. The general guideline: aim for at least 4–10× more chares than PEs.

System Utilities¶

Utility	Purpose
`CkPrintf(fmt, ...)`	Thread-safe printf
`ckout << ...`	Thread-safe iostream output
`CkExit()`	Terminate all PEs cleanly
`CkAbort(msg)`	Abort with error message
`CkMyPe()`	Index of the current PE
`CkNumPes()`	Total number of PEs
`CkWallTimer()`	Wall-clock time in seconds (double)
`CkArgMsg`	Carries argc/argv to the main chare

Summary¶

The structure of a Charm++ program:

.ci file: declare modules, chare classes, and entry methods.
charmc hello.ci: generates hello.decl.h and hello.def.h.
.C file: implement chare classes, bracketed by #include "hello.decl.h" and #include "hello.def.h".
Create chares with CProxy_Foo::ckNew(args) — returns a proxy, creation is asynchronous.
Invoke entry methods via proxy — non-blocking, may execute out of order.
Terminate with CkExit() from the last entry method that should run.

The most important habit to develop: think asynchronously. Design programs that are correct regardless of message delivery order. The callback-counting pattern shown above is a fundamental building block.