Multi-threading and globals

When designing a multi-threaded application, special attention must be paid to the use of global variables. When the globals are really meant to be shared among different threads, they can be protected with mutexes, for example. But sometimes a variable declared as global was not supposed to be part of the shared state. Maybe it was declared global to be shared between modules, not threads. I have encountered such a scenario when coding Pumpkin OS.

Pumpkin OS was designed to be multi-threaded from the start, so I have taken the necessary actions to protect the shared state. But there are portions of Pumpkin OS that depend on third-party projects that were never meant to be multi-threaded. One example is Musashi, a M68K emulator used to run native PalmOS applications inside Pumpkin OS. It does a superb job on running all those classic programs, but its use of global space means Pumpkin OS can run only one M68K application at a time. Mutexes would not help here: each M68k application should have its own copy the global state and should be able to run independently of the others. In other words, each application should have its own M68K processor running at full speed (as far as the Musashi code is concerned). I could try to refactor the code changing all function prototypes to include a pointer to a state variable. With hundreds of function to change, it is also a lot of work.

The solution I came up with uses something called thread local storage. Each thread has access to a private memory region that the main thread can setup in advance. When a deeply nested function needs to access global state, instead of using a global variable, it gets a pointer to its local storage. Each emulated M68K thread writes to its own M68K state, not interfering with another thread. And no function prototype needs to change. The first step was to identify all global variables used by the M68K emulator, which were surprisingly few:

m68ki_cpu_core s_m68ki_cpu;
int  m68ki_initial_cycles;
sint m68ki_remaining_cycles;
uint m68ki_tracing;
uint m68ki_address_space;
uint m68ki_aerr_address;
uint m68ki_aerr_write_mode;
uint m68ki_aerr_fc;
jmp_buf m68ki_bus_error_jmp_buf;

Then a C struct was created to encapsulate these variables:

typedef struct {
  m68ki_cpu_core s_m68ki_cpu;
  int  s_m68ki_initial_cycles;
  sint s_m68ki_remaining_cycles;
  uint s_m68ki_tracing;
  uint s_m68ki_address_space;
  uint s_m68ki_aerr_address;
  uint s_m68ki_aerr_write_mode;
  uint s_m68ki_aerr_fc;
  jmp_buf s_m68ki_bus_error_jmp_buf;
  /* and any other information necessary for running a m68k app */
} m68k_state_t;

A number of defines are used to allow transparent access to the members of an instance of this struct, as if they were still true globals:

#define m68ki_cpu               state->s_m68ki_cpu
#define m68ki_initial_cycles    state->s_m68ki_initial_cycles
#define m68ki_remaining_cycles  state->s_m68ki_remaining_cycles
#define m68ki_tracing           state->s_m68ki_tracing
#define m68ki_address_space     state->s_m68ki_address_space
#define m68ki_aerr_address      state->s_m68ki_aerr_address
#define m68ki_aerr_write_mode   state->s_m68ki_aerr_write_mode
#define m68ki_aerr_fc           state->s_m68ki_aerr_fc
#define m68ki_bus_error_jmp_buf state->s_m68ki_bus_error_jmp_buf

The main thread, before firing a new M68K thread, allocates this struct and passes its pointer to the new thread. A “key” used to identify the global state is initialized only once before any thread is created (this example uses the pthreads library available on most OS’es):

pthread_key_t key; // this key must be a true global!
pthread_t t;
m68k_state_t *state = calloc(1, sizeof(m68k_state_t));
pthread_key_create(&key, dummy_destructor);
pthread_create(&t, NULL, m68k_main, state)

m68k_main() is the thread entry point. It stores the state pointer received as argument into its local storage using the key initialized by the main thread:

void *m68k_main(void *arg) {
  m68k_state_t *state = (m68k_state_t *)arg;
  initialize_state_as_needed(state);
  pthread_set_specific(key, state);
  run_some_68k_code();
  ...
  return NULL;
}

To illustrate how this works, let’s take a sample function from Musashi that uses a global variable (in this case, m68ki_remaining_cycles):

static void m68k_op_stop(void) {
  if (FLAG_S) {
    uint new_sr = OPER_I_16();
    m68ki_trace_t0();        /* auto-disable (see m68kcpu.h) */
    CPU_STOPPED |= STOP_LEVEL_STOP;
    m68ki_set_sr(new_sr);
    if (m68ki_remaining_cycles >= CYC_INSTRUCTION[REG_IR])
      m68ki_remaining_cycles = CYC_INSTRUCTION[REG_IR];
    else
      USE_ALL_CYCLES();
    return;
  }
  m68ki_exception_privilege_violation();
}

The required change to this function is minimal. Just add a call to the GET_STATE macro as the first thing in the function:

static void m68k_op_stop(void) {
  GET_STATE;
  if (FLAG_S) {
    uint new_sr = OPER_I_16();
    m68ki_trace_t0();        /* auto-disable (see m68kcpu.h) */
    CPU_STOPPED |= STOP_LEVEL_STOP;
    m68ki_set_sr(new_sr);
    if (m68ki_remaining_cycles >= CYC_INSTRUCTION[REG_IR])
      m68ki_remaining_cycles = CYC_INSTRUCTION[REG_IR];
    else
      USE_ALL_CYCLES();
    return;
  }
  m68ki_exception_privilege_violation();
}

The GET_STATE macro fetches a pointer to the global state using the same key as before:

#define GET_STATE m68k_state_t *state = pthread_getspecific(key)

With the help of the member accessing macros defined before, the remaining code inside the function requires no change. The only thing to note is, instead of using a common name like “state” for the variable, it would probably be safer to choose a more unique name, to avoid clashing with any local variable name that any function could possibly use.

When adapting Musashi for use on Pumpkin OS, I ended up using a mix of two different techniques, depending on the balance between ease of change and good practices: either changing function prototypes to accept a state pointer, or using thread local storage like described in this post. Now Pumpkin OS can run multiple M68K applications at the same time, side by side with native applications (compiled from source to native code).

Advertisement