ARM64 OS Handbook
🔍

Chapter 38: User Space

What You Will Learn in This Chapter
  • What user space is and how the CPU enforces privilege separation
  • How the kernel enters user mode (ERET)
  • Setting up user stacks and argument passing
  • The syscall convention and user-space library (libc)
  • How system calls switch from EL0 to EL1 and back
  • Our user-space entry implementation

38.1 Privilege Separation

ARM64 provides Exception Level 0 (EL0) for user-space programs. Code running at EL0:

  • Cannot access system registers (TCR, MAIR, SCTLR, etc.)
  • Cannot access kernel memory (MMU enforces page permissions)
  • Cannot execute privileged instructions (MSR to system registers)
  • Can only access memory mapped as EL0-accessible in page tables

When a user program needs a privileged operation (opening a file, allocating memory, sending a network packet), it calls a system call (Chapter 14). The SVC instruction traps to EL1, the kernel performs the operation, and returns via ERET.

38.2 Entering User Mode

The kernel enters user mode after fork+exec. The eret instruction returns to EL0 with the saved SPSR_EL1 and ELR_EL1 values:

/* Prepare to enter user space for the first time */
void enter_user_space(uint64_t entry, uint64_t sp, uint64_t arg) {
    /* Set up the user state in the process's context */
    struct pcb *proc = get_current_process();

    /* Set exception link register to entry point */
    proc->context.elr = entry;

    /* Set stack pointer for user mode */
    proc->context.sp_el0 = sp;

    /* Set SPSR: EL0 in AArch64 mode, interrupts enabled */
    proc->context.spsr = 0;  /* SPSR_EL1 = 0 means EL0t, AArch64 */

    /* Put arguments in registers (x0 = argc, x1 = argv) */
    proc->context.regs[0] = 1;               /* argc */
    proc->context.regs[1] = sp + 8;          /* argv pointer */

    /* Return to user mode */
    restore_context_and_eret(&proc->context);
}

/* Assembly: ERET to user space */
// restore_context_and_eret:
//     ldr x0, [sp, #CONTEXT_ELR_OFF]
//     msr elr_el1, x0
//     ldr x0, [sp, #CONTEXT_SPSR_OFF]
//     msr spsr_el1, x0
//     ldr x0, [sp, #CONTEXT_SP_EL0_OFF]
//     msr sp_el0, x0
//     /* Restore x0-x30 */
//     ldp x0, x1, [sp, #CONTEXT_X0_OFF]
//     ldp x2, x3, [sp, #CONTEXT_X2_OFF]
//     ...
//     ldp x28, x29, [sp, #CONTEXT_X28_OFF]
//     ldr x30, [sp, #CONTEXT_X30_OFF]
//     eret  /* Jump to ELR_EL1 in EL0 mode */

38.3 User Stack and Arguments

When a program starts, the kernel sets up the user stack with the standard C runtime layout:

User stack (grows down from USER_STACK_TOP):
[high addresses]
  USER_STACK_TOP
  Auxiliary vectors (AT_PHDR, AT_ENTRY, etc.)
  Environment strings ("PATH=/bin\0", etc.)
  Argument strings ("program_name\0", "arg1\0", ...)
  Pointers to env strings (argv[argc] = NULL, envp[0], ...)
  Pointers to arg strings (argv[0], argv[1], ...)
  argc (8 bytes)
[low addresses] SP after ERET
#define USER_STACK_TOP  0x0000FFFFFFFFF000ULL
#define USER_STACK_SIZE (256 * 1024)  /* 256 KB */

/* Set up user stack with argc/argv/envp */
uint64_t setup_user_stack(struct pcb *proc, int argc, char **argv) {
    uint64_t sp = USER_STACK_TOP;

    /* Push environment strings (simplified: empty) */
    uint64_t envp_start = sp;

    /* Push argument strings */
    sp -= strlen(argv[0]) + 1;
    memcpy((void *)sp, argv[0], strlen(argv[0]) + 1);

    /* Align SP to 16 bytes */
    sp &= ~15;

    /* Push argv pointers (NULL-terminated) */
    sp -= 16;  /* 1 arg + NULL */
    uint64_t *arg_ptr = (uint64_t *)sp;
    arg_ptr[argc] = 0;  /* NULL terminator */
    for (int i = 0; i < argc; i++) {
        /* Store pointer to each arg string */
        arg_ptr[i] = sp + 16 + i * 16;  /* Simplified */
    }

    /* Push argc */
    sp -= 8;
    *(uint64_t *)sp = argc;

    return sp;
}

38.4 The C Runtime (crt0)

User programs start executing at _start, the C runtime entry point. The crt0 code sets up the C environment and calls main():

/* crt0.S: user-space program entry point */
.section .text
.global _start
_start:
    /* x0 = argc, x1 = argv */
    mov x29, #0        /* Clear frame pointer */
    mov x30, #0        /* Clear link register */

    /* Call main(argc, argv, envp) */
    /* envp is at x1 + (argc+1)*8 */
    add x2, x1, x0, lsl #3
    add x2, x2, #8     /* x2 = envp */
    bl main

    /* Call exit with return value */
    mov x1, x0
    mov x0, #93        /* __NR_exit */
    svc #0

    /* Never reached */
    b .

38.5 Syscall Wrappers

User-space programs cannot directly include kernel headers. Instead, a minimal libc provides wrapper functions:

/* libc: syscall wrapper for AArch64 */
static inline long syscall(long n, long a0, long a1, long a2,
                           long a3, long a4, long a5) {
    register long x0 asm("x0") = a0;
    register long x1 asm("x1") = a1;
    register long x2 asm("x2") = a2;
    register long x3 asm("x3") = a3;
    register long x4 asm("x4") = a4;
    register long x5 asm("x5") = a5;
    register long x8 asm("x8") = n;

    asm volatile("svc #0"
                 : "=r"(x0)
                 : "r"(x0), "r"(x1), "r"(x2),
                   "r"(x3), "r"(x4), "r"(x5), "r"(x8)
                 : "memory");

    return x0;
}

#define SYS_WRITE  64
#define SYS_OPEN   56
#define SYS_READ   63
#define SYS_EXIT   93

int write(int fd, const void *buf, size_t count) {
    return syscall(SYS_WRITE, fd, (long)buf, count, 0, 0, 0);
}

int open(const char *path, int flags) {
    return syscall(SYS_OPEN, (long)path, flags, 0, 0, 0, 0);
}

void exit(int code) {
    syscall(SYS_EXIT, code, 0, 0, 0, 0, 0);
    __builtin_unreachable();
}

38.6 Our Implementation

Our user space subsystem provides:

  • User mode entry: ERET to EL0 with proper SPSR, ELR, and SP_EL0
  • User stack setup: argc/argv/envp layout at USER_STACK_TOP
  • Minimal libc: syscall wrappers for write, read, open, exit, brk, etc.
  • crt0: assembly entry point that calls main() and exit()
  • User address space layout: 0x0000_0000_0001_0000 to 0x0000_00FF_FFFF_FFFF
  • First user process: /sbin/init (PID 1), started after kernel init

38.7 Exercises

Exercise 1: Print a String

Write the simplest possible user program in assembly: print "Hello World\n" to stdout using the write syscall. Assemble and link it.

Exercise 2: Implement getpid

Add a getpid syscall (syscall number 172 on ARM64). Write the kernel handler and user-space wrapper. Test by printing the PID of a user process.

38.8 Summary

User space provides a secure, isolated environment for running user programs at EL0. The kernel enters user mode via ERET, setting up the user stack with standard C runtime arguments. System calls (SVC) are the gateway to kernel services. A minimal crt0 and libc allow user programs to be written in C with familiar functions like main(), write(), and exit(). The first user process (/sbin/init) is spawned after kernel initialization.