操作系统中的虚拟内存技术及其实现代码

虚拟内存是现代操作系统普遍使用的一种技术。

虚拟内存的基本思想是，每个进程有用独立的逻辑地址空间，内存被分为大小相等的多个块,称为页(Page)。每个页都是一段连续的地址。对于进程来看，逻辑上貌似有很多内存空间，其中一部分对应物理内存上的一块(称为页框 page frame，通常页和页框大小相等)，还有一些没加载在内存中的对应在硬盘上。通过引入进程的逻辑地址，把进程地址空间与实际存储空间分离，增加存储管理的灵活性。

地址空间和存储空间两个基本概念的定义如下：

 

地址空间：将源程序经过编译后得到的目标程序，存在于它所限定的地址范围内，这个范围称为地址空间。地址空间是逻辑地址的集合。

 

存储空间：指主存中一系列存储信息的物理单元的集合，这些单元的编号称为物理地址存储空间是物理地址的集合。

由此衍生出的管理方式有三种：
页式存储管理、段式存储管理和段页式存储管理。这里主要介绍页式存储。

在页式系统中进程建立时，操作系统为进程中所有的页分配页框。当进程撤销时收回所有分配给它的页框。在程序的运行期间，如果允许进程动态地申请空间，操作系统还要为进程申请的空间分配物理页框。操作系统为了完成这些功能，必须记录系统内存中实际的页框使用情况。操作系统还要在进程切换时，正确地切换两个不同的进程地址空间到物理内存空间的映射。为了理解操作系统如何完成这些需求，我们先理解页表技术。先看张图，转载自51CTO：

页表中的条目被称为页表项（page table entry），一个页表项负责记录一段虚拟地址到物理地址的映射关系。

既然页表是存储在内存中的，那么程序每次完成一次内存读取时都至少会访问内存两次，相比于不使用MMU（MMU是Memory Management Unit的缩写，它代表集成在CPU内部的一个硬件逻辑单元，主要作用是给CPU提供从虚拟地址向物理地址转换的功能，从硬件上给软件提供一种内存保护的机制）时的一次内存访问，效率被大大降低了，如果所使用的内存的性能比较差的话，这种效率的降低将会更明显。因此，如何在发挥MMU优势的同时使系统消耗尽量减小，就成为了一个亟待解决的问题。

于是，TLB产生了。TLB是什么呢？我们叫它转换旁路缓冲器，它实际上是MMU中临时存放转换数据的一组重定位寄存器。既然TLB本质上是一组寄存器，那么不难理解，相比于访问内存中的页表，访问TLB的速度要快很多。因此如果页表的内容全部存放于TLB中，就可以解决访问效率的问题了。

然而，由于制造成本等诸多限制，所有页表都存储在TLB中几乎是不可能的。这样一来，我们只能通过在有限容量的TLB中存储一部分最常用的页表，从而在一定程度上提高MMU的工作效率。

这一方法能够产生效果的理论依据叫做存储器访问的局部性原理。它的意思是说，程序在执行过程中访问与当前位置临近的代码的概率更高一些。因此，从理论上我们可以说，TLB中存储了当前时间段需要使用的大多数页表项，所以可以在很大程度上提高MMU的运行效率。

我们这里所用的是二级页表的技术，何为二级页表，即是MMU采用二级查表的方法，即首先由虚拟地址索引出第一张表的某一段内容，然后再根据这段内容搜索第二张表，最后才能确定物理地址。这里的第一张表，我们叫它一级页表，第二张表被称为是二级页表。采用二级查表法的主要目的是减小页表自身占据的内存空间，但缺点是进一步降低了内存的寻址效率。

好了，前情介绍完毕，下面上干货，用哈佛大学开发的用于教学的OS161来实现VM，OS161基于MIP-I hardware。

代码位于github上：https://github.com/tian-jiang/OS161-VirtualMemory

首先看一段代码，kern/arch/mips/include/vm.h，物理内存的分配定义在此

/*
 * MIPS-I hardwired memory layout:
 *    0xc0000000 - 0xffffffff   kseg2 (kernel, tlb-mapped)
 *    0xa0000000 - 0xbfffffff   kseg1 (kernel, unmapped, uncached)
 *    0x80000000 - 0x9fffffff   kseg0 (kernel, unmapped, cached)
 *    0x00000000 - 0x7fffffff   kuseg (user, tlb-mapped)
 *
 * (mips32 is a little different)
 */

#define MIPS_KUSEG  0x00000000
#define MIPS_KSEG0  0x80000000
#define MIPS_KSEG1  0xa0000000
#define MIPS_KSEG2  0xc0000000

内存的分配用图表示如下

这张图展示了在OS161中物理内存的分配. 

让我们从头开始：kern/startup/man.c

    /* Early initialization. */
    ram_bootstrap();
        .......

    /* Late phase of initialization. */
    vm_bootstrap();
        ........

在操作系统启动的时候，调用raw_bootstrap()以及vm_bootstrap()来启动vm管理模块。那么这两个函数是在哪里定义和使用的呢，我们接着看下面的代码。

kern/include/vm.h和kern/arch/mips/include/vm.h

/* Initialization function */
void vm_bootstrap(void);
......

  /* Allocate/free kernel heap pages (called by kmalloc/kfree) */

  void frametable_bootstrap(void);

/*
 * Interface to the low-level module that looks after the amount of
 * physical memory we have.
 *
 * ram_getsize returns the lowest valid physical address, and one past
 * the highest valid physical address. (Both are page-aligned.) This
 * is the memory that is available for use during operation, and
 * excludes the memory the kernel is loaded into and memory that is
 * grabbed in the very early stages of bootup.
 *
 * ram_stealmem can be used before ram_getsize is called to allocate
 * memory that cannot be freed later. This is intended for use early
 * in bootup before VM initialization is complete.
 */

void ram_bootstrap(void);
paddr_t ram_stealmem(unsigned long npages);
void ram_getsize(paddr_t *lo, paddr_t *hi);

这两个function是定义在这里的，那么这两个function又是干什么事情的呢

kern/arch/mips/vm/ram.c, kern/arch/mips/vm/vm.c, kern/vm/frametable.c

vaddr_t firstfree;   /* first free virtual address; set by start.S */

static paddr_t firstpaddr;  /* address of first free physical page */
static paddr_t lastpaddr;   /* one past end of last free physical page */

/*
 * Called very early in system boot to figure out how much physical
 * RAM is available.
 */
void
ram_bootstrap(void)
{
    size_t ramsize;
    
    /* Get size of RAM. */
    ramsize = mainbus_ramsize();

    /*
     * This is the same as the last physical address, as long as
     * we have less than 508 megabytes of memory. If we had more,
     * various annoying properties of the MIPS architecture would
     * force the RAM to be discontiguous. This is not a case we 
     * are going to worry about.
     */
    if (ramsize > 508*1024*1024) {
        ramsize = 508*1024*1024;
    }

    lastpaddr = ramsize;

    /* 
     * Get first free virtual address from where start.S saved it.
     * Convert to physical address.
     */
    firstpaddr = firstfree - MIPS_KSEG0;

    kprintf("%uk physical memory available\n", 
        (lastpaddr-firstpaddr)/1024);
}

/*
 * Initialise the frame table
 */
void
vm_bootstrap(void)
{
    frametable_bootstrap();
}

/*
 * Make variables static to prevent it from other file's accessing
 */
static struct frame_table_entry *frame_table;
static paddr_t frametop, freeframe;

/*
 * initialise frame table
 */
void
frametable_bootstrap(void)
{
    struct frame_table_entry *p;
    paddr_t firsta, lasta, paddr;
    unsigned long framenum, entry_num, frame_table_size, i;
    
    // get the useable range of physical memory
    ram_getsize(&firsta, &lasta);
    KASSERT((firsta & PAGE_FRAME) == firsta);
    KASSERT((lasta & PAGE_FRAME) == lasta);
    
    framenum = (lasta - firsta) / PAGE_SIZE;
    
    // calculate the size of the whole framemap
    frame_table_size = framenum * sizeof(struct frame_table_entry);
    frame_table_size = ROUNDUP(frame_table_size, PAGE_SIZE);
    entry_num = frame_table_size / PAGE_SIZE;
    KASSERT((frame_table_size & PAGE_FRAME) == frame_table_size);
    
    frametop = firsta;
    freeframe = firsta + frame_table_size;
    
    if (freeframe >= lasta) {
        // This is impossible for most of the time
        panic("vm: framemap consume physical memory?\n");
    }
    
    // keep the frame state in the top of the useable range of physical memory
    // the free frame page address started from the end of the frame map
    frame_table = (struct frame_table_entry *) PADDR_TO_KVADDR(firsta);
    
    // Initialise the frame list, each entry corrsponding to a frame,
    // and each entry stores the address of the next free frame.
    // If the next frame address of this entry equals zero, means this current frame is allocated
    p = frame_table;
    for (i = 0; i < framenum-1; i++) {
        if (i < entry_num) {
            p->next_freeframe = 0;
            p += 1;
            continue;
        }
        paddr = frametop + (i+1) * PAGE_SIZE;
        p->next_freeframe = paddr;
        p += 1;
    }
}

kern/include/vm.h
struct frame_table_entry {
    // address of next free frame
    size_t          next_freeframe;
};

raw_bootstrap是系统初始化时用来查看有多少物理内存可以使用的。而vm_bootstrap只是简单的调用了frametable_bootstrap()，而frametable_bootstrap()则是将能用的物理内存分页，每页大小为4K，然后保存一个记录空白页的linked list在内存中，从free的内存的顶部开始存放，但是在存放之前，先要算出需要多少空间来存放这个frame table。所以代码的前段在计算frame table的大小，后面则是初始化frame table这个linked list。因为初始化的时候都是空的，所以直接指向下一个page的地址即可。

操作系统的vm初始化到此完毕。那vm是怎么使用的呢，请看下面

kern/include/vm.h

/* Fault handling function called by trap code */
int vm_fault(int faulttype, vaddr_t faultaddress);

vaddr_t alloc_kpages(int npages);
void free_kpages(vaddr_t addr);

kern/include/addrspace.h，实现在kern/vm/addrspace.c

/* 
 * Address space - data structure associated with the virtual memory
 * space of a process.
 *
 * You write this.
 */

/*
 * A linked list which defined to store the information for regions(code, text, bss...)
 */
struct as_region {
    vaddr_t as_vbase;    /* the started virtual address for one region */
    size_t as_npages;    /* how many pages this region occupied from the vbase */
    unsigned int as_permissions;    /* does this region readable? writable? executable? */
    struct as_region *as_next_region;    /* address of the following region */
};

struct addrspace {
#if OPT_DUMBVM
        vaddr_t as_vbase1;
        paddr_t as_pbase1;
        size_t as_npages1;
        vaddr_t as_vbase2;
        paddr_t as_pbase2;
        size_t as_npages2;
        paddr_t as_stackpbase;
#else
        /* Put stuff here for your VM system */
    struct as_region *as_regions_start;    /* header of the regions linked list */
    vaddr_t as_pagetable;               /* address of the first-level page table */
#endif
};

/*
 * The structure of PTE in page table:
 * |        address             |  PTE_VALID      |    PE_W        |    PF_R        |    PF_X
 *  the virtual address of frame | valid indicator | writeable flag | readable flag | executable flag 
 * I don't use structure to represent PTE, just use type vaddr_t, and becuase the last 12 bit is free 
 * for a virtual address of frame, some of they could be used for the flags
 */

/*
 * Functions in addrspace.c:
 *
 *    as_create - create a new empty address space. You need to make 
 *                sure this gets called in all the right places. You
 *                may find you want to change the argument list. May
 *                return NULL on out-of-memory error.
 *
 *    as_copy   - create a new address space that is an exact copy of
 *                an old one. Probably calls as_create to get a new
 *                empty address space and fill it in, but that's up to
 *                you.
 *
 *    as_activate - make the specified address space the one currently
 *                "seen" by the processor. Argument might be NULL, 
 *                meaning "no particular address space".
 *
 *    as_destroy - dispose of an address space. You may need to change
 *                the way this works if implementing user-level threads.
 *
 *    as_define_region - set up a region of memory within the address
 *                space.
 *
 *    as_prepare_load - this is called before actually loading from an
 *                executable into the address space.
 *
 *    as_complete_load - this is called when loading from an executable
 *                is complete.
 *
 *    as_define_stack - set up the stack region in the address space.
 *                (Normally called *after* as_complete_load().) Hands
 *                back the initial stack pointer for the new process.
 *
 *    as_zero_region - zero out a new allocated page.
 *
 *    as_destroy_regions - free all the space allocated for regions storeage.
 */

struct addrspace *as_create(void);
int               as_copy(struct addrspace *src, struct addrspace **ret);
void              as_activate(struct addrspace *);
void              as_destroy(struct addrspace *);

int               as_define_region(struct addrspace *as, 
                                   vaddr_t vaddr, size_t sz,
                                   int readable, 
                                   int writeable,
                                   int executable);
int               as_prepare_load(struct addrspace *as);
int               as_complete_load(struct addrspace *as);
int               as_define_stack(struct addrspace *as, vaddr_t *initstackptr);
void          as_zero_region(vaddr_t vaddr, unsigned npages);
void          as_destroy_regions(struct as_region *ar);

kern/vm/frametable.c

/*
 * Allocate n pages. 
 * Before frame table initialisation, using ram_stealmem
 */
static
paddr_t
getppages(int npages)
{
    paddr_t paddr;
    struct frame_table_entry *p;
    int i;
    
    spinlock_acquire(&frametable_lock);
    if (frame_table == 0)
        paddr = ram_stealmem(npages);
    else
    {
        if (npages > 1){
            spinlock_release(&frametable_lock);
            return 0;
        }
        
        // Freeframe equals zero means all the frames have been allocated
        // and there is no frame to use.
        if (freeframe == 0){
            spinlock_release(&frametable_lock);
            return 0;
        }
        
        // Get the current free frame's entry id 
        // and retrieve the next free frame 
        paddr = freeframe;
        i = (freeframe - frametop) / PAGE_SIZE;
        p = frame_table + i;
        
        freeframe = p->next_freeframe;
        p->next_freeframe = 0;
    }
    spinlock_release(&frametable_lock);
    
    return paddr;
}

/*
 * Allocation function for public accessing
 * Returning virtual address of frame
 */
vaddr_t
alloc_kpages(int npages)
{
    paddr_t paddr = getppages(npages);
    
    if(paddr == 0)
        return 0;
    
    return PADDR_TO_KVADDR(paddr);
}

/*
 * Free page
 * Stores the address of the current freeframe into the entry of the frame to be freed
 * and update the address of the freeframe.
 */
static
void
freeppages(paddr_t paddr)
{
    struct frame_table_entry *p;
    int i;
    spinlock_acquire(&frametable_lock);
    i = (paddr - frametop) / PAGE_SIZE;
    p = frame_table + i;
    p->next_freeframe = freeframe;
    freeframe = paddr;
    spinlock_release(&frametable_lock);
}

/*
 * Free page function for public accessing
 */
void
free_kpages(vaddr_t addr)
{
    KASSERT(addr >= MIPS_KSEG0);
    
    paddr_t paddr = KVADDR_TO_PADDR(addr);
    if (paddr <= frametop) {
        // memory leakage
    }
    else {
        freeppages(paddr);
    }
}

kern/arch/mips/vm

这是最关键的一个函数，当TLB里面找不到用户app需要的virtual page时，怎么处理

/*
 * When TLB miss happening, a page fault will be trigged.
 * The way to handle it is as follow:
 * 1. check what page fault it is, if it is READONLY fault, 
 *    then do nothing just pop up an exception and kill the process
 * 2. if it is a read fault or write fault
 *    1. first check whether this virtual address is within any of the regions
 *       or stack of the current addrspace. if it is not, pop up a exception and
 *       kill the process, if it is there, goes on. 
 *    2. then try to find the mapping in the page table, 
 *       if a page table entry exists for this virtual address insert it into TLB 
 *    3. if this virtual address is not mapped yet, mapping this address,
 *     update the pagetable, then insert it into TLB
 */
int
vm_fault(int faulttype, vaddr_t faultaddress)
{
    vaddr_t *vaddr1, *vaddr2, vaddr, vbase, vtop, faultadd = 0;
    paddr_t paddr;
    struct addrspace *as;
    struct as_region *s;
    uint32_t ehi, elo;
    int i, index1, index2, spl;
    unsigned int permis = 0;
    
    switch (faulttype) {
        case VM_FAULT_READONLY:
            return EFAULT;
        case VM_FAULT_READ:
        case VM_FAULT_WRITE:
            break;
        default:
            return EINVAL;
    }
    
    as = curthread -> t_addrspace;
    if (as == NULL) {
        return EFAULT;
    }
    
    // Align faultaddress
    faultaddress &= PAGE_FRAME;
    
    // Go through the link list of regions 
    // Check the validation of the faultaddress
    KASSERT(as->as_regions_start != 0);
    s = as->as_regions_start;
    while (s != 0) {
        KASSERT(s->as_vbase != 0);
        KASSERT(s->as_npages != 0);
        KASSERT((s->as_vbase & PAGE_FRAME) == s->as_vbase);
        vbase = s->as_vbase;
        vtop = vbase + s->as_npages * PAGE_SIZE;
        if (faultaddress >= vbase && faultaddress < vtop) {
            faultadd = faultaddress;
            permis = s->as_permissions;
            break;
        }
        s = s->as_next_region;
    }
    
    if (faultadd == 0) {
        vtop = USERSTACK;
        vbase = vtop - VM_STACKPAGES * PAGE_SIZE;
        if (faultaddress >= vbase && faultaddress < vtop) {
            faultadd = faultaddress;
            // Stack is readable, writable but not executable
            permis |= (PF_W | PF_R);
        }
        
        // faultaddress is not within any range of the regions and stack
        if (faultadd == 0) {
            return EFAULT;
        }
    }
    
    index1 = (faultaddress & TOP_TEN) >> 22;
    index2 = (faultaddress & MID_TEN) >> 12;

    vaddr1 = (vaddr_t *)(as->as_pagetable + index1 * 4);
    if (*vaddr1) {
        vaddr2 = (vaddr_t *)(*vaddr1 + index2 * 4);
        // If the mapping exits in page table,
        // get the address stores in PTE, 
        // translate it into physical address, 
        // check writeable flag,
        // and prepare the physical address for TLBLO
        if (*vaddr2 & PTE_VALID) {
            vaddr = *vaddr2 & PAGE_FRAME;
            paddr = KVADDR_TO_PADDR(vaddr);
            if (permis & PF_W) {
                paddr |= TLBLO_DIRTY;
            }
        }
        // If not exists, do the mapping, 
        // update the PTE of the second page table,
        // check writeable flag,
        // and prepare the physical address for TLBLO
        else {
            vaddr = alloc_kpages(1);
            KASSERT(vaddr != 0);
            
            as_zero_region(vaddr, 1);
            *vaddr2 |= (vaddr | PTE_VALID);
            
            paddr = KVADDR_TO_PADDR(vaddr);
            if (permis & PF_W) {
                paddr |= TLBLO_DIRTY;
            }
        }
    }
    // If second page table even doesn't exists, 
    // create second page table,
    // do the mapping,
    // update the PTE,
    // and prepare the physical address.
    else {
        *vaddr1 = alloc_kpages(1);
        KASSERT(*vaddr1 != 0);
        as_zero_region(*vaddr1, 1);
        
        vaddr2 = (vaddr_t *)(*vaddr1 + index2 * 4);
        vaddr = alloc_kpages(1);
        KASSERT(vaddr != 0);
        as_zero_region(vaddr, 1);
        *vaddr2 |= (vaddr | PTE_VALID);

        paddr = KVADDR_TO_PADDR(vaddr);
        if (permis & PF_W) {
            paddr |= TLBLO_DIRTY;
        }
    }
        
    spl = splhigh();
    
    // update TLB entry
    // if there still a empty TLB entry, insert new one in
    // if not, randomly select one, throw it, insert new one in
    for (i=0; i<NUM_TLB; i++) {
        tlb_read(&ehi, &elo, i);
        if (elo & TLBLO_VALID) {
            continue;
        }
        ehi = faultaddress;
        elo = paddr | TLBLO_VALID;
        tlb_write(ehi, elo, i);
        splx(spl);
        return 0;
    }
    
    // FIXME, TLB replacement algo.
    ehi = faultaddress;
    elo = paddr | TLBLO_VALID;
    tlb_random(ehi, elo);
    splx(spl);
    return 0;
}

在系统运行的过程中，会不断的产生page fault，这是因为，虽然系统给了运行的程序分配了页（分配的函数见kern/vm/frametable.c），但是这个TLB里面没有记录这个页面从虚拟地址到物理地址的映射，所以无法使用。所以在程序真正需要使用这个页的时候，需要首先访问TLB，从里面取出对应的物理地址。