memory(7)

NAME

memory − local memory

DESCRIPTION

Local memory is an optional architectural feature. Local memory was designed to improve the performance of multiprocessor systems by reducing the amount of system memory bus traffic and contention. This contention has been one of the factors that has traditionally limited the number of processors that can be effectively utilized in a tightly-coupled multiprocessing arrangement.

Local memory is a pool of memory physically located on the CPU board of the system (or on each CPU board of a multi-processor system). Each CPU has a data path to it’s local memory pool that does not require use of the system bus, and so does not incur any system bus arbitration delay when a memory access is performed. Each CPU also has access via the system bus to global memory, which is memory shared by all CPUs in the system. A CPU can also access the local memory of another CPU, which is called a foreign memory access. However, hardware bus and cache constraints make such foreign memory accesses slow, and their use is restricted by the kernel. In addition, use of the multiprocessor synchronizing instructions (the CAS, CAS2, TAS and XMEM, for example) to foreign local memory pools should be avoided since the indivisibility of their read-modify-write cycle is not guaranteed.

Local memory is one level in the hierarchy of memory subsystems available. For series 5000 the following memory hierarchy exists:

At the lowest level are the on-chip caches which are an architectural feature of the CPU. While small, these on-chip caches provide the fastest memory access times. There are separate instruction and data caches. Both global and local memory addresses may be cached in the on-chip caches.

Each CPU also has a secondary cache which functions as instruction and data caches for each CPU. This cache is much larger, although slower, than the on-chip cache. Again, both global and local memory addresses may be cached in the secondary caches.

Both the on-chip and secondary caches provide coherency with both local and global memory, even in multiple CPU configurations.

Foreign local memory accesses are not cached in either on-chip or secondary caches. In fact, if a memory access is made with the copyback cache mode, an exception will occur. Therefore, cached local memory accesses are usually only those accesses that are from the local memory pool that is local (non-foreign) to that CPU. In general, the operating system will not cache foreign local memory accesses.

At the next level of the memory hierarchy, local memory may be used to reduce the amount of global memory and frontplane contention. On series 5000 systems, a local memory pool is shared by all the CPUs that reside on the same CPU board with that local memory. In general, the operating system ensures that foreign local memory accesses are not cached. During process migrations on series 5000 systems, it is not necessary to move regions located in local memory if the new CPU also shares the same local memory pool as the previous CPU.

At the top of the memory subsystem hierarchy is global memory, which is shared by all the CPUs in the system.

For series 4000 systems, the following memory hierarchy exists:

At the lowest level are the Cache/Memory Management Units (CMMUs) which function as instruction and data caches for each CPU. The CMMUs provide the fastest memory access times. Both global and local memory data may be cached in the CMMUs, and certain CMMU cache modes provide cache coherency with both local and global memory, even in multiple CPU configurations. However, cache coherency is not provided with foreign local memory accesses. Therefore, local memory accesses cached by the CMMUs are usually only those accesses that are from the local memory pool that is local (non-foreign) to that CPU. In general, the operating system will not cache foreign local memory accesses in the CMMUs.

At the next level of the memory hierarchy, local memory may be used to reduce the amount of global memory and frontplane contention. On series 4000 systems, a local memory pool is shared by all the CPUs that reside on the same CPU board with that local memory. Therefore, on series 4000 HN4400 single CPU board systems, local memory is shared between all of the processors. Hence, foreign local memory accesses will never occur. On series 4000 HN4800 multiple CPU board systems, however, foreign local memory accesses may occur. In general, the operating system ensures that foreign local memory accesses are not cached in the CPU’s CMMUs. During process migrations on series 4000 systems, it is not necessary to move regions located in local memory if the new CPU also shares the same local memory pool as the previous CPU.

At the top of the memory subsystem hierarchy is global memory, which is shared by all the CPUs in the system.

Once configured by the console processor, each local memory pool is automatically initialized by the kernel as part of the system initialization procedure, though no local memory is actually consumed by the kernel. The mpadvise(2) system call can be used to return a mask of the CPUs which have local memory.

Each process which begins execution on the system has an address space which consists of three (3) regions for text, data and user-mode stack; a per-process area called the user area, or u-area; and zero or more shared memory regions. The contents of these regions are as follows:

Text Contains the executable instructions of the program. The text region is typically shared among multiple processes, though it need not be. The text region is fixed in size throughout the execution of the program.

Data Contains the initialized and uninitialized (bss) data of the program. The data region is not shared among multiple processes. The data region may expand and possibly contract during execution of the program if the program dynamically allocates and frees memory.

Stack Contains the user-mode stack of the program. This region is not shared among multiple processes. The stack region may grow during execution of the program.

PCB Contains the process control block , kernel-mode stack and u-area of the process. The PCB is where the context of the process is saved and restored when a context switch is performed. The kernel stack is used by the kernel during system calls. The u-area contains per-process information that must be accessible to the kernel.

Shared Memory These regions allow the sharing of data among multiple processes.

Each process which runs on the system has a memory flags field in it’s process table entry which specifies the binding of the regions of the program to local or global memory. The memory flags of a process are automatically inherited by any child processes generated by a fork(2) system call. Default memory bindings affecting all processes running on the system may be specified by the system administrator by altering the binding of the regions of the init(1M) process using rerun(1). This will typically be done as part of the /etc/rc script during system initialization. The binding of the text, data, stack, and PCB regions of a process to local memory is explicitly controlled by the memadvise(2) system call, or by using the run(1) or rerun(1) commands. memadvise(2) may also be used to query the current settings of the processes memory flags. The binding of shared memory regions to local memory is explicitly controlled by the value of the shmflg argument on a shmget(2) system call.

Binding a region to local memory always refers to the memory pool that is local to the CPU on which the process is executing. It is not possible to bind a region to a foreign memory. This implies that shared text, and by default, shared memory regions bound to local memory can only be shared by processes executing on CPUs that share the same local memory pool. Therefore, any region that is to be shared by processes that are to be executed on CPUs that do not share the same local memory pool should be loaded in global memory. It should be mentioned that there is a shmat(2) option which will allow a process to attach to a shared memory region that is bound to a foreign local memory pool. This option is not generally recommended, and is provided only as a special application support feature. If, during execution, a process changes the CPU on which it is executing via the mpadvise(2) system call, then the system must migrate any pages of regions attached by the process which were loaded in local memory to the local memory of the new CPU. A migrate of a shared text region from one local memory to another may require that a new region be created and new copies of the regions pages be loaded, thus mitigating some of the advantages of shared text regions. A shared memory region cannot be migrated. Therefore, migration of a process attached to a shared memory region will fail if the shared memory region is bound to local memory, unless the shmat(2) option which allows foreign memory attachments was previously specified by the process.

The binding of a particular region to local memory is considered to be either hard or soft. A soft binding allows pages to be allocated from within global memory when pages are not available for allocation within the specified local memory pool. If a binding for a particular region is "hard", then the calling process will be aborted if pages cannot be allocated from the local memory pool when needed. Use of hard bindings is recommended only when the contents of a given local memory pool are well known and strictly controlled, since failure of a memory allocation for a hard binding will abort the calling program.

Museum

Related Articles

memory(7)

NAME

DESCRIPTION

SEE ALSO