NAME

hrm − HVME Reflective Memory (HRM)

DESCRIPTION

HRM is a single sized HVME board which plugs into the standard HVME backplane.  HRM is a high speed reflective memory board that is used to "cluster" multiple systems together. 
 
The front edge of the HRM board connects to a flat cable assembly through a front row of connectors.  There are 3 cables that are used for high speed data writes between HRM boards.
 
Each HRM board can be thought of as a node on a network of HRM nodes, where each node contains its own physical copy of memory.  Data written to reflective memory will be broadcast on the cable, which causes memory on every HRM node to be updated.  Data read from a HRM node’s memory space will be accessed locally, without the need for any cable broadcasting.
 
Processes will read and write to reflective memory by attaching to a shared memory region that is bound to the physical memory on the HRM board.
 
As a method for synchronizing access to reflective memory, the HRM board supports an inter-node interrupt mechanism.  A HRM node may send an interrupt to any of the other HRM nodes that are connected on the same cable. The sending HRM node can send this interrupt to more than one HRM node at the same time.  The inter-node interrupt is sent/broadcast across the HRM cable in much the same way and order that reflective memory writes are sent. Therefore, if a process writes to reflective memory before sending an inter-node interrupt to the other HRM node(s), it is guaranteed that the HRM node(s) receiving the inter-node interrupt will have already received the reflective memory writes from the interrupting HRM node.  There are two ioctl(2) commands available for sending these interrupts, and one ioctl(2) command available which will allow a single process to specify a signal to be sent to itself when an inter-node interrupt arrives. 
 
The HRM device driver supports user-level interrupt routine connections to the inter-node interrupt via the IOCTLVECNUM ioctl () system service call.  For more information on user-level interrupt routines, see iconnect(2). 

Limitations

The following is a list of certain HRM limitations:

Although the read/modify/write type of instructions may be executed on data within reflective memory, these instructions will not be supported.  Since these instructions also hold the backplane until the entire read/write operation completes, there is a system wide performance penalty for using these instructions.  Therefore, use of these test and set instructions in reflective memory is greatly discouraged.  Examples of CX/UX features that make use of test and set instructions are binary semaphores (struct binsem) and user-level spin locks (struct spin_mutex). 

In general, writes to reflective memory will remain consistent across all HRM nodes in the configuration.  In other words, any particular location in reflective memory will usually contain the same value across all HRM nodes.  However, when two or more HRM nodes execute a write instruction to the same reflective memory location at relatively the same point in time, it is possible for the resulting values in that memory location to be different on various HRM nodes.  Therefore, it is recommended that applications coordinate write operations to memory locations that are written by multiple HRM nodes. 

A HRM board may not send an inter-node interrupt to itself while in normal (non-diagnostic) operational mode. 

Although the HRM board supports asynchronous block transfers (DMA), all transfers must use A32 addressing.  Therefore, only (H)VME controllers that support A32 addresses can be used to DMA directly into and out of reflective memory. 

A DMA read or write operation that spans across two HRM boards is not supported. 

Configuration Specifications

The following configuration specifications apply:

A maximum of eight HRM nodes per configuration. 

If one or more nodes in the configuration are not up, the rest of the nodes will continue to function properly. 

A maximum of two boards in one system.  (These boards must belong to different HRM networks.) 

A maximum total cable length of 180 feet. 

It is highly recommended that all nodes within a configuration be set to the same physical address range. 

Memory

HRM boards will contain either 4 or 16 mbytes of DRAM.  If both 4 and 16 mbyte HRM boards are used within the same network, then the bottom 4 mbytes of the 16 mbyte HRM boards will function properly with the 4 mbyte HRM boards.  Writes to addresses above 4 mbytes on the 16 mbyte HRM boards will not be written to the 4 mbyte HRM board’s memory. 
 
The HRM supports a configurable physical memory address range.  The address range is specified via a addrsel field in the HRM control register. The ranges are as follows:

4 mbyte HRM16 mbyte HRM
addrsel  address rangeaddrsel  address range
  0E2000000 - E23FFFFF  0E2000000 - E2FFFFFF
  1E2400000 - E27FFFFF  1E2000000 - E2FFFFFF
  2E2800000 - E2BFFFFF  2E2000000 - E2FFFFFF
  3E2C00000 - E2FFFFFF  3E2000000 - E2FFFFFF
  4E3000000 - E33FFFFF  4E3000000 - E3FFFFFF
  5E3400000 - E37FFFFF  5E3000000 - E3FFFFFF
  6E3800000 - E3BFFFFF  6E3000000 - E3FFFFFF
  7E3C00000 - E3FFFFFF  7E3000000 - E3FFFFFF

The system initialization code in the hrm driver will always set the hrm0 board to an address select value of 0 (E2000000) and the hrm1 board to an address select value of 4 (E3000000).  It is not usually necessary to modify these defaults, but it is possible to do so via the hrmconfig(1M) program. 
 

Errors

Error detection and correction will be supported for the reflective memory DRAMs on single bit errors (these errors are automatically corrected by the HRM board).  When single bit errors occur, the HRM board will generate a HRM parity error interrupt.  These errors will be logged to the system error log. 
 
The HRM board is also able to detect HRM cable parity errors.  If a cable parity error should occur, a HRM parity error interrupt will be generated, and this error will be logged to the system error log.  This error is fatal to the HRM subsystem.  Therefore, error messages will be output to the system console and the HRM board will be reset, disconnected from the HRM cable, and placed in a disable state.
 
The errpt utility will output any logged reflective memory or cable parity errors when the -a (all) or -dmem (memory) options are used (see errpt(1M) for details). 
 
Multiple bit errors (which are not correctable) will be reported via a system fault exception on Series 4000 HN4400 systems, via a m-bus error exception on Series 4000 HN4800 systems, and via a system fault exception on Series 5000 HN5800 systems.  In all cases, multiple bit errors are fatal to the system (a panicp() will occur). 
 

CONFIGURATION

HRM boards installed in the HVME backplane need to be defined in the system configuration file.  The following lines should be uncommented (or entered) in the system configuration file for the system:

...
driver hrm  cdev 84 vector hrmnode hrmintr
...

The cdev major device number value should be a unique value within the range specified in the configuration file for a HCSD character KIDDI driver. 

...
#
# HVME Reflective Memory
# Uncomment only those lines which represent a hrm device which
# actually exists on your system.
#
# device hrm0at vba? csr ?   vector hrmnode hrmintr
# device hrm1at vba? csr ?   vector hrmnode hrmintr
...

The driver and device hrm0 lines should always be uncommented, but the second device hrm1 line should be uncommented only if two HRM boards are plugged into the system.  When two HRM boards are plugged into the system, it is highly recommended that the "csr ?" fields be replaced by the actual slot numbers that each of the HRM boards are in.  This will provide a clear mapping between each HRM special file and HRM board.  If the "csr ?" fields are not replaced, then the HRM board in the lowest slot will correspond to hrm0 and the HRM board in the higher slot will correspond to hrm1. 

Special Files, Minor Device Numbers

HRM device special file names have the form /dev/hrm#, where the character "#" stands for the minor device number (0-1).  For HRM devices, the minor device number directly corresponds to a particular HRM board.  The mapping of special file to HRM device is specified in the device hrm[0-1] entries in the configuration file. 
 
For each HRM board plugged into the HVME backplane, there will need to be a corresponding special device file in the /dev directory.  This special file will allow /etc/hrmconfig (see hrmconfig(1M)) or other programs to open(2) a HRM board and issue the ioctl(2) configuration commands to the board.  The special device file(s) for reflective memory board(s) can be created by the following command sequence while in the /dev directory: MAKEDEV [-f configuration_file] hrm0 [hrm1]

The hrm1 parameter on the MAKEDEV command needs to be provided only if there are two reflective memory boards configured in the system. 

User Interface

A HRM board is controlled by open(2), close(2), and ioctl(2) system calls:

The open(2) call assigns a file descriptor to the HRM device, /dev/hrm#.  Only one assignment may be made to a HRM board at any given point in time. 

A close (2) call frees the file descriptor, and thus allows another process to assign to the HRM device. 

The ioctl(2) call provides a way to configure the HRM device, and to also obtain the current configuration information from the HRM device.  All ioctl(2) calls use the definitions in <sys/hrm.h>. 

Note that this device does not support the read (2) and write (2) system calls. 

Ioctl Calls

The following ioctl(2) calls have the format:

#include <sys/types.h>
#include <sys/hrm.h>
 ioctl(fildes, command, 0)
int fildes, command;

Where command may be:

HRM_SETSHORTCBLSet the short cable parameter on the HRM board.  If this command is specified, then the entire cable length of the configuration must be less than 80 ft.  Although this command will provide shorter arbitration times in the reflective memory configuration which may result in higher performance, if the cable length is actually greater than 80 ft., the HRM system will not function properly.  If this command is used, then all reflective memory boards connected to this cable should also be configured with this parameter. 

HRM_CLRSHORTCBLClear the short cable parameter on the HRM board.  (Total cable length is between 80 and 180 ft.)  After a system boot, the HRM board defaults to a long cable state. 

The following ioctl(2) calls have the following format:

#include <sys/types.h>
#include <sys/hrm.h>
 ioctl(fildes, command, arg)
int fildes, command;
int ∗arg;

Where command may be:

HRM_SETADDRThis command sets the addrsel (address select) bits on the HRM board.  This parameter controls the physical memory address range of the memory located on the HRM board.  The user address of the addrsel value is specified in the arg parameter on the ioctl call.  This user address should contain an integer ranging from 0 to 7.  After a system boot, this parameter defaults to an addrsel value of 0 for /dev/hrm0, and a value of 4 for /dev/hrm1.  The memory address ranges that a HRM board covers depends upon both the addrsel select value and the amount of memory that is located on the HRM board.  The possible addrsel values and their corresponding physical memory ranges are defined in the DESCRIPTION section on Memory.  If two boards are configured on a single system, the address ranges of the two boards should NOT overlap. 

HRM_SETNODEIDThis command sets the node identification parameter on the HRM board.  The arg parameter on the ioctl call defines the user address of the node id, where node id is a number between 0 and 7.  This parameter must be defined to the HRM board before the HRM board will be placed into an enabled and running state.  All HRM boards that are connected via the same set of reflective memory cables must have unique node id values. 

HRM_SENDINTThis command sends an interrupt to the HRM node(s) specified in the user address location contained in arg.  Bits 7-0 of this memory location are an inter-node bit mask, where bit 0 corresponds to node 0, and bit 7 corresponds to node 7.   Any HRM node whose bit is set in the arg mask will be sent an HRM inter-node interrupt. 

HRM_ATTACH_SIGNAL
This command defines a signal number that will be sent to the calling process whenever an inter-node interrupt arrives.  The arg parameter on the ioctl call defines the user address of the signal number, where signal number is a either a valid signal number or zero.  A value of zero disables the sending of the previously defined signal.  If the process executes a close(2) on the HRM device file descriptor, then the signal will no longer be sent to the process. 

The following ioctl(2) call has the following format:

#include <sys/types.h>
#include <sys/hrm.h>
 ioctl(fildes, command, arg)
int fildes, command;
struct hrm_info ∗arg;

Where command is:

HRM_GETSTATUSThis command returns the current configuration status for the HRM board in the hrm_info structure whose address is specified via the arg parameter.  See <sys/hrm.h> for details on the hrm_status structure contents. 

The following ioctl(2) call has the following format:

#include <sys/types.h>
#include <sys/hrm.h>
 ioctl(fildes, command, arg)
int fildes, command;
int ∗arg;

Where command is:

HRM_VECTORThis command returns the interrupt vector number that is being used for the inter-node interrupt.  This value is returned to the user in the user address specified by arg.  This information is also available as part of the information returned in the HRM_GETSTATUS ioctl(2) command.  For compatibility reasons, this command has the same value as the IOCTLVECNUM command. This command may be used to obtain the vector number to be used on an iconnect(2) system service call, which allows a process to directly connect a user-level routine to the inter-node interrupt. 

The following ioctl(2) call has the following format:

#include <sys/types.h>
#include <sys/hrm.h>
 ioctl(fildes, command, arg)
int fildes, command;
caddr_t ∗arg;

Where command is:

HRM_VECTORADRThis command returns the physical address of the HRM board’s inter-node interrupt register.  This value is returned in the user address specified in arg.  This command is provided so that a program may bind the inter-node interrupt register into its address space, and then write directly to this register to send interrupts to other HRM node(s). 

FILES

/dev/hrm[0-1]      HRM character device special files

ERROR CODES

In addition to the possible error codes that may be returned from open(2), close(2) or ioctl(2) calls, the following errors may be returned:

[ENXIO] The HRM board does not exist, or did not initialize properly. 

[EBUSY] The HRM device is currently in use (already opened). 

SEE ALSO

hrmconfig(1M), config(1M), open(2), close(2), ioctl(2), iconnect(2). 

CX/UX Administrator’s Reference