SCI: Scalable Coherent Interface

ion layer, but that it is also impossible to serve all fields of use at once. For example, MPI is a message passing standard that could still benefit from the low message passing overhead and latency that SCI provides. In the case of MPI, some interface or middle layer software is required to match the IEEE API to the MPI API. However, such middle layer or meta driver has to be implemented only once in order to gain complete hardware independence for any SCI based MPI system if the IEEE API is used as underlying software standard. Given this standard being submitted to IEEE and its scope as a hardware abstraction layer, it did not seem appropriate to try to define, for example, a standard address resolution protocol or an SCI network topology implementing failover functionality within the scope of this standard. It is left to the higher level software to implement those functions. However, the SCI Physical Layer API implements standard methods required to support the implementation of such functionality. 10.2 SCI Physical Layer API Architecture and Features This chapter gives an overview of the SCI Physical Layer API. For a detailed reference, please refer to the draft standard document itself. Figure 10.1 below shows its overview. Fig. 10.1. A sketch of the SCI Physical Layer API Architecture. The various data paths (1) through (5) are detailed in the text. A user level process interfaces to the API in order to perform all procedural functions. Depending on the operating system, the API may consist of user level routines and kernel level routines. For example, in the case of initialization and mapping calls, the user level API routines would call the protected kernel level API routines (3) which would perform any authentication and security checking in order to ensure that no unauthorized physical 194 V. Lindenstruth, D. Gustavson access is granted. Responses to API procedure calls and any exceptions are returned to the user level API routines (4). Initialization such as cold start and configuration of the SCI interfaces’ address maps is typically also executed in the API kernel routines (5). This is necessary in order to ensure that no user program can directly access the security relevant address translation tables in the interface. The SCI Physical Layer API requires functionality which is not supported by the hardware to be emulated in software. For example, should the interface not support DMA functionality, the API is required to implement that as programmed I/O. This functionality is expected to reside in the user level part of the API (2). Once a shared memory is set up, the user level process can perform shared memory transactions without involvement of the API as indicated by path (1). This feature allows the smallest overhead and latency. Two general classes of transactions are supported as indicated in Figure 10.2 below. Fig. 10.2. A sketch of a synchronous transaction (left) with a potential exception, and a sketch of an asynchronous transaction (right) Synchronous transactions such as read transactions (Figure 10.2, left) return after valid read data is available. This results in potential stalling of the host processor. If an error occurs, an exception is fired (refer to Section 10.2.1). The second class of transactions are asynchronous transactions, which return as soon as possible without awaiting the completion of the actual SCI transaction. This is illustrated in the right half of Figure 10.2. Upon completion of the asynchronous transaction, a specified call-back procedure is executed. The body of this procedure can be used to implement whatever synchronization method is desired. Examples of asynchronous transactions are posted writes or DMA transactions. In the case of posted writes the call-back procedure would typically act only if an error occurs. In the case of a DMA transaction, the call-back would notify the host about the DMA completion status. Before an SCI transaction can be executed, a certain amount of setup and control is required. For example, address translation tables may need to be configured appropriately in order to map an SCI address region into the 10. SCI Physical Layer API 195 process address space. This is implemented based on windows. A window is a contiguous address region with a defined set of default transaction attributes and configurations. Those default transaction attributes define whether write transactions may be posted, a window is write protected, or write transactions to this given window are executed as broadcast, etc. The appropriate address translation setup of both the operating system and the SCI interface is part of the configuration. 10.2.1 Exception Handling The SCI Physical Layer API interfaces to external hardware and consequently requires a method for asynchronously handling exceptions, such as link failures or asynchronous conditions such as failing posted writes, DMA completion, etc. Since this standard is required to be operating-system independent, a very simple method is supplied for implementing asynchronous attention handlers. It is based on the definition of a context structure, allowing the complete description of the state of the SCI hardware and software interface. For an asynchronous attention condition, a call-back procedure is executed which allows user supplied implementation of the exception handler. The appropriate procedure is supplied as a procedure pointer by the higher software layers and acts much like an interrupt handler. Some transaction specific exceptions cannot be traced to the calling process(or). For example, a posted write transaction exception may occur long after the write request is completed. Other posted writes may have been executed in the meantime. In this case it is not possible to determine the appropriate transaction specific call-back procedure. Therefore, a global exception handler is implemented which will be executed in such a case. However, the global attention handler can only identify the type of condition, based on the context structure. Therefore, in order to debug and trace these conditions, posted transparent writes must be disabled. Asynchronous transactions such as the chained-mode DMA use call-back procedures to provide a tool to synchronize with the host program. This is done by using whatever synchronization method is supported best by the given operating system (signals, events, semaphores etc.). In order to allow implementation of checkpoints, a synchronization transaction is provided, which stalls the calling process until all pending transactions have completed and all related potentially pending transaction handlers were executed, or a specified timeout expired.