| << Prev | - Up - | Next >> |
This chapter shows how to use the failure model to build robust distributed applications. We first present basic fault-tolerant versions of common language operations. Then we present fault-tolerant versions of the server examples. We conclude with a bigger example: reliable objects with recovery.
We will show how to define a simple open fault-tolerant broadcast channel. The example uses almost every concept seen in the former chapter. This is a useful abstraction; for example it can be used as the heart of a chat tool such as IRC. The service has a client/server structure and is aware of permanent crashes of clients or the server. In case of a client crash, the system continues to work. In case of a server crash, the service will no longer be available. Clients receive notification of this.
Users access the broadcast service through a local client. The user creates a client object that is initialized with a ticket provided by the server. The client object has a method sendMessage for broadcasting a message. When the client object receives a message or is notified of a client or server crash, it informs the user by calling a user-defined procedure with one argument. The following events are possible:
message(UserID Mess): receive the message Mess from client UserID.
registered(UserID): the client identified by UserID has registered to the channel.
unregistered(UserID): the client identified by UserID has unregistered from the channel.
permClient(UserID): the client identified by UserID has crashed.
permServer: the broadcast channel server has crashed.
We give an example of how the broadcast channel is used, and we follow this by showing its implementation. We first show how to use and implement a non-fault-tolerant broadcast channel, and then we show the small extensions needed for it to detect client and server crashes.
First we create the channel server. The server is a port. To connect with clients, the server is made available via a ticket with unlimited connection ability. The ticket is available through a publicly-accessible URL.
local
S={NewChannelServer}
in
{Pickle.save {Connection.offerMany S} "/usr/staff/pvr/public_html/chat"}
end
A client can be created on another site. We first define on the client's site a procedure HandleIncomingMessage that will handle incoming messages from the broadcast channel. Then we access to the channel by its URL. Finally, we create a local client and give it our handler procedure.
local
proc {HandleIncomingMessage M}
{Show {VirtualString.toString
case M
of message(From Content) then From#' : '#Content
[] registered(UserID) then UserID#' joined us'
[] unregistered(UserID) then UserID#' left us'
end}}
end
S={Connection.take {Pickle.load "http://www.info.ucl.ac.be/~pvr/chat"}}
C={New ChannelClient init(S 'Raphael' HandleIncomingMessage)}
in
{For 1 1000 1
proc {$ I}
{C sendMessage('hello'#I)}
{Delay 800}
end}
{C close}
endIn this example we send 1000 messages of the form 'hello'#I, where I takes successive values from 1 to 1000. Then we close the client.
Note that the client needs the code of the class ChannelClient. It is not difficult to avoid that requirement by making the server provide a functor, that the client applies, and which instantiates a ChannelClient. Please consult the Application Programming tutorial to learn more about that.
The whole client-server communication is implemented with ports. They have a well-defined behavior in the case of a permanent crash: the operation Send has no more effect. We will show the service's implementation in two steps. First, we show how it is written without taking fault tolerance into account. Second, we complete the example by adding fault handling code. This is easy; it amounts to add a few threads monitoring the fault state of the clients and server ports.
The client is an object with the following structure, while the server is a port created by the function NewChannelServer.
class ChannelClient
feat client userID
attr server
<Client methods>
end
fun {NewChannelServer}
Stream
Server={NewPort Stream}
<Server's serving procedures>
in
thread {Serve Stream} end
Server
end
The client has three methods. The first one, init, initializes a client with a server port, a user identifier, and a message handler procedure. The second one, close, deregisters the client from the service. The last one, sendMessage, sends a message for broadcast.
meth init(Server UserID MsgHandler)
Ms
in
@server = Server
self.client={NewPort Ms}
self.userID=UserID
{Send Server register(self.client self.userID)}
thread {ForAll Ms MsgHandler} end
<Client's server failure handler>
end
meth close()
{Send @server unregister(self.client self.userID)}
server := unit
end
meth sendMessage(M)
{Send @server broadcast(self.userID M)}
endThe client keeps a reference to the server, to itself for unregistering, and to its user identification. The user-defined handler procedure is directly applied to the stream of incoming messages from the server. In our case, the server only sends events as described above. As we can see, the server must be able to process three kinds of messages: register and unregister for managing a client's connection to the server, and broadcast for sending messages to all clients.
There is one statement in method init that is in charge of handling server failures. Currently we assume that statement is skip, which is the semantics of the handler in case of no failure.
A user should access the broadcast channel only through a client.
The server's procedure Serve must process client messages (at least). In order to avoid inter-client dependencies when forwarding messages, the server creates one thread per registered client, that processes incoming messages to the server, and sends events to that client only. That thread is created when the server observes a registration message from a client. This is possible because of the simplicity of the service.
proc {Serve X|T}
case X of register(Client UserID) then
{ServeClient Client UserID X|T}
else skip end
{Serve T}
end
<Server's simple ServeClient>
The procedure ServeClient creates one thread that sends appropriate event messages to a given client. Currently the thread processes the three messages sent by clients, namely register, unregister, and broadcast. Note how the client thread terminates automatically when its client unregisters: an exception is raised in the processing loop, and caught at the outer level. This makes the registration management implicit in the server.
proc {ServeClient Client UserID L}
proc {Loop X|T}
case X
of broadcast(U M) then {Send Client message(U M)}
[] register(C U) then {Send Client registered(C U)}
[] unregister(C U) andthen C==Client then raise done end
[] unregister(C U) then {Send Client unregistered(C U)}
end
{Loop T}
end
in
thread
try {Loop L} catch done then skip end
end
end
Note that clients can receive server messages at different rates. The server does not wait for client acknowledgement when sending messages. However, the order of messages is preserved because the ServeClient threads all read from the same sequential stream.
Note that clients are identified uniquely by references to the client object Client, and not by the client's user ID UserID. This is visible in the processing of the message unregister. This means that the channel will work correctly even if there are clients with the same user ID. The users may get confused, but the channel will not.
The fault-tolerant channel can be used in exactly the same way as the non-fault-tolerant version. The only difference is that the user-defined handler procedure can receive two extra messages, permClient and permServer, to indicate client and server crashes:
proc {UserMessageHandler Msg}
{Show {VirtualString.toString
case Msg
of message(From Content) then From#' : '#Content
[] registered(UserID) then UserID#' joined us'
[] unregistered(UserID) then UserID#' left us'
[] permClient(UserID) then UserID#' has crashed'
[] permServer then 'Server has crashed'
end}}
end
The non-fault-tolerant version of Section 5.1.2 is easily extended to detect client and server crashes.
This definition extends the definition given in Section 5.1.2. The client creates a concurrent failure handler that monitors the server's port with its fault stream. Whenever the server's port reaches state permFail, the client sends the message permServer to its own port. This message will be seen by the user message handler.
thread
if {List.member permFail {DP.getFaultStream S}} then
{Send self.client permServer}
server := unit
end
end
Notice that the attribute server is set to unit, like in method close. This makes every subsequent call to sendMessage on the client object fail with an exception.
We extend the server's definition to handle permanent failures from clients. For that purpose we modify a bit the Loop procedure in ServeClient, and add a failure handler that monitors the client's port.
proc {ServeClient Client UserID L}
proc {Loop X|T}
case X
of broadcast(U M) then {Send Client message(U M)}
[] register(C U) then {Send Client registered(C U)}
[] unregister(C U) andthen C==Client then raise done end
[] unregister(C U) then {Send Client unregistered(C U)}
[] permClient(C U) andthen C==Client then raise done end
[] permClient(C U) then {Send Client permClient(U)}
end
{Loop T}
end
in
thread
try {Loop L} catch done then skip end
end
thread
if {List.member permFail {DP.getFaultStream Client}} then
{Send Server permClient(Client UserID)}
end
end
end
The client thread in the server now also processes the message permClient. The thread that recognizes its own client port in the message raises the exception done to stop sending messages to that client, just like if the client was unregistered. Otherwise it simply notifies its client with the appropriate event. The failure handler created by ServeClient is responsible for sending the message permClient to the server's port. The message will be seen by all client threads on the server.
| << Prev | - Up - | Next >> |