HotSwap Code using Scala and Actors

Update: In this article I am showing an even more powerful way of doing hotswap in Scala. ---- In this post I will show you how you can do code hotswap in the same fashion as in Erlang using Scala and its Actors package. An actor is an abstraction that implements Message-Passing Concurrency. Actors have no shared state and are communicating by sending and receiving messages. This is a paradigm that provides a very different and much simpler concurrency model than Shared-State Concurrency (the scheme adopted by C, Java, C# etc.) and is avoiding all of the latter one's problems with deadlocks, live locks, thread starvation etc. This makes it possible to write code that is deterministic and side-effect-free, something that makes easier to write, test, understand and reason about. Each actor has a mailbox in which it receives incoming messages and can use pattern matching on the messages to decide if a message is interesting and which action to take. The most well known and successful implementation of actors can be found in the Erlang language (and the OTP platform) where it has been used to implement extremely fault tolerant (99.9999999% reliability - 9 nines) and massively concurrent systems (with hundreds of thousand simultaneous actors). Let's start by writing a little server. We implement this in the form of a trait, which is Scala's mixin construct. Traits allows you to build up your components using so-called mixin composition which is something that can give you a very high grade of reuse and flexibility. This trait only defines a single method named status which prints out info about the enclosing instance. Completely useless and not much for a server, but it will give you the idea. Then we subclass this mixin and define the ServerOne concrete server class (with the status method mixed in).

// servers

trait Server {
  def status = println("current server: " + this)
}
class ServerOne extends Server

Let's instantiate the ServerOne class and see what the status method it will print out. Here we are doing it interactively through Scala's REPL (read-eval-print-loop).

$ scala -cp .

scala> val server = new ServerOne
server: ServerOne = ServerOne@7be75d

scala> server status
current server: ServerOne@7be75d

Now, before we write the actor we have to define the messages it responds to. Here Scala is using something called case classes which are similar to normal classes but with some enhancements. First you can match on them, e.g. use pattern matching similar to the one found in Erlang. They also have some syntactic sugar, f.e. you can create them without using new, the compiler generates getters and setters for the constructor arguments, equality is not based on object id but on meaning/content (something that makes them ideal to use for value objects, but that is another story). We define two different messages; Status and HotSwap.

// triggers the status method

case object Status

// triggers hotswap - carries the new server to be hotswapped to

case class HotSwap(s: Server)

Ok, now it is time for the actual actor. Actor is a base class in the Scala library and we can choose to either extend it explicitly or to create an anonymous actor through the actor {...} construct. When we subclass the actor we have to implement the method called act which is the callback method that is invoked when the actor is started up. Scala comes with two different implementations; one that is based on Java threads in that each actor is getting its own thread, while the other one is based on events and very lightweight allowing hundreds of thousands of actors to run simultaneously. Here we will use the event-based version (which is done by using the react method instead of the receive method for receiving messages). The trick to do hotswap by using actors is to loop recursively and pass on the state in each recursive call. This is a very common idiom in functional programming. The beauty of it is that we do not update any mutual state but our execution is side effect free which makes it easier to test and reason about. In this case our state is the actual server. We start the loop by instantiating ServerOne. The pattern matching is happening in react statement in which we have three different cases (pattern matchers). The first one matches our Status, when we receive this message we simply invoke the status method on our server and then taking another round in the loop passing along the server. The second one matches our HotSwap message. It is here things are starting to get interesting. Now we can take the new replacement server (here called newServer), which is passed to us as an argument to the HotSwap message, and pass it in to the call to the loop method. Voila, we have updated our server at runtime. Now all subsequent messages will act on our new server instance. This will work since the react method will in fact never return but infinitely recur. Infinite recursion would have been a problem in f.e. Java since each recursion would consume a new stack frame until we run out of memory. But recursion is one of the most powerful and commonly used idioms in functional programming and the Scala compiler optimizes tail-recursive algorithms and turns them into regular loops. At the end we have also added a match-all pattern that does nothing, this is defined by the case _ => ... clause. Let's take a look at the code.

class ServerActor extends Actor {
  def act = {
    println("starting server actor...")
    loop(new ServerOne)
  }

  def loop(server: Server) {
    react {
      case Status =>
        server.status
        loop(server)

      case HotSwap(newServer) =>
        println("hot swapping code...")
        loop(newServer)

      case _ => loop(server)
    }
  }
}

Finally we will follow one of Scala's idioms and create a companion object for our ServerActor class. In this object, which is a singleton but should be seen upon as a module for functions and immutable state, we define an immutable handle to an instantiated and started actor. Worth to note is that the val holding our actor is initialized when the enclosing object is first used, and since we are starting up the actor in the initialization block of the val, the actor will not be started until it is used for the first time.

// actor companion object

object ServerActor {
  val actor = {
    val a = new ServerActor
    a.start; a
  }
}

Let's try to run it in the Scala REPL. The Scala function ! (pronounced bang) means "send a message". So act ! msg means send message msg to actor act.

$ scala -cp .

scala> import hotswap._
import hotswap._

scala> val actor = ServerActor.actor
starting actor...
actor: hotswap.ServerActor = hotswap.ServerActor@528ed7

scala> actor ! Status
current server: hotswap.ServerOne@226445

scala> class ServerTwo extends Server {
     | override def status = println("hotswapped server: " + this)
     | }
defined class ServerTwo

scala> actor ! HotSwap(new ServerTwo)
hot swapping code...

scala> actor ! Status
hotswapped server: line5$object$$iw$$iw$$iw$ServerTwo@b556

Pretty cool, right? This would be even more cool if Scala came with an SSH server that could provide this REPL remotely (like we have in Erlang OTP). Then we could connect to our application from the outside and change its behavior, fixing bugs, upgrade the server etc. Another solution would be to make use of the remote actors in the Scala distribution, but that is something for another post. --- Update: This is a slightly simplified version of the GenericServer code (as discussed here), allowing hotswap of arbitrary pieces of pattern matching code:

// the actor's main loop

def act = loop { react { genericBase orElse actorBase } }

// should we go with the hotswapped impl or the regular server impl (body)

private def actorBase: PartialFunction[Any, Unit] = hotswap getOrElse body

// the hotswapped impl

private var hotswap: Option[PartialFunction[Any, Unit]] = None

// generic functionality

private val genericBase: PartialFunction[Any, Unit] = {
  case HotSwap(code) => hotswap = code
}

// the regular server implementation

def body: PartialFunction[Any, Unit] = {
  ...
}

This code can be used like this:

server ! HotSwap(Some({
  case Ping =>
    println("Ping")
}))

---