OpenLambda

Quickstart

Because the Docker operations require root privileges, it is recommended to use OpenLambda in a virtual environment. The setup script has been tested in only Ubuntu 14.04. Once in your virtual environment as the root user, run these terminal commands (commands in bold may take minutes to finish). Start by grabbing the code and installing some prereqs:

wget https://github.com/open-lambda/open-lambda/archive/v0.1.0.tar.gz
tar -xf v0.1.0.tar.gz
cd open-lambda-0.1.0
bash ./tools/quickstart/startup.sh

Now build the Lambda engine, and start running it in a local cluster:

make imgs/lambda-node
./util/start-local-cluster.py

Finally, start a simple chat application:

./util/setup.py pychat chat.py
cat applications/pychat/static/config.json

Copy the URL from from the above command, and use it to send requests to the application. For example, try the following (replace "URL"):

curl -w "\n" URL -d '{"op":"init"}'
curl -w "\n" URL -d '{"op":"msg", "msg": "hello"}'
curl -w "\n" URL -d '{"op":"updates", "ts": 0}'

If you want to host static content with nginx (recommended): run the suggested command at the end of the previous command's output. To learn how to interact with the application, see the Issuing RPCs section of this documentation.

Architecture

The simplest OpenLambda deployment has three physical machines (currently Docker containers). One physical machine has an Nginx instance that serves as the load balancer. The second physical machine holds a registry of lambdas, which are Docker images containing the Python handler written by the developer. The third machine holds a lambda node, or node, which has a worker, a Docker instance, and a RethinkDB database shard. The web developer supplies Python handlers, which perform certain computations based on RPCs from the client-side of the web application. This Python, along with any data the web application may need, is stored in a Docker image that sits in the registry. When a user visits the website and does something to trigger the handler, an RPC is sent via AJAX from the JavaScript code. This RPC is routed through the Nginx load balancer, which will choose a node to handle the request. The node's worker is the recipient of the request. The worker pulls the appropriate lambda from the registry to the node (if the lambda is not already on the node) and then starts a Docker container from the lambda. The data from the RPC are fed to the lambda function (the Python code the developer wrote), and the appropriate answer is returned. Lastly, the worker responds to the initial RPC by sending information back to the client side. For more information on the OpenLambda architecture, see these slides for some helpful diagrams. Slides 56-64 show the OpenLambda architecture.

For Contributors (What's in the Github repo?)

Several of the directories in the Github repository are pertinent to understanding OpenLambda:

applications: contains applications written to showcase and provide a benchmark suite for the OpenLambda system. Currently, three applications are functional: pychat, pyocr, and pytix

lambda-generator: has the Python code for generating lambda nodes, including builder.py

nginx: contains a build of nginx for load balancing. It's also possible to serve an app's static content (web interface) with nginx

node: contains the Python code that builds the nodes of the OpenLambda network, namely builder.py

testing: contains some code for unit testing

tools: contains scripts to initialize environments in which OpenLambda can run

util: has scripts for starting a local cluster of nodes, stopping a local cluster of nodes, and building a lambda

worker: contains the Go code for the server-like part (worker) of a node

For Lambda Writers (How do apps work?)

Using the pychat app (/applications/pychat) as an example can help illustrate how apps run on OpenLambda. The pychat directory has a file called chat.py (the pychat app's lambda function), which contains a function called handler(). When a client-side RPC is made, this handler() function gets the data from the RPC, chooses the correct function to carry out the necessary computations, and returns the result. Looking at chat.py will help clarify this process. The pychat directory also has a subdirectory called static that contains the developer-written JavaScript code and HTML for the application web interface. Users can interact with the application via the web interface (by using Nginx to host static content) or via the terminal tool curl. For more on that, see the Issuing RPCs section. For more on the specifics of how OpenLambda handles RPCs, see these slides for some helpful diagrams. Slides 56-64 show the OpenLambda architecture, while slides 97-108 show how OpenLambda runs on one physical machine (like your computer).

Issuing RPCs (curl, browser)

There are two ways to interact with apps running on OpenLambda via RPCs: the command line tool curl, or the JavaScript webpage.

Curl

Curl is a command line tool for issuing RPCs. A sample curl call is

curl -w "\n" localhost:loadbalancer/runLambda/ imgname -d '{"op":"init"}'

where the bold parts of the address need to be replaced with specific information as follows. localhost needs to be replaced with the IP address of your localhost. loadbalancer needs to be replaced with the port that the Nginx load balancer is connected to. This port can be found by typing docker ps into the terminal and then looking at the second nginx container under the PORTS column, in the last of the three listings there. The port number should be around 32770. imgname can be found by typing docker images into the terminal and looking in the topmost row in the REPOSITORY column for a string of 12 random characters, which is imgname. Different apps take different RPC data (the section after the -d flag), and you can figure out how those work from the application's lambda function.

Browser

After setting up static content hosting with nginx using the command suggested at the end of the output from ./util/setup.py pychat chat.py, go to your web browser and enter IP:80, where IP is replaced with the IP address of your localhost. You'll see the web interface specified by the main.js and index.html files in the static subdirectory of the app that you are running, and you can experiment with the app.