MochiWeb Walkthrough

Purpose

"MochiWeb is an Erlang library for building lightweight HTTP servers."

The following is guided tour of the MochiWeb source code. By the end, you should understand how MochiWeb works, and how MochiWeb uses OTP. Naturally, you should then have a well-educated guess on how to build applications using MochiWeb.

Errors and Contribution

Please let me know if you have any errors to report or additions to make using the feedback form. I'll be very grateful to you for making this a more valuable resource, and I'll cite your contribution.

Changelog

2010.11.04: added feedback form, links to apps, and reworded a few sentences. published on drfloob.com today.
2010.10.02: this document is unfinished, but hopefully useful as is.

Mise en Place

It is assumed that you:

have a copy of the MochiWeb source code, and
are working at a bash shell, on a linux system

Most of the following will probably apply to non-Linux systems, but I have not tested anything in those environments. YMMV.

For your reference, this walkthrough is based on git commit 9a53dbd7b2c52eb5b9d4e90088ab471cac7b8ae9, from July 24th, 2010.

Getting Started

This section covers the new_mochiweb.erl script. If you're not interested, feel free to skip to the next section, where we begin discussing MochiWeb's server internals.

Creating a new mochiweb application is made easy by the scripts/new_mochiweb.erl script. It essentially copies the contents of the priv/skel dir into a new project dir, renaming the files and their contents to match the project name you give it at the command line.

For example, if we want to create a new mochiweb project called demo_mochiweb in the ~/proj folder:

aj@fattie:~/proj/mochiweb$ ./scripts/new_mochiweb.erl demo_mochiweb ~/proj
proj/demo_mochiweb/
proj/demo_mochiweb/support/
    include.mk
    run_tests.escript
    Makefile
proj/demo_mochiweb/src/
    skel.erl
    skel_app.erl
    skel.app
    skel.hrl
    skel_deps.erl
    Makefile
    skel_sup.erl
    skel_web.erl
    start-dev.sh
proj/demo_mochiweb/priv/
proj/demo_mochiweb/priv/www/
    index.html
    start.sh

This simple script is written in escript. The script's execution, from beginning to end, looks something like:

scripts/new_mochiweb.erl

Checks to see if mochiweb has been compiled yet, and if not, compiles it
Calls mochiweb_skel:skelcopy(Dest, Name)

mochiweb_skel:skelcopy/2

Recursively copies each file from priv/skel to Dest (the location you specified on the command line), while renaming files and replacing in each file skel with Name (the name you gave on the command line).
Symlinks the deps/mochiweb-src to the actual MochiWeb source.

cool erlang trick: In scripts/new_mochiweb.erl's main([Name, Dest]) function, the mochiweb source code is dynamically built and loaded without stopping/restarting the node itself. Look for code:rehash().

Assuming all went well, you now have a mochiweb project in ~/proj/demo_mochiweb, and we're ready to see it in action.

The Skeleton Application

The skeleton application starts a MochiWeb server on port 8000, and does nothing but display "MochiWeb running." We're now going to find out exactly how it accomplishes that feat.

In your project folder, you have two scripts that start your sever: start-dev.sh and start.sh

start.sh

Adds the */ebin folders to the path
starts SASL
calls demo_mochiweb:start/0

start-dev.sh

does everything start.sh does
calls make to compile the source before starting the applications
also calls reloader:start/0 before starting the demo_mochiweb application

The mochiweb skeleton comes with a simple Makefile that builds your erlang source and places it in the correct folder. If you're unfamiliar, you should read a bit about GNU's make tool sometime soon.

Excluding the make step, start-dev.sh accomplishes all of its goals using this one-liner:

exec erl -pa $PWD/ebin $PWD/deps/*/ebin -boot start_sasl -s reloader -s demo_mochiweb

Taking a few bits straight from The Erlang Emulator's man page, the options used are:

-pa Dir1 Dir2 ...: Adds the specified directories to the beginning of the code search path.
-boot File: Specifies the name of the boot file, File.boot, which is used to start the system. See init(3). Unless File contains an absolute path, the system searches for File.boot in the current and $ROOT/bin directories.
start_sasl is a boot script that come with Erlang/OTP. From the System Principles Guide, using the -boot start_sasl option "[l]oads the code for and starts the applications Kernel, STDLIB and SASL."
-s Mod [Func [Arg1, Arg2, ...]]: Makes init call the specified function. Func defaults to start. If no arguments are provided, the function is assumed to be of arity 0.

Execution

What's Running: `-boot start_sasl`

I don't fully understand the value of SASL yet. You're best off learning about it yourself ;) Here's the SASL User's Guide and Reference Manual @todo learn about sasl

What's Running: `-s reloader`

This calls reloader:start/0. Reloader is a gen_server whose stated goal is "automatically reloading modified modules during development." reloader:start/0 effectively starts the reloader, a gen_server outside any supervision tree, which polls for newly-compiled versions of every loaded module's source code, every 1 second, and loads new code if found. The code is very readable, and not complex; if you're interested, you should definitely read the code.

What's Running: `-s demo_mochiweb`

Finally, our code is getting executed! Specifically, demo_mochiweb:start/0 is called. The call stack looks roughly as follows:

demo_mochiweb:start/0

calls demo_mochiweb_deps:ensure/0, which adds any ebin or include folders that are not already on the path to the VM's code path.
makes sure that the crypto application is started (exits badly otherwise).
Starts the demo_mochiweb application (our app!)

demo_mochiweb.app

Instructs OTP to start the demo_mochiweb_app defined in demo_mochiweb_app.erl

demo_mochiweb_app:start/2 (application)

(once again) ensures all dependency ebins are on the code path @todo find reason for calling ensure again? find discussions? file bug report?
starts the demo_mochiweb_sup Supervisor

demo_mochiweb_sup:start_link/0 (supervisor)

instructs OTP to call demo_mochiweb_web:start/0 (which is not a gen_server, as it turns out) as a permanent child process, one_for_one restart policy, with a treshold of 10 restarts in 10 seconds (exit time threshold of 5000ms before it is mercifully killed). This is also where the web server is configured, namely given the port and docroot to operate on.

demo_mochiweb_web:start/1

defines a callback function Loop for mochiweb. In our case, this loop function calls demo_mochiweb_erl:loop/2, which responds by serving files out of the docroot. Can you guess where "MochiWeb running" came from now? Answer
calls mochiweb_http:start/1 with our callback function, among other options.

What's Running: MochiWeb Code

where we're at: Our application started the main supervisor, which then spawned demo_mochiweb_web:start/1 (which is not a gen_server, recall). Now, control is being passed to the MochiWeb source.

mochiweb_http:start/1

parses options, setting defaults for missing vars.
wraps our 'Loop' callback function in a call to mochiweb_http:loop(Socket, HttpLoop), where HttpLoop is our 'Loop' function
returns the value of a call to mochiweb_socket_server:start/1, called with the set of parsed options.

mochiweb_socket_server:start/1 (gen_server)

parses the options into a #mochiweb_socket_server record.
returns the value of a call to :start_server/1, called with the #mochiweb_socket_server record it built from our options

mochiweb_socket_server:start_server/1

if using SSL, starts crypto @todo, wasn't that done already? why again?, public_key, and ssl applications if ssl is used
returns the call into gen_server:start_link/3 that starts itself.

mochiweb_socket_server:init/1

traps exits
configures options for :listen/3 (@todo fill in purpose of options)
calls :listen/3, and decides what to do/return based on its output. @todo quickly describe the decision and its effects. unprivileged ports = stop gen_server.

mochiweb_socket_server:listen/3

calls mochiweb_socket:listen/4
if that returned alright, returns a newly created acceptor pool (see below).
otherwise, the server is stopped.

mochiweb_socket:listen/4

Returns a socket created via gen_tcp:listen/2 (or ssl:listen/2 if using SSL)

mochiweb_socket_server:new_acceptor_pool/2

creates N 'acceptors' by calling mochiweb_acceptor:start_link/3, where N in this case is the default Size is set in the record definition as 16.
Returns the new "State" where acceptor_pool is the set of Pids of the newly created acceptors

mochiweb_acceptor:start_link/3

spawn_links to :init/3 @todo why proc_lib:spawn_link instead of spawn_link or gen_server:start_link?

mochiweb_acceptor:init/3

listens on the socket for 2 seconds via gen_tcp:accept/1 (ssl:ssl_accept if ssl)
if no connection arrives, the process exits
if a connection arrives, our gen_server from mochiweb_socket_server is cast the {accepted, ...} message, and the socket is then parsed into a request via :call_loop/2, which effectively returns the value from a call to our 'Loop' function.

Is your head spinning yet?

Your MochiWeb Skeleton app consists of a single application, supervisor, and gen_server.

The gen_server spawns a bunch of little connection acceptors that live for around 2 seconds.

Once a connection acceptor finishes its job (or times out), it lets the gen_server know it's done, and then dies.

The gen_server listens for acceptor deaths and respawns new acceptors under normal circumstances.

Code Listing

Referenced Erlang Modules you may not have toyed with yet

gen_tcp
sets

Request Processing

Recall that the body of our application, the 'Loop' function, was wrapped in a call to mochiweb_http:loop/2. From this wrapper function, the stream of incoming data is parsed into manageable data structures, and finally our 'Loop' is called.

Clearly, this is not finished yet

acknowledgements and thanks

Essays 1743 (discovered via Mark Pilgrim's Dive Into HTML5), Bob Ippolito (author of MochiWeb), Linux, GNU, emacs, git, inkscape.