MochiWeb Walkthrough

Purpose

From the MochiWeb library's README:

"MochiWeb is an Erlang library for building lightweight HTTP servers."

The following is guided tour of the MochiWeb source code. By the end, you should understand how MochiWeb works, and how MochiWeb uses OTP. Naturally, you should then have a well-educated guess on how to build applications using MochiWeb.


Errors and Contribution

Please let me know if you have any errors to report or additions to make using the feedback form. I'll be very grateful to you for making this a more valuable resource, and I'll cite your contribution.

Changelog

2010.11.04
added feedback form, links to apps, and reworded a few sentences. published on drfloob.com today.
2010.10.02
this document is unfinished, but hopefully useful as is.

Mise en Place

It is assumed that you:

Most of the following will probably apply to non-Linux systems, but I have not tested anything in those environments. YMMV.

For your reference, this walkthrough is based on git commit 9a53dbd7b2c52eb5b9d4e90088ab471cac7b8ae9, from July 24th, 2010.

Getting Started

This section covers the new_mochiweb.erl script. If you're not interested, feel free to skip to the next section, where we begin discussing MochiWeb's server internals.

Creating a new mochiweb application is made easy by the scripts/new_mochiweb.erl script. It essentially copies the contents of the priv/skel dir into a new project dir, renaming the files and their contents to match the project name you give it at the command line.

For example, if we want to create a new mochiweb project called demo_mochiweb in the ~/proj folder:

aj@fattie:~/proj/mochiweb$ ./scripts/new_mochiweb.erl demo_mochiweb ~/proj
proj/demo_mochiweb/
proj/demo_mochiweb/support/
    include.mk
    run_tests.escript
    Makefile
proj/demo_mochiweb/src/
    skel.erl
    skel_app.erl
    skel.app
    skel.hrl
    skel_deps.erl
    Makefile
    skel_sup.erl
    skel_web.erl
    start-dev.sh
proj/demo_mochiweb/priv/
proj/demo_mochiweb/priv/www/
    index.html
    start.sh
    

This simple script is written in escript. The script's execution, from beginning to end, looks something like:

scripts/new_mochiweb.erl
  1. Checks to see if mochiweb has been compiled yet, and if not, compiles it
  2. Calls mochiweb_skel:skelcopy(Dest, Name)
mochiweb_skel:skelcopy/2
  1. Recursively copies each file from priv/skel to Dest (the location you specified on the command line), while renaming files and replacing in each file skel with Name (the name you gave on the command line).
  2. Symlinks the deps/mochiweb-src to the actual MochiWeb source.

cool erlang trick: In scripts/new_mochiweb.erl's main([Name, Dest]) function, the mochiweb source code is dynamically built and loaded without stopping/restarting the node itself. Look for code:rehash().

Assuming all went well, you now have a mochiweb project in ~/proj/demo_mochiweb, and we're ready to see it in action.

The Skeleton Application

The skeleton application starts a MochiWeb server on port 8000, and does nothing but display "MochiWeb running." We're now going to find out exactly how it accomplishes that feat.

In your project folder, you have two scripts that start your sever: start-dev.sh and start.sh

start.sh
start-dev.sh

The mochiweb skeleton comes with a simple Makefile that builds your erlang source and places it in the correct folder. If you're unfamiliar, you should read a bit about GNU's make tool sometime soon.

Excluding the make step, start-dev.sh accomplishes all of its goals using this one-liner:

exec erl -pa $PWD/ebin $PWD/deps/*/ebin -boot start_sasl -s reloader -s demo_mochiweb

Taking a few bits straight from The Erlang Emulator's man page, the options used are:

-pa Dir1 Dir2 ...
Adds the specified directories to the beginning of the code search path.
-boot File
Specifies the name of the boot file, File.boot, which is used to start the system. See init(3). Unless File contains an absolute path, the system searches for File.boot in the current and $ROOT/bin directories.

start_sasl is a boot script that come with Erlang/OTP. From the System Principles Guide, using the -boot start_sasl option "[l]oads the code for and starts the applications Kernel, STDLIB and SASL."

-s Mod [Func [Arg1, Arg2, ...]]
Makes init call the specified function. Func defaults to start. If no arguments are provided, the function is assumed to be of arity 0.

Execution

What's Running: -boot start_sasl

I don't fully understand the value of SASL yet. You're best off learning about it yourself ;) Here's the SASL User's Guide and Reference Manual @todo learn about sasl

What's Running: -s reloader

This calls reloader:start/0. Reloader is a gen_server whose stated goal is "automatically reloading modified modules during development." reloader:start/0 effectively starts the reloader, a gen_server outside any supervision tree, which polls for newly-compiled versions of every loaded module's source code, every 1 second, and loads new code if found. The code is very readable, and not complex; if you're interested, you should definitely read the code.

What's Running: -s demo_mochiweb

Finally, our code is getting executed! Specifically, demo_mochiweb:start/0 is called. The call stack looks roughly as follows:

demo_mochiweb:start/0
demo_mochiweb.app
demo_mochiweb_app:start/2 (application)
demo_mochiweb_sup:start_link/0 (supervisor)
demo_mochiweb_web:start/1

What's Running: MochiWeb Code

where we're at: Our application started the main supervisor, which then spawned demo_mochiweb_web:start/1 (which is not a gen_server, recall). Now, control is being passed to the MochiWeb source.

mochiweb_http:start/1
mochiweb_socket_server:start/1 (gen_server)
mochiweb_socket_server:start_server/1
mochiweb_socket_server:init/1
mochiweb_socket_server:listen/3
mochiweb_socket:listen/4
mochiweb_socket_server:new_acceptor_pool/2
mochiweb_acceptor:start_link/3
mochiweb_acceptor:init/3

Is your head spinning yet?

Your MochiWeb Skeleton app consists of a single application, supervisor, and gen_server.

The gen_server spawns a bunch of little connection acceptors that live for around 2 seconds.

Once a connection acceptor finishes its job (or times out), it lets the gen_server know it's done, and then dies.

The gen_server listens for acceptor deaths and respawns new acceptors under normal circumstances.

Code Listing

Referenced Erlang Modules you may not have toyed with yet


Request Processing

Recall that the body of our application, the 'Loop' function, was wrapped in a call to mochiweb_http:loop/2. From this wrapper function, the stream of incoming data is parsed into manageable data structures, and finally our 'Loop' is called.




Clearly, this is not finished yet




acknowledgements and thanks

Essays 1743 (discovered via Mark Pilgrim's Dive Into HTML5), Bob Ippolito (author of MochiWeb), Linux, GNU, emacs, git, inkscape.