My Project: HTTP/1.1 Compliance Testing and User-Agent Development for the Mojo Web Framework

[Note: The following is a slightly abbreviated version of the proposal I submitted.]

Abstract

When building Web applications, developers expect the tools they use to be fully interoperable through clean protocol implementations. This project aims to ensure full HTTP/1.1 compliance in the Mojo Web Framework, through both whitebox and blackbox testing. Time permitting, Mojo’s client code will also be exercised through the development of a smart User-Agent similar to LWP’s.

Benefits to the Perl/Open Source Community

Ensuring protocol compliance through extensive testing will help speed the adoption of Mojo, the next generation Web framework, by boosting developer confidence in the code’s correctness and robustness. Frameworks like Mojo [1] and Catalyst [2] bring cutting edge Web development to Perl, and enrich the palette of tools that the Open Source movement offers to Web developers. TIMTOWTDI, and this project will strengthen the position of Mojo, Perl and the Open Source movement in the world of Web development.

Deliverables

1. Whitebox tests using Test::More [3] and integrating into Mojo’s current testing framework. Most tests will be concentrated on the Mojo::Message class and focus on HTTP/1.1 [4] content parsing, with emphasis on edge cases.
2. A blackbox test suite to run against Mojo’s built-in server using an appropriate HTTP/1.1 client library (most likely libcurl [5]). Tests to include some cases of adversarial stance (e.g. deliberately malformed requests).
3. If necessary, patches to Mojo that enable it to pass the test suites.
4. If time permits, a smart User-Agent class (MojoX::UserAgent) similar to LWP’s [6] and exercising all features of HTTP/1.1 supported by the Mojo::Client class.
5. If time permits, a port of the test suite implemented in "2." (above) using the new MojoX::UserAgent.

Project Details

Description & Goals
When it comes to protocol implementations, the devil is in the details, and edge cases can trigger bugs that are very difficult to trace for end-users. Therefore, thorough test coverage is paramount to instill user confidence. Weighing in at 176 pages, RFC2616 (the HTTP/1.1 specification document) defines a fairly complex protocol. As a result, many implementations are limited to a small subset of the functionality defined in the specification. However, the Mojo Web development framework explicitly aims to provide a “[f]ull stack HTTP 1.1 client/server implementation” [7].

This project’s primary goal is thus to increase the test coverage of Mojo against the HTTP/1.1 specification, through both whitebox and blackbox testing. A secondary goal is to leverage the advanced protocol features implemented by Mojo to develop a compelling User-Agent class that may (hopefully) outperform other similar Perl solutions.

Preliminary Investigations
Preliminary investigations for this project were conducted in the week of March 23-30th. It was found that Mojo already implements a nice whitebox testing framework (using Test::More), but that the coverage of the test cases needs to be improved. I have already submitted a small patch and test case to implement an HTTP/1.1 "SHOULD" requirement. Preliminary blackbox testing has uncovered a major issue with request pipelining that has lead Sebastian Riedel to embark into a significant refactoring of Mojo’s HTTP handling core.

Risks & Risk Mitigation
This project is dependent on the refactoring of some of Mojo’s core code, which has been undertaken by Sebastian Riedel in order to implement the main feature found to be broken in preliminary investigations (see above). This refactoring both makes the need for thorough testing more important, but also introduces the slight risk that refactoring might still be ongoing when the project starts. Also, since it has been a long time since I worked as a software developer (see Bio section), it is difficult for me to reliably make scheduling estimates.

In order to mitigate these risks, I have planned a fairly flexible schedule, with a focus on the more pressing needs first (i.e. testing) with new and exciting – but less central – features coming second. It is my hope that I will continue working on MojoX::UserAgent long after the GSoC 2009 is over. Even if that part of the work were to get pushed back, the project as a whole should still make a significant contribution to Mojo’s robustness and correctness.

Project Schedule

• Preparation Stage (before May 23rd)
  Refresh my memory on RFC2616 (HTTP/1.1 spec); start playing with Mojo, git & the Perl debugger; hang on Mojo’s IRC channel and read its mailing list.
• May 23rd - June 6th (two weeks)
  Write whitebox tests against Mojo::Message for proper request and response parsing. Leverage test framework in message.t using Test::More.
• June 6th
  Submit code changes through Github.
• June 7th - June 27th (three weeks)
  Find and evaluate suitability of HTTP/1.1 client libraries (LWP, libcurl, ??) for writing a blackbox test suite exercising all protocol-level features of Mojo. Implement blackbox test suite.
• June 28th - July 6th (9 days)
  Patch Mojo code so that it passes all tests.
• July 6th
  Submit code changes through Github.
  Midterm evaluation checkpoint.
• July 6th - 27th (3 weeks)
  Start implementing MojoX::UserAgent class. This class will, at least initially, only support the HTTP protocol, but will include such features as persistent connection management, transparent redirection handling, request pipelining and session cookie support.
• July 27th - August 10th (2 weeks)
  Port the blackbox test suite to MojoX::UserAgent.
• August 10th
  'Pencils down' date. Submit code into Github.

Bio

My story likely diverges significantly from the average GSoC applicant profile, but then TIMTOWTDI, right? After working in (closed source) software development for over eight years, I opted several years ago to reorient myself towards... anthropology! I have completed a master’s degree in that field, and embarked on a PhD, but I am now getting the urge to code again.

I have always been a FOSS advocate and a Perl fan, so TPF seems like a natural fit for me to put my skills where my values are, and start contributing to Open Source. I am especially well-suited to work on HTTP/1.1 testing for Mojo, since one of my most vividly remembered accomplishments from my previous life as a software developer involved upgrading a Web security application based on an HTTP proxy model to support HTTP/1.1.

References

[1] http://mojolicious.org/
[2] http://www.catalystframework.org/
[3] http://search.cpan.org/~mschwern/Test-Simple/lib/Test/More.pm
[4] http://www.ietf.org/rfc/rfc2616.txt
[5] http://curl.haxx.se/
[6] http://search.cpan.org/~gaas/libwww-perl-5.825/lib/LWP.pm
[7] http://mojolicious.org/