Add lots of documentation

Author: carols10cents

docs/ARCHITECTURE.md

@@ -0,0 +1,132 @@
+# Architecture of Crates.io
+
+This document is a tour of the codebase in this repo. If you want to work on a bug or a feature,
+hopefully after reading this doc, you'll have a good idea of where to start looking for the code
+you want to change.
+
+This is a work in progress. Pull requests and issues to improve this document are very welcome!
+
+## Backend
+
+The backend of crates.io is written in Rust. Most of that code lives in the *src* directory. It
+serves a JSON API over HTTP, and the HTTP server interface is provided by the [conduit][] crate and
+related crates.
+
+[conduit]: https://crates.io/crates/conduit
+
+### Server
+
+The code to actually run the server is in *src/bin/server.rs*. This is where most of the pieces of
+the system are instantiated and configured, and can be thought of as the "entry point" to crates.io.
+
+The server does the following things:
+
+1. Initialize logging
+2. Check out the index git repository, if it isn't already checked out
+3. Reads values from environment variables to configure a new instance of `cargo_registry::App`
+4. Adds middleware to the app by calling `cargo_registry::middleware`
+5. Syncs the categories defined in *src/categories.toml* with the categories in the database
+6. Starts a [civet][] `Server` that uses the `cargo_registry::App` instance
+7. Tells Nginx on Heroku that the application is ready to receive requests, if running on Heroku
+8. Blocks forever (or until the process is killed) waiting to receive messages on a channel that no
+   messages are ever sent to, in order to outive the civet `Server` threads
+
+[civet]: https://crates.io/crates/civet
+
+### Routes
+
+The API URLs that the server responds to (aka "routes") are defined in
+*src/lib.rs*.
+
+All of the `api_router` routes are mounted under the `/api/v1` path (see the
+lines that look like `router.get("/api/v1/*path", R(api_router.clone()));`).
+
+Each API route definition looks like this:
+
+```rust
+api_router.get("/crates", C(krate::index));
+```
+
+This line defines a route that responds to a GET request made to
+`/api/v1/crates` with the results of calling the `krate::index` function. `C`
+is a struct that holds a function and implements the [`conduit::Handler`][]
+trait so that the results of the function are the response if the function
+succeeds, and that the server returns an error response if the function doesn't
+succeed. The `C` struct's purpose is to reduce some boilerplate.
+
+[`conduit::Handler`]: https://docs.rs/conduit/0.8.1/conduit/trait.Handler.html
+
+### Code having to do with running a web application
+
+These modules could *maybe* be refactored into another crate. Maybe not. But their primary purpose
+is supporting the running of crates.io's web application parts, and they don't have much to do with
+the crate registry purpose of the application.
+
+#### The `app` module
+
+This contains the `App` struct, which holds a `Config` instance plus a few more application
+components such as:
+
+- The database connection pools (there are two until we finish migrating the app to use Diesel
+  everywhere)
+- The GitHub OAuth configuration
+- The cookie session key given to [conduit-cookie][]
+- The `git2::Repository` instance for the index repo checkout
+- The `Config` instance
+
+This module also contains `AppMiddleware`, which implements the `Middleware` trait in order to
+inject the `app` instance into every request. That way, we can call `req.app()` to get to any of
+these components.
+
+[conduit-cookie]: https://crates.io/crates/conduit-cookie
+
+#### The `config` module
+
+#### The `db` module
+
+#### The `dist` module
+
+#### The `http` module
+
+#### The `model` module
+
+#### The `schema` module
+
+#### The `utils` module
+
+### Code having to do with managing a registry of crates
+
+These modules are specific to the domain of being a crate registry. These concepts would exist no
+matter what language or framework crates.io was implemented in.
+
+#### The `krate` module
+
+#### The `users` module
+
+#### The `badge` module
+
+#### The `categories` module
+
+#### The `category` module
+
+#### The `dependency` module
+
+#### The `download` module
+
+#### The `git` module
+
+#### The `keyword` module
+
+#### The `owner` module
+
+#### The `upload` module
+
+#### The `uploaders` module
+
+#### The `version` module
+
+### Database
+
+### Tests
+
+### Deploying

src/app.rs

@@ -1,3 +1,5 @@
+//! Application-wide components in a struct accessible from each request
+
 use std::env;
 use std::error::Error;
 use std::path::PathBuf;
@@ -18,14 +20,19 @@ pub struct App {
     /// The database connection pool
     pub database: db::Pool,
 
-    /// The database connection pool
+    /// The diesel database connection pool
     pub diesel_database: db::DieselPool,
 
     /// The GitHub OAuth2 configuration
     pub github: oauth2::Config,
 
+    /// A unique key used with conduit_cookie to generate cookies
     pub session_key: String,
+
+    /// The crate index git repository
     pub git_repo: Mutex<git2::Repository>,
+
+    /// The location on disk of the checkout of the crate index git repository
     pub git_repo_checkout: PathBuf,
 
     /// The server configuration
@@ -38,14 +45,20 @@ pub struct AppMiddleware {
 }
 
 impl App {
+    /// Creates a new `App` with a given `Config`
+    ///
+    /// Configures and sets up:
+    ///
+    /// - GitHub OAuth
+    /// - Database connection pools
+    /// - A `git2::Repository` instance from the index repo checkout (that server.rs ensures exists)
     pub fn new(config: &Config) -> App {
         let mut github = oauth2::Config::new(
             &config.gh_client_id,
             &config.gh_client_secret,
             "https://github.com/login/oauth/authorize",
             "https://github.com/login/oauth/access_token",
         );
-
         github.scopes.push(String::from("read:org"));
 
         let db_pool_size = match (env::var("DB_POOL_SIZE"), config.env) {
@@ -66,6 +79,7 @@ impl App {
             _ => 1,
         };
 
+        // We need two connection pools until we finish transitioning everything to use diesel.
         let db_config = r2d2::Config::builder()
             .pool_size(db_pool_size)
             .min_idle(db_min_idle)
@@ -78,6 +92,7 @@ impl App {
             .build();
 
         let repo = git2::Repository::open(&config.git_repo_checkout).unwrap();
+
         App {
             database: db::pool(&config.db_url, db_config),
             diesel_database: db::diesel_pool(&config.db_url, diesel_db_config),
@@ -89,6 +104,11 @@ impl App {
         }
     }
 
+    /// Returns a handle for making HTTP requests to upload crate files.
+    ///
+    /// The handle will go through a proxy if the uploader being used has specified one, which
+    /// is only done in test mode in order to be able to record and inspect the HTTP requests
+    /// that tests make.
     pub fn handle(&self) -> Easy {
         let mut handle = Easy::new();
         if let Some(proxy) = self.config.uploader.proxy() {

src/bin/server.rs

@@ -17,10 +17,15 @@ use std::sync::mpsc::channel;
 
 #[allow(dead_code)]
 fn main() {
+    // Initialize logging
     env_logger::init().unwrap();
+
+    // If there isn't a git checkout containing the crate index repo at the path specified
+    // by `GIT_REPO_CHECKOUT`, delete that directory and clone the repo specified by `GIT_REPO_URL`
+    // into that directory instead. Uses the credentials specified in `GIT_HTTP_USER` and
+    // `GIT_HTTP_PWD` via the `cargo_registry::git::credentials` function.
     let url = env("GIT_REPO_URL");
     let checkout = PathBuf::from(env("GIT_REPO_CHECKOUT"));
-
     let repo = match git2::Repository::open(&checkout) {
         Ok(r) => r,
         Err(..) => {
@@ -36,11 +41,15 @@ fn main() {
                 .unwrap()
         }
     };
+
+    // All commits to the index registry made through crates.io will be made by bors, the Rust
+    // community's friendly GitHub bot.
     let mut cfg = repo.config().unwrap();
     cfg.set_str("user.name", "bors").unwrap();
     cfg.set_str("user.email", "[email protected]").unwrap();
 
     let api_protocol = String::from("https");
+
     let mirror = if env::var("MIRROR").is_ok() {
         Replica::ReadOnlyMirror
     } else {
@@ -56,7 +65,9 @@ fn main() {
 
     let uploader = match (cargo_env, mirror) {
         (Env::Production, Replica::Primary) => {
-            // `env` panics if these vars are not set
+            // `env` panics if these vars are not set, and in production for a primary instance,
+            // that's what we want since we don't want to be able to start the server if the server
+            // doesn't know where to upload crates.
             Uploader::S3 {
                 bucket: s3::Bucket::new(
                     env("S3_BUCKET"),
@@ -69,8 +80,14 @@ fn main() {
             }
         }
         (Env::Production, Replica::ReadOnlyMirror) => {
-            // Read-only mirrors don't need access key or secret key,
-            // but they might have them. Definitely need bucket though.
+            // Read-only mirrors don't need access key or secret key since by definition,
+            // they'll only need to read from a bucket, not upload.
+            //
+            // Read-only mirrors might have access key or secret key, so use them if those
+            // environment variables are set.
+            //
+            // Read-only mirrors definitely need bucket though, so that they know where
+            // to serve crate files from.
             Uploader::S3 {
                 bucket: s3::Bucket::new(
                     env("S3_BUCKET"),
@@ -82,8 +99,13 @@ fn main() {
                 proxy: None,
             }
         }
+        // In Development mode, either running as a primary instance or a read-only mirror
         _ => {
             if env::var("S3_BUCKET").is_ok() {
+                // If we've set the `S3_BUCKET` variable to any value, use all of the values
+                // for the related S3 environment variables and configure the app to upload to
+                // and read from S3 like production does. All values except for bucket are
+                // optional, like production read-only mirrors.
                 println!("Using S3 uploader");
                 Uploader::S3 {
                     bucket: s3::Bucket::new(
@@ -96,6 +118,9 @@ fn main() {
                     proxy: None,
                 }
             } else {
+                // If we don't set the `S3_BUCKET` variable, we'll use a development-only
+                // uploader that makes it possible to run and publish to a locally-running
+                // crates.io instance without needing to set up an account and a bucket in S3.
                 println!("Using local uploader, crate files will be in the dist directory");
                 Uploader::Local
             }
@@ -110,13 +135,15 @@ fn main() {
         gh_client_secret: env("GH_CLIENT_SECRET"),
         db_url: env("DATABASE_URL"),
         env: cargo_env,
-        max_upload_size: 10 * 1024 * 1024,
+        max_upload_size: 10 * 1024 * 1024, // 10 MB default file upload size limit
         mirror: mirror,
         api_protocol: api_protocol,
     };
     let app = cargo_registry::App::new(&config);
     let app = cargo_registry::middleware(Arc::new(app));
 
+    // On every server restart, ensure the categories available in the database match
+    // the information in *src/categories.toml*.
     cargo_registry::categories::sync().unwrap();
 
     let port = if heroku {
@@ -131,7 +158,11 @@ fn main() {
     let mut cfg = civet::Config::new();
     cfg.port(port).threads(threads).keep_alive(true);
     let _a = Server::start(cfg, app);
+
     println!("listening on port {}", port);
+
+    // Creating this file tells heroku to tell nginx that the application is ready
+    // to receive traffic.
     if heroku {
         File::create("/tmp/app-initialized").unwrap();
     }