Epic(experiment): Better support for multi-repo set ups

[loujaybee]

Summary

A discovery epic for investments into documentation, and product improvements related to understanding and documenting use-cases for multi-repository set-ups. The epic is concerned with documenting any existing approaches, rather than implementing product features.

Context

Currently, Gitpod is built around some key product assumptions:

• A remote dev environment is configured in code, that is versioned in a git repository
• Dev environments can be started from repository contexts (branches, PRs, issues)
• Dev environments are ephemeral, so should have a great degree of automation. Do not invest in a single workspace through manual configuration.

This idea works great for single repo projects (mono-repos with multiple components in it or monoliths), but is not straight-forward with projects that consist of multiple git repositories.

That said, there are currently two main solutions for allowing Gitpod to work well for multi-repo set-ups.

🅰️ Multiple Workspaces

Depending on how many repositories a project consists of and how the components communicate with each other the preferred solution is to, configure each repository individually and then start workspaces for the needed repositories and bridge the network.

Example: A project consists of two repositories: "Backend" and "Frontend". Each repository has its own .gitpod.yml. The backend repository doesn't even have a dependency on frontend code, so people can code on it without starting a workspace for Frontend if they wish. If you work on both, you start two workspaces one for "Frontend" and one for "Backend" and just code on them through two distinct IDE windows as devs would do locally. The network bridging makes sure your Frontend workspace uses the services exposed by the "Backend" workspace.

Caveat: If you have a large number of small services, starting a large number of workspaces and maintaining the automatic workspace timeout is inconvenient.

Benefits: This approach nicely keeps your repositories separated including their configuration (no big single dev env image but a small dedicated one per repo). Also, it allows starting workspaces on repository contexts for all your components.

🅱️ Meta Repository

Introducing one main meta-repository that contains the configuration for a single large dev environment that clones all the repos. This kind of mimics a mono-repo and if possible I'd go down the mono-repo route instead of this, because the drawbacks mentioned below:

Caveat: Starting a dev environment from the sub repositories doesn't work, because they don't have a configuration (you can add links to the Readme, though). Contexts don't work, i.e. starting a dev environment on a branch or PR.

Benefits: Kind of is the same as local, where you have one machine that is containing all the dependencies for the various components. More convenient than individual workspaces when you have a highly distributed application with lots of small services sitting in individual repositories.

Hypothesis

If we better document / show users how to convert from their current workflows (specifically more "advanced" use-cases like cross-repo and multi-container) to the Gitpod workflow, users are more likely to adopt, retain, and use Gitpod for doing significant project work.

Value

• Better understand how Gitpod fits with different architectural "patterns"
• It will be clearer to Gitpod users how they can adapt their current local dev workflow to Gitpod

Acceptance Criteria

• The main "use-cases" are well documented
• Any future epics/issues are created for required product changes

Growth Area

• Acquisition - Potentially will impact the number of users that perceive Gitpod as valid for their use-case before trying the product.
• Activation - Some users may be more likely to adopt Gitpod if they are aware of the best ways to configure their dev environments.
• Expansion - With more clarity around how Gitpod solves some of these architectural patterns, some teams may be able to progress

Measurement

• Pageviews on new documentation pages
• Sentiment based feedback on new documentation

Persona(s)

This epic is concerned with helping to define user persona's rather than impact existing ones.

In Scope

• Documenting existing product features and patterns

Out Of Scope

• Implementing new product features and patterns

Internal slack conversation [1]
Internal slack conversation [2]
Internal slack conversation [3]

[shaal]

TLDR: DrupalPod use a repo for the Gitpod environment setup, and another repo for the Drupal code that is being developed.

The DrupalPod project, was created to allow starting Drupal contributions in 1 click.

Setting up Drupal project is a complex task that can take a few hours for first timers.

DrupalPod use the following technologies to achieve its goal:

• custom browser extension - analyze drupal.org issue pages, to send the correct arguments to Gitpod environment
• https://github.com/shaal/DrupalPod - Gitpod environment setup + bash scripts to prepare everything according to arguments from browser extension
• ddev - manage local environment using Docker for PHP projects (allow same setup for local dev on computer or in Gitpod)
• external Drupal repository - Drupal core / module / theme / profile

[rfay]

ddev-gitpod-launcher uses its own repo with environment variables to load another specified repo. That way everything is all set up for the other repo. It's a pretty convoluted technique, https://github.com/drud/ddev-gitpod-launcher/blob/main/.gitpod/start_repo.sh. But amazing that it's do-able.

[axonasif]

Here's an idea:

Users will be able to specify a bunch of sub-repos in their .gitpod.yml (similar pattern to specifying extensions:). The sub-repos should get cloned under /workspace after cloning parent repo. The sub-repos can have their own .gitpod.yml. Sub-repos will be listed in the VSCODE sidebar, from which the user can open another VSCODE instance in a new browser-tab (similar to Remote Explorer mechanism)

============

Here is something I made to achieve a similar thing but nothing like having this feature built-in and well integrated 🤟
https://github.com/axonasif/scripts4gp/tree/main/multi-repo

Thanks!

[hobgoblina]

+1 for including repo declarations in .gitpod.yml and having them cloned into /workspace. That's basically the pattern I'm using for a meta-repository. Just using a separate repos.yml with a simple script to clone the repos before sending a sync-done that allows tasks to run for the cloned repositories - then run code -add /workspace/* during startup. (put up a lil example here)

Ofc would need a more-robust/less-dependent implementation than that example, but it'd be great to just declare repo objects in a repositories section in .gitpod.yml. And maybe as an alternative to cd-ing to the repo's directory for tasks, being able to declare tasks within those repository objects, or add repo/directory params to task objects, or running .gitpod.ymls in the cloned repositories, if they exist... porque no los tres?

[ThePaulMcBride]

This sounds like a great idea. Where I work, our app is powered by a handful of separate repos. A Next.js front end, a Ruby on Rails server, a background worker, and a few other micro services. When added a new feature, or especially when modifying an existing feature, we have to work on more than one of those repos at a time.

Maybe a configuration that works a bit like docker compose where you can specify other repos that your current project depends on?

For example, I could work on the rails app, which depends on a database without needing our frontend app running. But to work on the front end app on anything non-trivial it depends on the rails api, which in turn depends on a database.

Right now, some of our app are dockerised and some are not. I would expect to have to dockerise them to get this to work. Local development at the minute is a case of spinning up each of the parts of the app we need, and allowing them to speak to each other by configuring their addresses/ports as environment variables. IE. we tell the front end that the rails app is at localhost:3000 and our uploader service is at localhost:5000 etc

[JohannesLandgraf]

I think there is a shorter, simpler solution to the problem. I suppose that in the status quo if you have multiple repos for Front - and Backend you also have multiple IDE windows opened on your local setup. Both services can talk to each other because they run on the same machine.

In the Gitpod world you can spin up workspaces based on the individual repos that also essentially become two IDE windows. However, as the workloads are running on the cloud in having the services speak with each other becomes a networking problem.

Instead of using submodules, creating meta repos and other git acrobatic the easy solution would be to leverage Tailscale:

• https://tailscale.com/kb/1161/gitpod-io/?q=gitpod
• https://www.gitpod.io/docs/configure/tailscale

cc @svenefftinge because we spoke about this today

[ThePaulMcBride]

If this is the kind of thing that could be configured as a project and automated, that would be amazing. Locally they are in different repos and are worked on in separate VSCode windows.

[loujaybee]

Added an issue to update the website docs to call out the current solutions more clearly for users: https://github.com/gitpod-io/website/issues/1461