Skip to content

Snapshots Support #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamziel opened this issue Feb 14, 2024 · 4 comments
Open

Snapshots Support #9

adamziel opened this issue Feb 14, 2024 · 4 comments

Comments

@adamziel
Copy link
Collaborator

Let's define the relationship between Snapshots and Blueprints.

  • When writing Blueprints, it's useful to include some files, e.g. private plugins. If Blueprints required shipping them in wp-content/plugins, that starts resembling a Snapshot.
  • A Snapshot exported from WordPress could include a Blueprint like {"theme": "twentytwentythree"} instead of shipping the actual files. That's a smaller archive and an up-to-date theme import.

Blueprints and Snapshots are a spectrum, not distinct ideas

The smallest WordPress setup description is an empty Blueprint {}.

The most extensive description is a full Snapshot + Playground binary bundled as a double click executable – click, and that exact site runs on your computer.

There's so much in between, though:

  • A Blueprint file with some plugins, themes, and site options
  • A Blueprint file + a wp-config.php
  • A Blueprint file with a few plugins + wp-content with more plugins and themes
  • A full snapshot with a Blueprint to update the WordPress version
@adamziel
Copy link
Collaborator Author

adamziel commented Nov 13, 2024

Here's an article I once wrote about Snapshots:

Playground as the WordPress export format

I propose Playground-powered WordPress Snapshots as the canonical site export format for WordPress core.

A Playground export already enables what no other export format may ever be able to do:

  • Preview the exported site
  • Edit the exported data without breaking it...
  • ...using the editors and UI built specifically for that
  • Preserves data, files, plugins, custom database tables
  • Removes any PHP and database version compatibility issues
  • Keeps producing what was there at the time the export was created
  • Is usable in a vanilla browser, local dev env, or even on a mobile device

What inside the Playground export today?

An exported Playground site is a zip with wp-config.php and the wp-content directory. You can import that zip into Playground, run it locally in wp-now, or turn it into a real site using Playground CLI.

I think it would be more useful as a complete WordPress site with all WordPress core files in it. This way, that snapshot would just work without any additional downloads. It could be bundled with Playground as a single executable and would continue to work 10 years from now.

What else is possible with more work?

Playground itself, complete with the portable PHP binary, could be bundled with the exported Snapshot. This would make the export self-contained and usable locally, on any device, and independently from playground.wordpress.net.

WordPress could export itself into the Snapshot format. This would instantly unlock spinning a "temporary test copy" of your site.

WordPress could import from Snapshots to enable easy site transfers and easy "committing" of the changes from the test site.

The above features would also unlock "site templates" that you could turn into real WordPress sites with a few clicks.

What's wrong with WXR, XML, CSV, JSON, Markdown, etc?

Nothing. These are great formats for exporting content.

However, they aren't well suited for exporting entire sites. Exporting an entire site as XML or JSON is lossy and requires more work than just exporting an entire site.

What about advanced setups that require, e.g., Redis?

They won't be supported initially. Eventually, WebAssembly may replace Docker containers enough that we'll be able to bundle more parts of the stack.

What about really large sites, like 1GB or 300GB?

They won't be supported initially, but with additional work in Playground and WordPress core they absolutely can be supported:

  • Playground will have to read data directly from the local disk (it already does) without loading it all into memory (to do).
  • WordPress will need to support exports that can be paused, resumed, and streamed into the browser using multiple requests and zip files.

As a side note, a direct transfer may be a better fit for those huge sites than an export file. More about that on Trac.

@adamziel
Copy link
Collaborator Author

Here's another one:

What Blueprints and WordPress Snapshots are good and bad at

Are you considering Blueprints for your project? Have you considered WordPress Snapshots? This post will give you a good idea of the strengths and weaknesses of both of these formats.

Blueprints are recipes for building WordPress Snapshots. A WordPress Snapshot = Blueprint + Build step. In Docker analogy, these are Dockerfiles.

WordPress Snapshots are collections of, optionally zipped, files. They contain all the PHP source code, database data, static assets, and metadata necessary to start a WordPress website -- for example, in WordPress Playground or wp-now. In Docker analogy, these are Docker Images.

Snapshots are data, Blueprints are computations.

Let's also define a WordPress Website. It is a running, living organism where the source code, the PHP runtime, the database connection, and the hosting setup all work together so that you can use a web browser to read (or publish) that cool news article. In Docker terms, sites are Containers.

Things both Blueprints and WordPress Snapshots are good at

  • Creating disposable websites -- WordPress Snapshots + Runtime = WordPress Website. You can easily dispose of the website and run a new one from the Snapshot, e.g. by refreshing the Playground website. Blueprints may enable building that Snapshot on the fly, but what ultimately becomes the website is the Snapshot.
  • Version Control -- Tracking Blueprints is trivial as they're just text files. Tracking Snapshots takes extracting ZIP bundles and turning the SQLite blob into a plaintext SQL dump which could be made trivial by developer tooling.
  • Forking and changing -- You can update a Blueprint with a text editor, and you can update an Snapshot via wp-admin or with a file editor.
  • Undo -- Working on the copy of either a Blueprint or a Snapshot enables you to discard your changes and revert to the original version when anything breaks.

WordPress Snapshots are useful for users, designers, testers, and developers

WordPress Snapshots strengths

  • Easy to change for users -- Run the Snapshot, go to wp-admin, write posts, import content, install plugins, and do anything else you might need. It's just WordPress.
  • Familiar for developers -- A Snapshot is just WordPress. You can call a PHP function, copy a plugin to wp-content, or create a custom CSS file as usual.
  • AI-friendly -- LLMs already know how to update site title, register a custom post type, and create a three-column layout using WordPress APIs. You run a WordPress Snapshot, ask AI to do things, and apply each response directly to the running website while showing the user a live preview of the changes.
  • Zero build time -- Snapshots can be instantly started as all the information is embedded inside. Plug-in the runtime, done.
  • No network required -- Snapshots are self--contained so you can use them on a busy workshop, a conference talk, or in a plane.
  • Doesn't require special tools -- A Snapshot can be extracted and hosted without going through any specialized WordPress Playground tooling. It's just WordPress
  • Reproducible and self-contained -- The same Snapshot will yield the same WordPress website today and 20 years from now. There are no external dependencies that could break. This is great for archiving websites.
  • WordPress sites can be exported as and restored from a Snapshot -- It means simply packaging up the data, code, and uploads. See Playground as the WordPress export format for more details. Some tools to run the export/import process already exist and they might become a part of WordPress core in the future.

WordPress Snapshots weaknesses

  • Size -- Source code, data and static assets can take megabytes or, sometimes, gigabytes.
  • Opaque -- You can never be certain what's baked into the WordPress Snapshot unless you inspect its contents.
  • Outdated dependencies -- A Snapshot from last year bundles last year's plugins and themes.

Blueprints are for advanced developers with specific needs

You would typically write a Blueprint if a) you're a developer and b) you need a WordPress build script. One-off Websites are so easy to create with wp-admin that most people wouldn't even think about Blueprints, but simply start clicking.

Blueprints strengths

  • Flexibility -- With Blueprints, developers can make surgical adjustments to the build process.
  • Lightweight -- Blueprints are tiny text files that cost almost nothing to store or transfer.
  • Transparency -- A Blueprint includes all the commands needed to build a WordPress Snapshot. You can read through it and understand exactly how the site is built.
  • Up-to-date dependencies -- A typical Blueprint fetches the latest version of WordPress, a particular plugin, and, perhaps, a theme. The resulting Snapshot is then up to date with the latest features and security fixes.
  • WordPress.org integration -- Writing a Blueprint enables you to demo your plugin in the WordPress plugin directory.
  • Spinning a development environment -- A new developer in the team could just download the Blueprint, run a hypothetical wp up command, and get a fresh dev env. The CI scripts could reuse the same method, and even the production build could be based on a Blueprint.

Blueprints weaknesses

  • A WordPress site cannot be exported as a Blueprint -- Going from a WordPress site to a build script can't be automated. It is a manual effort that takes trial and error. The same goes for recording user interactions as a Blueprint. The way to export a WordPress site it to bundle it as a WordPress Snapshot.
  • Difficult for AI -- LLMs don't know how to use Blueprints. Even if you teach themvia prompting or function calling API, they'd need to output all the setup steps in one go and that's a challenge in itself. Also, each change in a Blueprint means you have to build a new Snapshot (and wait).
  • High entry barrier -- People know WordPress, not Blueprints. People are familiar with the UI of wp-admin and the structure of wp-content. Defining a build pipeline requires time, training, and a different way of thinking.
  • Build time -- The more build steps, the longer the build takes. The Tag Concierge plugin demo took enough time to boot, it was eventually migrated to a pre-built WordPress Snapshot that only needs to be downloaded.
  • Non-reproducible -- Using recent dependencies means a build could fail some day as BC breaks, plugins get abandoned, and API endpoints go offline.
  • Requires a build tool -- You can't just run a Blueprint. You need WordPress Playground.

@adamziel
Copy link
Collaborator Author

adamziel commented Nov 13, 2024

@bph – would either of the writeups above make sense as a documentation page or a make.wp.org post?

@bph
Copy link

bph commented Nov 13, 2024

@adamziel I'll review this when I get back next week.

adamziel added a commit that referenced this issue May 30, 2025
## Blueprints v2 runner

> [!WARNING]  
> This is an early alpha version. Proceed with caution!

This PR ships a reference implementation of a subset of the [Blueprints
version
2](https://github.com/Automattic/php-toolkit/blob/19aef3f7273da85ef0927937827c642bf9aa9fb9/components/Blueprints/Versions/Version2/json-schema/wsp/wsp-1-blueprint-v2-schema/proposal.md)
spec. It consists of:

* A CLI runner tool (bundled as .phar) for easy consumption
* A PHP library that can be integrated in other PHP software

These tools are compatible with PHP 7.2+ on Mac, Linux, Windows and
require no PHP extensions. Libraries for processing ZIP, HTTP, XML,
HTML, and Git are included.

This huge PR also:

* Downgrades the codebase to PHP 7.2
* Gets the unit tests to run
* Resolves a number of bugs in most components (HttpClient, Zip,
Filesystem, ByteStream, and more)
* Introduces an all-php TCP server for testing

Splitting it into atomic changes and documenting them could easily take
a week. Therefore I apologize, but I am not going to do that. Let's
treat it as the new initial commit and apply atomic changes from this
point forward.

## Remaining work

First and foremost: **stabilize this runner with a large and meaningful
suite of tests**.

Other than that:

### In this PR

- [x] Remove the buggy, experimental HTTP cache support from this PR

### Independently of this PR

* Consolidate and prioritize the `@TODO` comments in a way that can be
discussed and tracked
* A large set of unit tests for the HTTP client covering all sorts of
use-cases
* (nice to have) An actual, reliable HTTP cache implementation for
faster execution
* A unit test suite running the 100 most popular Blueprints

### Out of scope until the baseline functionality works

* Integrate Data Liberation data importers. The runner uses the
[HumanMade/WordPress-importer](https://github.com/humanmade/WordPress-Importer/)
for now.

## Using it

Download blueprints.phar from [the latest alpha
release](https://github.com/Automattic/php-toolkit/releases/tag/v0.0.3-alpha)
and follow the instructions there.

For running from source, use this command:

```
> php ./components/Blueprints/bin/blueprint.php
WordPress Blueprint Runner
Usage: php blueprint.php exec <blueprint> --site-url=<url> --site-path=<path> [options]

Positional arguments:
  blueprint            Path / URL / DataReference to the blueprint (required)

Options:
  -u, --site-url <value>           Public site URL (https://example.com)
      --site-path <value>          Target directory with WordPress install context) (required)
  -x, --execution-context <value>  Source directory with Blueprint context files
  -m, --mode <value>               Execution mode (create|apply) (default 'create')
  -d, --db-engine <value>          Database engine (mysql|sqlite) (default 'mysql')
      --db-host <value>            MySQL host (default 'localhost')
      --db-user <value>            MySQL user (default 'root')
      --db-pass <value>            MySQL password (default '')
      --db-name <value>            MySQL database (default 'wordpress')
  -p, --db-path <value>            SQLite file path (default 'wp.db')
      --dry-run                    Don’t change anything, just validate (default false)
  -h, --help                       Show full help (default false)
  -V, --version                    Show version (default false)

Examples:
  php blueprint.php exec my-blueprint.json --site-url https://mysite.test --site-path /var/www/mysite.com
  php blueprint.php exec my-blueprint.json --execution-context /var/www --site-url https://mysite.test --mode apply --site-path ./site
```

Here's an example of using it to create a new site (with a complex v1
Blueprint to showcase compatibility):

```
> php ./components/Blueprints/bin/blueprint.php exec https://github.com/wordpress/blueprints/blob/trunk/blueprints/stylish-press/blueprint.json --site-path=./newsite/ --site-url=http://127.0.0.1:4356 --db-engine=sqlite --mode create
Creating a new site
  Site URL:  http://127.0.0.1:4356
  Site path: /www/php-toolkit/untracked/newsite
  Blueprint: /www/php-toolkit/untracked/blueprint.json
[100%] Complete                                                                
✔ Blueprint successfully executed.
```


https://github.com/user-attachments/assets/5565037c-28f0-40cc-9e6d-19e1b86d66dd

The command above created a new WordPress site in a `newsite` directory.
Here's what it did under the hood:

* Validated Blueprint v1 and transpiled it to v2 schema
* Reported live progress
* Downloaded files in parallel
* Imported content and media files
* Ran code in subprocesses

With another Blueprint, you could also create new post types, source
data directly from a git repo, and more.

## Scripts

### Bundle `blueprints.phar`

To build a single `blueprints.phar` executable, install
[box](https://github.com/box-project/box) and run:

```bash
composer build-blueprints-phar
# Under the hood it runs:
#    box compile -c phar-box.json
# and recreates dist/blueprints.phar
```

### Regenerate JSON Schema based on TypeScript types

This PR ships a copy of the [Blueprints v2
proposal](Automattic/wordpress-playground-private#109)
including the TypeScript types. To regenerate the JSON schema used to
validate input Blueprints, run:

```bash
# Install dependencies
npm install
composer regenerate-json-schema

# Under the hood, it runs:
#     node components/Blueprints/json-schema/regenerate-schema.ts
# and recreates components/Blueprints/json-schema/blueprint-v2-schema.json
```

---------

Co-authored-by: Brandon Payton <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants