VPR: Adjust tooling to improve workflow for adding new integration plugin docs

[karenzone]

Related:

• Here's one possible approach for consideration: VPR: Revisit allowing a non-standard landing page (use case: new integrations) #89
• Here's a sample checklist created from the draft internal doc notes: Add logstash-integration-aws to default plugin list logstash#14370

Adding new integration plugin docs is tricky, nuanced, and time consuming. order of operations are important, and a misstep creates a "chicken-and-egg" situation that requires some effort to unlock.

TL;DR: We can't create files and link to those files in the same operation.

Traditional plugin docs (inputs, output, filters, and codecs) use a standard header file. Integration plugins mimic traditional plugins at docgen time, while being a parent to traditional plugins. They require a special header file. These two header files introduce linking that points back and forth between the Logstash Reference and the Versioned Plugin Reference. We can't create files and link to those files in the same operation. The need to publish plugins and the docgen process further complicates matters.

I've captured some notes to illustrate and add to our internal docs. Better yet? Let's fix the tooling problem.

[karenzone]

@roaksoax, can we get this prioritized, please? I can provide additional details if needed.

[karenzone]

@mashhurs, thanks for taking on docs tooling again.

TL;DR: We need to avoid the situation in which missing files causes the VPR docgen job to fail with no output. When that happens, we are gridlocked and can't generate the files we need to work our way out of the situation.

If the docgen job can complete successfully, we can fix the bad cross-reference links in the PR and move forward.

With changes we've made to docs tooling, can we get a successful docgen run, albeit with broken links that will be caught at the PR stage by docs-CI?

[karenzone]

The last time I remember being gridlocked with not way to get output was in May 2021. I'm wondering if changes in #58 might have had a positive side effect of eliminating this issue.

Can we figure out a way to test this?

[mashhurs]

Thank you @karenzone for your time today. This is a short wrap note what we discussed and would be a best practice for new plugin docs. I will organize issues and TODO lists more detailed but this is so far high level needs to track/make-it-happen:

• add a new plugin to skip list
  • this will be so far simple manual process that we need to add new plugin to PLUGIN_SKIP_LIST of versioned_plugins.rb
• add index/placeholder page
  • I am thinking to introduce a workflow under logstash-docs to add an empty placeholder with a flexibility that any type of plugin can be placed
• develop and publish plugin
  • this will include plugin docs (example, index.asciidoc, {type}-index.asciidoc files of the plugin, including integration type plugin).
  • current logstash-integration-logstash plugin docs commented links which are not necessary now, we can uncomment (I will update it)
• generate docs
  • current process, no changes but important note is this process fills the content of empty placeholder

[karenzone]

@mashhurs, good ideas in flight here. I love the github actions idea for adding an index placeholder.
Even though it's implied by the name of the PR, will you please update workflow (in bullets 1, 2) to clarify that in most cases, we would need this only for integration plugins. That step won't be necessary for standard inputs, outputs, filters, codecs.