[plugin] org-roam.nvim progress & discussion

[chipsenkbeil]

About this task

This issue started discussing a potential bug, and ended up evolving into a rundown of the ongoing work on org-roam.nvim, a plugin to implement Org-roam in neovim.

Scroll further down to see details about the plugin, code pointers and highlights, etc.

The original bug report is kept below for clarity.

Describe the bug

I am writing a plugin that creates some buffers with a buftype of nofile. The purpose of these is to generate some immutable orgmode content that you can navigate. In particular, I want to take advantage of the fantastic folding that orgmode offers.

Unfortunately, when I go to collapse a section, I get an error about the current file is not found or not an org file, caused by

orgmode/lua/orgmode/files/init.lua

Line 106 in 261c987

assert(orgfile, 'Current file not found or not an org file')

Steps to reproduce

1. Create a scratch buffer via vim.api.nvim_create_buf(false, true)
2. Set the filetype to org via vim.api.nvim_buf_set_option(bufnr, "filetype", "org")
3. Populate with some org headings via vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, {"* Heading 1-a", "** Heading 2", "* Heading 1-b"})
4. Navigate to the buffer and hit tab to try to collapse or expand a section

Expected behavior

Section is properly folded.

Emacs functionality

No response

Minimal init.lua

-- Import orgmode using the minimal init example

Screenshots and recordings

No response

OS / Distro

MacOS 14.4

Neovim version/commit

0.9.5

Additional context

Using orgmode on master branch at commit 651078a.

[kristijanhusak]

File is considered valid only if it has an org or org_archive extension. Try setting a name for a buffer that has an org extension with nvim_buf_set_name

  local bufnr = vim.api.nvim_create_buf(false, true)
  vim.api.nvim_buf_set_option(bufnr, 'filetype', 'org')
  vim.api.nvim_buf_set_name(bufnr, 'somename.org')
  vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, { '* Heading 1-a', '** Heading 2', '* Heading 1-b' })
  vim.cmd('b'..bufnr)

[chipsenkbeil]

@kristijanhusak that solved the error about not being a current file! I'm hitting an issue where the plugin still doesn't detect a fold. There may be something wrong with my buffer or setup. Applies on any heading.

[kristijanhusak]

What would you expect to be folded here? Did you try doing zx to recalculate folds?
I see you are trying to implement org-roam. I had similar idea but didn't catch time to start working on it.

[chipsenkbeil]

The first heading. If I create a file manually and reproduce the contents, I can fold it.

I had to make the repo private while I got permission to work on it on personal time. Now that I have it, here's the current plugin: https://github.com/chipsenkbeil/org-roam.nvim

About the plugin

Database

I wrote a simplistic implementation of a database with indexing supported that provides an easy search for links and backlinks. It isn't as full-fledged as SQL and I'm sure will struggle with larger roam collections, but for me this is enough.

Parser

Leveraging the orgmode treesitter parser for me to find the details I need to build the above database.

Completion of node under cursor

Covers both completing within a link and under a cursor. Essentially does the complete everywhere out of the box. Here's a demo:

example-org-roam-completion.mp4

Quickfix jump to backlinks

I like the quickfix list, so while Emacs doesn't have this and uses an org-roam buffer, this was easy to whip up for neovim:

example-org-roam-quickfix.mp4

Org buffer

I was generating an org file instead of writing a custom buffer. I may switch to the custom buffer highlighting and format anyway because I need to capture link navigation to open in a different buffer versus the current one.

Id navigation

Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently. So I'll need to write an org-roam variant to support opening id links. Shouldn't be hard at all. I also built out id generation for a uuid-v4 format in pure Lua. My test nodes aren't using it, though.

[kristijanhusak]

Thanks for opening it up. I just skimmed through it, and it seems you did a lot of custom stuff.
A recent refactor was also done to support org-roam with some of the internals. This is what I had in mind:

1. Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51
2. Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though
3. Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23
4. Use orgmode capture class with custom templates only for org-roam where it would append the properties with id

Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently

In the issue description you wrote you are using commit 651078a. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.

[chipsenkbeil]

Thanks for the pointers! When I looked at the plugin's API and data structures, it was (and I believe still is) missing crucial information I'd need for a fully-functional org-roam implementation. Would it make sense to move this discussion to a separate issue? I know you have the plugin API pinned issue, but there's a lot of different questions and dialog I'd want to have about needs and usage that feels better for a separate issue.

Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51

Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under cursor, etc.

Wonder how much of that has changed or was misinformed from my first look.

I also didn't check to see how you're loading files. I tried to keep most of my logic async - in the sense of leveraging neovim's scheduler and callbacks - so I can (in the future) easily report loading a large set of files, monitoring file changes and reloading, etc.

Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though

I don't know on this one. The database I wrote is a way to both track outgoing links (ones you'd find in a file) and incoming links (aka backlinks into a file). I like the design I've got, so I'll most likely keep this.

Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.

Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23

Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.

Use orgmode capture class with custom templates only for org-roam where it would append the properties with id

I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.

In the issue description you wrote you are using commit 651078a. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

Good to know. I've got different versions on different machines. Given I started fiddling with this a couple of months back, I guess the org id was implemented after.

Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.

I want to reduce the hacks and redundancy where possible. My implementation is meant to build on top of your plugin to supply the org-roam features, but when I started it looked like there was enough missing that I ended up only leveraging the treesitter language tree to get the information I needed.

Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.

[kristijanhusak]

We can either have a separate issue or rename this one.

Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under the cursor, etc.

You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files
Files are loaded asynchronously using luv.
Most methods in file and headline provide the element content and the node containing location information (range).
For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.

Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.

This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.

I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.

I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.

Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.

Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.

Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.

I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.

[chipsenkbeil]

We can either have a separate issue or rename this one.

Renamed this one.

You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files Files are loaded asynchronously using luv. Most methods in file and headline provide the element content and the node containing location information (range). For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.

I'll give these a look in a week or two to see how they operate.

This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.

I don't know if it makes sense to add it to that method, but the locations of links within a file are needed for org-roam in order to support listing multiple references to the same backlink and to be able to pull in a sample of the content that linked to a node. I think it's easier to see from this person's tutorial of the Emacs usage: https://youtu.be/AyhPmypHDEw?si=mLGsAdosnKjTZ-1f&t=1690

I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.

I was really hoping that I could reuse it, so this makes me happy to hear. :) Will be giving that a look as soon as I reach that point.

Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.

Got it. Yes, I definitely want to use your code to handle omnicompletion. The selector is more like a built-in telescope interface, which I wanted to have similar to what is shown in the emacs tutorial above when it comes to selecting nodes where you can not only select between choices but also filter the choices further. So having both is my plan.

I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.

I don't really mind where my plugin lives, so if you would be interested in taking this in at some point in the future once we remove as much of the custom logic as makes sense, then I'd be happy to hand it over to you. I need something like this for work, and it didn't exist, which is why I'm building it now.

[kristijanhusak]

I added range property to links in c4eeb3d that holds the position of the links, including [[ and ]] markers.

We can add a few more things as we go if you find them missing, just let me know.

[chipsenkbeil]

@kristijanhusak are these indexed starting at 0 or 1?

[kristijanhusak]

It's 1.

[chipsenkbeil]

@kristijanhusak I just updated to c4eeb3d and when trying to open an id link using org_open_at_point (bound to a local leader mapping), I'm still getting the issue about "No headline found with id: ...". I'm assuming this is because having a file-level id is unique to org-roam and not orgmode, which I "think" only has the concept of ids in property drawers at the heading level. But I'm not sure if global ids follow that logic or not.

From https://orgmode.org/manual/Handling-Links.html

If the headline has a ‘CUSTOM_ID’ property, store a link to this custom ID. In addition or alternatively, depending on the value of org-id-link-to-org-use-id, create and/or use a globally unique ‘ID’ property for the link 28. So using this command in Org buffers potentially creates two links: a human-readable link from the custom ID, and one that is globally unique and works even if the entry is moved from file to file. The ‘ID’ property can be either a UUID (default) or a timestamp, depending on org-id-method. Later, when inserting the link, you need to decide which one to use.

Given this fact and your desire to maintain only core orgmode functionality within this plugin, I'm assuming that unless the global ID at a file level is part of core orgmode, I will need to write my own open logic for links that navigates to the correct id that supports file-level ids. The thought process is to have that function fall back to your implementation if the link is not an id link. Thoughts?