Markdown files are corrupted

[bronson]

I stored my Markdown notes in Nextcloud. After making a few edits using the Nextcloud web interface, they've been ruined. I need to restore them from backup and try to remember the edits I made.

Is this behavior intentional?

For example, what used to be this:

Is now this:

Steps to Reproduce

1. In the Nextcloud browser, open any typical Markdown file that includes tables, references, line breaks, frontmatter, or related formatting
2. Make a one character edit (for example, remove the extra space after a period)
3. The entire file has been changed. Previously valid Markdown is no longer valid.
4. Restore from backup

Expected behaviour

These are text files. I'd expect my changes to be saved and the rest of the file to remain as it was.

Actual behaviour

The files are completely changed, top to bottom.

Server configuration detail

Operating system: Linux 5.4.10-x86_64-linode132 #1 SMP PREEMPT Thu Jan 9 21:17:12 UTC 2020 x86_64

Webserver: Apache/2.4.29 (Ubuntu) (apache2handler)

Database: mysql 5.7.29

PHP version:

7.2.24-0ubuntu0.18.04.3
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, sodium, session, standard, apache2handler, mysqlnd, PDO, xml, apcu, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, intl, json, exif, mysqli, pdo_mysql, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 18.0.3 - 18.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status

Array
(
)

List of activated apps

Enabled:
 - accessibility: 1.4.0
 - activity: 2.11.0
 - admin_audit: 1.8.0
 - cloud_federation_api: 1.1.0
 - comments: 1.8.0
 - contacts: 3.2.0
 - dav: 1.14.0
 - encryption: 2.6.0
 - federatedfilesharing: 1.8.0
 - federation: 1.8.0
 - files: 1.13.1
 - files_external: 1.9.0
 - files_pdfviewer: 1.7.0
 - files_rightclick: 0.15.2
 - files_sharing: 1.10.1
 - files_trashbin: 1.8.0
 - files_versions: 1.11.0
 - files_videoplayer: 1.7.0
 - firstrunwizard: 2.7.0
 - gpxpod: 4.2.1
 - issuetemplate: 0.6.0
 - logreader: 2.3.0
 - lookup_server_connector: 1.6.0
 - maps: 0.1.6
 - nextcloud_announcements: 1.7.0
 - notifications: 2.6.0
 - oauth2: 1.6.0
 - password_policy: 1.8.0
 - photos: 1.0.0
 - previewgenerator: 2.3.0
 - privacy: 1.2.0
 - provisioning_api: 1.8.0
 - recommendations: 0.6.0
 - serverinfo: 1.8.0
 - settings: 1.0.0
 - sharebymail: 1.8.0
 - spreed: 8.0.7
 - support: 1.1.0
 - survey_client: 1.6.0
 - systemtags: 1.8.0
 - theming: 1.9.0
 - twofactor_backupcodes: 1.7.0
 - updatenotification: 1.8.0
 - viewer: 1.2.0
 - workflowengine: 2.0.0
Disabled:
 - notes
 - text
 - user_ldap

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "cloud.ifying.com"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/cloud.ifying.com",
    "dbtype": "mysql",
    "version": "18.0.3.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "htaccess.RewriteBase": "\/",
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "filelocking.enabled": "true",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "timeout": 0
    },
    "maintenance": false,
    "theme": "",
    "loglevel": 2,
    "updater.release.channel": "stable",
    "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
    "enable_previews": true,
    "enabledPreviewProviders": [
        "OC\\Preview\\PNG",
        "OC\\Preview\\JPEG",
        "OC\\Preview\\GIF",
        "OC\\Preview\\HEIC",
        "OC\\Preview\\BMP",
        "OC\\Preview\\XBitmap",
        "OC\\Preview\\MP3",
        "OC\\Preview\\TXT",
        "OC\\Preview\\MarkDown",
        "OC\\Preview\\PDF",
        "OC\\Preview\\OpenDocument",
        "OC\\Preview\\MSOffice2003",
        "OC\\Preview\\MSOffice2007",
        "OC\\Preview\\MSOfficeDoc",
        "OC\\Preview\\PDF",
        "OC\\Preview\\Postscript",
        "OC\\Preview\\Image",
        "OC\\Preview\\Photoshop",
        "OC\\Preview\\TIFF",
        "OC\\Preview\\SVG",
        "OC\\Preview\\Font",
        "OC\\Preview\\Movie",
        "OC\\Preview\\MKV",
        "OC\\Preview\\MP4",
        "OC\\Preview\\AVI"
    ]
}

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15

Operating system: Macintosh

Logs

Web server error log

Insert your web server log here 

Nextcloud log

Insert your Nextcloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[bronson]

Just noticed in the README that the Text app advertises an:

Open format: Files are saved as Markdown,
so you can edit them from any other text app too.

This isn't true until the Text app preserves Markdown.

Right now it's only safe to either:

• edit Markdown 100% with the Text app, or
• avoid using it completely (so there's no risk of your files being reformatted)

[2q2q]

This problem makes the Text Editor completely unusable for editing any markdown files.

Example
Markdown Header before Editing:

---
Title: any
---

Markdown Header after Editing:

---

## Title: any

[mcgodwin]

Same issue here. I am trying to use Pico CMS app, which needs .md files to remain in a fixed layout, with clean and consistent YAML headers. The Text app messes with the YAML header's formatting, preventing Pico to render any page edited with Text. This is a big issue for me, as we will be a team of people collaborating on a website made with Pico CMS for Nextcloud, as developed in this issue: nextcloud/cms_pico#116

[almereyda]

To refrain from the negative impetus in this issue's title, can we safely rename it to YAML frontmatter support and assign another of the labels, like feature: integration?

It appears this is not an issue of the Text app alone, but stems from the wider ecosystem of unaligned Markdown implementations. Not even CommonMark 0.29 mentions frontmatter or YAML anywhere. It appears hard for me to consider this a bug, since the design documents nowhere state that the app supports a (YAML) frontmatter.

[mcgodwin]

Hi @almereyda and thanks for your comment.
Focusing back on usability, it appears that prior to v.18, the default text editor was not only preserving frontmatter data but could be plugged in with a Markdown editor. Since v.18, no available app allows to view a .md file's source, and the new default app alters frontmatter data.

So yes, from the usability point of view, it is a bug. A crucial feature (on which some apps rely) has been removed.

[lunar-debian]

I have been bitten by this bug too. My solution was to change the extension to .txt so I could still use Text as a collaborative editor while keeping the original formatting and all other characters.
Which now gives me an idea of a small step that could reduce user pain. Please hear me out:

1. Load the file.
2. Parse it as Markdown.
3. Make an export and compare it with the original file.
4. If the export and the original file are the same, start the rich-text editor ; otherwise keep the simple text editor with no extra parsing.

This should prevent any data loss.

[bronson]

@almereyda I'm happy to change the title. Do you have a suggestion? (edited to @ the right person, sorry!)

This issue isn't limited to frontmatter. I don't use frontmatter and it affected me (example above). It clobbers a number of markdown constructs.

[bronson]

Just to be clear: the issue isn't what markdown it supports. The issue is that it reformats markdown that it doesn't understand. No other Markdown editor (that I've used) does that.

[almereyda]

Indeed, I forgot to consider the case where decidedly formatted Markdown tables got distorted, as per the original posting.

Either these two are one case of general support for higher-order grammars of Markdown, then I suggest a title like "Support for extended Markdown features broken".

Else I would suggest to track the cases independently as:

• Frontmatter support gone
  which could include a mention of agnosticism against different formats that we find in the wild, like YAML, JSON or TOML
• Markdown table support rudimentary
  which could mention that we aim at recreating human-readable Markdown tables

[lunar-debian]

@almereyda I respectfully disagree with your classification. Opening a preexisting file, making a change, and losing information (it's not mere presentation, it's structure) should be seen as data loss. It's not only a regression from the previous editor in term of features.

[bronson]

@almereyda I'm not sure about your suggested title. I don't mind at all if this editor chooses not to support support any particular Markdown feature. That's A-OK. The problem is that it wrecks content that it doesn't understand.

If we were to track them separately, maybe we'd open issues for the following?

• frontmatter
• tables
• references
• line breaks
• setext headers
• line blocks
• definition lists

And probably find a few others used in data science and publishing: https://rmarkdown.rstudio.com/authoring_pandoc_markdown.html%23raw-tex

I'm happy to do it if you think that's the way to go.

[almereyda]

I'm unable to make a meaningful consideration at this point, and would leave a decision about classification to the maintainers.

…

On Wed, 29 Apr 2020 at 02:26, Scott Bronson ***@***.***> wrote: @almereyda <https://github.com/almereyda> I'm not sure about your suggested title. I don't mind at all if this editor chooses not to support support any particular Markdown feature. That's A-OK. The problem is that it wrecks content that it doesn't understand. If we were to track them separately, maybe we'd open issues for the following? - frontmatter - tables - references - line breaks - setext headers - line blocks - definition lists And probably find a few others used in data science and publishing: https://rmarkdown.rstudio.com/authoring_pandoc_markdown.html%23raw-tex I'm happy to do it if you think that's the way to go. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#772 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMRV7FDCVWKHOLU67W6KZDRO5X2JANCNFSM4MIAKUVA> .

[e-alfred]

The old files_texteditor app is back on the App Store (https://apps.nextcloud.com/apps/files_texteditor), a rough workaround would be to use it (again) and files_markdown and not use Text for these type of files.

[mcgodwin]

@e-alfred thanks! 🙌