Carriage Return characters (\r) are incorrectly replaced with Newline characters (\n)

[ericdrobinson]

• Are you running the latest version?
• Have you included sample input, output, error, and expected output?
• Have you checked if you are using correct configuration?
• Did you try online tool?

Description

Carriage Return characters (\r) found within parsed XML are incorrectly converted to newline characters (\n).

I did a quick scan of the source code and found a likely culprit:

const parseXml = function(xmlData) {
  xmlData = xmlData.replace(/\r\n?/g, "\n"); //TODO: remove this line

That pretty clearly replaces any \r character (and possibly \r\n pair) with \n.

The online tool does not exhibit this issue because browser node access APIs encode Carriage Return "strings" (the character \ followed by r) as \\r. The regular expression no longer matches. See:

> "1) \r 2) \\r 3) \n 4) \\n 5) \r\n 6) \\r\\n".replace(/\r\n?/g, "\n")
  '1) \n 2) \\r 3) \n 4) \\n 5) \n 6) \\r\\n'

Input

The XML that exhibits this issue is of the form:

<properties object="" engine="">
    <property type="string" name="x" state="changed">
        <![CDATA[This is a carriage return \r...]]>
    </property>
    <property type="string" name="y" state="changed">
        <![CDATA[\r]]>
    </property>
</properties>

Code

The code is pretty straightforward.

const XML_OPTIONS_NO_TAG_PARSE: fastXMLParser.X2jOptionsOptional = {
    attributeNamePrefix: "@",
    ignoreAttributes: false,
    parseAttributeValue: false,
    parseTagValue: false,
    textNodeName: "#value",
};
const XML_PARSER_NO_TAG_PARSE = new fastXMLParser.XMLParser(XML_OPTIONS_NO_TAG_PARSE);

// ...

const parsed = XML_PARSER_NO_TAG_PARSE.parse(xmlData);

After that code runs, the parsed text node content has \n instead of the expected \r.

Output

Running the above results in the following JSON:

{
    "properties": {
        "property": [
            {
                "#value": "This is a carriage return \n...",
                "@type": "string",
                "@name": "x",
                "@state": "changed"
            },
            {
                "#value": "\n",
                "@type": "string",
                "@name": "y",
                "@state": "changed"
            },
        ],
        "@object": "",
        "@engine": ""
    }
}

Expected Data

I expect the following output:

{
    "properties": {
        "property": [
            {
                "#value": "This is a carriage return \r...",
                "@type": "string",
                "@name": "x",
                "@state": "changed"
            },
            {
                "#value": "\r",
                "@type": "string",
                "@name": "y",
                "@state": "changed"
            },
        ],
        "@object": "",
        "@engine": ""
    }
}

Would you like to work on this issue?

• Yes
• No

[github-actions]

I'm glad you find this repository helpful. I'll try to address your issue ASAP. You can watch the repo for new changes or star it.

[ericdrobinson]

I have just confirmed that this line:

const parseXml = function(xmlData) {
  xmlData = xmlData.replace(/\r\n?/g, "\n"); //TODO: remove this line

is responsible for the bug. (Verified by stepping through with a debugger.)

[ericdrobinson]

I have also determined that the reason the online tool doesn't exhibit this issue is because the XML string returned by browser node access APIs (used internally by CodeMirror) returns carriage return characters as \\r instead of \r.

Depending upon the purpose of the problematic line of code that I identified, this may actually cause a bug in the online tool.

Regardless, this is clearly not operating as expected.

[nkappler]

this seems to only happen when trimValues is turned on, at least in my use case, I get a lot more new lines from the text node when I turn it on.

[amitguptagwl]

can you check with v5 if it works?

[nkappler]

is v5 not available as an npm module? How do I install it and where do I get it?

[amitguptagwl]

https://github.com/NaturalIntelligence/fast-xml-parser/blob/master/docs/v5/1.%20Getting%20Started.md

[yelly]

This issue is also affecting us as we need to parse XML files which do not encode carriage returns.
We can't really move to v5 because we are also building XML files which isn't supported yet (I guess we could parse with v5 and build with v4 but that seems a bit cumbersome).
I know that replacing all CRLFs or CRs with LFs is in the XML spec, but it would be nice to be able to turn this off (at least within values, between tags newlines are ignored anyway).
In the meantime as a workaround we are manually encoding all CRs before calling parse and then decoding them in the tagValueProcessor which may lead to bugs in some strange edge cases.

[amitguptagwl]

@yelly , I can understand the trouble you're experiencing. But unfortunately, I'll not be able to do any changes in the code for next 2-3 months. I would take v5 ahead for new changes and build the other components like validator, builder etc for v5. I just wanted to take a feedback on v5 before I do any big change.

[livehigh]

will it be fixed in v4?

[ericdrobinson]

can you check with v5 if it works?

@amitguptagwl I've not yet tested v5 but the source code still contains the exact line that caused the issue as I pointed out here. See:

The main branch at time of writing (v5.0.8):

fast-xml-parser/src/xmlparser/OrderedObjParser.js

Lines 186 to 187 in af4f1d2

	const parseXml = function(xmlData) {
	xmlData = xmlData.replace(/\r\n?/g, "\n"); //TODO: remove this line

The content when I opened this ticket (v4.0.13):

fast-xml-parser/src/xmlparser/OrderedObjParser.js

Lines 177 to 178 in 007d638

	const parseXml = function(xmlData) {
	xmlData = xmlData.replace(/\r\n?/g, "\n"); //TODO: remove this line

It's unchanged. Replacing \r characters with \n is the first thing that the parser does when called. Is there some other mechanism that handles this that you had in mind when asking if the v5 code would work?

[amitguptagwl]

@ericdrobinson previously called v5 is now v6 as I had to release v5 for ESM module

• code
• docs