Very Large String crashes PS Editor Services

[brettmillerb]

Started the Advent of Code 2017 challenge and pasted in the input string into a .ps1 editor windows.

This crashed Editor Services.

Was able to reproduce.

Input string: https://adventofcode.com/2017/day/1/input

Log
04/12/2017 17:33:37 [NORMAL] - Method "StartLogging" at line 144 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Host\EditorServicesHost.cs

PowerShell Editor Services Host v1.5.1.0 starting (pid 17852)...

  Host application details:

    Name:      Visual Studio Code Host
    ProfileId: Microsoft.VSCode
    Version:   1.5.1
    Arch:      64-bit

  Operating system details:

    Version: Microsoft Windows NT 10.0.14393.0
    Arch:    64-bit

04/12/2017 17:33:37 [NORMAL] - Method "StartLanguageService" at line 180 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Host\EditorServicesHost.cs

Language service started, listening on port 15230

04/12/2017 17:33:37 [NORMAL] - Method "StartDebugService" at line 254 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Host\EditorServicesHost.cs

Debug service started, listening on port 27356

04/12/2017 17:33:37 [NORMAL] - Method "Initialize" at line 220 of C:\projects\powershelleditorservices\src\PowerShellEditorServices\Session\PowerShellContext.cs

PowerShell runtime version: 5.1.14393.1884, edition: Desktop

04/12/2017 17:33:37 [NORMAL] - Method "FindPSScriptAnalyzerModule" at line 354 of C:\projects\powershelleditorservices\src\PowerShellEditorServices\Analysis\AnalysisService.cs

PSScriptAnalyzer found at C:\Users\brettm\.vscode\extensions\ms-vscode.powershell-1.5.1\modules\PSScriptAnalyzer\PSScriptAnalyzer.psd1

04/12/2017 17:33:38 [NORMAL] - Method "Register" at line 36 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.VSCode\ComponentRegistration.cs

PowerShell Editor Services VS Code module loaded.

04/12/2017 17:33:38 [ERROR] - Method "DispatchMessage" at line 147 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Protocol\MessageProtocol\MessageDispatcher.cs

MessageDispatcher: No handler registered for Event type 'initialized'

04/12/2017 17:33:38 [ERROR] - Method "DispatchMessage" at line 147 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Protocol\MessageProtocol\MessageDispatcher.cs

MessageDispatcher: No handler registered for Event type '$/setTraceNotification'

04/12/2017 17:33:38 [ERROR] - Method "OnListenTaskCompleted" at line 391 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Protocol\MessageProtocol\ProtocolEndpoint.cs

ProtocolEndpoint message loop terminated due to unhandled exception:

System.AggregateException: One or more errors occurred. ---> System.ArgumentOutOfRangeException: Index and length must refer to a location within the string.
Parameter name: length
   at System.String.Substring(Int32 startIndex, Int32 length)
   at Microsoft.PowerShell.EditorServices.ScriptRegion.Create(IScriptExtent scriptExtent)
   at Microsoft.PowerShell.EditorServices.ScriptFileMarker.FromParseError(ParseError parseError)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Microsoft.PowerShell.EditorServices.ScriptFile.ParseFileContents()
   at Microsoft.PowerShell.EditorServices.Workspace.GetFileBuffer(String filePath, String initialBuffer)
   at Microsoft.PowerShell.EditorServices.Protocol.Server.LanguageServer.HandleDidOpenTextDocumentNotification(DidOpenTextDocumentParams openParams, EventContext eventContext)
   at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.MessageDispatcher.<>c__DisplayClass6_0`2.<SetEventHandler>b__0(Message eventMessage, MessageWriter messageWriter)
   at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.MessageDispatcher.<DispatchMessage>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.ProtocolEndpoint.<ListenForMessages>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerShell.EditorServices.Utility.AsyncContext.Start(Func`1 asyncMainFunc, ILogger logger)
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.ArgumentOutOfRangeException: Index and length must refer to a location within the string.
Parameter name: length
   at System.String.Substring(Int32 startIndex, Int32 length)
   at Microsoft.PowerShell.EditorServices.ScriptRegion.Create(IScriptExtent scriptExtent)
   at Microsoft.PowerShell.EditorServices.ScriptFileMarker.FromParseError(ParseError parseError)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Microsoft.PowerShell.EditorServices.ScriptFile.ParseFileContents()
   at Microsoft.PowerShell.EditorServices.Workspace.GetFileBuffer(String filePath, String initialBuffer)
   at Microsoft.PowerShell.EditorServices.Protocol.Server.LanguageServer.HandleDidOpenTextDocumentNotification(DidOpenTextDocumentParams openParams, EventContext eventContext)
   at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.MessageDispatcher.<>c__DisplayClass6_0`2.<SetEventHandler>b__0(Message eventMessage, MessageWriter messageWriter)
   at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.MessageDispatcher.<DispatchMessage>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.ProtocolEndpoint.<ListenForMessages>d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerShell.EditorServices.Utility.AsyncContext.Start(Func`1 asyncMainFunc, ILogger logger)
   at System.Threading.Tasks.Task.Execute()<---

04/12/2017 17:33:38 [ERROR] - Method "ProtocolEndpoint_UnhandledException" at line 418 of C:\projects\powershelleditorservices\src\PowerShellEditorServices.Host\EditorServicesHost.cs

PowerShell Editor Services is terminating due to an unhandled exception, see previous logs for details.

[TylerLeonhardt]

I'll look into this. I seem to only be able to repro when that string is a number.

For example, take the input number:

21752342814933766938172121674976879111362417653261522357855816893656462449168377359285244818489723869987861247912289729579296691684761143544956991583942215236568961875851755854977946147178746464675227699149925227227137557479769948569788884399379821111382536722699575759474473273939756348992714667963596189765734743169489599125771443348193383566159843593541134749392569865481578359825844394454173219857919349341442148282229689541561169341622222354651397342928678496478671339383923769856425795211323673389723181967933933832711545885653952861879231537976292517866354812943192728263269524735698423336673735158993853556148833861327959262254756647827739145283577793481526768156921138428318939361859721778556264519643435871835744859243167227889562738712953651128317624673985213525897522378259178625416722152155728615936587369515254936828668564857283226439881266871945998796488472249182538883354186573925183152663862683995449671663285775397453876262722567452435914777363522817594741946638986571793655889466419895996924122915777224499481496837343194149123735355268151941712871245863553836953349887831949788869852929147849489265325843934669999391846286319268686789372513976522282587526866148166337215961493536262851512218794139272361292811529888161198799297966893366553115353639298256788819385272471187213579185523521341651117947676785341146235441411441813242514813227821843819424619974979886871646621918865274574538951761567855845681272364646138584716333599843835167373525248547542442942583122624534494442516259616973235858469131159773167334953658673271599748942956981954699444528689628848694446818825465485122869742839711471129862632128635779658365756362863627135983617613332849756371986376967117549251566281992964573929655589313871976556784849231916513831538254812347116253949818633527185174221565279775766742262687713114114344843534958833372634182176866315441583887177759222598853735114191874277711434653854816841589229914164681364497429324463193669337827467661773833517841763711156376147664749175267212562321567728575765844893232718971471289841171642868948852136818661741238178676857381583155547755219837116125995361896562498721571413742

If you wrap it in " the issue no longer repros.

[TylerLeonhardt]

Update - still looking into this.

Here's what I've found out. I'm pretty sure the issue exists here:
https://github.com/PowerShell/PowerShellEditorServices/blob/master/src/PowerShellEditorServices/Workspace/ScriptRegion.cs#L86
which goes up to:
https://github.com/PowerShell/PowerShellEditorServices/blob/master/src/PowerShellEditorServices/Workspace/ScriptFile.cs#L584

I've got a repro in PowerShell:

Add-Type -Path C:\Users\tyleonha\PowerShellEditorServices\src\PowerShellEditorServices\bin\Debug\net451\System.Management.Automation.dll
$in = "1234765123875213874521321543211283641982364198327461324987214987621349876321487621348761234987612398476129836124369817326498716432987164329876321498761432987698743213218746123987461432121423987321432141234123443522345234523452345234523452435234553524352345234523452345243523452345234523452345234524352435234524352345243524353"  # Really long string that just contains a number
[System.Management.Automation.Language.Parser]::ParseInput($in, "C:\Users\tyleonha\Desktop\repro.ps1", [ref] $tokens, [ref] $parseErrors)

Note at this point there is one item in parseErrors:

$parseErrors[0].Message
The numeric constant 1234765123875213874521321543211283641982364198327461324987214987621349876321487621348761234987612398476129836124369817326498716432987164329876321498761432987698743213218746123987461432121423987321432141234123443522345234523452345234523452435234553524352345234523452345243523452345234523452345234524352435234524352345243524353 is not valid.

Continuing the example to the failure:

Add-Type -Path C:\Users\tyleonha\PowerShellEditorServices\src\PowerShellEditorServices\bin\Debug\net451\Microsoft.PowerShell.EditorServices.dll
[Microsoft.PowerShell.EditorServices.ScriptRegion]::Create($parseErrors[0].Extent)

This gives us:

Exception calling "Create" with "1" argument(s): "Index and length must refer to a location within the string.
Parameter name: length"
At line:1 char:1
+ [Microsoft.PowerShell.EditorServices.ScriptRegion]::Create($parseErro ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ArgumentOutOfRangeException

Not the most useful error but I think it might have something to do with the IScriptExtent that ScriptRegion extends.

@rkeithhill or @daviwil : do either of you guys have any thoughts?

[rkeithhill]

This is not something you'd expect but accessing the IScriptExtent.Text property generates an exception in this case. We should catch that exception but it seems like a bug in PowerShell to me. Of course, what string do we use for the Text property in this case?

[TylerLeonhardt]

@rkeithhill on Windows PowerShell (W10) the Text property is null for me. The full extent:

File                : C:\Users\tyleonha\Desktop\repro.ps1
StartScriptPosition : System.Management.Automation.Language.InternalScriptPosition
EndScriptPosition   : System.Management.Automation.Language.InternalScriptPosition
StartLineNumber     : 1
StartColumnNumber   : 326
EndLineNumber       : 1
EndColumnNumber     : 327
Text                : 
StartOffset         : 325
EndOffset           : 326

I don't get any exception - are you seeing something else?

[SeeminglyScience]

PowerShell ignores exceptions from properties. You can see it if you call the getter directly with $extent.get_Text()

[rkeithhill]

Yup. I was staring at the impl of ScriptRegion.Create in PSES and thinking WTF, there is no call to Substring in this method. That's when I realized it had to be one of IScriptExtent properties. Once I repro'd the error under the VS debugger, it became obvious it was the Text property.

[TylerLeonhardt]

Thanks for that @SeeminglyScience 😄

So to figure out what Text should be, I used the string: $f = { for a case where parseErrors has something in it but does not cause a crash.

In this case, Text = {

repro good case:

$in = '$f = {'
 [System.Management.Automation.Language.Parser]::ParseInput($in, "C:\Users\tyleonha\Desktop\repro.ps1", [ref] $tokens, [ref] $parseErrors)

Note, at this point $parseErrors[0].Extent is:

File                : C:\Users\tyleonha\Desktop\repro.ps1
StartScriptPosition : System.Management.Automation.Language.InternalScriptPosition
EndScriptPosition   : System.Management.Automation.Language.InternalScriptPosition
StartLineNumber     : 1
StartColumnNumber   : 6
EndLineNumber       : 1
EndColumnNumber     : 7
Text                : {
StartOffset         : 5
EndOffset           : 6

And ScriptRegion.Create succeeds:

[Microsoft.PowerShell.EditorServices.ScriptRegion]::Create($parseErrors[0].Extent)


File                : C:\Users\tyleonha\Desktop\repro.ps1
Text                : {
StartLineNumber     : 1
StartColumnNumber   : 6
StartOffset         : 5
EndLineNumber       : 1
EndColumnNumber     : 7
EndOffset           : 6
StartScriptPosition :
EndScriptPosition   :

Looks like we just need to catch that error. I'm guessing that when we access Extent.Text at all it throws otherwise we could do a null check and set Text in the ScriptRegion constructor to some default value?