Limit size of ConsoleHost_history.txt file

[jdthorpe]

Recently, accessing command history via the up-arrow has been quite slow (3-5 seconds to respond), and today, when I opened PowerShell, the console was non-responsive and after about 3 mintues would throw the error below:

Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\Users\me\Documents>
Oops, something went wrong.  Please report this bug with the details below.
Report on GitHub: https://github.com/lzybkr/PSReadLine/issues/new
-----------------------------------------------------------------------
Last 0 Keys:


Exception:
System.OutOfMemoryException: Array dimensions exceeded supported range.
   at System.Collections.Generic.List`1.set_Capacity(Int32 value)
   at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
   at System.Collections.Generic.List`1.Add(T item)
   at System.IO.File.InternalReadAllLines(String path, Encoding encoding)
   at Microsoft.PowerShell.PSConsoleReadLine.<ReadHistoryFile>b__67_0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.DelayedOneTimeInitialize()
   at Microsoft.PowerShell.PSConsoleReadLine.Initialize(Runspace runspace, EngineIntrinsics engineIntrinsics)
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics)
-----------------------------------------------------------------------

With little digging I found that the console history file found here:

C:\Users\me\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost_history.txt

had grown to 13MB and contained about 3 months of commands, and deleting the file fixed both issues.

How can the size of this file be moderated?

Environment data

PS version: 5.1.14393.1198
PSReadline version: 1.2
os: 10.0.14393.0 (rs1_release.160715-1616)
PS file version: 10.0.14393.0 (rs1_release.160715-1616)

[rkeithhill]

I recommend setting these options:

Set-PSReadlineOption -MaximumHistoryCount 10000 -HistoryNoDuplicates 
Set-PSReadlineOption -AddToHistoryHandler {
    param([string]$line)
    # Do not save any command line unless it has more than 3 characters.  Prevents storing gci, gps, etc.
    return $line.Length -gt 3
}

To wrangle your existing history file to a reasonable size, try this script I created to optimize PSReadLine history. You will need to exit all PowerShell instances and then start a PowerShell -NoProfile to make sure the history file isn't in use when you run the script -

https://gist.github.com/rkeithhill/4099bfd8420eed0e6dbc

You can run this script on your history file with -WhatIf and it should give you an estimate of how many duplicate commands it can remove. If you run without -Whatif, it will create a backup of your original history file. That said, it wouldn't hurt if you made your own backup of the file first before running the script i.e. no warranties, guarantees, expressed or implied, yada, yada, yada.

[lzybkr]

@rkeithhill - your script has some serious problems - it doesn't read multi-line input correctly.

[rkeithhill]

Doh! I thought I fixed that. Let me check to see if I didn't update that Gist.

[rkeithhill]

Hmm, I'm stepping through the script in the VSCode debugger and so far I'm not seeing any problems reading multi-line input.

Let me make sure I've got the rules right for multi-line input though. If a line is encountered that ends with a backtick then that line is part of multi-line input. And that same multi-line input continues until we hit a line that doesn't end with a backtick, right?

Also on Linux, how do you handle duplicate entry checking? Case-sensitive, right? So gci and GCI will be duplicates but paths that varying only by case won't get eliminated by the dupe check.

[rkeithhill]

@lzybkr OK, spent time debugging this and comparing history files before and after. Found (and fixed) a few issues and updated the Gist.

1. I assumed that empty lines were meaningless - wrong! Those are frequently used to end multi-line input. This is likely what you were seeing.
2. If you don't tell Get-Content that the encoding of the history file is UTF8 it bungles smart quotes, non-breaking spaces, etc. The history file does not have a UTF8 BOM but I'm still a big surprised that .NET didn't detect the encoding correctly. FWIW I also write back out as UTF8 no BOM.
3. There was a pretty serious perf issue saving the history file back out.
4. Had Ordinal / OrdinalIgnoreCase comparer set wrong (opposite) for Linux vs Windows/macOS.

All of these issues have been fixed. Try it again and see what you think.

[lzybkr]

It looks like I had an old version and you fixed the problem I encountered on Mar 3, 2017.

Maybe you should just add your script to this repo.

[rkeithhill]

Sure. Where should I put it?

[rkeithhill]

Hmm, so even though my profile sets the -MaximumHistoryCount like so:

Set-PSReadlineOption -MaximumHistoryCount 10000 -HistorySearchCursorMovesToEnd -HistoryNoDuplicates -HistorySavePath $PSScriptRoot\PSReadLine_history.txt

When I Get-PSReadlineOption it shows that MaxiumHistoryCount is set to 4096.

Huh? Is there a maximum value for MaximumHistoryCount parameter? If so, shouldn't the Set-PSReadlineOption validate the value?

[lzybkr]

That would be a bug. There is some weirdness around setting that value from $MaximumHistoryCount so I'm guessing there is a flaw in the logic there somewhere.

[mrahhal]

Also faced this issue. In my case the "ConsoleHost_history.txt" file was 3.19 GB in size! That's just insane!

Anyone mind telling what that file is even used for? Is it important? I deleted it for now. And I'm guessing I'll have to look at it again in the future.

[lzybkr]

Just like other shells (like bash), when you start a new shell, you can immediately use commands you entered from a previous session. For example, open a new instance of PowerShell, type Ctrl+r cd and you'll see the most recent cd command you used from another PowerShell instance.

[vors]

Got the same exception for a different reason on macOS:

Error reading or writing history file '/Users/svorobev/.local/share/powershell/PSReadLine/ConsoleHost_history.txt': Too many open files in system
Exception of type 'System.OutOfMemoryException' was thrown.

[JamesLear92]

My ConsoleHost_history file was 3.7GB. Just how?
MaximumHistoryCount is set to 4096.

[lzybkr]

@Omniusz - sorry about that - I think you hit a rare race condition which I had only heard about once before. See #524 - the symptom was different but I believe the fix is the same.

[dennisroche]

I have just found a 5.7GB ConsoleHost_history.txt. 😱

Scanning the file and it appears 10k blocks of the same line so something went a bit crazy. I just deleted the file and system returned to normal, but damn that was unexpected.

[lzybkr]

Sorry about that @dennisroche. You're the third to report this issue, fortunately it is fixed in 2.0.0-beta1.

[dennisroche]

No problem; thanks for getting it fixed.

[R-Studio]

Sorry about that @dennisroche. You're the third to report this issue, fortunately it is fixed in 2.0.0-beta1.

I use PowerShell in Version 5.1.14393.2636, but i still have the issue "the limitation is not working".
So what do you mean "it is fixed in 2.0.0-beta1" ? Which Version is that?

[lzybkr]

As mentioned above, there was a race condition that caused some history files to grow unreasonably. That race condition is fixed in all 2.0 versions.

This issue remains open because there is still no limit to the size of the history file.

[R-Studio]

As mentioned above, there was a race condition that caused some history files to grow unreasonably. That race condition is fixed in all 2.0 versions.

This issue remains open because there is still no limit to the size of the history file.

Ok thanks, but do I have to update manually to version 2.0 or will the issue by fixed with windows updates?

[lzybkr]

A version of PSReadLine 2.0 does ship in Windows 10 1809. It's nearly the same as beta3 I think, but I'm not sure as I'm not responsible for what ships in Windows.

Older versions of Windows will likely not see updates to PSReadLine.

[rably1960]

I know this is an old incident, but have there been any thoughts on implementing a way to limit the size of the history file? "MaximumHistoryCount" is only how many lines are read from the file and put into your history. It has nothing to do with the file size, so the file will continue to get bigger forever. A multi-GB history file serves no purpose and is just a waste of space.