docs: document the new encoding options on ProcessExecuter.run_with_capture

Author: jcouball

README.md

@@ -26,13 +26,16 @@ then click the "Documentation" link.
 - Compatible with MRI 3.1+, TruffleRuby 24+, and JRuby 9.4+
 - Works on Mac, Linux, and Windows platforms
 
-## Table of Contents
+## Table of contents
 
 - [Requirements](#requirements)
-- [Table of Contents](#table-of-contents)
+- [Table of contents](#table-of-contents)
 - [Usage](#usage)
-  - [Key Methods](#key-methods)
+  - [Key methods](#key-methods)
   - [ProcessExecuter::MonitoredPipe](#processexecutermonitoredpipe)
+  - [Encoding](#encoding)
+    - [Encoding summary](#encoding-summary)
+    - [Encoding details](#encoding-details)
 - [Breaking Changes](#breaking-changes)
   - [2.x](#2x)
     - [`ProcessExecuter.spawn`](#processexecuterspawn)
@@ -61,7 +64,7 @@ then click the "Documentation" link.
 [Full YARD documentation](https://rubydoc.info/gems/process_executer/) for this gem
 is hosted on RubyGems.org. Read below for an overview and several examples.
 
-### Key Methods
+### Key methods
 
 ℹ️ See [the ProcessExecuter module
   documentation](https://rubydoc.info/gems/process_executer/ProcessExecuter) for
@@ -104,7 +107,7 @@ This class's initializer accepts any compatible redirection destination supporte
 In addition to the standard redirection destinations, `MonitoredPipe` also
 supports these additional types of destinations:
 
-- **Arbitrary Writers**
+- **Arbitrary writers**
 
   You can redirect subprocess output to any Ruby object that implements the
   `#write` method. This is particularly useful for:
@@ -114,14 +117,88 @@ supports these additional types of destinations:
   - processing with a streaming parser to parse and process command output as the
     command runs
 
-- **Multiple Destinations**
+- **Multiple destinations**
 
   MonitoredPipe supports duplicating (or "teeing") output to multiple
   destinations simultaneously. This is achieved by providing an array in the
   format `[:tee, destination1, destination2, ...]`, where each `destination` can
   be any value that `MonitoredPipe` itself supports (including another tee or
   MonitoredPipe).
 
+### Encoding
+
+#### Encoding summary
+
+The gem's core (`MonitoredPipe`) passes through raw bytes from the subprocess without
+attempting to interpret or transcode them. `ProcessExecuter.run_with_capture` allows
+text encodings to be specified for the captured stdout and stderr (defaulting to
+`UTF-8`). For these outputs, the raw bytes are interpreted as being in that specified
+encoding. The original byte sequence is preserved and the resulting captured string
+is tagged with the target encoding. No transcoding between different text encodings
+(e.g., `Latin-1` to `UTF-8`) is performed.
+
+#### Encoding details
+
+`ProcessExecuter::MonitoredPipe` is encoding agnostic. Bytes pass through this class
+from the subprocesses output to the destination object as a stream of unaltered
+bytes. No transcoding is applied. Strings written to the destination are tagged for
+the ASCII-8BIT (aka BINARY) encoding.
+
+`ProcessExecuter` methods `.spawn_with_timeout`, `.run`, and `.run_with_capture` are
+also encoding agnostic except with one exception: the user can specify the assumed
+encoding for strings returned from `ResultWithCapture#stdout` and
+`ResultWithCapture#stderr`.
+
+As a convenience, the captured output is assumed to be UTF-8 by default:
+
+```ruby
+result = ProcessExecuter.run_with_capture('pwd')
+result.stdout #=> "/Users/James/projects/process_executer\n"
+result.stdout.encoding #=> #<Encoding::UTF-8>
+```
+
+You can changed the assumed encoding for the captured stdout and stderr via options
+passed to `#run_with_capture`:
+
+```ruby
+# Set the assumed encoding for both stdout and stderr
+result = ProcessExecuter.run_with_capture('pwd', encoding: Encoding::BINARY)
+result.stdout #=> "/Users/James/projects/process_executer\n"
+result.stdout.encoding #=> #<Encoding:BINARY (ASCII-8BIT)>
+
+# You can set the assumed encoding separately for stdout and stderr
+# Encoding may be different for each
+result = ProcessExecuter.run_with_capture('pwd', stdout_encoding: 'BINARY', stderr_encoding: 'UTF-8')
+result.stdout.encoding #=> #<Encoding:BINARY (ASCII-8BIT)>
+result.stderr.encoding #=> #<Encoding:UTF-8>
+```
+
+It is possible that the bytes captured are not valid in the given encoding. The user
+will need to check the `#valid_encoding?` method to know for sure.
+
+```ruby
+File.binwrite('output.txt', "\xFF\xFE") # little-endian BOM marker is not valid UTF-8
+result = ProcessExecuter.run_with_capture('cat output.txt')
+result.stdout #=> "\xFF\xFE"
+result.stdout.encoding #=> #<Encoding:UTF-8>
+result.stdout.valid_encoding? #=> false
+```
+
+Encoding options accept any encoding objects returned by `Encoding.list` or their
+String equivalent given by `#to_s`:
+
+```ruby
+Encoding::UTF_8.to_s #=> 'UTF-8'
+```
+
+Changing the assumed encoding DOES NOT cause transcoding. It simply interprets the
+bytes captured as the given encoding.
+
+These encoding options ONLY affect the internally captured stdout and stderr for
+`ProcessExecuter::run_with_capture`. If you give an `out:` or `err:` option, these
+will result in BINARY encoded strings and you will need to handle setting the right
+encoding or transcoding after collecting the output.
+
 ## Breaking Changes
 
 ### 2.x

lib/process_executer.rb

@@ -258,32 +258,69 @@ def self.run(*command, **options_hash)
   #
   # Accepts all [Process.spawn execution
   # options](https://docs.ruby-lang.org/en/3.4/Process.html#module-Process-label-Execution+Options),
-  # the additional options defined by {spawn_with_timeout} and {run}, and the additional
-  # option `merge_output`:
-  #
-  # * `merge_output: true` merges stdout and stderr into a single capture buffer
-  #   (default is false)
+  # the additional options defined by {spawn_with_timeout} and {run}, and the
+  # additional options `merge_output`, `encoding`, `stdout_encoding`, and
+  # `stderr_encoding`:
+  #
+  # * `merge_output: <Boolean>` if true merges stdout and stderr into a single
+  #   capture buffer (default is false)
+  # * `encoding: <Encoding>` sets the encoding for both stdout and stderr captures
+  #   (default is `Encoding::UTF_8`)
+  # * `stdout_encoding: <Encoding>` sets the encoding for the stdout capture and, if
+  #   not nil, overrides the `encoding` option for stdout (default is nil)
+  # * `stderr_encoding: <Encoding>` sets the encoding for the stderr capture and, if
+  #   not nil, overrides the `encoding` option for stderr (default is nil)
   #
   # The captured output is accessed in the returned object's `#stdout` and `#stderr`
   # methods. Merged output (if the `merged_output: true` option is given) is accessed
   # in the `#stdout` method.
   #
-  # stdout and stderr redirection options may be given by the user. User-supplied
-  # redirections will receive the output in addition to the internal capture.
+  # stdout and stderr redirection destinations may be given by the user (e.g. `out:
+  # <destination>` or `err: <destination>`). These redirections will receive the
+  # output in addition to the internal capture.
+  #
+  # Unless told otherwise, the internally captured output is assumed to be in UTF-8
+  # encoding. This assumption can be changed with the `encoding`,
+  # `stdout_encoding`, or `stderr_encoding` options. These options accept any
+  # encoding objects returned by `Encoding.list` or their String equivalent given by
+  # `#to_s`.
+  #
+  # The bytes captured are not transcoded. They are interpreted as being in the
+  # specified encoding. The user will have to check the validity of the
+  # encoding by calling `#valid_encoding?` on the captured output (e.g.,
+  # `result.stdout.valid_encoding?`).
   #
   # A `ProcessExecuter::ArgumentError` will be raised if both an options object and
   # an options_hash are given.
   #
   # @example capture stdout and stderr
-  #   result = ProcessExecuter.run_with_capture('echo HELLO; echo ERROR >&2')
-  #   result.stdout #=> "HELLO\n"
-  #   result.stderr #=> "ERROR\n"
+  #   result =
+  #   ProcessExecuter.run_with_capture('echo HELLO; echo ERROR >&2')
+  #   result.stdout #=> "HELLO\n" result.stderr #=> "ERROR\n"
   #
   # @example merge stdout and stderr
   #   result = ProcessExecuter.run_with_capture('echo HELLO; echo ERROR >&2', merge_output: true)
   #   # order of output is not guaranteed
-  #   result.stdout #=> "HELLO\nERROR\n"
-  #   result.stderr #=> ""
+  #   result.stdout #=> "HELLO\nERROR\n" result.stderr #=> ""
+  #
+  # @example default encoding
+  #   result = ProcessExecuter.run_with_capture('echo HELLO')
+  #   result.stdout #=> "HELLO\n"
+  #   result.stdout.encoding #=> #<Encoding:UTF-8>
+  #   result.stdout.valid_encoding? #=> true
+  #
+  # @example custom encoding
+  #   result = ProcessExecuter.run_with_capture('echo HELLO', encoding: Encoding::ISO_8859_1)
+  #   result.stdout #=> "HELLO\n"
+  #   result.stdout.encoding #=> #<Encoding:ISO-8859-1>
+  #   result.stdout.valid_encoding? #=> true
+  #
+  # @example custom encoding with invalid bytes
+  #   File.binwrite('output.txt', "\xFF\xFE") # little-endian BOM marker is not valid UTF-8
+  #   result = ProcessExecuter.run_with_capture('cat output.txt')
+  #   result.stdout #=> "\xFF\xFE"
+  #   result.stdout.encoding #=> #<Encoding:UTF-8>
+  #   result.stdout.valid_encoding? #=> false
   #
   # @overload run_with_capture(*command, **options_hash)
   #
@@ -301,6 +338,24 @@ def self.run(*command, **options_hash)
   #   @option options_hash [Boolean] :merge_output if true, stdout and stderr will be
   #     merged into a single capture buffer
   #
+  #   @option options_hash [Encoding, String] :encoding the encoding to assume for
+  #     the internal stdout and stderr captures
+  #
+  #     The default is `Encoding::UTF_8`. This option is overridden by the `stdout_encoding`
+  #     and `stderr_encoding` options if they are given and not nil.
+  #
+  #   @option options_hash [Encoding, String, nil] :stdout_encoding the encoding to
+  #     assume for the internal stdout capture
+  #
+  #     The default is nil, which means the `encoding` option is used. If this option is
+  #     is not nil, it is used instead of the `encoding` option.
+  #
+  #   @option options_hash [Encoding, String, nil] :stderr_encoding the encoding to
+  #     assume for the internal stderr capture
+  #
+  #     The default is nil, which means the `encoding` option is used. If this option
+  #     is not nil, it is used instead of the `encoding` option.
+  #
   # @overload run_with_capture(*command, options)
   #
   #   @param command [Array<String>] see [Process module, Argument `command_line` or
@@ -337,6 +392,9 @@ def self.run(*command, **options_hash)
   #
   # @return [ProcessExecuter::ResultWithCapture]
   #
+  #   Where `#stdout` and `#stderr` are strings whose encoding is determined by the
+  #   `:encoding`, `:stdout_encoding`, or `:stderr_encoding` options.
+  #
   # @api public
   #
   def self.run_with_capture(*command, **options_hash)