Skip to content

Add First filter #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 29, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 24 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,16 @@ That was almost too easy! So let's pass in a list of files on the command line,
script.Args().Concat().Match("Error").Stdout()
```

What's that? You want to append that output to a file instead of printing it to the terminal? No problem:
Maybe we're only interested in the first 10 matches. No problem:

```go
script.Args().Concat().Match("Error").AppendFile("/var/log/errors.txt")
script.Args().Concat().Match("Error").First(10).Stdout()
```

What's that? You want to append that output to a file instead of printing it to the terminal? _You've got some attitude, mister_.

```go
script.Args().Concat().Match("Error").First(10).AppendFile("/var/log/errors.txt")
```

## How does it work?
Expand Down Expand Up @@ -323,6 +329,14 @@ fmt.Println(output)
// Output: hello world
```

### First

`First()` reads its input and passes on the first N lines of it (like Unix [`head`](examples/head/main.go)):

```go
Stdin().First(10).Stdout()
```

### Join

`Join()` reads its input and replaces newlines with spaces, preserving a terminating newline if there is one.
Expand Down Expand Up @@ -567,11 +581,12 @@ These are some ideas I'm playing with for additional features. If you feel like

### Filters

* Ideas welcome!
* `Column()` reads columnar (whitespace-separated) data and cuts the specified column, like Unix `cut`
* `CountFreq()` counts the frequency of input lines, and prepends each unique line with its frequency (like Unix `uniq -c`). The results are sorted in descending numerical order (that is, most frequent lines first).

### Sinks

* Ideas equally welcome!
* [Ideas welcome!](https://github.com/bitfield/script/issues/new)

### Examples

Expand All @@ -580,10 +595,13 @@ Since `script` is designed to help you write system administration programs, a f
* [cat](examples/cat/main.go) (copies stdin to stdout)
* [cat 2](examples/cat2/main.go) (takes a list of files on the command line and concatenates their contents to stdout)
* [grep](examples/grep/main.go)
* [head](examples/head/main.go)
* [echo](examples/echo/main.go)

More examples would be welcome!
[More examples would be welcome!](https://github.com/bitfield/script/pulls)

### Use cases

The best libraries are designed to satisfy real use cases. If you have a sysadmin task which you'd like to implement with `script`, let me know by opening an issue.
The best libraries are designed to satisfy real use cases. If you have a sysadmin task which you'd like to implement with `script`, let me know by [opening an issue](https://github.com/bitfield/script/issues/new) - I'd love to hear from you.

If you use `script` for real work (or, for that matter, real play), I'm always very interested to hear about it. Drop me a line to [email protected] and tell me how you're using `script` and what you think of it!
17 changes: 17 additions & 0 deletions examples/head/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package main

import (
"log"
"os"
"strconv"

"github.com/bitfield/script"
)

func main() {
lines, err := strconv.Atoi(os.Args[1])
if err != nil {
log.Fatal(err)
}
script.Stdin().First(lines).Stdout()
}
23 changes: 23 additions & 0 deletions filters.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,26 @@ func (p *Pipe) Concat() *Pipe {
})
return p.WithReader(io.MultiReader(readers...))
}

// First reads from the pipe, and returns a new pipe containing only the first N
// lines. If there is an error reading the pipe, the pipe's error status is also
// set.
func (p *Pipe) First(lines int) *Pipe {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently negative values are treated the same as 0. head has a special interpretation of negative numbers, which could be added here as well.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GNU head allows negative arguments for -n option, meaning to print all but the last - argument value counted - lines of each input file

That sounds like effectively it would behave like Last()—well, we don't have that yet, but when we do, we won't need to worry about this :)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it is not like Last(), because it prints everything BUT the last n lines, where Last() prints ONLY the last n lines. So in fact it is more like an inverted Last().
A similar option is available in tail als well.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, I wasn't reading that properly. Well, it's a neat feature, but I don't have a use case for it—so I'll wait until there is one.

if p == nil || p.Error() != nil {
return p
}
scanner := bufio.NewScanner(p.Reader)
output := strings.Builder{}
for i := 0; i < lines; i++ {
if !scanner.Scan() {
break
}
output.WriteString(scanner.Text())
output.WriteRune('\n')
}
err := scanner.Err()
if err != nil {
p.SetError(err)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this implementation behave, if you pass a very big file and you use the First() filter? With the unix commands, the pipe is closed by head after 10 lines, which allows all the commands in front of head in the pipe to end as well, without the need of reading the whole file.
Would this work here as well, if we would close the pipe?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a very good point. We don't read any more of the pipe than we need to, but we also don't close it. I'd better add that...

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8921a7d

return Echo(output.String())
}
36 changes: 36 additions & 0 deletions filters_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -219,3 +219,39 @@ func TestConcat(t *testing.T) {
t.Errorf("want %q, got %q", want, got)
}
}

func TestFirst(t *testing.T) {
t.Parallel()
want, err := ioutil.ReadFile("testdata/first10.golden.txt")
if err != nil {
t.Fatal(err)
}
got, err := File("testdata/first.input.txt").First(10).Bytes()
if err != nil {
t.Error(err)
}
if !bytes.Equal(got, want) {
t.Errorf("want %q, got %q", want, got)
}
// First(0) should return zero lines
zero := File("testdata/first.input.txt").First(0)
gotZero, err := zero.CountLines()
if err != nil {
t.Fatal(err)
}
if gotZero != 0 {
t.Errorf("want 0 lines, got %d lines", gotZero)
}
// First(N) where the input has less than N lines, should just return the input.
want, err = File("testdata/first.input.txt").Bytes()
if err != nil {
t.Fatal(err)
}
got, err = File("testdata/first.input.txt").First(100).Bytes()
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, want) {
t.Errorf("want %q, got %q", want, got)
}
}
5 changes: 3 additions & 2 deletions pipes.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,9 @@ func (p *Pipe) ExitStatus() int {
return status
}

// WithReader takes an io.Reader which does not need to be closed after reading,
// and associates the pipe with that reader.
// WithReader takes an io.Reader, and associates the pipe with that reader. If
// necessary, the reader will be automatically closed once it has been
// completely read.
func (p *Pipe) WithReader(r io.Reader) *Pipe {
if p == nil {
return p
Expand Down
2 changes: 2 additions & 0 deletions pipes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ func doMethodsOnPipe(t *testing.T, p *Pipe, kind string) {
p.WithError(nil)
action = "Read()"
p.Read([]byte{})
action = "First()"
p.First(1)
}

func TestNilPipes(t *testing.T) {
Expand Down
18 changes: 18 additions & 0 deletions testdata/first.input.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
The tao that can be told
is not the eternal Tao
The name that can be named
is not the eternal Name.

The unnamable is the eternally real.
Naming is the origin
of all particular things.

Free from desire, you realize the mystery.
Caught in desire, you see only the manifestations.

Yet mystery and manifestations
arise from the same source.
This source is called darkness.

Darkness within darkness.
The gateway to all understanding.
10 changes: 10 additions & 0 deletions testdata/first10.golden.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
The tao that can be told
is not the eternal Tao
The name that can be named
is not the eternal Name.

The unnamable is the eternally real.
Naming is the origin
of all particular things.

Free from desire, you realize the mystery.