Skip to content

feat: store time as timespec #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 8, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
docs
package-lock.json
yarn.lock
.nyc_output

# Logs
logs
Expand Down
24 changes: 22 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ The UnixFS spec can be found inside the [ipfs/specs repository](http://github.co
- [get total fileSize](#get-total-filesize)
- [marshal and unmarshal](#marshal-and-unmarshal)
- [is this UnixFS entry a directory?](#is-this-unixfs-entry-a-directory)
- [has an mtime been set?](#has-an-mtime-been-set)
- [Contribute](#contribute)
- [License](#license)

Expand Down Expand Up @@ -116,7 +117,12 @@ message Data {
optional uint64 hashType = 5;
optional uint64 fanout = 6;
optional uint32 mode = 7;
optional int64 mtime = 8;
optional UnixTime mtime = 8;
}

message UnixTime {
required int64 Seconds = 1;
optional fixed32 FractionalNanoseconds = 2;
}

message Metadata {
Expand All @@ -142,7 +148,7 @@ const data = new UnixFS([options])
- data (Buffer): The optional data field for this node
- blockSizes (Array, default: `[]`): If this is a `file` node that is made up of multiple blocks, `blockSizes` is a list numbers that represent the size of the file chunks stored in each child node. It is used to calculate the total file size.
- mode (Number, default `0644` for files, `0755` for directories/hamt-sharded-directories) file mode
- mtime (Date, default `0`): The modification time of this node
- mtime (Date, { secs, nsecs }, { Seconds, FractionalNanoseconds }, [ secs, nsecs ], default { secs: 0 }): The modification time of this node

#### add and remove a block size to the block size list

Expand Down Expand Up @@ -177,6 +183,20 @@ const file = new Data({ type: 'file' })
file.isDirectory() // false
```

#### has an mtime been set?

If no modification time has been set, no `mtime` property will be present on the `Data` instance:

```JavaScript
const file = new Data({ type: 'file' })
file.mtime // undefined

Object.prototype.hasOwnProperty.call(file, 'mtime') // false

const dir = new Data({ type: 'dir', mtime: new Date() })
dir.mtime // { secs: Number, nsecs: Number }
```

## Contribute

Feel free to join in. All welcome. Open an [issue](https://github.com/ipfs/js-ipfs-unixfs/issues)!
Expand Down
6 changes: 4 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"release": "aegir release",
"release-minor": "aegir release --type minor",
"release-major": "aegir release --type major",
"coverage": "aegir coverage"
"coverage": "nyc -s aegir test -t node && nyc report --reporter=html"
},
"repository": {
"type": "git",
Expand All @@ -38,9 +38,11 @@
"devDependencies": {
"aegir": "^20.4.1",
"chai": "^4.2.0",
"dirty-chai": "^2.0.1"
"dirty-chai": "^2.0.1",
"nyc": "^15.0.0"
},
"dependencies": {
"err-code": "^2.0.0",
"protons": "^1.1.0"
},
"contributors": [
Expand Down
118 changes: 107 additions & 11 deletions src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
const protons = require('protons')
const pb = protons(require('./unixfs.proto'))
const unixfsData = pb.Data
const errcode = require('err-code')

const types = [
'raw',
Expand Down Expand Up @@ -45,6 +46,84 @@ function parseArgs (args) {
return args[0]
}

function parseMtime (mtime) {
if (mtime == null) {
return undefined
}

// { secs, nsecs }
if (Object.prototype.hasOwnProperty.call(mtime, 'secs')) {
mtime = {
secs: mtime.secs,
nsecs: mtime.nsecs
}
}

// UnixFS TimeSpec
if (Object.prototype.hasOwnProperty.call(mtime, 'Seconds')) {
mtime = {
secs: mtime.Seconds,
nsecs: mtime.FractionalNanoseconds
}
}

// process.hrtime()
if (Array.isArray(mtime)) {
mtime = {
secs: mtime[0],
nsecs: mtime[1]
}
}

// Javascript Date
if (mtime instanceof Date) {
const ms = mtime.getTime()
const secs = Math.floor(ms / 1000)

mtime = {
secs: secs,
nsecs: (ms - (secs * 1000)) * 1000
}
}

/*
TODO: https://github.com/ipfs/aegir/issues/487

// process.hrtime.bigint()
if (typeof mtime === 'bigint') {
const secs = mtime / BigInt(1e9)
const nsecs = mtime - (secs * BigInt(1e9))

mtime = {
secs: parseInt(secs),
nsecs: parseInt(nsecs)
}
}
*/

if (!Object.prototype.hasOwnProperty.call(mtime, 'secs')) {
return undefined
}

if (mtime.nsecs < 0 || mtime.nsecs > 999999999) {
throw errcode(new Error('mtime-nsecs must be within the range [0,999999999]'), 'ERR_INVALID_MTIME_NSECS')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@achingbrain what did we decide on the range? Your thumbs up at ipfs/specs#236 (comment) seems to indicate you agreed with [1,999999999]...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is lagging a little behind the spec discussion.

That said, I think we can reject (or omit) 0 as a nanosecond value when serializing to a protobuf but accept it at this level as a nicety to the developer.

Be liberal in what you accept and conservative in what you send, etc, etc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, if the user passes the output of process.hrtime() as the mtime (e.g. [secs, nsecs] and it just so happens to fall exactly on the second with a 0 nanosecond fragment, it seems silly to reject the value as it creates a really subtle timing bug.

Instead we can just Do The Right Thing and omit the 0 value when writing out to a protobuf.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@achingbrain on the user-facing API side: I absolutely agree. Whenever I say "reject" I mean in the cases when a block is being decoded.

I.e. I make a distinction between "developer is constructing stuff to send to the network" and "we are decoding stuff we received from the network"

}

return mtime
}

function parseMode (mode) {
if (mode == null) {
return undefined
}

if (typeof mode === 'string' || mode instanceof String) {
mode = parseInt(mode, 8)
}

return mode & 0xFFF
}

class Data {
// decode from protobuf https://github.com/ipfs/specs/blob/master/UNIXFS.md
static unmarshal (marshaled) {
Expand All @@ -55,7 +134,7 @@ class Data {
data: decoded.hasData() ? decoded.Data : undefined,
blockSizes: decoded.blocksizes,
mode: decoded.hasMode() ? decoded.mode : undefined,
mtime: decoded.hasMtime() ? new Date(decoded.mtime * 1000) : undefined
mtime: decoded.hasMtime() ? decoded.mtime : undefined
})
}

Expand All @@ -71,25 +150,35 @@ class Data {
} = parseArgs(args)

if (!types.includes(type)) {
throw new Error('Type: ' + type + ' is not valid')
throw errcode(new Error('Type: ' + type + ' is not valid'), 'ERR_INVALID_TYPE')
}

this.type = type
this.data = data
this.hashType = hashType
this.fanout = fanout
this.blockSizes = blockSizes || []
this.mtime = mtime || new Date(0)
this.mode = mode || mode === 0 ? (mode & 0xFFF) : undefined
this._originalMode = mode

const parsedMode = parseMode(mode)

if (parsedMode !== undefined) {
this.mode = parsedMode
}

if (this.mode === undefined && type === 'file') {
this.mode = DEFAULT_FILE_MODE
}

if (this.mode === undefined && this.isDirectory()) {
this.mode = DEFAULT_DIRECTORY_MODE
}

const parsedMtime = parseMtime(mtime)

if (parsedMtime) {
this.mtime = parsedMtime
}
}

isDirectory () {
Expand Down Expand Up @@ -135,7 +224,7 @@ class Data {
case 'symlink': type = unixfsData.DataType.Symlink; break
case 'hamt-sharded-directory': type = unixfsData.DataType.HAMTShard; break
default:
throw new Error(`Unkown type: "${this.type}"`)
throw errcode(new Error('Type: ' + type + ' is not valid'), 'ERR_INVALID_TYPE')
}

let data = this.data
Expand All @@ -152,8 +241,8 @@ class Data {

let mode

if (this.mode || this.mode === 0) {
mode = (this._originalMode & 0xFFFFF000) | (this.mode & 0xFFF)
if (this.mode != null) {
mode = (this._originalMode & 0xFFFFF000) | parseMode(this.mode)

if (mode === DEFAULT_FILE_MODE && this.type === 'file') {
mode = undefined
Expand All @@ -166,11 +255,18 @@ class Data {

let mtime

if (this.mtime) {
mtime = Math.round(this.mtime.getTime() / 1000)
if (this.mtime != null) {
const parsed = parseMtime(this.mtime)

if (parsed) {
mtime = {
Seconds: parsed.secs,
FractionalNanoseconds: parsed.nsecs
}

if (mtime === 0) {
mtime = undefined
if (mtime.FractionalNanoseconds === 0) {
delete mtime.FractionalNanoseconds
}
}
}

Expand Down
7 changes: 6 additions & 1 deletion src/unixfs.proto.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,12 @@ message Data {
optional uint64 hashType = 5;
optional uint64 fanout = 6;
optional uint32 mode = 7;
optional int64 mtime = 8;
optional UnixTime mtime = 8;
}

message UnixTime {
required int64 Seconds = 1;
optional fixed32 FractionalNanoseconds = 2;
}

message Metadata {
Expand Down
Loading