[safetensors] better RE_SAFETENSORS_SHARD_FILE #605
Conversation
| @@ -14,7 +14,7 @@ export const SAFETENSORS_INDEX_FILE = "model.safetensors.index.json"; | |||
| /// but in some situations safetensors weights have different filenames. | |||
| export const RE_SAFETENSORS_FILE = /\.safetensors$/; | |||
| export const RE_SAFETENSORS_INDEX_FILE = /\.safetensors\.index\.json$/; | |||
| export const RE_SAFETENSORS_SHARD_FILE = /\d{5}-of-\d{5}\.safetensors$/; | |||
| export const RE_SAFETENSORS_SHARD_FILE = /[-_]?(\d{5})-of-(\d{5})\.safetensors$/; | |||
There was a problem hiding this comment.
I don't get [-_]?, since it's optional it doesn't do anything?
Maybe remove the ?
There was a problem hiding this comment.
I don't get [-_]?, since it's optional it doesn't do anything?
src here
on example above, I'd like to think that - is part of -xxxxx-of-xxxxx.safetensors rather than model-. The reason of this decision is that, if in the future, we wanna show common name of those tensors files it should be model rather than model- & with this regex we can achieve that by str.slice(0, -str.match(RE_SAFETENSORS_SHARD_FILE).length)
Maybe remove the ?
- or _ are optional. So we need to have ?. For example there can be model00002-of-00072.safetensors
There was a problem hiding this comment.
hm ok but that's a tenuous dependency between hub.js and the hub. It should be made an explicit function extractShardTensorsData(filename) that returns everything (the root, the numbers, ...) in a typed JSON.
Otherwise assumptions made in one codebase about another codebase can easily be broken, especially if there's no comment / warning that some things are there for a particular reason
There was a problem hiding this comment.
i don't think [_-] can really be optional, i've always seen either of them
|
Superceded by #622 |
Follow up to #593
Two updates:
-or_character present at the beginning of shard substring