Skip to content

Commit cfc4d75

Browse files
authored
doc: fix outdated default value of batch size (#6336)
* doc: fix outdated default value of batch size * doc: add doc for ubatch-size
1 parent 6902cb7 commit cfc4d75

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

examples/main/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -296,7 +296,9 @@ These options help improve the performance and memory usage of the LLaMA models.
296296

297297
### Batch Size
298298

299-
- `-b N, --batch-size N`: Set the batch size for prompt processing (default: 512). This large batch size benefits users who have BLAS installed and enabled it during the build. If you don't have BLAS enabled ("BLAS=0"), you can use a smaller number, such as 8, to see the prompt progress as it's evaluated in some situations.
299+
- `-b N, --batch-size N`: Set the batch size for prompt processing (default: `2048`). This large batch size benefits users who have BLAS installed and enabled it during the build. If you don't have BLAS enabled ("BLAS=0"), you can use a smaller number, such as 8, to see the prompt progress as it's evaluated in some situations.
300+
301+
- `-ub N`, `--ubatch-size N`: physical maximum batch size. This is for pipeline parallelization. Default: `512`.
300302

301303
### Prompt Caching
302304

0 commit comments

Comments
 (0)