Skip to content

Refactor Scheduler and add BloomFilter for duplicate removing #118

Closed
@code4craft

Description

@code4craft

The Scheduler use hashset for duplicate removing. It will take a lot of memory when number of urls is huge. Add BloomFilter for less memory usage.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions