Skip to content

Conversation

sibiryakov
Copy link
Member

@sibiryakov sibiryakov commented May 10, 2016

Handling of scoring log events in strategy worker was refactored. The major changes are:

  • FrontierManager is passed to strategy class,
  • MessageBusScheduler is introduced to buffer update_score events during strategy worker execution.
  • StatesContext is introduced to allow proper creation of Frontera request object (with instant state retrieval from storage) using BaseCrawlingStrategy.create_request(url) method.
  • Scheduling is done using BaseCrawlingStrategy.schedule(request) method. Previous mechanism is eliminated.

@sibiryakov sibiryakov force-pushed the strategy-improvements branch from 6fa87a0 to d23e9be Compare May 17, 2016 09:17
@sibiryakov sibiryakov force-pushed the strategy-improvements branch from 7bfa211 to 496c03a Compare May 17, 2016 12:52
@sibiryakov sibiryakov merged commit 035be29 into master May 17, 2016
@sibiryakov sibiryakov deleted the strategy-improvements branch January 24, 2018 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant