-
Notifications
You must be signed in to change notification settings - Fork 9
Dev #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
test and fix location and account API
improve README
Refractor Object decoder
ilyazub
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @jvmvik!
We with @dimitryzub reviewed this PR and made some fixes that we've placed in a separate PR to this one: #2
| decoder + ", available: json, html, object") | ||
|
|
||
|
|
||
| class Client(HttpClient): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Client extends HttpClient is a bit confusing.
README.md.erb
Outdated
| First things first, import the serpapi module: | ||
|
|
||
| ```python | ||
| import serpapi | ||
| ``` | ||
| You'll need a client instance to make a search. This object handles all of the details of connection pooling and thread safety so that you don't have to: | ||
|
|
||
| ```python | ||
| client = serpapi.Client() | ||
| ``` | ||
| To make a search using SerpApi.com: | ||
|
|
||
| ```python | ||
| parameter = { | ||
| api_key: "secret_api_key", # from serpapi.com | ||
| engine: "google", # search engine | ||
| q: "coffee", # search topic | ||
| location: "Austin,TX" # location | ||
| } | ||
| results = searpapi.search(parameter) | ||
| ``` | ||
| Putting everything together. | ||
| ```python | ||
| import serpapi | ||
|
|
||
| parameter = { | ||
| api_key: "secret_api_key", # from serpapi.com | ||
| engine: "google", # search engine | ||
| q: "coffee", # search topic | ||
| location: "Austin,TX" # location | ||
| } | ||
| results = searpapi.search(parameter) | ||
| print(results) | ||
| ``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| First things first, import the serpapi module: | |
| ```python | |
| import serpapi | |
| ``` | |
| You'll need a client instance to make a search. This object handles all of the details of connection pooling and thread safety so that you don't have to: | |
| ```python | |
| client = serpapi.Client() | |
| ``` | |
| To make a search using SerpApi.com: | |
| ```python | |
| parameter = { | |
| api_key: "secret_api_key", # from serpapi.com | |
| engine: "google", # search engine | |
| q: "coffee", # search topic | |
| location: "Austin,TX" # location | |
| } | |
| results = searpapi.search(parameter) | |
| ``` | |
| Putting everything together. | |
| ```python | |
| import serpapi | |
| parameter = { | |
| api_key: "secret_api_key", # from serpapi.com | |
| engine: "google", # search engine | |
| q: "coffee", # search topic | |
| location: "Austin,TX" # location | |
| } | |
| results = searpapi.search(parameter) | |
| print(results) | |
| ``` | |
| The following example runs a search for `"coffee"` using your secret API key which you can find at [SerpApi Dashboard](https://serpapi.com/manage-api-key) page. The `searpapi.search` object handles all of the details of connection pooling and thread safety so that you don't have to: | |
| ```python | |
| import serpapi | |
| parameter = { | |
| api_key: "secret_api_key", # from serpapi.com | |
| engine: "google", # search engine | |
| q: "coffee", # search topic | |
| location: "Austin,TX" # location | |
| } | |
| results = searpapi.search(parameter) | |
| print(results) |
README.md.erb
Outdated
| Optionally, rhe HTTP connection can be tuned: | ||
| - timeout : connection timeout by default 60s | ||
| - retries : attempt to reconnect if the connection failed by default: False. | ||
| serpapi is reliable at 99.99% but your company network might not be as stable. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but your company network might not be as stable
Sounds too self-confident 🤣
README.md.erb
Outdated
|
|
||
| ### Advanced settings | ||
| SerpApi Client uses urllib3 under the hood. | ||
| Optionally, rhe HTTP connection can be tuned: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Optionally, rhe HTTP connection can be tuned: | |
| Optionally, the HTTP connection can be tuned: |
README.md.erb
Outdated
| ``` | ||
|
|
||
| ### Advanced settings | ||
| SerpApi Client uses urllib3 under the hood. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| SerpApi Client uses urllib3 under the hood. | |
| SerpApi Client uses `urllib3` under the hood. |
README.md.erb
Outdated
| [](https://github.com/serpapi/serpapi-python/actions/workflows/ci.yml) | ||
|
|
||
| [SerpApi]](https://serpapi.com) allows to scrape any search engine results. | ||
| It's easy, fast, easy, feature rich, cost effective, scalable and reliable. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds too self-confident :-)
| It's easy, fast, easy, feature rich, cost effective, scalable and reliable. | |
| It's easy, fast, feature rich API. |
serpapi/serpapi.py
Outdated
| if 'timeout' in parameter: | ||
| self.timeout = parameter['timeout'] | ||
| else: | ||
| # 60s default | ||
| self.timeout = 60.0 | ||
|
|
||
| # no HTTP retry | ||
| if 'retries' in parameter: | ||
| self.retries = parameter['retries'] | ||
| else: | ||
| self.retries = False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are 3 required fields to the serpapi.Client constructor: timeout, retries, and api_key.
Others are passed to the search or other methods.
Here's what seems more convenient to me:
import serpapi
client = serpapi.Client(
retries = 5, # or urllib3.util.Retry(connect=5, read=2)
timeout = 4.2,
api_key = "secret_api_key", # from serpapi.com
)
parameters = {
"q": "coffee",
}
results = client.search(parameters)
print(results)| # merge parameter defaults and overrides | ||
| fields = self.parameter.copy() | ||
| fields.update(parameter) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class hierarchy is very confusing while reviewing. We with @dimitryzub had to jump up and down to find the flow of parameters from constructors, and search to start method.
| if response.status != 200: | ||
| try: | ||
| raw = response.data.decode('utf-8') | ||
| payload = json.loads(raw) | ||
| raise SerpApiException(payload['error']) | ||
| except Exception as ex: | ||
| raise SerpApiException(raw) from ex | ||
|
|
||
| # HTTP success 200 | ||
| payload = response.data.decode('utf-8') | ||
|
|
||
| # successful response decoding | ||
| if decoder == 'json': | ||
| return json.loads(payload) | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
response.data.decode('utf-8') and json.loads(payload) is called twice. So, our library slows down developer experience.
call to review new implementation, documentation, examples and more