Skip to content

Force https for urls in scraper #2222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 1, 2025
Merged

Force https for urls in scraper #2222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d60f615
force http to https
shanbady Apr 30, 2025
d51b706
fixing util method to convert to https
shanbady Apr 30, 2025
9af6840
adding test mode field
shanbady May 1, 2025
2ff2ac9
adding filters to pull in test_mode resources
shanbady May 1, 2025
2646c1b
Revert "adding test mode field"
shanbady May 1, 2025
18d49ec
Revert "adding filters to pull in test_mode resources"
shanbady May 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions learning_resources/site_scrapers/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@


def scraper_for_site(url):
url = url.replace("http://", "https://")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW http:// is legal in query parameters, so theoretically this could change more than the protocol. (E.g., our login URL is https://api.rc.learn.mit.edu/learn/login?next=https://rc.learn.mit.edu/)

Very likely those sorts of URLs don't matter for this scraping (and maybe changing http to https in query params would be desirable, too).

for pattern in SITE_SCRAPER_MAP:
if re.search(pattern, url):
return SITE_SCRAPER_MAP[pattern](url)
Expand Down
19 changes: 19 additions & 0 deletions learning_resources/site_scrapers/utils_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,22 @@ def test_scraper_for_site(mocker, url, expected_scraper_class):

scraper = scraper_for_site(url)
assert isinstance(scraper, expected_scraper_class)


@pytest.mark.parametrize(
"url",
[
"http://example.com",
"http://micromasters.mit.edu/ds/",
"http://unknownsite.com",
"http://executive.mit.edu/course/innovation-executive-academy/a05U1000005l8nFIAQ.html",
],
)
def test_scraper_forces_https(mocker, url):
"""
Test that the scraper class forces https for the start url
"""

scraper = scraper_for_site(url)
assert "http://" not in scraper.start_url
assert "https://" in scraper.start_url
Loading