Move sponsors to DB (#9512)

* Create Sponsor model

* Service column can be empty

* Command to load sponsors to database

* initdb should load sponsors

* Add sqlalchemy-utils as a dependency

* Use URL type as fields and refactor to have a single migration

* Populate images URL using current pypi static as the URL prefix

* Fix code lint warnings

I added #noqa for a few of the activity strings from the sponsors.

* Add direnv's .envrc to gitignore

I'm used to use direnv to configure env variables. I had to create it to
always have WAREHOUSE_IPYTHON_SHELL enabled.

* Change sponsors page to read data from db instead of HTML include

* Fix routes test

* Unit test sponsors page

* Better name for url field

* Replace pickle by pg's array type

* Update migration file

* Revert "Unit test sponsors page"

This reverts commit 8d5b1cc.

* Revert "Fix routes test"

This reverts commit 582581f.

* Revert "Change sponsors page to read data from db instead of HTML include"

This reverts commit c5f0d7a.

* Test utils for sponsors app

* Add sponsors list to all requests (except for static files)

* Update templates to read sponsors from request variable

* Reformat

* Camoify logos

* Add minimal sponsor list to admin

* Add link to sponsors list

* Add minimal edit page

* Reformat

* Simple admin view to create sponsors

* Unit test sponsors admin

* Update migration to define activity as a markdown field

* Update code do handle activity as markdown

* Add missing pretend calls to admin routes test

* Fix flake8 lint warnings

* Add new is_active flag to control sponsors display

* Only display active sponsors

* Add is_active boolean to sponsors admin

* Add feedback message after sponsor update

* Color logo is required by the admin

* Do not validate footer sponsor without white logo

* Implement new view to delete a sponsor

* Add modal to confirm sponsor before deletion

* Run make translations to fix CI

* Footer sponsors can be only infra sponsors as well

* Increase test coverage for sponsors/models.py

* 100% test coverage at cli/sponsors.py

* Add simple unit test to make explicit validation done by readme lib

* Add sponsors app to the docs

* Add new boolean flag is_psf_staff to user model

* Introduce PSF Staff group and permission

* Configure permissions considering PSF staff members

* Use SQL false function instead

* Refactor permissions introducing a base one to guarantee admin access

* Rename admin dashboard access permission

Co-authored-by: Ee W. Durbin III <[email protected]>

Author: berinhard

.gitignore

@@ -8,6 +8,7 @@
 .coverage
 .state
 .idea
+.envrc
 
 docs/_build/
 build/

Makefile

@@ -147,6 +147,7 @@ initdb:
 	xz -d -f -k dev/$(DB).sql.xz --stdout | docker-compose run --rm web psql -h db -d warehouse -U postgres -v ON_ERROR_STOP=1 -1 -f -
 	docker-compose run --rm web python -m warehouse db upgrade head
 	$(MAKE) reindex
+	docker-compose run web python -m warehouse sponsors populate-db
 
 reindex:
 	docker-compose run --rm web python -m warehouse search reindex

docs/application.rst

@@ -110,6 +110,7 @@ Directories within the repository:
   - `rss/ <https://github.com/pypa/warehouse/tree/master/warehouse/rss>`_ - RSS feeds: :doc:`api-reference/feeds`
   - `search/ <https://github.com/pypa/warehouse/tree/master/warehouse/search>`_ - utilities for building and querying the search index
   - `sitemap/ <https://github.com/pypa/warehouse/tree/master/warehouse/sitemap>`_ - site maps
+  - `sponsors/ <https://github.com/pypa/warehouse/tree/master/warehouse/sponsors>`_ - sponsors management
   - `templates/ <https://github.com/pypa/warehouse/tree/master/warehouse/templates>`_ - Jinja templates for web pages, emails, etc.
   - `utils/ <https://github.com/pypa/warehouse/tree/master/warehouse/utils>`_ - various utilities Warehouse uses
 

requirements/main.in

@@ -50,6 +50,7 @@ sentry-sdk
 setuptools
 sqlalchemy>=0.9,<1.4.0  # https://github.com/pypa/warehouse/pull/9228
 sqlalchemy-citext
+sqlalchemy-utils
 stdlib-list
 structlog
 transaction

requirements/main.txt

@@ -875,11 +875,16 @@ six==1.15.0 \
     #   pyopenssl
     #   python-dateutil
     #   readme-renderer
+    #   sqlalchemy-utils
     #   tenacity
     #   webauthn
 sqlalchemy-citext==1.8.0 \
     --hash=sha256:a1740e693a9a334e7c8f60ae731083fe75ce6c1605bb9ca6644a6f1f63b15b77
-    # via -r requirements/main.in
+    # via -r main.in
+sqlalchemy-utils==0.37.2 \
+    --hash=sha256:042e08454ee7b822b1e2f2b7c20f76fe7b8255de10354718a11e68ced1a64643 \
+    --hash=sha256:c60b8b43c9ef809d147b0bc571cfbfe0992f854ec242bc01ab7a562f76113743
+    # via -r main.in
 sqlalchemy==1.3.23 \
     --hash=sha256:040bdfc1d76a9074717a3f43455685f781c581f94472b010cd6c4754754e1862 \
     --hash=sha256:1fe5d8d39118c2b018c215c37b73fd6893c3e1d4895be745ca8ff6eb83333ed3 \
@@ -924,6 +929,7 @@ sqlalchemy==1.3.23 \
     #   alembic
     #   paginate-sqlalchemy
     #   sqlalchemy-citext
+    #   sqlalchemy-utils
     #   zope.sqlalchemy
 stdlib-list==0.8.0 \
     --hash=sha256:2ae0712a55b68f3fbbc9e58d6fa1b646a062188f49745b495f94d3310a9fdd3e \

tests/common/db/accounts.py

@@ -30,6 +30,7 @@ class Meta:
     is_active = True
     is_superuser = False
     is_moderator = False
+    is_psf_staff = False
     date_joined = factory.fuzzy.FuzzyNaiveDateTime(
         datetime.datetime(2005, 1, 1), datetime.datetime(2010, 1, 1)
     )

tests/common/db/base.py

@@ -46,3 +46,26 @@ def fuzz(self):
         chars = string.ascii_letters + string.digits
         username = "".join(random.choice(chars) for i in range(12))
         return "@".join([username, self.domain])
+
+
+class FuzzyList(fuzzy.BaseFuzzyAttribute):
+    def __init__(self, item_factory, item_kwargs=None, size=1, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.item_factory = item_factory
+        self.item_kwargs = item_kwargs or {}
+        self.size = size
+
+    def fuzz(self):
+        return [self.item_factory(**self.item_kwargs).fuzz() for i in range(self.size)]
+
+
+class FuzzyUrl(fuzzy.BaseFuzzyAttribute):
+    def __init__(self, domain="example.com", is_secure=False, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.domain = domain
+        self.protocol = "https" if is_secure else "http"
+
+    def fuzz(self):
+        chars = string.ascii_letters
+        path = "".join(random.choice(chars) for i in range(12))
+        return f"{self.protocol}://{self.domain}/{path}"

tests/common/db/sponsors.py

@@ -0,0 +1,37 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from factory import fuzzy
+
+from warehouse.sponsors.models import Sponsor
+
+from .base import FuzzyUrl, WarehouseFactory
+
+
+class SponsorFactory(WarehouseFactory):
+    class Meta:
+        model = Sponsor
+
+    name = fuzzy.FuzzyText(length=12)
+    service = fuzzy.FuzzyText(length=12)
+    activity_markdown = fuzzy.FuzzyText(length=12)
+
+    link_url = FuzzyUrl()
+    color_logo_url = FuzzyUrl()
+    white_logo_url = FuzzyUrl()
+
+    is_active = True
+    footer = True
+    psf_sponsor = True
+    infra_sponsor = False
+    one_time = False
+    sidebar = True

tests/unit/accounts/test_core.py

@@ -227,16 +227,61 @@ def test_via_basic_auth_compromised(
 
 class TestAuthenticate:
     @pytest.mark.parametrize(
-        ("is_superuser", "is_moderator", "expected"),
+        ("is_superuser", "is_moderator", "is_psf_staff", "expected"),
         [
-            (False, False, []),
-            (True, False, ["group:admins", "group:moderators"]),
-            (False, True, ["group:moderators"]),
-            (True, True, ["group:admins", "group:moderators"]),
+            (False, False, False, []),
+            (
+                True,
+                False,
+                False,
+                [
+                    "group:admins",
+                    "group:moderators",
+                    "group:psf_staff",
+                    "group:with_admin_dashboard_access",
+                ],
+            ),
+            (
+                False,
+                True,
+                False,
+                ["group:moderators", "group:with_admin_dashboard_access"],
+            ),
+            (
+                True,
+                True,
+                False,
+                [
+                    "group:admins",
+                    "group:moderators",
+                    "group:psf_staff",
+                    "group:with_admin_dashboard_access",
+                ],
+            ),
+            (
+                False,
+                False,
+                True,
+                ["group:psf_staff", "group:with_admin_dashboard_access"],
+            ),
+            (
+                False,
+                True,
+                True,
+                [
+                    "group:moderators",
+                    "group:psf_staff",
+                    "group:with_admin_dashboard_access",
+                ],
+            ),
         ],
     )
-    def test_with_user(self, is_superuser, is_moderator, expected):
-        user = pretend.stub(is_superuser=is_superuser, is_moderator=is_moderator)
+    def test_with_user(self, is_superuser, is_moderator, is_psf_staff, expected):
+        user = pretend.stub(
+            is_superuser=is_superuser,
+            is_moderator=is_moderator,
+            is_psf_staff=is_psf_staff,
+        )
         service = pretend.stub(get_user=pretend.call_recorder(lambda userid: user))
         request = pretend.stub(find_service=lambda iface, context: service)
 

tests/unit/admin/test_routes.py

@@ -160,4 +160,24 @@ def test_includeme():
             "/admin/verdicts/{verdict_id}/review",
             domain=warehouse,
         ),
+        pretend.call(
+            "admin.sponsor.list",
+            "/admin/sponsors/",
+            domain=warehouse,
+        ),
+        pretend.call(
+            "admin.sponsor.create",
+            "/admin/sponsors/create/",
+            domain=warehouse,
+        ),
+        pretend.call(
+            "admin.sponsor.delete",
+            "/admin/sponsors/{sponsor_id}/delete/",
+            domain=warehouse,
+        ),
+        pretend.call(
+            "admin.sponsor.edit",
+            "/admin/sponsors/{sponsor_id}/",
+            domain=warehouse,
+        ),
     ]