implement type-hints for all (some) our functions #54

igalic · 2017-09-01T10:48:11Z

gradually add types on all functions and variables.
this effort is guided by mypy.

gronke · 2017-09-01T10:55:03Z

libiocage/lib/Distribution.py

@@ -27,7 +27,7 @@ class DistributionGenerator:
    def __init__(self, host, zfs=None, logger=None):
        self.logger = libiocage.lib.helpers.init_logger(logger)
        self.zfs = libiocage.lib.helpers.init_zfs(zfs)
-        libiocage.lib.helpers.init_host(self, host)
+        self.host = libiocage.lib.helpers.init_host(host, self.logger)


I prefer to use named arguments - at least for logger and those kind of dependencies. It's much easier to recognize broken interfaces when being explicit.

ack, will change

igalic

some types still not working

igalic · 2017-09-01T11:47:48Z

libiocage/lib/events.py

@@ -15,7 +17,7 @@ class IocageEvent:
    Base class for all other iocage events
    """

-    HISTORY = []
+    HISTORY: List[Any] = []


this is not ideal, but i have idea how to tell it the type of this class

Guess I can explain that: List[libiocage.lib.events.IocageEvent]

The class attribute, not the instance attribute contains a list of all events occurred. This way every event has access to the full history.

python (or mypy) can't find the type libiocage.lib.events.IocageEvent

Within this file it's just IocageEvent, not the fqdn

yeah, that's what I thought too, but that doesn't work either.

igalic · 2017-09-01T11:49:29Z

libiocage/lib/helpers.py



 def get_zfs():
    return libzfs.ZFS(history=True, history_prefix="<iocage>")


-def init_host(self, host=None):
+def init_host(host: libiocage.lib.Host.Host=None, logger:
+              libiocage.lib.Logger.Logger=None) -> libiocage.lib.Host.Host:


this return type is now incompatible

We're returning a HostGenerator. Internally we don't use the wrappers that flatten the generator to a result list (like Host, Jail, etc)

HostGenerator should be the type returned.

ack, thanks

igalic · 2017-09-01T11:50:29Z

libiocage/lib/helpers.py

-            self.host = self._class_host(logger=logger)
-        except:
-            self.host = libiocage.lib.Host.HostGenerator(logger=logger)
+    return libiocage.lib.Host.HostGenerator(logger=logger)


with this return

gronke

I tend to use the full fqdn for typings for better readability. (e.g. typing.List instead of List). Beside that and the few comments, this looks pretty good !

gronke · 2017-09-01T11:57:31Z

libiocage/lib/Distribution.py

-        libiocage.lib.helpers.init_host(self, host)
+        self.logger = libiocage.lib.helpers.init_logger(logger)
+        self.zfs = libiocage.lib.helpers.init_zfs(zfs)
+        self.host = libiocage.lib.helpers.init_host(host, self.logger)


Please use named attributes for everything, but the first argument.

gronke · 2017-09-01T11:58:56Z

libiocage/lib/Jails.py

+        self.logger = libiocage.lib.helpers.init_logger(logger)
+        self.host = libiocage.lib.helpers.init_host(host, self.logger)
+
+        # this could be wrong..er.. now?:


What's this comment for? Do you suspect something to be wrong with the following lines?

so, we init_zfs(), and then we run the equivalent of the get_zfs() function, and if that's meant to be happening consistently, we may wanna just always do that in init_zfs()

gronke · 2017-09-01T12:00:42Z

libiocage/lib/errors.py

@@ -388,7 +388,8 @@ def __init__(self, *args, **kwargs):

 class MissingFeature(IocageException, NotImplementedError):

-    def __init__(self, feature_name: str, plural: bool=False, *args, **kwargs):
+    def __init__(self, feature_name: str, plural: bool=False,
+                 *args, **kwargs) -> None:


Please break every argument into a new line. Like such:

def __init__( self feature_name: str plural: bool=False, *args **kwargs ) -> None: pass

yapf doesn't work for python 3.6, and standard indent can't seem to be able to read minds 😛

I'm running this syntax on python3.6. Why should it not work?

gronke · 2017-09-01T12:01:57Z

libiocage/lib/helpers.py



 def get_zfs():
    return libzfs.ZFS(history=True, history_prefix="<iocage>")


-def init_host(self, host=None):
+def init_host(host: libiocage.lib.Host.Host=None, logger:
+              libiocage.lib.Logger.Logger=None) -> libiocage.lib.Host.Host:


HostGenerator should be the type returned.

gronke · 2017-09-02T23:26:48Z

libiocage/lib/Distribution.py

@@ -36,7 +36,7 @@

 class DistributionGenerator:

-    _class_release = libiocage.lib.Release.ReleaseGenerator
+    _class_release = 'libiocage.lib.Release.ReleaseGenerator'


This should not be a string. Later on we instantiate this class.

gronke · 2017-09-02T23:28:37Z

libiocage/lib/Host.py

@@ -91,4 +89,4 @@ def processor(self):

 class Host(HostGenerator):

-    class_distribution = libiocage.lib.Distribution.DistributionGenerator
+    class_distribution = 'libiocage.lib.Distribution.DistributionGenerator'


it should have been libiocage.lib.Distribution.Distribution here. Line 35 should not be deleted as well.

n.b.: In class HostGenerator this property was _class_distribution (yay ☭) but here it's without the leading _.

gronke · 2017-09-02T23:30:29Z

libiocage/lib/Host.py

        self.datasets = libiocage.lib.Datasets.Datasets(
            root=root_dataset,
            logger=self.logger,
            zfs=self.zfs
        )
-        self.distribution = self._class_distribution(
+        self.distribution = libiocage.lib.Distribution.DistributionGenerator(


It is important to instantiate from the class attribute _class_distribution. If you use a non-generator class like Host, you don't expect a generator distribution.

@igalic as we discussed offline, this change should be reverted

gronke · 2017-09-02T23:31:44Z

libiocage/lib/JailConfig.py

@@ -79,8 +79,7 @@ def __init__(self,

        dict.__init__(self)

-        libiocage.lib.helpers.init_logger(self, logger)
-
+        self.logger = libiocage.lib.helpers.init_logger(logger)


I like to use the output from the helper to store explicitly as self.logger, but we should remain passing self as first attribute. This way we do not need to pass other named arguments explicitly later on.

gronke · 2017-09-02T23:33:02Z

libiocage/lib/Jails.py

-        libiocage.lib.helpers.init_zfs(self, zfs)
-        libiocage.lib.helpers.init_host(self, host)
+        self.logger = libiocage.lib.helpers.init_logger(logger)
+        self.host = libiocage.lib.helpers.init_host(host, logger=self.logger)


Here's one example why we should pass self as first argument to the init_ helpers.

gronke · 2017-09-02T23:38:41Z

libiocage/lib/errors.py

+                 feature_name: str,
+                 plural: bool=False,
+                 *args,
+                 **kwargs) -> None:


Please use this style for multi-line method interfaces

def __init__(self, feature_name: str, plural: bool=False, *args, **kwargs ) -> None:

igalic

not sure this is correct

igalic · 2017-09-04T09:12:37Z

libiocage/lib/Host.py

@@ -91,4 +91,4 @@ def processor(self):

 class Host(HostGenerator):

-    class_distribution = libiocage.lib.Distribution.DistributionGenerator
+    _class_distribution = libiocage.lib.Distribution.DistributionGenerator


is this supposed to be a generator?
is it supposed to be class_distribution and not _class_distribution

As rule of thumb: when the class is not a generator it is not supposed to expose or use generators at all. So, this should be a Distribution, not DistributionGenerator.

Even tho we're using mypy to guide us in this endavour, we don't add it as dependency, because it's still considered experimental. But we do allow travis to check our types for us!

use 'forward references' of types that haven't been properly imported yet Including IocageEvent - which is referenced from from itself! fix distribu forward ref

this confuses the type-checker ;)

gronke · 2017-09-04T13:26:05Z

@igalic besides that init_* functions should remain having self as first argument, so that we don't need to pass other attributes (like logger, host, etc) on every helpers.init_* call. It's easier to update on inter-dependency changes in future updates as well.

no need to modify self!

igalic mentioned this pull request Sep 1, 2017

implement type-hints for all our functions #42

Closed

igalic force-pushed the type-hints-rev branch from a49fad5 to 72b05d2 Compare September 1, 2017 10:52

gronke reviewed Sep 1, 2017

View reviewed changes

igalic commented Sep 1, 2017

View reviewed changes

gronke requested changes Sep 1, 2017

View reviewed changes

igalic force-pushed the type-hints-rev branch 13 times, most recently from 63db46b to 309de59 Compare September 2, 2017 16:14

gronke requested changes Sep 2, 2017

View reviewed changes

igalic force-pushed the type-hints-rev branch 5 times, most recently from fc1be12 to c02d426 Compare September 4, 2017 08:54

igalic commented Sep 4, 2017

View reviewed changes

implement type-hints for all our functions

2f0fa99

Even tho we're using mypy to guide us in this endavour, we don't add it as dependency, because it's still considered experimental. But we do allow travis to check our types for us!

igalic force-pushed the type-hints-rev branch from c02d426 to 435cd35 Compare September 4, 2017 12:46

igalic added 2 commits September 4, 2017 15:17

add type annotations for when mypy feels confused

3c5d63b

use 'forward references' of types that haven't been properly imported yet Including IocageEvent - which is referenced from from itself! fix distribu forward ref

dont overload (reassign/mutate!) a variable

7e38593

this confuses the type-checker ;)

igalic force-pushed the type-hints-rev branch from 435cd35 to 2c95ed8 Compare September 4, 2017 13:18

igalic changed the title ~~implement type-hints for all our functions~~ implement type-hints for all (some) our functions Sep 4, 2017

igalic added 2 commits September 4, 2017 16:17

convert helpers/init_* into pure functions

8ece3b0

no need to modify self!

correctly ignore lib/ so ag/rg/ack will work

bd8214d

igalic force-pushed the type-hints-rev branch from c11f61b to bd8214d Compare September 4, 2017 14:18

gronke approved these changes Sep 4, 2017

View reviewed changes

gronke merged commit 572fac9 into bsdci:master Sep 4, 2017

igalic deleted the type-hints-rev branch September 4, 2017 14:26

igalic mentioned this pull request Sep 4, 2017

Use type-hints for all function arguments & return values #39

Closed

implement type-hints for all (some) our functions #54

implement type-hints for all (some) our functions #54

Uh oh!

Conversation

igalic commented Sep 1, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

igalic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gronke Sep 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gronke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

igalic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gronke commented Sep 4, 2017

Uh oh!

Uh oh!

gronke Sep 1, 2017 •

edited

Loading