Python Typing: Copy `**kwargs` from on function to another

Question

It is common pattern in Python extend functions and use **kwargs to pass all keyword arguments to the extended function.

i.e. take

class A:
    def bar(self, *, a: int, b: str, c: float) -> str:
       return f"{a}_{b}_{c}"
   

class B:
    def bar(self, **kwargs):
        return f"NEW_{super().bar(**kwargs)}"


def base_function(*, a: int, b: str, c: float) -> str:
    return f"{a}_{b}_{c}"


def extension(**kwargs):
    return f"NEW_{super().bar(**kwargs)}"

Now calling extension(no_existing="a") would lead to a TypeError, that could be detected by static type checkers.

How can I annotate my extension in order to detect this problem before I run my code?

This annotation would be also helpful for IDE's to give me the correct suggestions for extension.

score 1 · Answer 1 · answered Apr 22 '22 at 12:03

PEP 612 introduced the ParamSpec (see Documentation) Type.

We can exploit this to generate a decorator that tells our type checker, that the decorated functions has the same arguments as the given function:

from typing import Callable, ParamSpec, TypeVar, cast, Any, Type

# Our test function
def source_func(*, foo: str, bar: int) -> str:
    return f"{foo}_{bar}"

# Define some specification, see documentation
P = ParamSpec("P")
T = TypeVar("T")

# For a help about decorator with parameters see 
# https://stackoverflow.com/questions/5929107/decorators-with-parameters
def copy_kwargs(kwargs_call: Callable[P, Any], target: Type[T]) -> Callable[[Callable], Callable[P, T]]:
    """Decorator does nothing but returning the casted original function"""
    def return_func(func: Callable[..., T]) -> Callable[P, T]:
        return cast(Callable[P, T], func)

    return return_func


@copy_kwargs(source_func, float)
def kwargs_test(**kwargs) -> float:
    print(source_func(**kwargs))
    return 1.2

# define some expected return values
okay: float
broken_kwargs: float
broken_return: str

okay = kwargs_test(foo="a", bar=1)
broken_kwargs = kwargs_test(foo=1, bar="2")
broken_return = kwargs_test(foo="a", bar=1)

Checking this file with pyre gives the correct warnings:

ƛ Found 3 type errors!
src/kwargs.py:30:28 Incompatible parameter type [6]: In anonymous call, for 1st parameter `foo` expected `str` but got `int`.
src/kwargs.py:30:35 Incompatible parameter type [6]: In anonymous call, for 2nd parameter `bar` expected `int` but got `str`.
src/kwargs.py:31:0 Incompatible variable type [9]: broken_return is declared to have type `str` but is used as type `float`.

MyPy just recently (7th April 2022) merged a first implementation for ParamSpec that I did not check yet.

According to the related typedshed Issue, PyCharm should support ParamSpec but did not correctly detect the copied **kwargs but complained that okay = kwargs_test(foo="a", bar=1) would have invalid arguments.

Related Issues:

Mypy: Allow using TypedDict for more precise typing of **kwds
Mypy: PEP 612 tracking issue
Pyright Prototype support for typed **kwargs

It would be nice to see this ability incorporated into functools as a capability of update_wrapper — Jeremy, Apr 22 '22 at 12:19
Indeed. But maybe even with more functionality. I.e. automatically detecting the return value, and allowing to combine the arguments of multiple functions using [Concatenate](https://docs.python.org/3/library/typing.html#typing.Concatenate) Also maybe this should go into tpying and not functools? But yeah probably worth a PEP request IMHO. — Kound, Apr 22 '22 at 12:26

score 0 · Answer 2 · answered Apr 24 '22 at 15:39

Based on @kound answer.

To remain DRY, we can do the same without re-declaring return type. Type variable T will be deduced later (not when copy_kwargs is called, but when its returned function is), but it doesn't affect further type checking.

from typing import Callable, ParamSpec, TypeVar, cast, Any

# Our test function
def source_func(*, foo: str, bar: int) -> str:
    return f"{foo}_{bar}"

# Define some specification, see documentation
P = ParamSpec("P")
T = TypeVar("T")

# For a help about decorator with parameters see 
# https://stackoverflow.com/questions/5929107/decorators-with-parameters
def copy_kwargs(kwargs_call: Callable[P, Any]) -> Callable[[Callable[..., T]], Callable[P, T]]:
    """Decorator does nothing but returning the casted original function"""
    def return_func(func: Callable[..., T]) -> Callable[P, T]:
        return cast(Callable[P, T], func)

    return return_func


@copy_kwargs(source_func)
def kwargs_test(**kwargs) -> float:
    print(source_func(**kwargs))
    return 1.2

reveal_type(kwargs_test(foo="a", bar=1))
reveal_type(kwargs_test(foo=1, bar="2"))

And here's mypy playground link to look at this in action.

Nice that is works with the up to date MyPy. But it struggles with `PyRe`: `src/kwargs_dry.py:12:50 Invalid type variable [34]: The type variable 'Variable[T]' isn't present in the function's parameters.` (pyre 0.9.11) — Kound, Apr 26 '22 at 09:15
Wow, thanks! It's interesting to know. I'll file a report maybe, because here the type variable is actually bound (within another `Callable`), so it's a bug in `PyRe`. — SUTerliakov, Apr 26 '22 at 18:24

Python Typing: Copy `**kwargs` from on function to another

2 Answers2