Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

final TypedDict #7981

Open
crusaderky opened this issue Nov 20, 2019 · 14 comments
Open

final TypedDict #7981

crusaderky opened this issue Nov 20, 2019 · 14 comments

Comments

@crusaderky
Copy link
Contributor

Reopened from #7845 as requested

mypy seems to ignore the @final decorator when it is applied to a TypedDict. This issue becomes material when one invokes the items() or values() methods, or passes a generic string to get(); then the output will be incorrectly upcast to object since mypy thinks it may be dealing with a subclass.

from typing_extensions import final, TypedDict

@final
class Point(TypedDict):
    x: float
    y: float
    z: float


p: Point
x: str
reveal_type(p.values())
reveal_type(p.items())
reveal_type(p.get(x, 0.0))

Output:

12: note: Revealed type is 'typing.ValuesView[builtins.object*]'
13: note: Revealed type is 'typing.AbstractSet[Tuple[builtins.str*, builtins.object*]]'
14: note: Revealed type is 'builtins.object*'

Expected output:

12: note: Revealed type is 'typing.ValuesView[builtins.float*]'
13: note: Revealed type is 'typing.ItemsView[builtins.str*, builtins.float*]'
14: note: Revealed type is 'builtins.float*'
@crusaderky
Copy link
Contributor Author

Related: #7849, #7865

@JukkaL
Copy link
Collaborator

JukkaL commented Nov 21, 2019

The proposed semantics seem to be that a final TypedDict object doesn't have any extra keys beyond those included in the definition. Also, a final TypedDict can't be used as a base to define derived TypedDicts.

I haven't thought about this carefully, but it may be possible to define this in a sound fashion. A final TypedDict would only be compatible with another final TypedDict, and they must have the same keys and required keys, and the key types must be compatible.

Since this seems to work at runtime, this could be implemented without changes to typing. However, I'm not sure how useful this would be. I'd be interested in hearing if anybody has real-world use cases where this would be helpful.

@ilevkivskyi
Copy link
Member

A final TypedDict would only be compatible with another final TypedDict, and they must have the same keys and required keys, and the key types must be compatible.

Exactly, it looks like this is the only way to make this sound. This however may be not very useful, as one would need to have same typed dict all the way down the call stack. On the other hand this may be OK for new codebases.

@crusaderky
Copy link
Contributor Author

The same problem is on the keys. .keys() and .__iter__() yield str according to mypy, but the final decorator could make it change that to Literal.

Real life use case:

from typing_extensions import TypedDict, final


@final
class Counters(TypedDict):
    counter_1: int
    blah_blah: int
    something_else: int
    and_another: int
    one_more: int


def reset_counters(c: Counters) -> None:
    for k in c:
        c[k] = 0  # error: TypedDict key must be a string literal; expected one of ('counter_1', 'blah_blah', 'something_else', 'and_another', 'one_more')

@intgr
Copy link
Contributor

intgr commented Apr 16, 2020

I would find the implementation propsed by @JukkaL quite useful. I have a TypedDict whose all values are List[something] and I would like to iterate over the dict's items() in a generic manner.

@JosiahKane
Copy link

JosiahKane commented Jun 12, 2020

I would also find this useful in annotating real world code, as indicated in the linked closed issue.

My actual usecase is a Computer Vision type problem where we're using TypedDict to package together corresponding images. For example, there might be an RGB image and a binary mask indicating the foreground of that image.

class Image:
    def save(self, filename: str) -> None:
        ...

class ColourImage(Image):
    ...

class BitmaskImage(Image):
    ...

@final
class ImageAnnotation(TypedDict):
    img: ColourImage
    foreground: BitmaskImage
    # Although these are all Images, we don't want to use a general Dict[str, Image] because we'd lose the specialisations.

def save_all(img_pack: ImageAnnotation, prefix: str) -> None:
    for k, v in img_pack.items():
        # Currently this fails because save is not defined for object. 
        v.save(f"{prefix}_{k}.png")
        # It would instead be necessary to say 
        # for k, v in cast(Mapping[str, Image], img_pack).items():

A particularly elegant construction that might be enabled by marking a TypedDict closed for extension is the ability to create a new one of the same kind with a comprehension. For example

ImageT = TypeVar("ImageT", bound=Image)

def resize(img: ImageT, scale: float) -> ImageT:
    ...

def resize_all(img_pack: ImageAnnotation, scale: float) -> ImageAnnotation:
    return {k: resize(v, scale) for k, v in img_pack.items()}
    # This wouldn't work even with a cast

@wmdrichards
Copy link

wmdrichards commented Aug 15, 2020

I've also just run into this issue -- use case is I have a number of TypedDicts that are all Mapping[str, T], but for each TypedDict I'd like to restrict the keys to a known set. Besides declaring @final, one other option that comes to mind is to make TypedDict a generic class, so that the following syntax would work:

class Point(TypedDict[float]):
    x: float
    y: float
    z: float

This would I think solve the use cases in this thread, without disallowing extension

@richardxia
Copy link

However, I'm not sure how useful this would be. I'd be interested in hearing if anybody has real-world use cases where this would be helpful.

I just hit a case where it would be useful to have a final TypedDict. I have a utility function that accepts a list[dict[str, str]] and writes it out to a CSV file using csv.DictWriter. I'd like to be able to pass arbitrary TypedDicts to the utility function where I know the type of all the keys in the TypedDict is str. Currently, it fails because the TypedDicts don't conform to dict[str, str], presumably because non-final TypedDicts don't preclude other keys with different values. Since I am constructing the TypedDicts all within Python, I can guarantee that there are no other keys present, but there's no way for me to communicate that to the type system.

I suppose I could use a dataclass, but one thing that is useful about using TypedDicts this way is that I can use keys that are not valid Python identifiers (e.g. containing spaces), which I can directly serialize as a header row of the CSV, making it easier for non-technical users to read the CSVs.

@intgr
Copy link
Contributor

intgr commented Aug 27, 2021

I think most use cases mentioned here would be better served by intersection types: python/typing#213

So if you want to express that all values of a dict are float and it also conforms to a TypedDict, you would write SomeTypedDict & dict[str, float] or Intersection[SomeTypedDict, dict[str, float]]. And you wouldn't need to artificially restrict subtyping of SomeTypedDict.

@smurzin
Copy link

smurzin commented Nov 9, 2021

This now also makes mypy diverge with pylance - microsoft/pyright#1899
They've decided to allow final for TypedDict in order to be able to provide type narrowing in Union[TypedDict] case.

Which is problematic for projects that use pylance for IDE/intellisense and mypy in CI.

@97littleleaf11
Copy link
Collaborator

Jukka‘s comment about this: #12266 (comment)

@AlexWaygood AlexWaygood added the topic-final PEP 591 label Mar 28, 2022
@godlygeek
Copy link
Contributor

godlygeek commented Aug 18, 2022

pylance ... decided to allow final for TypedDict in order to be able to provide type narrowing in Union[TypedDict] case.

I find myself in exactly this case. Imagine I'm hitting an API that returns a JSON response. If my request succeeded, the service responds with something like

{"transactionDetails": {"transactionId": "1234-56-7890"}}

but if my request failed the service response with something like

{"error": {"code": 1234, "description": "Oops!"}}

I can annotate each of those possibilities as a TypedDict:

class TransactionDetails(TypedDict):
    transactionId: str

class TransactionDetailsResponse(TypedDict):
    transactionDetails: TransactionDetails

class Error(TypedDict):
    code: int
    description: str

class ErrorResponse(TypedDict):
    error: Error

Response = ErrorResponse | TransactionDetailsResponse

At this point, what I'd like mypy to understand is that something typed as Response either has a "transactionDetails" key (in which case its type should narrow to TransactionDetailsResponse) or it has a "error" key (in which case its type should narrow to ErrorResponse).

Unfortunately, I'm instead stuck with casts:

def handle_response(response: Response):
    if "transactionDetails" in response:
        print(f"Transaction {cast(TransactionDetailsResponse, response)['transactionDetails']['transactionId']} submitted!")
    else:
        print(f"Got error {cast(ErrorResponse, response)['error']['code']}: {cast(ErrorResponse, response)['error']['description']}")

I really don't want that cast. I want to say that "transactionDetails" in response is a TypeGuard[TransactionDetailsResponse], and that seems to be exactly what pyright implemented. It's not onerous to me to mark these TypedDict's as @final, since they are required to have distinct keys in order for me to distinguish which type of response I got - that's a part of the service's API contract, and it doesn't make sense for there to be subtypes for that reason.

godlygeek added a commit to godlygeek/mypy that referenced this issue Aug 30, 2022
Allow a `TypedDict` to be decorated with `@final`. Like a regular class,
it is a type error to subclass from a final `TypedDict`.

Relates-to: python#7981
godlygeek added a commit to godlygeek/mypy that referenced this issue Aug 30, 2022
Allow a `TypedDict` to be decorated with `@final`. Like a regular class,
mypy will emit an error if a final `TypedDict` is subclassed.

Relates-to: python#7981
godlygeek added a commit to godlygeek/mypy that referenced this issue Aug 30, 2022
Allow a `TypedDict` to be decorated with `@final`. Like a regular class,
mypy will emit an error if a final `TypedDict` is subclassed.

Relates-to: python#7981
hauntsaninja pushed a commit that referenced this issue Sep 8, 2022
Allow a `TypedDict` to be decorated with `@final`. Like a regular class,
mypy will emit an error if a final `TypedDict` is subclassed.

Allow `@final` to be applied to a `TypedDict`, and have mypy emit an
error if class is derived from a final `TypedDict`. This goes some way
towards closing #7981 and closing a feature gap with pyright, though not
the whole way, as #7981 also asks for additional type narrowing for a
final `TypedDict`.
@Hnasar
Copy link
Contributor

Hnasar commented Apr 26, 2024

the output will be incorrectly upcast to object since mypy thinks it may be dealing with a subclass.

I ran into this limitation and realized mypy's behavior (with assuming `object) is consistent with PEP 589 - TypedDict and the TypedDict Typing Spec.

A TypedDict with all int values is not consistent with Mapping[str, int], since there may be additional non-int values not visible through the type, due to structural subtyping. These can be accessed using the values() and items() methods in Mapping, for example.

Currently the above Typing Spec page, and the one for @final don't clarify the interaction between these two features.

@erictraut
Copy link

Currently the above Typing Spec page, and the one for @final don't clarify the interaction between these two features.

The @final class decorator indicates that a class cannot be subclassed. This makes sense for classes that define nominal types. However, TypedDict is a structural type, similar to a Protocol. That means two TypedDict classes with different names but the same field definitions are equivalent types. Their names and hierarchies don't matter for determining type consistency. For that reason, @final has no impact on a TypedDict type consistency rules, nor should it change the behavior of items or values.

What you're looking for is a new concept referred to as a "closed" TypedDict. This concept is introduced in draft PEP 728. It allows one to specify that a TypedDict cannot have any extra fields beyond the ones that are defined. (Alternatively, it allows for additional extra fields that are constrained to a particular type.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests