python assignment self

Demystifying the self Parameter: A Complete Guide to Using self in Python Classes

The self parameter is a special parameter that is passed to methods in Python classes. It binds the instance of the class to the method, allowing the method to access and modify the attributes and methods of the class instance. Properly utilizing self is key to writing effective Python classes. This guide will provide a deep dive into self - what it is, when it’s used, and how to leverage it to create well-designed class methods in Python.

Table of Contents

What is self , 1. instance methods, 2. constructor methods, how does self work, when is self not needed, getting attributes, setting attributes, calling methods, modifying multiple instances, caveats of using self, data classes, game characters, networking with sockets.

In Python, self refers to the instance of the class that a method is being called on. By convention, self is always the first parameter passed to methods in a class. For example:

Here, when greet() is called on the john instance, john gets passed automatically as the self argument. This allows greet() to access the name attribute to print the greeting message specific to the john instance.

In a nutshell, self allows a method to have access to the attributes and methods of the class instance it is called on. Without self , methods would not know which instance they are interacting with.

When is self used?

The self parameter is used in two primary cases:

In instance methods - methods that are called on an instance of a class.

In the __init__ constructor method to initialize instance attributes.

Essentially, any method defined inside a class declaration should have self as the first parameter if you want to access attributes and methods of the class instance.

Let’s expand on these two main use cases:

Instance methods are methods that are designed to be called on instances of a class. That is, they are not standalone functions, but rather behaviors that a class instance can exhibit.

Instance methods always take self as the first parameter. By convention, it is named self but could technically be named anything (though self is universally preferred).

When calling an instance method, Python binds the instance to self automatically so you do not need to pass it explicitly.

For example:

This allows area() to access the width and height of the rect instance through self.width and self.height .

Without self , the method would not know which rectangle instance to operate on.

The __init__ method is a special constructor method that is called automatically when an instance of a class is created. It is used to initialize the attributes of a class instance.

__init__ always takes self as the first parameter, along with any other parameters needed to instantiate the class.

Here, when creating the sam instance, __init__ is called automatically with sam passed as self and the name and breed passed as the other two parameters.

This allows the name and breed attributes to be initialized on the sam instance properly.

So in summary, self is utilized in __init__ to set up the new class instance.

Behind the scenes, when a method is called on an instance, Python automatically passes the instance as the first argument to the method.

Is equivalent to:

So self gives us a way to access the object instance via a parameter to the method call. This is what enables instance methods to operate on instance state.

Without the use of self , code inside a method would not be able to access or modify attributes of the instance it was called on. Attempting to access self.some_attr would raise an error about that attribute being undefined.

There are some cases in Python where self is not explicitly required as a method parameter:

Static methods - These behave like regular functions, not operating on an instance or its state. They have the @staticmethod decorator.

Class methods - These operate at the class level rather than instance level. They take cls as the first arg instead of self . Denoted with @classmethod .

Functions outside a class - Regular functions defined outside a class definition do not take self or any instance, they operate independently.

In these cases, self is not necessary because the method or function does not operate on an instance state.

Accessing Instance Attributes with self

Now that we understand what self represents, let’s explore how we can leverage it to access attributes and methods of a class instance:

We can access any attribute defined on an instance via self in a method.

This accesses the name and age attributes of the john instance using self.name and self.age

We can also set or update attributes using assignment via self :

Here birthday() increments john ’s age using self.age += 1 .

In addition to attributes, self allows a method to call other methods on the instance:

This allows methods to reuse behavior and operate on the instance.

A key benefit of using self is that we can define methods that uniformly operate on attributes across all instances of a class.

The increment() method will properly increment the counter for each instance it is called on. Without self , the counters would incorrectly share state and not work properly.

This allows us to write reusable methods that can operate on any instance of a class in a encapsulated manner.

While self is useful, there are some caveats to keep in mind:

Methods that use self can only be called on class instances, not the class itself. Attempting to call Class.method() will raise an error if self is used.

All attributes must be accessed via self . Code like count += 1 would modify a local variable, not the instance attribute.

It is easy to accidentally overwrite self in a method with a local variable, losing access to the instance. Avoid reassigning self .

Methods that do not need access to instance state should be made static or class methods to avoid unnecessary use of self .

self can be confusing for developers coming from other languages. Take time to properly learn Python instance methods and self usage.

With disciplined coding, self enables writing reusable classes that operate on encapsulated instance state. But it takes practice to use properly and effectively.

Example Use Cases

To illustrate practical applications of self in Python, let’s walk through some examples:

self allows custom data classes to store state and define constructors, string representations, and comparisons:

This provides convenient class functionality building on self .

In a game, self can be used to track character stats and call actions:

This keeps each character’s state encapsulated and reusable.

For network programming with sockets:

self is leveraged to track each connection and send/receive data accordingly.

The examples showcase just a fraction of how self can be utilized for clean class design across domains like data science, games, networking, and more.

To summarize proper usage of the self parameter in Python:

self refers to the instance a method is called on and is automatically passed by Python.

It provides access to attributes and methods on the instance via self.attr and self.method() .

self should be the first parameter in instance methods and __init__ constructors.

It enables read/write access to state across instances separately.

Avoid reassigning self accidentally inside a method definition.

Use self to write reusable, encapsulated classes in Python.

With this knowledge, you are now equipped to leverage self effectively in your own Python classes and unlock the full power of object-oriented programming in Python. The self parameter ties together key concepts like encapsulation, access control, and polymorphism. Mastering its use will level up your Python class design skills.

•     python
•     object-oriented-programming-oop

Understanding Python self Variable with Examples

Python self variable is used to bind the instance of the class to the instance method. We have to explicitly declare it as the first method argument to access the instance variables and methods. This variable is used only with the instance methods.

In most of the Object-Oriented programming languages, you can access the current object in a method without the need for explicitly having it as a method parameter. For example, we can use “ this ” keyword to access the current object in Java program. But, in Python we have to explicitly declare the object instance as “self” variable.

Python self is a keyword?

Python self variable is not a reserved keyword . But, it’s the best practice and convention to use the variable name as “self” to refer to the instance.

Python self variable Example

Let’s say we have a Dog class defined as below.

Output : Labrador is barking.

• The __init__() function is defined with two variables but when we are creating the Dog instance, we have to provide only one argument. The “self” is automatically assigned to the newly created instance of Dog class.
• The bark() method has only one argument – “self” – which gets bind to the Dog instance that calls this method. That’s why we are not passing any argument when calling the bark() method.
• If we have to access any instance variable in the function, we can use the dot operator.

Can we skip “self” variable?

What if the instance method doesn’t need to access instance variables. Can we skip the self variable in this case?

Let’s find out with a simple example.

If you will run the above code, there won’t be any error. But, we are not calling the bark() method. Let’s see what happens when we try to call the bark() method.

We are getting error as the bark() method accepts 0 argument but we provided 1. It’s because when we are calling d.bark() , the “d” instance is automatically passed as the first argument to the bark() instance method.

But, if we access the bark() instance method through class reference then it will work fine. So, calling Dog.bark() will not cause any errors.

Similar variables for Class Method and Static Method?

The same behavior is present with the Class methods too. The only difference is that the convention is to use “ cls ” as the variable name for the Class reference.

However, it’s not required with a static method. Because, the static methods are self sufficient functions and they can’t access any of the class variables or functions directly.

Let’s look at a complete example with self and cls variables and a static method without any arguments.

Quick Example to Break the Convention

This example is just to show you that it’s not mandatory to use the variable name as “self” and “cls”. In real programming, always stick to this convention.

The “self” variable is bound to the current instance

The self variable gives us access to the current instance properties. We can confirm this with a simple example by creating two different instances of the Dog class.

Why not make “self” variable implicit?

There had been a lot of suggestions to make the “self” variable a reserved keyword and implicitly available to the instance method. But, the suggestion was rejected by “Guido van Rossum”. You can read about them here and here .

Python Tutorial

File handling, python modules, python numpy, python pandas, python matplotlib, python scipy, machine learning, python mysql, python mongodb, python reference, module reference, python how to, python examples, python assignment operators.

Assignment operators are used to assign values to variables:

Related Pages

COLOR PICKER

Contact Sales

If you want to use W3Schools services as an educational institution, team or enterprise, send us an e-mail: [email protected]

Report Error

If you want to report an error, or if you want to make a suggestion, send us an e-mail: [email protected]

Top Tutorials

Top references, top examples, get certified.

Member-only story

What Is Python’s “Self” Argument, Anyway?

A behind-the-scenes look into this well-known argument.

Martin Heinz

Better Programming

Every Python developer is familiar with the self argument, which is present in every* method declaration of every class. We all know how to use it, but do you really know what it is, why it's there, and how it works under the hood?

What We Already Know

Written by Martin Heinz

CKA | RHCE | DevOps Engineer | Working with Python, Kubernetes, Linux and more | https://martinheinz.dev/ | https://ko-fi.com/martinheinz

Text to speech

Python Enhancement Proposals

• Python »
• PEP Index »

PEP 572 – Assignment Expressions

The importance of real code, exceptional cases, scope of the target, relative precedence of :=, change to evaluation order, differences between assignment expressions and assignment statements, specification changes during implementation, _pydecimal.py, datetime.py, sysconfig.py, simplifying list comprehensions, capturing condition values, changing the scope rules for comprehensions, alternative spellings, special-casing conditional statements, special-casing comprehensions, lowering operator precedence, allowing commas to the right, always requiring parentheses, why not just turn existing assignment into an expression, with assignment expressions, why bother with assignment statements, why not use a sublocal scope and prevent namespace pollution, style guide recommendations, acknowledgements, a numeric example, appendix b: rough code translations for comprehensions, appendix c: no changes to scope semantics.

This is a proposal for creating a way to assign to variables within an expression using the notation NAME := expr .

As part of this change, there is also an update to dictionary comprehension evaluation order to ensure key expressions are executed before value expressions (allowing the key to be bound to a name and then re-used as part of calculating the corresponding value).

During discussion of this PEP, the operator became informally known as “the walrus operator”. The construct’s formal name is “Assignment Expressions” (as per the PEP title), but they may also be referred to as “Named Expressions” (e.g. the CPython reference implementation uses that name internally).

Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts.

Additionally, naming sub-parts of a large expression can assist an interactive debugger, providing useful display hooks and partial results. Without a way to capture sub-expressions inline, this would require refactoring of the original code; with assignment expressions, this merely requires the insertion of a few name := markers. Removing the need to refactor reduces the likelihood that the code be inadvertently changed as part of debugging (a common cause of Heisenbugs), and is easier to dictate to another programmer.

During the development of this PEP many people (supporters and critics both) have had a tendency to focus on toy examples on the one hand, and on overly complex examples on the other.

The danger of toy examples is twofold: they are often too abstract to make anyone go “ooh, that’s compelling”, and they are easily refuted with “I would never write it that way anyway”.

The danger of overly complex examples is that they provide a convenient strawman for critics of the proposal to shoot down (“that’s obfuscated”).

Yet there is some use for both extremely simple and extremely complex examples: they are helpful to clarify the intended semantics. Therefore, there will be some of each below.

However, in order to be compelling , examples should be rooted in real code, i.e. code that was written without any thought of this PEP, as part of a useful application, however large or small. Tim Peters has been extremely helpful by going over his own personal code repository and picking examples of code he had written that (in his view) would have been clearer if rewritten with (sparing) use of assignment expressions. His conclusion: the current proposal would have allowed a modest but clear improvement in quite a few bits of code.

Another use of real code is to observe indirectly how much value programmers place on compactness. Guido van Rossum searched through a Dropbox code base and discovered some evidence that programmers value writing fewer lines over shorter lines.

Case in point: Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code, e.g. instead of writing:

they would write:

Another example illustrates that programmers sometimes do more work to save an extra level of indentation:

This code tries to match pattern2 even if pattern1 has a match (in which case the match on pattern2 is never used). The more efficient rewrite would have been:

Syntax and semantics

In most contexts where arbitrary Python expressions can be used, a named expression can appear. This is of the form NAME := expr where expr is any valid Python expression other than an unparenthesized tuple, and NAME is an identifier.

The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:

There are a few places where assignment expressions are not allowed, in order to avoid ambiguities or user confusion:

This rule is included to simplify the choice for the user between an assignment statement and an assignment expression – there is no syntactic position where both are valid.

Again, this rule is included to avoid two visually similar ways of saying the same thing.

This rule is included to disallow excessively confusing code, and because parsing keyword arguments is complex enough already.

This rule is included to discourage side effects in a position whose exact semantics are already confusing to many users (cf. the common style recommendation against mutable default values), and also to echo the similar prohibition in calls (the previous bullet).

The reasoning here is similar to the two previous cases; this ungrouped assortment of symbols and operators composed of : and = is hard to read correctly.

This allows lambda to always bind less tightly than := ; having a name binding at the top level inside a lambda function is unlikely to be of value, as there is no way to make use of it. In cases where the name will be used more than once, the expression is likely to need parenthesizing anyway, so this prohibition will rarely affect code.

This shows that what looks like an assignment operator in an f-string is not always an assignment operator. The f-string parser uses : to indicate formatting options. To preserve backwards compatibility, assignment operator usage inside of f-strings must be parenthesized. As noted above, this usage of the assignment operator is not recommended.

An assignment expression does not introduce a new scope. In most cases the scope in which the target will be bound is self-explanatory: it is the current scope. If this scope contains a nonlocal or global declaration for the target, the assignment expression honors that. A lambda (being an explicit, if anonymous, function definition) counts as a scope for this purpose.

There is one special case: an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as “comprehensions”) binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope.

The motivation for this special case is twofold. First, it allows us to conveniently capture a “witness” for an any() expression, or a counterexample for all() , for example:

Second, it allows a compact way of updating mutable state from a comprehension, for example:

However, an assignment expression target name cannot be the same as a for -target name appearing in any comprehension containing the assignment expression. The latter names are local to the comprehension in which they appear, so it would be contradictory for a contained use of the same name to refer to the scope containing the outermost comprehension instead.

For example, [i := i+1 for i in range(5)] is invalid: the for i part establishes that i is local to the comprehension, but the i := part insists that i is not local to the comprehension. The same reason makes these examples invalid too:

While it’s technically possible to assign consistent semantics to these cases, it’s difficult to determine whether those semantics actually make sense in the absence of real use cases. Accordingly, the reference implementation [1] will ensure that such cases raise SyntaxError , rather than executing with implementation defined behaviour.

This restriction applies even if the assignment expression is never executed:

For the comprehension body (the part before the first “for” keyword) and the filter expression (the part after “if” and before any nested “for”), this restriction applies solely to target names that are also used as iteration variables in the comprehension. Lambda expressions appearing in these positions introduce a new explicit function scope, and hence may use assignment expressions with no additional restrictions.

Due to design constraints in the reference implementation (the symbol table analyser cannot easily detect when names are re-used between the leftmost comprehension iterable expression and the rest of the comprehension), named expressions are disallowed entirely as part of comprehension iterable expressions (the part after each “in”, and before any subsequent “if” or “for” keyword):

A further exception applies when an assignment expression occurs in a comprehension whose containing scope is a class scope. If the rules above were to result in the target being assigned in that class’s scope, the assignment expression is expressly invalid. This case also raises SyntaxError :

(The reason for the latter exception is the implicit function scope created for comprehensions – there is currently no runtime mechanism for a function to refer to a variable in the containing class scope, and we do not want to add such a mechanism. If this issue ever gets resolved this special case may be removed from the specification of assignment expressions. Note that the problem already exists for using a variable defined in the class scope from a comprehension.)

See Appendix B for some examples of how the rules for targets in comprehensions translate to equivalent code.

The := operator groups more tightly than a comma in all syntactic positions where it is legal, but less tightly than all other operators, including or , and , not , and conditional expressions ( A if C else B ). As follows from section “Exceptional cases” above, it is never allowed at the same level as = . In case a different grouping is desired, parentheses should be used.

The := operator may be used directly in a positional function call argument; however it is invalid directly in a keyword argument.

Some examples to clarify what’s technically valid or invalid:

Most of the “valid” examples above are not recommended, since human readers of Python source code who are quickly glancing at some code may miss the distinction. But simple cases are not objectionable:

This PEP recommends always putting spaces around := , similar to PEP 8 ’s recommendation for = when used for assignment, whereas the latter disallows spaces around = used for keyword arguments.)

In order to have precisely defined semantics, the proposal requires evaluation order to be well-defined. This is technically not a new requirement, as function calls may already have side effects. Python already has a rule that subexpressions are generally evaluated from left to right. However, assignment expressions make these side effects more visible, and we propose a single change to the current evaluation order:

• In a dict comprehension {X: Y for ...} , Y is currently evaluated before X . We propose to change this so that X is evaluated before Y . (In a dict display like {X: Y} this is already the case, and also in dict((X, Y) for ...) which should clearly be equivalent to the dict comprehension.)

Most importantly, since := is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions.

Conversely, assignment expressions don’t support the advanced features found in assignment statements:

• Multiple targets are not directly supported: x = y = z = 0 # Equivalent: (z := (y := (x := 0)))
• Single assignment targets other than a single NAME are not supported: # No equivalent a [ i ] = x self . rest = []
• Priority around commas is different: x = 1 , 2 # Sets x to (1, 2) ( x := 1 , 2 ) # Sets x to 1
• Iterable packing and unpacking (both regular or extended forms) are not supported: # Equivalent needs extra parentheses loc = x , y # Use (loc := (x, y)) info = name , phone , * rest # Use (info := (name, phone, *rest)) # No equivalent px , py , pz = position name , phone , email , * other_info = contact
• Inline type annotations are not supported: # Closest equivalent is "p: Optional[int]" as a separate declaration p : Optional [ int ] = None
• Augmented assignment is not supported: total += tax # Equivalent: (total := total + tax)

The following changes have been made based on implementation experience and additional review after the PEP was first accepted and before Python 3.8 was released:

• for consistency with other similar exceptions, and to avoid locking in an exception name that is not necessarily going to improve clarity for end users, the originally proposed TargetScopeError subclass of SyntaxError was dropped in favour of just raising SyntaxError directly. [3]
• due to a limitation in CPython’s symbol table analysis process, the reference implementation raises SyntaxError for all uses of named expressions inside comprehension iterable expressions, rather than only raising them when the named expression target conflicts with one of the iteration variables in the comprehension. This could be revisited given sufficiently compelling examples, but the extra complexity needed to implement the more selective restriction doesn’t seem worthwhile for purely hypothetical use cases.

Examples from the Python standard library

env_base is only used on these lines, putting its assignment on the if moves it as the “header” of the block.

• Current: env_base = os . environ . get ( "PYTHONUSERBASE" , None ) if env_base : return env_base
• Improved: if env_base := os . environ . get ( "PYTHONUSERBASE" , None ): return env_base

Avoid nested if and remove one indentation level.

• Current: if self . _is_special : ans = self . _check_nans ( context = context ) if ans : return ans
• Improved: if self . _is_special and ( ans := self . _check_nans ( context = context )): return ans

Code looks more regular and avoid multiple nested if. (See Appendix A for the origin of this example.)

• Current: reductor = dispatch_table . get ( cls ) if reductor : rv = reductor ( x ) else : reductor = getattr ( x , "__reduce_ex__" , None ) if reductor : rv = reductor ( 4 ) else : reductor = getattr ( x , "__reduce__" , None ) if reductor : rv = reductor () else : raise Error ( "un(deep)copyable object of type %s " % cls )
• Improved: if reductor := dispatch_table . get ( cls ): rv = reductor ( x ) elif reductor := getattr ( x , "__reduce_ex__" , None ): rv = reductor ( 4 ) elif reductor := getattr ( x , "__reduce__" , None ): rv = reductor () else : raise Error ( "un(deep)copyable object of type %s " % cls )

tz is only used for s += tz , moving its assignment inside the if helps to show its scope.

• Current: s = _format_time ( self . _hour , self . _minute , self . _second , self . _microsecond , timespec ) tz = self . _tzstr () if tz : s += tz return s
• Improved: s = _format_time ( self . _hour , self . _minute , self . _second , self . _microsecond , timespec ) if tz := self . _tzstr (): s += tz return s

Calling fp.readline() in the while condition and calling .match() on the if lines make the code more compact without making it harder to understand.

• Current: while True : line = fp . readline () if not line : break m = define_rx . match ( line ) if m : n , v = m . group ( 1 , 2 ) try : v = int ( v ) except ValueError : pass vars [ n ] = v else : m = undef_rx . match ( line ) if m : vars [ m . group ( 1 )] = 0
• Improved: while line := fp . readline (): if m := define_rx . match ( line ): n , v = m . group ( 1 , 2 ) try : v = int ( v ) except ValueError : pass vars [ n ] = v elif m := undef_rx . match ( line ): vars [ m . group ( 1 )] = 0

A list comprehension can map and filter efficiently by capturing the condition:

Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:

Note that in both cases the variable y is bound in the containing scope (i.e. at the same level as results or stuff ).

Assignment expressions can be used to good effect in the header of an if or while statement:

Particularly with the while loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value.

An example from the low-level UNIX world:

Rejected alternative proposals

Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above.

A previous version of this PEP proposed subtle changes to the scope rules for comprehensions, to make them more usable in class scope and to unify the scope of the “outermost iterable” and the rest of the comprehension. However, this part of the proposal would have caused backwards incompatibilities, and has been withdrawn so the PEP can focus on assignment expressions.

Broadly the same semantics as the current proposal, but spelled differently.

Since EXPR as NAME already has meaning in import , except and with statements (with different semantics), this would create unnecessary confusion or require special-casing (e.g. to forbid assignment within the headers of these statements).

(Note that with EXPR as VAR does not simply assign the value of EXPR to VAR – it calls EXPR.__enter__() and assigns the result of that to VAR .)

Additional reasons to prefer := over this spelling include:

• In if f(x) as y the assignment target doesn’t jump out at you – it just reads like if f x blah blah and it is too similar visually to if f(x) and y .
• import foo as bar
• except Exc as var
• with ctxmgr() as var

To the contrary, the assignment expression does not belong to the if or while that starts the line, and we intentionally allow assignment expressions in other contexts as well.

• NAME = EXPR
• if NAME := EXPR

reinforces the visual recognition of assignment expressions.

This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow y <- f(x) is not possible in Python, as it would be interpreted as less-than and unary minus.) This syntax has a slight advantage over ‘as’ in that it does not conflict with with , except and import , but otherwise is equivalent. But it is entirely unrelated to Python’s other use of -> (function return type annotations), and compared to := (which dates back to Algol-58) it has a much weaker tradition.

This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable’s scope is encoded into its name, making refactoring harder.

Execution order is inverted (the indented body is performed first, followed by the “header”). This requires a new keyword, unless an existing keyword is repurposed (most likely with: ). See PEP 3150 for prior discussion on this subject (with the proposed keyword being given: ).

This syntax has fewer conflicts than as does (conflicting only with the raise Exc from Exc notation), but is otherwise comparable to it. Instead of paralleling with expr as target: (which can be useful but can also be confusing), this has no parallels, but is evocative.

One of the most popular use-cases is if and while statements. Instead of a more general solution, this proposal enhances the syntax of these two statements to add a means of capturing the compared value:

This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return '' when done), and completely useless in more complicated cases (e.g. where the condition is f(x) < 0 and you want to capture the value of f(x) ). It also has no benefit to list comprehensions.

Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction of possible use-cases, even in if / while statements.

Another common use-case is comprehensions (list/set/dict, and genexps). As above, proposals have been made for comprehension-specific solutions.

This brings the subexpression to a location in between the ‘for’ loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, where reads the most cleanly, but also has the greatest potential for conflict (e.g. SQLAlchemy and numpy have where methods, as does tkinter.dnd.Icon in the standard library).

As above, but reusing the with keyword. Doesn’t read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into “longhand” for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why “with NAME = EXPR:” cannot be used as a statement on its own.

As per option 2, but using as rather than an equals sign. Aligns syntactically with other uses of as for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of with inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax.

Regardless of the spelling chosen, this introduces a stark difference between comprehensions and the equivalent unrolled long-hand form of the loop. It is no longer possible to unwrap the loop into statement form without reworking any name bindings. The only keyword that can be repurposed to this task is with , thus giving it sneakily different semantics in a comprehension than in a statement; alternatively, a new keyword is needed, with all the costs therein.

There are two logical precedences for the := operator. Either it should bind as loosely as possible, as does statement-assignment; or it should bind more tightly than comparison operators. Placing its precedence between the comparison and arithmetic operators (to be precise: just lower than bitwise OR) allows most uses inside while and if conditions to be spelled without parentheses, as it is most likely that you wish to capture the value of something, then perform a comparison on it:

Once find() returns -1, the loop terminates. If := binds as loosely as = does, this would capture the result of the comparison (generally either True or False ), which is less useful.

While this behaviour would be convenient in many situations, it is also harder to explain than “the := operator behaves just like the assignment statement”, and as such, the precedence for := has been made as close as possible to that of = (with the exception that it binds tighter than comma).

Some critics have claimed that the assignment expressions should allow unparenthesized tuples on the right, so that these two would be equivalent:

(With the current version of the proposal, the latter would be equivalent to ((point := x), y) .)

However, adopting this stance would logically lead to the conclusion that when used in a function call, assignment expressions also bind less tight than comma, so we’d have the following confusing equivalence:

The less confusing option is to make := bind more tightly than comma.

It’s been proposed to just always require parentheses around an assignment expression. This would resolve many ambiguities, and indeed parentheses will frequently be needed to extract the desired subexpression. But in the following cases the extra parentheses feel redundant:

Frequently Raised Objections

C and its derivatives define the = operator as an expression, rather than a statement as is Python’s way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between if (x == y) and if (x = y) belies their drastically different semantics. Thus this proposal uses := to clarify the distinction.

The two forms have different flexibilities. The := operator can be used inside a larger expression; the = statement can be augmented to += and its friends, can be chained, and can assign to attributes and subscripts.

Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from for loops or other constructs, and can be solved the same way: del the name once it is no longer needed, or prefix it with an underscore.

(The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2] )

As expression assignments can sometimes be used equivalently to statement assignments, the question of which should be preferred will arise. For the benefit of style guides such as PEP 8 , two recommendations are suggested.

• If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent.
• If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead.

The authors wish to thank Alyssa Coghlan and Steven D’Aprano for their considerable contributions to this proposal, and members of the core-mentorship mailing list for assistance with implementation.

Appendix A: Tim Peters’s findings

Here’s a brief essay Tim Peters wrote on the topic.

I dislike “busy” lines of code, and also dislike putting conceptually unrelated logic on a single line. So, for example, instead of:

instead. So I suspected I’d find few places I’d want to use assignment expressions. I didn’t even consider them for lines already stretching halfway across the screen. In other cases, “unrelated” ruled:

is a vast improvement over the briefer:

The original two statements are doing entirely different conceptual things, and slamming them together is conceptually insane.

In other cases, combining related logic made it harder to understand, such as rewriting:

as the briefer:

The while test there is too subtle, crucially relying on strict left-to-right evaluation in a non-short-circuiting or method-chaining context. My brain isn’t wired that way.

But cases like that were rare. Name binding is very frequent, and “sparse is better than dense” does not mean “almost empty is better than sparse”. For example, I have many functions that return None or 0 to communicate “I have nothing useful to return in this case, but since that’s expected often I’m not going to annoy you with an exception”. This is essentially the same as regular expression search functions returning None when there is no match. So there was lots of code of the form:

I find that clearer, and certainly a bit less typing and pattern-matching reading, as:

It’s also nice to trade away a small amount of horizontal whitespace to get another _line_ of surrounding code on screen. I didn’t give much weight to this at first, but it was so very frequent it added up, and I soon enough became annoyed that I couldn’t actually run the briefer code. That surprised me!

There are other cases where assignment expressions really shine. Rather than pick another from my code, Kirill Balunov gave a lovely example from the standard library’s copy() function in copy.py :

The ever-increasing indentation is semantically misleading: the logic is conceptually flat, “the first test that succeeds wins”:

Using easy assignment expressions allows the visual structure of the code to emphasize the conceptual flatness of the logic; ever-increasing indentation obscured it.

A smaller example from my code delighted me, both allowing to put inherently related logic in a single line, and allowing to remove an annoying “artificial” indentation level:

That if is about as long as I want my lines to get, but remains easy to follow.

So, in all, in most lines binding a name, I wouldn’t use assignment expressions, but because that construct is so very frequent, that leaves many places I would. In most of the latter, I found a small win that adds up due to how often it occurs, and in the rest I found a moderate to major win. I’d certainly use it more often than ternary if , but significantly less often than augmented assignment.

I have another example that quite impressed me at the time.

Where all variables are positive integers, and a is at least as large as the n’th root of x, this algorithm returns the floor of the n’th root of x (and roughly doubling the number of accurate bits per iteration):

It’s not obvious why that works, but is no more obvious in the “loop and a half” form. It’s hard to prove correctness without building on the right insight (the “arithmetic mean - geometric mean inequality”), and knowing some non-trivial things about how nested floor functions behave. That is, the challenges are in the math, not really in the coding.

If you do know all that, then the assignment-expression form is easily read as “while the current guess is too large, get a smaller guess”, where the “too large?” test and the new guess share an expensive sub-expression.

To my eyes, the original form is harder to understand:

This appendix attempts to clarify (though not specify) the rules when a target occurs in a comprehension or in a generator expression. For a number of illustrative examples we show the original code, containing a comprehension, and the translation, where the comprehension has been replaced by an equivalent generator function plus some scaffolding.

Since [x for ...] is equivalent to list(x for ...) these examples all use list comprehensions without loss of generality. And since these examples are meant to clarify edge cases of the rules, they aren’t trying to look like real code.

Note: comprehensions are already implemented via synthesizing nested generator functions like those in this appendix. The new part is adding appropriate declarations to establish the intended scope of assignment expression targets (the same scope they resolve to as if the assignment were performed in the block containing the outermost comprehension). For type inference purposes, these illustrative expansions do not imply that assignment expression targets are always Optional (but they do indicate the target binding scope).

Let’s start with a reminder of what code is generated for a generator expression without assignment expression.

• Original code (EXPR usually references VAR): def f (): a = [ EXPR for VAR in ITERABLE ]
• Translation (let’s not worry about name conflicts): def f (): def genexpr ( iterator ): for VAR in iterator : yield EXPR a = list ( genexpr ( iter ( ITERABLE )))

Let’s add a simple assignment expression.

• Original code: def f (): a = [ TARGET := EXPR for VAR in ITERABLE ]
• Translation: def f (): if False : TARGET = None # Dead code to ensure TARGET is a local variable def genexpr ( iterator ): nonlocal TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Let’s add a global TARGET declaration in f() .

• Original code: def f (): global TARGET a = [ TARGET := EXPR for VAR in ITERABLE ]
• Translation: def f (): global TARGET def genexpr ( iterator ): global TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Or instead let’s add a nonlocal TARGET declaration in f() .

• Original code: def g (): TARGET = ... def f (): nonlocal TARGET a = [ TARGET := EXPR for VAR in ITERABLE ]
• Translation: def g (): TARGET = ... def f (): nonlocal TARGET def genexpr ( iterator ): nonlocal TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Finally, let’s nest two comprehensions.

• Original code: def f (): a = [[ TARGET := i for i in range ( 3 )] for j in range ( 2 )] # I.e., a = [[0, 1, 2], [0, 1, 2]] print ( TARGET ) # prints 2
• Translation: def f (): if False : TARGET = None def outer_genexpr ( outer_iterator ): nonlocal TARGET def inner_generator ( inner_iterator ): nonlocal TARGET for i in inner_iterator : TARGET = i yield i for j in outer_iterator : yield list ( inner_generator ( range ( 3 ))) a = list ( outer_genexpr ( range ( 2 ))) print ( TARGET )

Because it has been a point of confusion, note that nothing about Python’s scoping semantics is changed. Function-local scopes continue to be resolved at compile time, and to have indefinite temporal extent at run time (“full closures”). Example:

This document has been placed in the public domain.

Source: https://github.com/python/peps/blob/main/peps/pep-0572.rst

Last modified: 2023-10-11 12:05:51 GMT

Python In-Place Assignment Operators

In-place assignment operators (also called compound assignment operators) perform an operation in-place on a variable provided as first operand. They overwrite the value of the first operand variable with the result of the operation when performing the operator without assignment. For example, x += 3 is the same as x = x + 3 of first calculating the result of x + 3 and then assigning it to the variable x.

You can watch me go over all of these operators in the following video:

We’ll rush over all in-place operators one-by-one next!

Python In-Place Addition

Python provides the operator x += y to add two objects in-place by calculating the sum x + y and assigning the result to the first operands variable name x . You can set up the in-place addition behavior for your own class by overriding the magic “dunder” method __iadd__(self, other) in your class definition.

The expression x += y is syntactical sugar for the longer-form x = x + y :

Related Tutorial: Python In-Place Addition

Python In-Place Subtraction

Python provides the operator x -= y to subtract two objects in-place by calculating the difference x - y and assigning the result to the first operands variable name x . You can set up the in-place subtraction behavior for your own class by overriding the magic “dunder” method __isub__(self, other) in your class definition.

The expression x -= y is syntactical sugar for the longer-form x = x - y :

Related Tutorial: Python In-Place Subtraction

Python In-Place Multiplication

Python provides the operator x *= y to multiply two objects in-place by calculating the product x * y and assigning the result to the first operands variable name x . You can set up the in-place multiplication behavior for your own class by overriding the magic “dunder” method __imul__(self, other) in your class definition.

The expression x *= y is syntactical sugar for the longer-form x = x * y :

Related Tutorial: Python In-Place Multiplication

Python In-Place Division

Python’s in-place division operator x /= y divides two objects in-place by calculating x / y and assigning the result to the first operands variable name x . Set up in-place division for your own class by overriding the magic “dunder” method __truediv__(self, other) in your class definition.

The expression x /= y is syntactical sugar for the longer-form x = x / y :

Related Tutorial: Python In-Place Division

Python In-Place Modulo

Python provides the operator x %= y to calculate the modulo operation x % y , and assign the result in-place to the first operands variable x . You can set up the in-place modulo behavior for your own class by overriding the magic “dunder” method __imod__(self, other) in your class definition.

The expression x %= y is syntactical sugar for the longer-form x = x % y :

Related Tutorial: Python In-Place Modulo

Python In-Place Integer Division

Python’s in-place integer division operator x //= y divides two objects in-place by calculating x // y and assigning the result to the first operands variable name x . Set up in-place integer (or floor) division for your own class by overriding the magic “dunder” method __floordiv__(self, other) in your class definition.

Related Tutorial: Python In-Place Integer Division

Python In-Place Exponentiation

Python provides the in-place exponentiation operator x **= y that raises x to the power of y using x ** y and assigns the result to the first operands variable name x . You can set up the in-place exponentiation behavior for your own class by overriding the magic “dunder” method __ipow__(self, other) in your class definition.

The expression x **= y is syntactical sugar for the longer-form x = x ** y :

Related Tutorial: Python In-Place Exponentiation

Python In-Place Bitwise AND

Python’s in-place bitwise AND operator x &= y calcualtes bitwise-and x & y and assigns the result to the first operand x . To set it up for your own class, override the magic “dunder” method __iand__(self, other) in your class definition.

The expression x &= y is syntactical sugar for the longer-form x = x & y :

Related Tutorial: Python In-Place Bitwise AND

Python In-Place Bitwise OR

Python’s A |= B applies the | operator in place. Thus, it is semantically identical to the longer-form version A = A | B of first performing the operation A | B and then assigning the result to the variable A .

The following minimal example creates two Boolean variables A and B and performs the in-place B |= A operation to perform a logical OR operation B | A and assigning the result to the first operand B that becomes True :

In this example, you’ve seen this in-place operation on Boolean operands. But the | operator is overloaded in Python. The three most frequent use cases for the | and |= operators are the following:

• Python Sets : set union operator
• Python Dictionaries : dictionary update operator
• Python Booleans : logical OR operator

Related Tutorial: Python In-Place Bitwise OR

Python In-Place Bitwise XOR

Python’s in-place bitwise XOR operator x ^= y calcualtes bitwise XOR x ^ y and assigns the result to the first operand x . To set this up for your own class, override the magic “dunder” method __ixor__(self, other) in your class definition.

The expression x ^ = y is syntactical sugar for the longer-form x = x ^ y :

Related Tutorial: Python In-Place Bitwise XOR

Python In-Place Bitwise Right-Shift

Python’s in-place bitwise right-shift operator x >>= y calculates the right-shift operation x >> y , and assigns the result to the first operands variable name x . You can set up the in-place right-shift behavior in your own class by overriding the magic “dunder” method __irshift__(self, other) in your class definition.

The expression x >>= y is syntactical sugar for the longer-form x = x >> y :

Related Tutorial: Python In-Place Bitwise Right-Shift

Python In-Place Bitwise Left-Shift

Python’s in-place bitwise left-shift operator x <<= y calculates the left-shift operation x << y , and assigns the result to the first operands variable name x . You can set up the in-place left-shift behavior in your own class by overriding the magic “dunder” method __ilshift__(self, other) in your class definition.

The expression x <<= y is syntactical sugar for the longer-form x = x << y :

Related Tutorial: Python In-Place Bitwise Left-Shift

Python In-Place Magic Methods

The following table provides the names of the magic methods you need to define to enable in-place operators on your custom class:

In the following code example, we create a custom class Data and define our “magic” double-underscore methods so that we can perform in-place computations on objects of this class.

Let’s try these out!

While working as a researcher in distributed systems, Dr. Christian Mayer found his love for teaching computer science students.

To help students reach higher levels of Python success, he founded the programming education website Finxter.com that has taught exponential skills to millions of coders worldwide. He’s the author of the best-selling programming books Python One-Liners (NoStarch 2020), The Art of Clean Code (NoStarch 2022), and The Book of Dash (NoStarch 2022). Chris also coauthored the Coffee Break Python series of self-published books. He’s a computer science enthusiast, freelancer , and owner of one of the top 10 largest Python blogs worldwide.

His passions are writing, reading, and coding. But his greatest passion is to serve aspiring coders through Finxter and help them to boost their skills. You can join his free email academy here.

• Python »
• 3.12.4 Documentation »
• The Python Standard Library »
• Development Tools »
• typing — Support for type hints
• Theme Auto Light Dark |

typing — Support for type hints ¶

Added in version 3.5.

Source code: Lib/typing.py

The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers , IDEs, linters, etc.

This module provides runtime support for type hints.

Consider the function below:

The function moon_weight takes an argument expected to be an instance of float , as indicated by the type hint earth_weight: float . The function is expected to return an instance of str , as indicated by the -> str hint.

While type hints can be simple classes like float or str , they can also be more complex. The typing module provides a vocabulary of more advanced type hints.

New features are frequently added to the typing module. The typing_extensions package provides backports of these new features to older versions of Python.

A quick overview of type hints (hosted at the mypy docs)

The Python typing system is standardised via PEPs, so this reference should broadly apply to most Python type checkers. (Some parts may still be specific to mypy.)

Type-checker-agnostic documentation written by the community detailing type system features, useful typing related tools and typing best practices.

Specification for the Python Type System ¶

The canonical, up-to-date specification of the Python type system can be found at “Specification for the Python type system” .

Type aliases ¶

A type alias is defined using the type statement, which creates an instance of TypeAliasType . In this example, Vector and list[float] will be treated equivalently by static type checkers:

Type aliases are useful for simplifying complex type signatures. For example:

The type statement is new in Python 3.12. For backwards compatibility, type aliases can also be created through simple assignment:

Or marked with TypeAlias to make it explicit that this is a type alias, not a normal variable assignment:

Use the NewType helper to create distinct types:

The static type checker will treat the new type as if it were a subclass of the original type. This is useful in helping catch logical errors:

You may still perform all int operations on a variable of type UserId , but the result will always be of type int . This lets you pass in a UserId wherever an int might be expected, but will prevent you from accidentally creating a UserId in an invalid way:

Note that these checks are enforced only by the static type checker. At runtime, the statement Derived = NewType('Derived', Base) will make Derived a callable that immediately returns whatever parameter you pass it. That means the expression Derived(some_value) does not create a new class or introduce much overhead beyond that of a regular function call.

More precisely, the expression some_value is Derived(some_value) is always true at runtime.

It is invalid to create a subtype of Derived :

However, it is possible to create a NewType based on a ‘derived’ NewType :

and typechecking for ProUserId will work as expected.

See PEP 484 for more details.

Recall that the use of a type alias declares two types to be equivalent to one another. Doing type Alias = Original will make the static type checker treat Alias as being exactly equivalent to Original in all cases. This is useful when you want to simplify complex type signatures.

In contrast, NewType declares one type to be a subtype of another. Doing Derived = NewType('Derived', Original) will make the static type checker treat Derived as a subclass of Original , which means a value of type Original cannot be used in places where a value of type Derived is expected. This is useful when you want to prevent logic errors with minimal runtime cost.

Added in version 3.5.2.

Changed in version 3.10: NewType is now a class rather than a function. As a result, there is some additional runtime cost when calling NewType over a regular function.

Changed in version 3.11: The performance of calling NewType has been restored to its level in Python 3.9.

Annotating callable objects ¶

Functions – or other callable objects – can be annotated using collections.abc.Callable or typing.Callable . Callable[[int], str] signifies a function that takes a single parameter of type int and returns a str .

For example:

The subscription syntax must always be used with exactly two values: the argument list and the return type. The argument list must be a list of types, a ParamSpec , Concatenate , or an ellipsis. The return type must be a single type.

If a literal ellipsis ... is given as the argument list, it indicates that a callable with any arbitrary parameter list would be acceptable:

Callable cannot express complex signatures such as functions that take a variadic number of arguments, overloaded functions , or functions that have keyword-only parameters. However, these signatures can be expressed by defining a Protocol class with a __call__() method:

Callables which take other callables as arguments may indicate that their parameter types are dependent on each other using ParamSpec . Additionally, if that callable adds or removes arguments from other callables, the Concatenate operator may be used. They take the form Callable[ParamSpecVariable, ReturnType] and Callable[Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable], ReturnType] respectively.

Changed in version 3.10: Callable now supports ParamSpec and Concatenate . See PEP 612 for more details.

The documentation for ParamSpec and Concatenate provides examples of usage in Callable .

Since type information about objects kept in containers cannot be statically inferred in a generic way, many container classes in the standard library support subscription to denote the expected types of container elements.

Generic functions and classes can be parameterized by using type parameter syntax :

Or by using the TypeVar factory directly:

Changed in version 3.12: Syntactic support for generics is new in Python 3.12.

Annotating tuples ¶

For most containers in Python, the typing system assumes that all elements in the container will be of the same type. For example:

list only accepts one type argument, so a type checker would emit an error on the y assignment above. Similarly, Mapping only accepts two type arguments: the first indicates the type of the keys, and the second indicates the type of the values.

Unlike most other Python containers, however, it is common in idiomatic Python code for tuples to have elements which are not all of the same type. For this reason, tuples are special-cased in Python’s typing system. tuple accepts any number of type arguments:

To denote a tuple which could be of any length, and in which all elements are of the same type T , use tuple[T, ...] . To denote an empty tuple, use tuple[()] . Using plain tuple as an annotation is equivalent to using tuple[Any, ...] :

The type of class objects ¶

A variable annotated with C may accept a value of type C . In contrast, a variable annotated with type[C] (or typing.Type[C] ) may accept values that are classes themselves – specifically, it will accept the class object of C . For example:

Note that type[C] is covariant:

The only legal parameters for type are classes, Any , type variables , and unions of any of these types. For example:

type[Any] is equivalent to type , which is the root of Python’s metaclass hierarchy .

User-defined generic types ¶

A user-defined class can be defined as a generic class.

This syntax indicates that the class LoggedVar is parameterised around a single type variable T . This also makes T valid as a type within the class body.

Generic classes implicitly inherit from Generic . For compatibility with Python 3.11 and lower, it is also possible to inherit explicitly from Generic to indicate a generic class:

Generic classes have __class_getitem__() methods, meaning they can be parameterised at runtime (e.g. LoggedVar[int] below):

A generic type can have any number of type variables. All varieties of TypeVar are permissible as parameters for a generic type:

Each type variable argument to Generic must be distinct. This is thus invalid:

Generic classes can also inherit from other classes:

When inheriting from generic classes, some type parameters could be fixed:

In this case MyDict has a single parameter, T .

Using a generic class without specifying type parameters assumes Any for each position. In the following example, MyIterable is not generic but implicitly inherits from Iterable[Any] :

User-defined generic type aliases are also supported. Examples:

For backward compatibility, generic type aliases can also be created through a simple assignment:

Changed in version 3.7: Generic no longer has a custom metaclass.

Changed in version 3.12: Syntactic support for generics and type aliases is new in version 3.12. Previously, generic classes had to explicitly inherit from Generic or contain a type variable in one of their bases.

User-defined generics for parameter expressions are also supported via parameter specification variables in the form [**P] . The behavior is consistent with type variables’ described above as parameter specification variables are treated by the typing module as a specialized type variable. The one exception to this is that a list of types can be used to substitute a ParamSpec :

Classes generic over a ParamSpec can also be created using explicit inheritance from Generic . In this case, ** is not used:

Another difference between TypeVar and ParamSpec is that a generic with only one parameter specification variable will accept parameter lists in the forms X[[Type1, Type2, ...]] and also X[Type1, Type2, ...] for aesthetic reasons. Internally, the latter is converted to the former, so the following are equivalent:

Note that generics with ParamSpec may not have correct __parameters__ after substitution in some cases because they are intended primarily for static type checking.

Changed in version 3.10: Generic can now be parameterized over parameter expressions. See ParamSpec and PEP 612 for more details.

A user-defined generic class can have ABCs as base classes without a metaclass conflict. Generic metaclasses are not supported. The outcome of parameterizing generics is cached, and most types in the typing module are hashable and comparable for equality.

The Any type ¶

A special kind of type is Any . A static type checker will treat every type as being compatible with Any and Any as being compatible with every type.

This means that it is possible to perform any operation or method call on a value of type Any and assign it to any variable:

Notice that no type checking is performed when assigning a value of type Any to a more precise type. For example, the static type checker did not report an error when assigning a to s even though s was declared to be of type str and receives an int value at runtime!

Furthermore, all functions without a return type or parameter types will implicitly default to using Any :

This behavior allows Any to be used as an escape hatch when you need to mix dynamically and statically typed code.

Contrast the behavior of Any with the behavior of object . Similar to Any , every type is a subtype of object . However, unlike Any , the reverse is not true: object is not a subtype of every other type.

That means when the type of a value is object , a type checker will reject almost all operations on it, and assigning it to a variable (or using it as a return value) of a more specialized type is a type error. For example:

Use object to indicate that a value could be any type in a typesafe manner. Use Any to indicate that a value is dynamically typed.

Nominal vs structural subtyping ¶

Initially PEP 484 defined the Python static type system as using nominal subtyping . This means that a class A is allowed where a class B is expected if and only if A is a subclass of B .

This requirement previously also applied to abstract base classes, such as Iterable . The problem with this approach is that a class had to be explicitly marked to support them, which is unpythonic and unlike what one would normally do in idiomatic dynamically typed Python code. For example, this conforms to PEP 484 :

PEP 544 allows to solve this problem by allowing users to write the above code without explicit base classes in the class definition, allowing Bucket to be implicitly considered a subtype of both Sized and Iterable[int] by static type checkers. This is known as structural subtyping (or static duck-typing):

Moreover, by subclassing a special class Protocol , a user can define new custom protocols to fully enjoy structural subtyping (see examples below).

Module contents ¶

The typing module defines the following classes, functions and decorators.

Special typing primitives ¶

Special types ¶.

These can be used as types in annotations. They do not support subscription using [] .

Special type indicating an unconstrained type.

Every type is compatible with Any .

Any is compatible with every type.

Changed in version 3.11: Any can now be used as a base class. This can be useful for avoiding type checker errors with classes that can duck type anywhere or are highly dynamic.

A constrained type variable .

Definition:

AnyStr is meant to be used for functions that may accept str or bytes arguments but cannot allow the two to mix.

Note that, despite its name, AnyStr has nothing to do with the Any type, nor does it mean “any string”. In particular, AnyStr and str | bytes are different from each other and have different use cases:

Special type that includes only literal strings.

Any string literal is compatible with LiteralString , as is another LiteralString . However, an object typed as just str is not. A string created by composing LiteralString -typed objects is also acceptable as a LiteralString .

LiteralString is useful for sensitive APIs where arbitrary user-generated strings could generate problems. For example, the two cases above that generate type checker errors could be vulnerable to an SQL injection attack.

See PEP 675 for more details.

Added in version 3.11.

Never and NoReturn represent the bottom type , a type that has no members.

They can be used to indicate that a function never returns, such as sys.exit() :

Or to define a function that should never be called, as there are no valid arguments, such as assert_never() :

Never and NoReturn have the same meaning in the type system and static type checkers treat both equivalently.

Added in version 3.6.2: Added NoReturn .

Added in version 3.11: Added Never .

Special type to represent the current enclosed class.

This annotation is semantically equivalent to the following, albeit in a more succinct fashion:

In general, if something returns self , as in the above examples, you should use Self as the return annotation. If Foo.return_self was annotated as returning "Foo" , then the type checker would infer the object returned from SubclassOfFoo.return_self as being of type Foo rather than SubclassOfFoo .

Other common use cases include:

classmethod s that are used as alternative constructors and return instances of the cls parameter.

Annotating an __enter__() method which returns self.

You should not use Self as the return annotation if the method is not guaranteed to return an instance of a subclass when the class is subclassed:

See PEP 673 for more details.

Special annotation for explicitly declaring a type alias .

TypeAlias is particularly useful on older Python versions for annotating aliases that make use of forward references, as it can be hard for type checkers to distinguish these from normal variable assignments:

See PEP 613 for more details.

Added in version 3.10.

Deprecated since version 3.12: TypeAlias is deprecated in favor of the type statement, which creates instances of TypeAliasType and which natively supports forward references. Note that while TypeAlias and TypeAliasType serve similar purposes and have similar names, they are distinct and the latter is not the type of the former. Removal of TypeAlias is not currently planned, but users are encouraged to migrate to type statements.

Special forms ¶

These can be used as types in annotations. They all support subscription using [] , but each has a unique syntax.

Union type; Union[X, Y] is equivalent to X | Y and means either X or Y.

To define a union, use e.g. Union[int, str] or the shorthand int | str . Using that shorthand is recommended. Details:

The arguments must be types and there must be at least one.

Unions of unions are flattened, e.g.:

Unions of a single argument vanish, e.g.:

Redundant arguments are skipped, e.g.:

When comparing unions, the argument order is ignored, e.g.:

You cannot subclass or instantiate a Union .

You cannot write Union[X][Y] .

Changed in version 3.7: Don’t remove explicit subclasses from unions at runtime.

Changed in version 3.10: Unions can now be written as X | Y . See union type expressions .

Optional[X] is equivalent to X | None (or Union[X, None] ).

Note that this is not the same concept as an optional argument, which is one that has a default. An optional argument with a default does not require the Optional qualifier on its type annotation just because it is optional. For example:

On the other hand, if an explicit value of None is allowed, the use of Optional is appropriate, whether the argument is optional or not. For example:

Changed in version 3.10: Optional can now be written as X | None . See union type expressions .

Special form for annotating higher-order functions.

Concatenate can be used in conjunction with Callable and ParamSpec to annotate a higher-order callable which adds, removes, or transforms parameters of another callable. Usage is in the form Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable] . Concatenate is currently only valid when used as the first argument to a Callable . The last parameter to Concatenate must be a ParamSpec or ellipsis ( ... ).

For example, to annotate a decorator with_lock which provides a threading.Lock to the decorated function, Concatenate can be used to indicate that with_lock expects a callable which takes in a Lock as the first argument, and returns a callable with a different type signature. In this case, the ParamSpec indicates that the returned callable’s parameter types are dependent on the parameter types of the callable being passed in:

PEP 612 – Parameter Specification Variables (the PEP which introduced ParamSpec and Concatenate )

• Annotating callable objects

Special typing form to define “literal types”.

Literal can be used to indicate to type checkers that the annotated object has a value equivalent to one of the provided literals.

Literal[...] cannot be subclassed. At runtime, an arbitrary value is allowed as type argument to Literal[...] , but type checkers may impose restrictions. See PEP 586 for more details about literal types.

Added in version 3.8.

Changed in version 3.9.1: Literal now de-duplicates parameters. Equality comparisons of Literal objects are no longer order dependent. Literal objects will now raise a TypeError exception during equality comparisons if one of their parameters are not hashable .

Special type construct to mark class variables.

As introduced in PEP 526 , a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. Usage:

ClassVar accepts only types and cannot be further subscribed.

ClassVar is not a class itself, and should not be used with isinstance() or issubclass() . ClassVar does not change Python runtime behavior, but it can be used by third-party type checkers. For example, a type checker might flag the following code as an error:

Added in version 3.5.3.

Special typing construct to indicate final names to type checkers.

Final names cannot be reassigned in any scope. Final names declared in class scopes cannot be overridden in subclasses.

There is no runtime checking of these properties. See PEP 591 for more details.

Special typing construct to mark a TypedDict key as required.

This is mainly useful for total=False TypedDicts. See TypedDict and PEP 655 for more details.

Special typing construct to mark a TypedDict key as potentially missing.

See TypedDict and PEP 655 for more details.

Special typing form to add context-specific metadata to an annotation.

Add metadata x to a given type T by using the annotation Annotated[T, x] . Metadata added using Annotated can be used by static analysis tools or at runtime. At runtime, the metadata is stored in a __metadata__ attribute.

If a library or tool encounters an annotation Annotated[T, x] and has no special logic for the metadata, it should ignore the metadata and simply treat the annotation as T . As such, Annotated can be useful for code that wants to use annotations for purposes outside Python’s static typing system.

Using Annotated[T, x] as an annotation still allows for static typechecking of T , as type checkers will simply ignore the metadata x . In this way, Annotated differs from the @no_type_check decorator, which can also be used for adding annotations outside the scope of the typing system, but completely disables typechecking for a function or class.

The responsibility of how to interpret the metadata lies with the tool or library encountering an Annotated annotation. A tool or library encountering an Annotated type can scan through the metadata elements to determine if they are of interest (e.g., using isinstance() ).

Here is an example of how you might use Annotated to add metadata to type annotations if you were doing range analysis:

Details of the syntax:

The first argument to Annotated must be a valid type

Multiple metadata elements can be supplied ( Annotated supports variadic arguments):

It is up to the tool consuming the annotations to decide whether the client is allowed to add multiple metadata elements to one annotation and how to merge those annotations.

Annotated must be subscripted with at least two arguments ( Annotated[int] is not valid)

The order of the metadata elements is preserved and matters for equality checks:

Nested Annotated types are flattened. The order of the metadata elements starts with the innermost annotation:

Duplicated metadata elements are not removed:

Annotated can be used with nested and generic aliases:

Annotated cannot be used with an unpacked TypeVarTuple :

This would be equivalent to:

where T1 , T2 , etc. are TypeVars . This would be invalid: only one type should be passed to Annotated.

By default, get_type_hints() strips the metadata from annotations. Pass include_extras=True to have the metadata preserved:

At runtime, the metadata associated with an Annotated type can be retrieved via the __metadata__ attribute:

The PEP introducing Annotated to the standard library.

Added in version 3.9.

Special typing construct for marking user-defined type guard functions.

TypeGuard can be used to annotate the return type of a user-defined type guard function. TypeGuard only accepts a single type argument. At runtime, functions marked this way should return a boolean.

TypeGuard aims to benefit type narrowing – a technique used by static type checkers to determine a more precise type of an expression within a program’s code flow. Usually type narrowing is done by analyzing conditional code flow and applying the narrowing to a block of code. The conditional expression here is sometimes referred to as a “type guard”:

Sometimes it would be convenient to use a user-defined boolean function as a type guard. Such a function should use TypeGuard[...] as its return type to alert static type checkers to this intention.

Using -> TypeGuard tells the static type checker that for a given function:

The return value is a boolean.

If the return value is True , the type of its argument is the type inside TypeGuard .

If is_str_list is a class or instance method, then the type in TypeGuard maps to the type of the second parameter after cls or self .

In short, the form def foo(arg: TypeA) -> TypeGuard[TypeB]: ... , means that if foo(arg) returns True , then arg narrows from TypeA to TypeB .

TypeB need not be a narrower form of TypeA – it can even be a wider form. The main reason is to allow for things like narrowing list[object] to list[str] even though the latter is not a subtype of the former, since list is invariant. The responsibility of writing type-safe type guards is left to the user.

TypeGuard also works with type variables. See PEP 647 for more details.

Typing operator to conceptually mark an object as having been unpacked.

For example, using the unpack operator * on a type variable tuple is equivalent to using Unpack to mark the type variable tuple as having been unpacked:

In fact, Unpack can be used interchangeably with * in the context of typing.TypeVarTuple and builtins.tuple types. You might see Unpack being used explicitly in older versions of Python, where * couldn’t be used in certain places:

Unpack can also be used along with typing.TypedDict for typing **kwargs in a function signature:

See PEP 692 for more details on using Unpack for **kwargs typing.

Building generic types and type aliases ¶

The following classes should not be used directly as annotations. Their intended purpose is to be building blocks for creating generic types and type aliases.

These objects can be created through special syntax ( type parameter lists and the type statement). For compatibility with Python 3.11 and earlier, they can also be created without the dedicated syntax, as documented below.

Abstract base class for generic types.

A generic type is typically declared by adding a list of type parameters after the class name:

Such a class implicitly inherits from Generic . The runtime semantics of this syntax are discussed in the Language Reference .

This class can then be used as follows:

Here the brackets after the function name indicate a generic function .

For backwards compatibility, generic classes can also be declared by explicitly inheriting from Generic . In this case, the type parameters must be declared separately:

Type variable.

The preferred way to construct a type variable is via the dedicated syntax for generic functions , generic classes , and generic type aliases :

This syntax can also be used to create bound and constrained type variables:

However, if desired, reusable type variables can also be constructed manually, like so:

Type variables exist primarily for the benefit of static type checkers. They serve as the parameters for generic types as well as for generic function and type alias definitions. See Generic for more information on generic types. Generic functions work as follows:

Note that type variables can be bound , constrained , or neither, but cannot be both bound and constrained.

The variance of type variables is inferred by type checkers when they are created through the type parameter syntax or when infer_variance=True is passed. Manually created type variables may be explicitly marked covariant or contravariant by passing covariant=True or contravariant=True . By default, manually created type variables are invariant. See PEP 484 and PEP 695 for more details.

Bound type variables and constrained type variables have different semantics in several important ways. Using a bound type variable means that the TypeVar will be solved using the most specific type possible:

Type variables can be bound to concrete types, abstract types (ABCs or protocols), and even unions of types:

Using a constrained type variable, however, means that the TypeVar can only ever be solved as being exactly one of the constraints given:

At runtime, isinstance(x, T) will raise TypeError .

The name of the type variable.

Whether the type var has been explicitly marked as covariant.

Whether the type var has been explicitly marked as contravariant.

Whether the type variable’s variance should be inferred by type checkers.

Added in version 3.12.

The bound of the type variable, if any.

Changed in version 3.12: For type variables created through type parameter syntax , the bound is evaluated only when the attribute is accessed, not when the type variable is created (see Lazy evaluation ).

A tuple containing the constraints of the type variable, if any.

Changed in version 3.12: For type variables created through type parameter syntax , the constraints are evaluated only when the attribute is accessed, not when the type variable is created (see Lazy evaluation ).

Changed in version 3.12: Type variables can now be declared using the type parameter syntax introduced by PEP 695 . The infer_variance parameter was added.

Type variable tuple. A specialized form of type variable that enables variadic generics.

Type variable tuples can be declared in type parameter lists using a single asterisk ( * ) before the name:

Or by explicitly invoking the TypeVarTuple constructor:

A normal type variable enables parameterization with a single type. A type variable tuple, in contrast, allows parameterization with an arbitrary number of types by acting like an arbitrary number of type variables wrapped in a tuple. For example:

Note the use of the unpacking operator * in tuple[T, *Ts] . Conceptually, you can think of Ts as a tuple of type variables (T1, T2, ...) . tuple[T, *Ts] would then become tuple[T, *(T1, T2, ...)] , which is equivalent to tuple[T, T1, T2, ...] . (Note that in older versions of Python, you might see this written using Unpack instead, as Unpack[Ts] .)

Type variable tuples must always be unpacked. This helps distinguish type variable tuples from normal type variables:

Type variable tuples can be used in the same contexts as normal type variables. For example, in class definitions, arguments, and return types:

Type variable tuples can be happily combined with normal type variables:

However, note that at most one type variable tuple may appear in a single list of type arguments or type parameters:

Finally, an unpacked type variable tuple can be used as the type annotation of *args :

In contrast to non-unpacked annotations of *args - e.g. *args: int , which would specify that all arguments are int - *args: *Ts enables reference to the types of the individual arguments in *args . Here, this allows us to ensure the types of the *args passed to call_soon match the types of the (positional) arguments of callback .

See PEP 646 for more details on type variable tuples.

The name of the type variable tuple.

Changed in version 3.12: Type variable tuples can now be declared using the type parameter syntax introduced by PEP 695 .

Parameter specification variable. A specialized version of type variables .

In type parameter lists , parameter specifications can be declared with two asterisks ( ** ):

For compatibility with Python 3.11 and earlier, ParamSpec objects can also be created as follows:

Parameter specification variables exist primarily for the benefit of static type checkers. They are used to forward the parameter types of one callable to another callable – a pattern commonly found in higher order functions and decorators. They are only valid when used in Concatenate , or as the first argument to Callable , or as parameters for user-defined Generics. See Generic for more information on generic types.

For example, to add basic logging to a function, one can create a decorator add_logging to log function calls. The parameter specification variable tells the type checker that the callable passed into the decorator and the new callable returned by it have inter-dependent type parameters:

Without ParamSpec , the simplest way to annotate this previously was to use a TypeVar with bound Callable[..., Any] . However this causes two problems:

The type checker can’t type check the inner function because *args and **kwargs have to be typed Any .

cast() may be required in the body of the add_logging decorator when returning the inner function, or the static type checker must be told to ignore the return inner .

Since ParamSpec captures both positional and keyword parameters, P.args and P.kwargs can be used to split a ParamSpec into its components. P.args represents the tuple of positional parameters in a given call and should only be used to annotate *args . P.kwargs represents the mapping of keyword parameters to their values in a given call, and should be only be used to annotate **kwargs . Both attributes require the annotated parameter to be in scope. At runtime, P.args and P.kwargs are instances respectively of ParamSpecArgs and ParamSpecKwargs .

The name of the parameter specification.

Parameter specification variables created with covariant=True or contravariant=True can be used to declare covariant or contravariant generic types. The bound argument is also accepted, similar to TypeVar . However the actual semantics of these keywords are yet to be decided.

Changed in version 3.12: Parameter specifications can now be declared using the type parameter syntax introduced by PEP 695 .

Only parameter specification variables defined in global scope can be pickled.

Concatenate

Arguments and keyword arguments attributes of a ParamSpec . The P.args attribute of a ParamSpec is an instance of ParamSpecArgs , and P.kwargs is an instance of ParamSpecKwargs . They are intended for runtime introspection and have no special meaning to static type checkers.

Calling get_origin() on either of these objects will return the original ParamSpec :

The type of type aliases created through the type statement.

The name of the type alias:

The module in which the type alias was defined:

The type parameters of the type alias, or an empty tuple if the alias is not generic:

The type alias’s value. This is lazily evaluated , so names used in the definition of the alias are not resolved until the __value__ attribute is accessed:

Other special directives ¶

These functions and classes should not be used directly as annotations. Their intended purpose is to be building blocks for creating and declaring types.

Typed version of collections.namedtuple() .

This is equivalent to:

To give a field a default value, you can assign to it in the class body:

Fields with a default value must come after any fields without a default.

The resulting class has an extra attribute __annotations__ giving a dict that maps the field names to the field types. (The field names are in the _fields attribute and the default values are in the _field_defaults attribute, both of which are part of the namedtuple() API.)

NamedTuple subclasses can also have docstrings and methods:

NamedTuple subclasses can be generic:

Backward-compatible usage:

Changed in version 3.6: Added support for PEP 526 variable annotation syntax.

Changed in version 3.6.1: Added support for default values, methods, and docstrings.

Changed in version 3.8: The _field_types and __annotations__ attributes are now regular dictionaries instead of instances of OrderedDict .

Changed in version 3.9: Removed the _field_types attribute in favor of the more standard __annotations__ attribute which has the same information.

Changed in version 3.11: Added support for generic namedtuples.

Helper class to create low-overhead distinct types .

A NewType is considered a distinct type by a typechecker. At runtime, however, calling a NewType returns its argument unchanged.

The module in which the new type is defined.

The name of the new type.

The type that the new type is based on.

Changed in version 3.10: NewType is now a class rather than a function.

Base class for protocol classes.

Protocol classes are defined like this:

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example:

See PEP 544 for more details. Protocol classes decorated with runtime_checkable() (described later) act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures.

Protocol classes can be generic, for example:

In code that needs to be compatible with Python 3.11 or older, generic Protocols can be written as follows:

Mark a protocol class as a runtime protocol.

Such a protocol can be used with isinstance() and issubclass() . This raises TypeError when applied to a non-protocol class. This allows a simple-minded structural check, very similar to “one trick ponies” in collections.abc such as Iterable . For example:

runtime_checkable() will check only the presence of the required methods or attributes, not their type signatures or types. For example, ssl.SSLObject is a class, therefore it passes an issubclass() check against Callable . However, the ssl.SSLObject.__init__ method exists only to raise a TypeError with a more informative message, therefore making it impossible to call (instantiate) ssl.SSLObject .

An isinstance() check against a runtime-checkable protocol can be surprisingly slow compared to an isinstance() check against a non-protocol class. Consider using alternative idioms such as hasattr() calls for structural checks in performance-sensitive code.

Changed in version 3.12: The internal implementation of isinstance() checks against runtime-checkable protocols now uses inspect.getattr_static() to look up attributes (previously, hasattr() was used). As a result, some objects which used to be considered instances of a runtime-checkable protocol may no longer be considered instances of that protocol on Python 3.12+, and vice versa. Most users are unlikely to be affected by this change.

Changed in version 3.12: The members of a runtime-checkable protocol are now considered “frozen” at runtime as soon as the class has been created. Monkey-patching attributes onto a runtime-checkable protocol will still work, but will have no impact on isinstance() checks comparing objects to the protocol. See “What’s new in Python 3.12” for more details.

Special construct to add type hints to a dictionary. At runtime it is a plain dict .

TypedDict declares a dictionary type that expects all of its instances to have a certain set of keys, where each key is associated with a value of a consistent type. This expectation is not checked at runtime but is only enforced by type checkers. Usage:

To allow using this feature with older versions of Python that do not support PEP 526 , TypedDict supports two additional equivalent syntactic forms:

Using a literal dict as the second argument:

Using keyword arguments:

Deprecated since version 3.11, will be removed in version 3.13: The keyword-argument syntax is deprecated in 3.11 and will be removed in 3.13. It may also be unsupported by static type checkers.

The functional syntax should also be used when any of the keys are not valid identifiers , for example because they are keywords or contain hyphens. Example:

By default, all keys must be present in a TypedDict . It is possible to mark individual keys as non-required using NotRequired :

This means that a Point2D TypedDict can have the label key omitted.

It is also possible to mark all keys as non-required by default by specifying a totality of False :

This means that a Point2D TypedDict can have any of the keys omitted. A type checker is only expected to support a literal False or True as the value of the total argument. True is the default, and makes all items defined in the class body required.

Individual keys of a total=False TypedDict can be marked as required using Required :

It is possible for a TypedDict type to inherit from one or more other TypedDict types using the class-based syntax. Usage:

Point3D has three items: x , y and z . It is equivalent to this definition:

A TypedDict cannot inherit from a non- TypedDict class, except for Generic . For example:

A TypedDict can be generic:

To create a generic TypedDict that is compatible with Python 3.11 or lower, inherit from Generic explicitly:

A TypedDict can be introspected via annotations dicts (see Annotations Best Practices for more information on annotations best practices), __total__ , __required_keys__ , and __optional_keys__ .

Point2D.__total__ gives the value of the total argument. Example:

This attribute reflects only the value of the total argument to the current TypedDict class, not whether the class is semantically total. For example, a TypedDict with __total__ set to True may have keys marked with NotRequired , or it may inherit from another TypedDict with total=False . Therefore, it is generally better to use __required_keys__ and __optional_keys__ for introspection.

Point2D.__required_keys__ and Point2D.__optional_keys__ return frozenset objects containing required and non-required keys, respectively.

Keys marked with Required will always appear in __required_keys__ and keys marked with NotRequired will always appear in __optional_keys__ .

For backwards compatibility with Python 3.10 and below, it is also possible to use inheritance to declare both required and non-required keys in the same TypedDict . This is done by declaring a TypedDict with one value for the total argument and then inheriting from it in another TypedDict with a different value for total :

If from __future__ import annotations is used or if annotations are given as strings, annotations are not evaluated when the TypedDict is defined. Therefore, the runtime introspection that __required_keys__ and __optional_keys__ rely on may not work properly, and the values of the attributes may be incorrect.

See PEP 589 for more examples and detailed rules of using TypedDict .

Changed in version 3.11: Added support for marking individual keys as Required or NotRequired . See PEP 655 .

Changed in version 3.11: Added support for generic TypedDict s.

Protocols ¶

The following protocols are provided by the typing module. All are decorated with @runtime_checkable .

An ABC with one abstract method __abs__ that is covariant in its return type.

An ABC with one abstract method __bytes__ .

An ABC with one abstract method __complex__ .

An ABC with one abstract method __float__ .

An ABC with one abstract method __index__ .

An ABC with one abstract method __int__ .

An ABC with one abstract method __round__ that is covariant in its return type.

ABCs for working with IO ¶

Generic type IO[AnyStr] and its subclasses TextIO(IO[str]) and BinaryIO(IO[bytes]) represent the types of I/O streams such as returned by open() .

Functions and decorators ¶

Cast a value to a type.

This returns the value unchanged. To the type checker this signals that the return value has the designated type, but at runtime we intentionally don’t check anything (we want this to be as fast as possible).

Ask a static type checker to confirm that val has an inferred type of typ .

At runtime this does nothing: it returns the first argument unchanged with no checks or side effects, no matter the actual type of the argument.

When a static type checker encounters a call to assert_type() , it emits an error if the value is not of the specified type:

This function is useful for ensuring the type checker’s understanding of a script is in line with the developer’s intentions:

Ask a static type checker to confirm that a line of code is unreachable.

Here, the annotations allow the type checker to infer that the last case can never execute, because arg is either an int or a str , and both options are covered by earlier cases.

If a type checker finds that a call to assert_never() is reachable, it will emit an error. For example, if the type annotation for arg was instead int | str | float , the type checker would emit an error pointing out that unreachable is of type float . For a call to assert_never to pass type checking, the inferred type of the argument passed in must be the bottom type, Never , and nothing else.

At runtime, this throws an exception when called.

Unreachable Code and Exhaustiveness Checking has more information about exhaustiveness checking with static typing.

Ask a static type checker to reveal the inferred type of an expression.

When a static type checker encounters a call to this function, it emits a diagnostic with the inferred type of the argument. For example:

This can be useful when you want to debug how your type checker handles a particular piece of code.

At runtime, this function prints the runtime type of its argument to sys.stderr and returns the argument unchanged (allowing the call to be used within an expression):

Note that the runtime type may be different from (more or less specific than) the type statically inferred by a type checker.

Most type checkers support reveal_type() anywhere, even if the name is not imported from typing . Importing the name from typing , however, allows your code to run without runtime errors and communicates intent more clearly.

Decorator to mark an object as providing dataclass -like behavior.

dataclass_transform may be used to decorate a class, metaclass, or a function that is itself a decorator. The presence of @dataclass_transform() tells a static type checker that the decorated object performs runtime “magic” that transforms a class in a similar way to @dataclasses.dataclass .

Example usage with a decorator function:

On a base class:

On a metaclass:

The CustomerModel classes defined above will be treated by type checkers similarly to classes created with @dataclasses.dataclass . For example, type checkers will assume these classes have __init__ methods that accept id and name .

The decorated class, metaclass, or function may accept the following bool arguments which type checkers will assume have the same effect as they would have on the @dataclasses.dataclass decorator: init , eq , order , unsafe_hash , frozen , match_args , kw_only , and slots . It must be possible for the value of these arguments ( True or False ) to be statically evaluated.

The arguments to the dataclass_transform decorator can be used to customize the default behaviors of the decorated class, metaclass, or function:

eq_default ( bool ) – Indicates whether the eq parameter is assumed to be True or False if it is omitted by the caller. Defaults to True .

order_default ( bool ) – Indicates whether the order parameter is assumed to be True or False if it is omitted by the caller. Defaults to False .

kw_only_default ( bool ) – Indicates whether the kw_only parameter is assumed to be True or False if it is omitted by the caller. Defaults to False .

Indicates whether the frozen parameter is assumed to be True or False if it is omitted by the caller. Defaults to False .

field_specifiers ( tuple [ Callable [ ... , Any ] , ... ] ) – Specifies a static list of supported classes or functions that describe fields, similar to dataclasses.field() . Defaults to () .

**kwargs ( Any ) – Arbitrary other keyword arguments are accepted in order to allow for possible future extensions.

Type checkers recognize the following optional parameters on field specifiers:

At runtime, this decorator records its arguments in the __dataclass_transform__ attribute on the decorated object. It has no other runtime effect.

See PEP 681 for more details.

Decorator for creating overloaded functions and methods.

The @overload decorator allows describing functions and methods that support multiple different combinations of argument types. A series of @overload -decorated definitions must be followed by exactly one non- @overload -decorated definition (for the same function/method).

@overload -decorated definitions are for the benefit of the type checker only, since they will be overwritten by the non- @overload -decorated definition. The non- @overload -decorated definition, meanwhile, will be used at runtime but should be ignored by a type checker. At runtime, calling an @overload -decorated function directly will raise NotImplementedError .

An example of overload that gives a more precise type than can be expressed using a union or a type variable:

See PEP 484 for more details and comparison with other typing semantics.

Changed in version 3.11: Overloaded functions can now be introspected at runtime using get_overloads() .

Return a sequence of @overload -decorated definitions for func .

func is the function object for the implementation of the overloaded function. For example, given the definition of process in the documentation for @overload , get_overloads(process) will return a sequence of three function objects for the three defined overloads. If called on a function with no overloads, get_overloads() returns an empty sequence.

get_overloads() can be used for introspecting an overloaded function at runtime.

Clear all registered overloads in the internal registry.

This can be used to reclaim the memory used by the registry.

Decorator to indicate final methods and final classes.

Decorating a method with @final indicates to a type checker that the method cannot be overridden in a subclass. Decorating a class with @final indicates that it cannot be subclassed.

Changed in version 3.11: The decorator will now attempt to set a __final__ attribute to True on the decorated object. Thus, a check like if getattr(obj, "__final__", False) can be used at runtime to determine whether an object obj has been marked as final. If the decorated object does not support setting attributes, the decorator returns the object unchanged without raising an exception.

Decorator to indicate that annotations are not type hints.

This works as a class or function decorator . With a class, it applies recursively to all methods and classes defined in that class (but not to methods defined in its superclasses or subclasses). Type checkers will ignore all annotations in a function or class with this decorator.

@no_type_check mutates the decorated object in place.

Decorator to give another decorator the no_type_check() effect.

This wraps the decorator with something that wraps the decorated function in no_type_check() .

Decorator to indicate that a method in a subclass is intended to override a method or attribute in a superclass.

Type checkers should emit an error if a method decorated with @override does not, in fact, override anything. This helps prevent bugs that may occur when a base class is changed without an equivalent change to a child class.

There is no runtime checking of this property.

The decorator will attempt to set an __override__ attribute to True on the decorated object. Thus, a check like if getattr(obj, "__override__", False) can be used at runtime to determine whether an object obj has been marked as an override. If the decorated object does not support setting attributes, the decorator returns the object unchanged without raising an exception.

See PEP 698 for more details.

Decorator to mark a class or function as unavailable at runtime.

This decorator is itself not available at runtime. It is mainly intended to mark classes that are defined in type stub files if an implementation returns an instance of a private class:

Note that returning instances of private classes is not recommended. It is usually preferable to make such classes public.

Introspection helpers ¶

Return a dictionary containing type hints for a function, method, module or class object.

This is often the same as obj.__annotations__ , but this function makes the following changes to the annotations dictionary:

Forward references encoded as string literals or ForwardRef objects are handled by evaluating them in globalns , localns , and (where applicable) obj ’s type parameter namespace. If globalns or localns is not given, appropriate namespace dictionaries are inferred from obj .

None is replaced with types.NoneType .

If @no_type_check has been applied to obj , an empty dictionary is returned.

If obj is a class C , the function returns a dictionary that merges annotations from C ’s base classes with those on C directly. This is done by traversing C.__mro__ and iteratively combining __annotations__ dictionaries. Annotations on classes appearing earlier in the method resolution order always take precedence over annotations on classes appearing later in the method resolution order.

The function recursively replaces all occurrences of Annotated[T, ...] with T , unless include_extras is set to True (see Annotated for more information).

See also inspect.get_annotations() , a lower-level function that returns annotations more directly.

If any forward references in the annotations of obj are not resolvable or are not valid Python code, this function will raise an exception such as NameError . For example, this can happen with imported type aliases that include forward references, or with names imported under if TYPE_CHECKING .

Changed in version 3.9: Added include_extras parameter as part of PEP 593 . See the documentation on Annotated for more information.

Changed in version 3.11: Previously, Optional[t] was added for function and method annotations if a default value equal to None was set. Now the annotation is returned unchanged.

Get the unsubscripted version of a type: for a typing object of the form X[Y, Z, ...] return X .

If X is a typing-module alias for a builtin or collections class, it will be normalized to the original class. If X is an instance of ParamSpecArgs or ParamSpecKwargs , return the underlying ParamSpec . Return None for unsupported objects.

Get type arguments with all substitutions performed: for a typing object of the form X[Y, Z, ...] return (Y, Z, ...) .

If X is a union or Literal contained in another generic type, the order of (Y, Z, ...) may be different from the order of the original arguments [Y, Z, ...] due to type caching. Return () for unsupported objects.

Check if a type is a TypedDict .

Class used for internal typing representation of string forward references.

For example, List["SomeClass"] is implicitly transformed into List[ForwardRef("SomeClass")] . ForwardRef should not be instantiated by a user, but may be used by introspection tools.

PEP 585 generic types such as list["SomeClass"] will not be implicitly transformed into list[ForwardRef("SomeClass")] and thus will not automatically resolve to list[SomeClass] .

Added in version 3.7.4.

A special constant that is assumed to be True by 3rd party static type checkers. It is False at runtime.

The first type annotation must be enclosed in quotes, making it a “forward reference”, to hide the expensive_mod reference from the interpreter runtime. Type annotations for local variables are not evaluated, so the second annotation does not need to be enclosed in quotes.

If from __future__ import annotations is used, annotations are not evaluated at function definition time. Instead, they are stored as strings in __annotations__ . This makes it unnecessary to use quotes around the annotation (see PEP 563 ).

Deprecated aliases ¶

This module defines several deprecated aliases to pre-existing standard library classes. These were originally included in the typing module in order to support parameterizing these generic classes using [] . However, the aliases became redundant in Python 3.9 when the corresponding pre-existing classes were enhanced to support [] (see PEP 585 ).

The redundant types are deprecated as of Python 3.9. However, while the aliases may be removed at some point, removal of these aliases is not currently planned. As such, no deprecation warnings are currently issued by the interpreter for these aliases.

If at some point it is decided to remove these deprecated aliases, a deprecation warning will be issued by the interpreter for at least two releases prior to removal. The aliases are guaranteed to remain in the typing module without deprecation warnings until at least Python 3.14.

Type checkers are encouraged to flag uses of the deprecated types if the program they are checking targets a minimum Python version of 3.9 or newer.

Aliases to built-in types ¶

Deprecated alias to dict .

Note that to annotate arguments, it is preferred to use an abstract collection type such as Mapping rather than to use dict or typing.Dict .

This type can be used as follows:

Deprecated since version 3.9: builtins.dict now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to list .

Note that to annotate arguments, it is preferred to use an abstract collection type such as Sequence or Iterable rather than to use list or typing.List .

This type may be used as follows:

Deprecated since version 3.9: builtins.list now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to builtins.set .

Note that to annotate arguments, it is preferred to use an abstract collection type such as AbstractSet rather than to use set or typing.Set .

Deprecated since version 3.9: builtins.set now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to builtins.frozenset .

Deprecated since version 3.9: builtins.frozenset now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias for tuple .

tuple and Tuple are special-cased in the type system; see Annotating tuples for more details.

Deprecated since version 3.9: builtins.tuple now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to type .

See The type of class objects for details on using type or typing.Type in type annotations.

Deprecated since version 3.9: builtins.type now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Aliases to types in collections ¶

Deprecated alias to collections.defaultdict .

Deprecated since version 3.9: collections.defaultdict now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.OrderedDict .

Added in version 3.7.2.

Deprecated since version 3.9: collections.OrderedDict now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.ChainMap .

Added in version 3.6.1.

Deprecated since version 3.9: collections.ChainMap now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.Counter .

Deprecated since version 3.9: collections.Counter now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.deque .

Deprecated since version 3.9: collections.deque now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Aliases to other concrete types ¶

Deprecated since version 3.8, will be removed in version 3.13: The typing.io namespace is deprecated and will be removed. These types should be directly imported from typing instead.

Deprecated aliases corresponding to the return types from re.compile() and re.match() .

These types (and the corresponding functions) are generic over AnyStr . Pattern can be specialised as Pattern[str] or Pattern[bytes] ; Match can be specialised as Match[str] or Match[bytes] .

Deprecated since version 3.8, will be removed in version 3.13: The typing.re namespace is deprecated and will be removed. These types should be directly imported from typing instead.

Deprecated since version 3.9: Classes Pattern and Match from re now support [] . See PEP 585 and Generic Alias Type .

Deprecated alias for str .

Text is provided to supply a forward compatible path for Python 2 code: in Python 2, Text is an alias for unicode .

Use Text to indicate that a value must contain a unicode string in a manner that is compatible with both Python 2 and Python 3:

Deprecated since version 3.11: Python 2 is no longer supported, and most type checkers also no longer support type checking Python 2 code. Removal of the alias is not currently planned, but users are encouraged to use str instead of Text .

Aliases to container ABCs in collections.abc ¶

Deprecated alias to collections.abc.Set .

Deprecated since version 3.9: collections.abc.Set now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

This type represents the types bytes , bytearray , and memoryview of byte sequences.

Deprecated since version 3.9, will be removed in version 3.14: Prefer collections.abc.Buffer , or a union like bytes | bytearray | memoryview .

Deprecated alias to collections.abc.Collection .

Added in version 3.6.

Deprecated since version 3.9: collections.abc.Collection now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Container .

Deprecated since version 3.9: collections.abc.Container now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.ItemsView .

Deprecated since version 3.9: collections.abc.ItemsView now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.KeysView .

Deprecated since version 3.9: collections.abc.KeysView now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Mapping .

Deprecated since version 3.9: collections.abc.Mapping now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.MappingView .

Deprecated since version 3.9: collections.abc.MappingView now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.MutableMapping .

Deprecated since version 3.9: collections.abc.MutableMapping now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.MutableSequence .

Deprecated since version 3.9: collections.abc.MutableSequence now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.MutableSet .

Deprecated since version 3.9: collections.abc.MutableSet now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Sequence .

Deprecated since version 3.9: collections.abc.Sequence now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.ValuesView .

Deprecated since version 3.9: collections.abc.ValuesView now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Aliases to asynchronous ABCs in collections.abc ¶

Deprecated alias to collections.abc.Coroutine .

The variance and order of type variables correspond to those of Generator , for example:

Deprecated since version 3.9: collections.abc.Coroutine now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.AsyncGenerator .

An async generator can be annotated by the generic type AsyncGenerator[YieldType, SendType] . For example:

Unlike normal generators, async generators cannot return a value, so there is no ReturnType type parameter. As with Generator , the SendType behaves contravariantly.

If your generator will only yield values, set the SendType to None :

Alternatively, annotate your generator as having a return type of either AsyncIterable[YieldType] or AsyncIterator[YieldType] :

Deprecated since version 3.9: collections.abc.AsyncGenerator now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.AsyncIterable .

Deprecated since version 3.9: collections.abc.AsyncIterable now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.AsyncIterator .

Deprecated since version 3.9: collections.abc.AsyncIterator now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Awaitable .

Deprecated since version 3.9: collections.abc.Awaitable now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Aliases to other ABCs in collections.abc ¶

Deprecated alias to collections.abc.Iterable .

Deprecated since version 3.9: collections.abc.Iterable now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Iterator .

Deprecated since version 3.9: collections.abc.Iterator now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Callable .

See Annotating callable objects for details on how to use collections.abc.Callable and typing.Callable in type annotations.

Deprecated since version 3.9: collections.abc.Callable now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Generator .

A generator can be annotated by the generic type Generator[YieldType, SendType, ReturnType] . For example:

Note that unlike many other generics in the typing module, the SendType of Generator behaves contravariantly, not covariantly or invariantly.

If your generator will only yield values, set the SendType and ReturnType to None :

Alternatively, annotate your generator as having a return type of either Iterable[YieldType] or Iterator[YieldType] :

Deprecated since version 3.9: collections.abc.Generator now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Hashable .

Deprecated since version 3.12: Use collections.abc.Hashable directly instead.

Deprecated alias to collections.abc.Reversible .

Deprecated since version 3.9: collections.abc.Reversible now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to collections.abc.Sized .

Deprecated since version 3.12: Use collections.abc.Sized directly instead.

Aliases to contextlib ABCs ¶

Deprecated alias to contextlib.AbstractContextManager .

Added in version 3.5.4.

Deprecated since version 3.9: contextlib.AbstractContextManager now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecated alias to contextlib.AbstractAsyncContextManager .

Added in version 3.6.2.

Deprecated since version 3.9: contextlib.AbstractAsyncContextManager now supports subscripting ( [] ). See PEP 585 and Generic Alias Type .

Deprecation Timeline of Major Features ¶

Certain features in typing are deprecated and may be removed in a future version of Python. The following table summarizes major deprecations for your convenience. This is subject to change, and not all deprecations are listed.

Table of Contents

• Specification for the Python Type System
• Type aliases
• Annotating tuples
• The type of class objects
• User-defined generic types
• The Any type
• Nominal vs structural subtyping
• Special types
• Special forms
• Building generic types and type aliases
• Other special directives
• ABCs for working with IO
• Functions and decorators
• Introspection helpers
• Aliases to built-in types
• Aliases to types in collections
• Aliases to other concrete types
• Aliases to container ABCs in collections.abc
• Aliases to asynchronous ABCs in collections.abc
• Aliases to other ABCs in collections.abc
• Aliases to contextlib ABCs
• Deprecation Timeline of Major Features

Previous topic

Development Tools

pydoc — Documentation generator and online help system

• Report a Bug
• Show Source

Calling function on startup

hello, when I try to call the function getSingleRows(), I get the following error TypeError: Thirdwindow.getSingleRows() missing 1 required positional argument: ‘self’ please point out what I am missing

here are my files

crashcourse.kv

getSingleRows is an instance method of Thirdwindow (it has a parameter self that will refer to the instance/object), but you’re calling it as a plain method getSingleRows() , so no instance there.

if you want to call the method getSingleRows at start-up during an object instantiation, include an __init__ constructor method, and call the method in there, i.e:

If you want access to the attribute x in other parts of your class script, define it in the constructor so that it becomes an attribute of the instances that you create. I don’t think that there is a need for the global keyword here.

where should I add this function def init (self): self.getSingleRows()

please be patient because I am new to this

At the top of the class, as in:

For the x value, assign it to whatever value is required during initialization. I have set it to None in this example since I do not know its intended initial value. For all classes, if you need code to run during instantiation, include it in the __init__ constructor method.

Hope that helps.

Experiment. For example:

As you can see, if you assign a variable in the __init__ constructor method, you can access it by any instance method or instance object. There is no need for a global keyword. However, if you wish to have access to an attribute between many instance objects, then assign it as a class attribute.

thank you for your patience but when I edited my code with the code you provided it produced that says EXPECTED INDENTED BLOCK

The class word that defines Windowmanager(screen) has a red line unerneath and when I hover it, it says expected indented block

When experimenting with new code and concepts , always start small. That is, begin with the bare minimum so that you can test the proposition in isolation. Thereafter, slowly build out.

In the script that you provided, the method getSingleRows(self) is missing a body statement. You can’t declare a method and leave it blank.

add a statement to it:

Another way to think of this, is suppose that you are in a classroom. Your professor has a list/roster of all of the students that are attending that class. As he is taking roll call, he slowly looks at each name in the roster and links it to a student (a body). If at any point he comes upon a student that is missing (missing a body), he makes a note of it. This is essentially what is going on here. There is a method name that has been declared, but with a missing body. Anytime that you declare a method/function, it has to accompanied by a body statement. The same is true for classes.

thank you the error has gone, but what about the code that deals with the database? does it go inside def getSingleRows(self):? thank you again for your advice

Yes, it can go inside the method. However, this is where your creativity will have to come into play. How do YOU want to design your application? As I stated in a previous post, start small. Experiment with a concept/propostion. Don’t write 2-3 pages worth of code and then decide to test for verification. Test at every step of the development phase so that you verify that it is being developed according to your requirements. This also helps you understand your own code btw.

hello Paul again. I listed to your advice and started running the code for testing after adding print(“print something”), it produced this error

TypeError: Thirdwindow. init () got an unexpected keyword argument ‘__no_builder’

please, solve this problem

Related Topics

Join us and get access to thousands of tutorials and a community of expert Pythonistas.

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Assignment Expression Syntax

• Discussion (1)

For more information on concepts covered in this lesson, you can check out:

• Python Scope & the LEGB Rule: Resolving Names in Your Code | Real Python Article
• PEP 572 – Assignment Expressions: Exceptional Cases | Python.org

00:00 Walrus operator syntax. One of the main reasons assignments were not expressions in Python from the beginning is the visual likeness of the assignment operator ( = ) and the equality comparison operator ( == ). This could potentially lead to bugs.

00:18 When introducing assignment expressions, a great deal of thought was put into how to avoid similar bugs with the walrus operator. As mentioned previously, one important feature is the walrus operator is never allowed as a direct replacement for the assignment operator, and vice versa.

00:35 As you saw at the beginning of the course, you can’t use a plain assignment expression to assign a value. It is syntactically legal to use an assignment expression to only assign a value, but only if you add parentheses.

00:57 Even though this is possible, this really is a prime example of where you should stay away from the walrus operator and use a traditional assignment statement instead.

01:07 There are several other examples where the walrus operator is either illegal or discouraged. All of these examples raise a SyntaxError .

01:40 In all of these cases, you would be better served using the assignment operator instead. The next examples are all similar and they are all legal code. However, the walrus operator doesn’t improve your code in any of these cases.

02:20 None of these examples make your code more readable. You should instead do the extra assignment separately by using a traditional assignment statement. If you want to look at this in more depth, take a look at the original PEP which created the assignment expression operator.

02:39 There’s one use case where the := character sequence is already valid Python. In f-strings, a colon ( : ) is used to separate values from their format specification.

02:56 The := in this case does look like a walrus operator, but the effect is quite different. To interpret x:=8 inside the f-string, the expression is broken into three parts: the x , the colon ( : ), and the =8 . Here, x is a value, the colon ( : ) acts as a separator, and =8 is a format specification.

03:21 In this context, equals ( = ) specifies an alignment option. In this case, the value is padded with spaces in a field width of 8 .

03:31 To use assignment expressions inside f-strings, you need to add parentheses.

03:44 This updates the value of x as expected. However, you’re probably better off using traditional assignments outside of your f-strings instead.

03:55 Let’s look at some other situations where assignment expressions are illegal. Attribute and item assignment: You can only assign to simple names, not dotted or index names.

04:22 As you can see, this fails with a descriptive error message, and there’s no straightforward workaround. Iterable unpacking: You can’t unpack when using the walrus operator.

04:45 Augmented assignment: You can’t use the walrus operator combined with augmented assignment operators like plus equals ( += ). This raises a SyntaxError .

05:01 The easiest workaround here would be to do the augmentation explicitly. When you’re using the walrus operator, it will behave similarly to traditional assignment statements in many respects.

05:14 The scope of the assignment target is the same as for assignments. It will follow the LEGB rule. Typically, the assignment will happen in the local scope, but if the target name is already declared global or non-local, that is honored.

05:33 For more information on this topic, check out this Real Python course, Python Scope & the LEGB Rule: Resolving Names in Your Code. The precedence of the walrus operator can cause some confusion.

05:47 It binds less tightly than all other operators except the comma, so you might need parentheses to delimit the expression that’s assigned. As an example, note what happens when you don’t use parentheses.

06:12 Here, square is bound to the whole expression number ** 2 > 5 . In other words, square gets the value True and not the value of number ** 2 , which was the intention. In this case, you can delimit the expression with parentheses.

06:38 The parentheses make the if statement both clearer and actually correct. There’s one final gotcha. When assigning a tuple using the walrus operator, you always need to use parentheses around the tuple.

06:53 Compare the following assignments.

07:21 Note that in the second example, walrus takes the value 3.8 and not the whole tuple of (3.8, True) . That’s because the walrus operator binds more tightly than the comma.

07:33 This may seem a bit annoying. However, if the walrus operator bound less tightly than the comma, it wouldn’t be possible to use the walrus operator in function calls with more than one argument.

07:45 The style recommendations for the walrus operator are mostly the same as for the assignment operator. First, always add spaces around the walrus operator in your code.

07:57 Second, use parentheses around the expression as necessary but avoid adding extra parentheses that are not needed. The general design of assignment expressions is to make them easy to use when they’re helpful but to avoid overusing them when they might clutter up your code.

08:15 Next up, a look at some of the pitfalls that may meet you when you use the walrus operator.

dar3232 on May 5, 2023

I took a step further the unpacking example… 🤯

Become a Member to join the conversation.

Python Programming

Practice Python Exercises and Challenges with Solutions

Free Coding Exercises for Python Developers. Exercises cover Python Basics , Data structure , to Data analytics . As of now, this page contains 18 Exercises.

What included in these Python Exercises?

Each exercise contains specific Python topic questions you need to practice and solve. These free exercises are nothing but Python assignments for the practice where you need to solve different programs and challenges.

• All exercises are tested on Python 3.
• Each exercise has 10-20 Questions.
• The solution is provided for every question.
• Practice each Exercise in Online Code Editor

These Python programming exercises are suitable for all Python developers. If you are a beginner, you will have a better understanding of Python after solving these exercises. Below is the list of exercises.

Select the exercise you want to solve .

Basic Exercise for Beginners

Practice and Quickly learn Python’s necessary skills by solving simple questions and problems.

Topics : Variables, Operators, Loops, String, Numbers, List

Python Input and Output Exercise

Solve input and output operations in Python. Also, we practice file handling.

Topics : print() and input() , File I/O

Python Loop Exercise

This Python loop exercise aims to help developers to practice branching and Looping techniques in Python.

Topics : If-else statements, loop, and while loop.

Python Functions Exercise

Practice how to create a function, nested functions, and use the function arguments effectively in Python by solving different questions.

Topics : Functions arguments, built-in functions.

Python String Exercise

Solve Python String exercise to learn and practice String operations and manipulations.

Python Data Structure Exercise

Practice widely used Python types such as List, Set, Dictionary, and Tuple operations in Python

Python List Exercise

This Python list exercise aims to help Python developers to learn and practice list operations.

Python Dictionary Exercise

This Python dictionary exercise aims to help Python developers to learn and practice dictionary operations.

Python Set Exercise

This exercise aims to help Python developers to learn and practice set operations.

Python Tuple Exercise

This exercise aims to help Python developers to learn and practice tuple operations.

Python Date and Time Exercise

This exercise aims to help Python developers to learn and practice DateTime and timestamp questions and problems.

Topics : Date, time, DateTime, Calendar.

Python OOP Exercise

This Python Object-oriented programming (OOP) exercise aims to help Python developers to learn and practice OOP concepts.

Topics : Object, Classes, Inheritance

Python JSON Exercise

Practice and Learn JSON creation, manipulation, Encoding, Decoding, and parsing using Python

Python NumPy Exercise

Practice NumPy questions such as Array manipulations, numeric ranges, Slicing, indexing, Searching, Sorting, and splitting, and more.

Python Pandas Exercise

Practice Data Analysis using Python Pandas. Practice Data-frame, Data selection, group-by, Series, sorting, searching, and statistics.

Python Matplotlib Exercise

Practice Data visualization using Python Matplotlib. Line plot, Style properties, multi-line plot, scatter plot, bar chart, histogram, Pie chart, Subplot, stack plot.

Random Data Generation Exercise

Practice and Learn the various techniques to generate random data in Python.

Topics : random module, secrets module, UUID module

Python Database Exercise

Practice Python database programming skills by solving the questions step by step.

Use any of the MySQL, PostgreSQL, SQLite to solve the exercise

Exercises for Intermediate developers

The following practice questions are for intermediate Python developers.

If you have not solved the above exercises, please complete them to understand and practice each topic in detail. After that, you can solve the below questions quickly.

Exercise 1: Reverse each word of a string

Expected Output

• Use the split() method to split a string into a list of words.
• Reverse each word from a list
• finally, use the join() function to convert a list into a string

Steps to solve this question :

• Split the given string into a list of words using the split() method
• Use a list comprehension to create a new list by reversing each word from a list.
• Use the join() function to convert the new list into a string
• Display the resultant string

Exercise 2: Read text file into a variable and replace all newlines with space

Given : Assume you have a following text file (sample.txt).

Expected Output :

• First, read a text file.
• Next, use string replace() function to replace all newlines ( \n ) with space ( ' ' ).

Steps to solve this question : -

• First, open the file in a read mode
• Next, read all content from a file using the read() function and assign it to a variable.
• Display final string

Exercise 3: Remove items from a list while iterating

Description :

In this question, You need to remove items from a list while iterating but without creating a different copy of a list.

Remove numbers greater than 50

Expected Output : -

• Get the list's size
• Iterate list using while loop
• Check if the number is greater than 50
• If yes, delete the item using a del keyword
• Reduce the list size

Solution 1: Using while loop

Solution 2: Using for loop and range()

Exercise 4: Reverse Dictionary mapping

Exercise 5: display all duplicate items from a list.

• Use the counter() method of the collection module.
• Create a dictionary that will maintain the count of each item of a list. Next, Fetch all keys whose value is greater than 2

Solution 1 : - Using collections.Counter()

Solution 2 : -

Exercise 6: Filter dictionary to contain keys present in the given list

Exercise 7: print the following number pattern.

Refer to Print patterns in Python to solve this question.

• Use two for loops
• The outer loop is reverse for loop from 5 to 0
• Increment value of x by 1 in each iteration of an outer loop
• The inner loop will iterate from 0 to the value of i of the outer loop
• Print value of x in each iteration of an inner loop
• Print newline at the end of each outer loop

Exercise 8: Create an inner function

Question description : -

• Create an outer function that will accept two strings, x and y . ( x= 'Emma' and y = 'Kelly' .
• Create an inner function inside an outer function that will concatenate x and y.
• At last, an outer function will join the word 'developer' to it.

Exercise 9: Modify the element of a nested list inside the following list

Change the element 35 to 3500

Exercise 10: Access the nested key increment from the following dictionary

Under Exercises: -

Python Object-Oriented Programming (OOP) Exercise: Classes and Objects Exercises

Updated on:  December 8, 2021 | 51 Comments

Python Date and Time Exercise with Solutions

Updated on:  December 8, 2021 | 11 Comments

Python Dictionary Exercise with Solutions

Updated on:  May 6, 2023 | 57 Comments

Python Tuple Exercise with Solutions

Updated on:  December 8, 2021 | 95 Comments

Python Set Exercise with Solutions

Updated on:  October 20, 2022 | 27 Comments

Python if else, for loop, and range() Exercises with Solutions

Updated on:  September 6, 2021 | 292 Comments

Updated on:  August 2, 2022 | 153 Comments

Updated on:  September 6, 2021 | 108 Comments

Python List Exercise with Solutions

Updated on:  December 8, 2021 | 197 Comments

Updated on:  December 8, 2021 | 7 Comments

Python Data Structure Exercise for Beginners

Updated on:  December 8, 2021 | 116 Comments

Python String Exercise with Solutions

Updated on:  October 6, 2021 | 220 Comments

Updated on:  March 9, 2021 | 23 Comments

Updated on:  March 9, 2021 | 50 Comments

Updated on:  July 20, 2021 | 29 Comments

Python Basic Exercise for Beginners

Updated on:  August 31, 2023 | 490 Comments

Useful Python Tips and Tricks Every Programmer Should Know

Updated on:  May 17, 2021 | 23 Comments

Python random Data generation Exercise

Updated on:  December 8, 2021 | 13 Comments

Python Database Programming Exercise

Updated on:  March 9, 2021 | 17 Comments

• Online Python Code Editor

Updated on:  June 1, 2022 |

About PYnative

PYnative.com is for Python lovers. Here, You can get Tutorials, Exercises, and Quizzes to practice and improve your Python skills .

Explore Python

• Learn Python
• Python Basics
• Python Databases
• Python Exercises
• Python Quizzes
• Python Tricks

To get New Python Tutorials, Exercises, and Quizzes

Legal Stuff

We use cookies to improve your experience. While using PYnative, you agree to have read and accepted our Terms Of Use , Cookie Policy , and Privacy Policy .

Copyright © 2018–2024 pynative.com

• Python Basics
• Interview Questions
• Python Quiz
• Popular Packages
• Python Projects
• Practice Python
• AI With Python
• Learn Python3
• Python Automation
• Python Web Dev
• DSA with Python
• Python OOPs
• Dictionaries

Python Operators

Precedence and associativity of operators in python.

• Python Arithmetic Operators
• Difference between / vs. // operator in Python
• Python - Star or Asterisk operator ( * )
• What does the Double Star operator mean in Python?
• Division Operators in Python
• Modulo operator (%) in Python
• Python Logical Operators
• Python OR Operator
• Difference between 'and' and '&' in Python
• not Operator in Python | Boolean Logic

Ternary Operator in Python

• Python Bitwise Operators

Python Assignment Operators

Assignment operators in python.

• Walrus Operator in Python 3.8
• Increment += and Decrement -= Assignment Operators in Python
• Merging and Updating Dictionary Operators in Python 3.9
• New '=' Operator in Python3.8 f-string

Python Relational Operators

• Comparison Operators in Python
• Python NOT EQUAL operator
• Difference between == and is operator in Python
• Chaining comparison operators in Python
• Python Membership and Identity Operators
• Difference between != and is not operator in Python

In Python programming, Operators in general are used to perform operations on values and variables. These are standard symbols used for logical and arithmetic operations. In this article, we will look into different types of Python operators. 

• OPERATORS: These are the special symbols. Eg- + , * , /, etc.
• OPERAND: It is the value on which the operator is applied.

Types of Operators in Python

• Arithmetic Operators
• Comparison Operators
• Logical Operators
• Bitwise Operators
• Assignment Operators
• Identity Operators and Membership Operators

Arithmetic Operators in Python

Python Arithmetic operators are used to perform basic mathematical operations like addition, subtraction, multiplication , and division .

In Python 3.x the result of division is a floating-point while in Python 2.x division of 2 integers was an integer. To obtain an integer result in Python 3.x floored (// integer) is used.

Example of Arithmetic Operators in Python

Division operators.

In Python programming language Division Operators allow you to divide two numbers and return a quotient, i.e., the first number or number at the left is divided by the second number or number at the right and returns the quotient. 

There are two types of division operators: 

Float division

• Floor division

The quotient returned by this operator is always a float number, no matter if two numbers are integers. For example:

Example: The code performs division operations and prints the results. It demonstrates that both integer and floating-point divisions return accurate results. For example, ’10/2′ results in ‘5.0’ , and ‘-10/2’ results in ‘-5.0’ .

Integer division( Floor division)

The quotient returned by this operator is dependent on the argument being passed. If any of the numbers is float, it returns output in float. It is also known as Floor division because, if any number is negative, then the output will be floored. For example:

Example: The code demonstrates integer (floor) division operations using the // in Python operators . It provides results as follows: ’10//3′ equals ‘3’ , ‘-5//2’ equals ‘-3’ , ‘ 5.0//2′ equals ‘2.0’ , and ‘-5.0//2’ equals ‘-3.0’ . Integer division returns the largest integer less than or equal to the division result.

Precedence of Arithmetic Operators in Python

The precedence of Arithmetic Operators in Python is as follows:

• P – Parentheses
• E – Exponentiation
• M – Multiplication (Multiplication and division have the same precedence)
• D – Division
• A – Addition (Addition and subtraction have the same precedence)
• S – Subtraction

The modulus of Python operators helps us extract the last digit/s of a number. For example:

• x % 10 -> yields the last digit
• x % 100 -> yield last two digits

Arithmetic Operators With Addition, Subtraction, Multiplication, Modulo and Power

Here is an example showing how different Arithmetic Operators in Python work:

Example: The code performs basic arithmetic operations with the values of ‘a’ and ‘b’ . It adds (‘+’) , subtracts (‘-‘) , multiplies (‘*’) , computes the remainder (‘%’) , and raises a to the power of ‘b (**)’ . The results of these operations are printed.

Note: Refer to Differences between / and // for some interesting facts about these two Python operators.

Comparison of Python Operators

In Python Comparison of Relational operators compares the values. It either returns True or False according to the condition.

= is an assignment operator and == comparison operator.

Precedence of Comparison Operators in Python

In Python, the comparison operators have lower precedence than the arithmetic operators. All the operators within comparison operators have the same precedence order.

Example of Comparison Operators in Python

Let’s see an example of Comparison Operators in Python.

Example: The code compares the values of ‘a’ and ‘b’ using various comparison Python operators and prints the results. It checks if ‘a’ is greater than, less than, equal to, not equal to, greater than, or equal to, and less than or equal to ‘b’ .

Logical Operators in Python

Python Logical operators perform Logical AND , Logical OR , and Logical NOT operations. It is used to combine conditional statements.

Precedence of Logical Operators in Python

The precedence of Logical Operators in Python is as follows:

• Logical not
• logical and

Example of Logical Operators in Python

The following code shows how to implement Logical Operators in Python:

Example: The code performs logical operations with Boolean values. It checks if both ‘a’ and ‘b’ are true ( ‘and’ ), if at least one of them is true ( ‘or’ ), and negates the value of ‘a’ using ‘not’ . The results are printed accordingly.

Bitwise Operators in Python

Python Bitwise operators act on bits and perform bit-by-bit operations. These are used to operate on binary numbers.

Precedence of Bitwise Operators in Python

The precedence of Bitwise Operators in Python is as follows:

• Bitwise NOT
• Bitwise Shift
• Bitwise AND
• Bitwise XOR

Here is an example showing how Bitwise Operators in Python work:

Example: The code demonstrates various bitwise operations with the values of ‘a’ and ‘b’ . It performs bitwise AND (&) , OR (|) , NOT (~) , XOR (^) , right shift (>>) , and left shift (<<) operations and prints the results. These operations manipulate the binary representations of the numbers.

Python Assignment operators are used to assign values to the variables.

Let’s see an example of Assignment Operators in Python.

Example: The code starts with ‘a’ and ‘b’ both having the value 10. It then performs a series of operations: addition, subtraction, multiplication, and a left shift operation on ‘b’ . The results of each operation are printed, showing the impact of these operations on the value of ‘b’ .

Identity Operators in Python

In Python, is and is not are the identity operators both are used to check if two values are located on the same part of the memory. Two variables that are equal do not imply that they are identical. 

Example Identity Operators in Python

Let’s see an example of Identity Operators in Python.

Example: The code uses identity operators to compare variables in Python. It checks if ‘a’ is not the same object as ‘b’ (which is true because they have different values) and if ‘a’ is the same object as ‘c’ (which is true because ‘c’ was assigned the value of ‘a’ ).

Membership Operators in Python

In Python, in and not in are the membership operators that are used to test whether a value or variable is in a sequence.

Examples of Membership Operators in Python

The following code shows how to implement Membership Operators in Python:

Example: The code checks for the presence of values ‘x’ and ‘y’ in the list. It prints whether or not each value is present in the list. ‘x’ is not in the list, and ‘y’ is present, as indicated by the printed messages. The code uses the ‘in’ and ‘not in’ Python operators to perform these checks.

in Python, Ternary operators also known as conditional expressions are operators that evaluate something based on a condition being true or false. It was added to Python in version 2.5. 

It simply allows testing a condition in a single line replacing the multiline if-else making the code compact.

Syntax :   [on_true] if [expression] else [on_false] 

Examples of Ternary Operator in Python

The code assigns values to variables ‘a’ and ‘b’ (10 and 20, respectively). It then uses a conditional assignment to determine the smaller of the two values and assigns it to the variable ‘min’ . Finally, it prints the value of ‘min’ , which is 10 in this case.

In Python, Operator precedence and associativity determine the priorities of the operator.

Operator Precedence in Python

This is used in an expression with more than one operator with different precedence to determine which operation to perform first.

Let’s see an example of how Operator Precedence in Python works:

Example: The code first calculates and prints the value of the expression 10 + 20 * 30 , which is 610. Then, it checks a condition based on the values of the ‘name’ and ‘age’ variables. Since the name is “ Alex” and the condition is satisfied using the or operator, it prints “Hello! Welcome.”

Operator Associativity in Python

If an expression contains two or more operators with the same precedence then Operator Associativity is used to determine. It can either be Left to Right or from Right to Left.

The following code shows how Operator Associativity in Python works:

Example: The code showcases various mathematical operations. It calculates and prints the results of division and multiplication, addition and subtraction, subtraction within parentheses, and exponentiation. The code illustrates different mathematical calculations and their outcomes.

To try your knowledge of Python Operators, you can take out the quiz on Operators in Python . 

Python Operator Exercise Questions

Below are two Exercise Questions on Python Operators. We have covered arithmetic operators and comparison operators in these exercise questions. For more exercises on Python Operators visit the page mentioned below.

Q1. Code to implement basic arithmetic operations on integers

Q2. Code to implement Comparison operations on integers

Explore more Exercises: Practice Exercise on Operators in Python

Please Login to comment...

Similar reads.

• python-basics
• Python-Operators

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

Self-Attention Explained with Code

How large language models create rich, contextual embeddings.

Bradney Smith

Towards Data Science

Part 3 in the “LLMs from Scratch” series — a complete guide to understanding and building Large Language Models. If you are interested in learning more about how these models work I encourage you to read:

• Prelude: A Brief History of Large Language Models
• Part 1: Tokenization — A Complete Guide
• Part 2: Word Embeddings with word2vec from Scratch in Python
• Part 3: Self-Attention Explained with Code
• Part 4: A Complete Guide to BERT with Code

Introduction

The paper “Attention is All You Need” debuted perhaps the single largest advancement in Natural Language Processing (NLP) in the last 10 years: the Transformer [1]. This architecture massively simplified the complex designs of language models at the time while achieving unparalleled results. State-of-the-art (SOTA) models, such as those in the GPT, Claude, and Llama families, owe their success to this design, at the heart of which is self-attention. In this deep dive, we will explore how this mechanism works and how it is used by transformers to create contextually rich embeddings that enable these models to perform so well.

1 — Overview of the Transformer Embedding Process

2 — positional encoding, 3 — the self-attention mechanism, 4 — transformer embeddings in python, 5 — conclusion, 6 — further reading, 1.1 — recap on transformers.

In the prelude article of this series, we briefly explored the history of the Transformer and its impact on NLP. To recap: the Transformer is a deep neural network architecture that is the foundation for almost all LLMs today. Derivative models are often called Transformer-based models or transformers for short, and so these terms will be used interchangeably here. Like all machine learning models, transformers work with numbers and linear algebra rather than processing human language directly. Because of this, they must convert textual inputs from users into numerical representations through several steps. Perhaps the most important of these steps is applying the self-attention mechanism, which is the focus of this article. The process of representing text with vectors is called embedding (or encoding ), hence the numerical representations of the input text are known as transformer embeddings .

1.2 — The Issue with Static Embeddings

In Part 2 of this series , we explored static embeddings for language models using word2vec as an example. This embedding method predates transformers and suffers from one major drawback: the lack of contextual information. Words with multiple meanings (called polysemous words) are encoded with somewhat ambiguous representations since they lack the context needed for precise meaning. A classic example of a polysemous word is bank . Using a static embedding model, the word bank would be represented in vector space with some degree of similarity to words such as money and deposit and some degree of similarity to words such as river and nature . This is because the word will occur in many different contexts within the training data. This is the core problem with static embeddings: they do not change based on context — hence the term “static”.

1.3 — Fixing Static Embeddings

Transformers overcome the limitations of static embeddings by producing their own context-aware transformer embeddings. In this approach, fixed word embeddings are augmented with positional information (where the words occur in the input text) and contextual information (how the words are used). These two steps take place in distinct components in transformers, namely the positional encoder and the self-attention blocks, respectively. We will look at each of these in detail in the following sections. By incorporating this additional information, transformers can produce much more powerful vector representations of words based on their usage in the input sequence. Extending the vector representations beyond static embeddings is what enables Transformer-based models to handle polysemous words and gain a deeper understanding of language compared to previous models.

1.4 — Introducing Learned Embeddings

Much like the word2vec approach released four years prior, transformers store the initial vector representation for each token in the weights of a linear layer (a small neural network). In the word2vec model, these representations form the static embeddings, but in the Transformer context these are known as learned embeddings . In practice they are very similar, but using a different name emphasises that these representations are only a starting point for the transformer embeddings and not the final form.

The linear layer sits at the beginning of the Transformer architecture and contains only weights and no bias terms (bias = 0 for every neuron). The layer weights can be represented as a matrix of size V × d_model , where V is the vocabulary size (the number of unique words in the training data) and d_model is the number of embedding dimensions. In the previous article, we denoted d_model as N , in line with word2vec notation, but here we will use d_model which is more common in the Transformer context. The original Transformer was proposed with a d_model size of 512 dimensions, but in practice any reasonable value can be used.

1.5 — Creating Learned Embeddings

A key difference between static and learned embeddings is the way in which they are trained. Static embeddings are trained in a separate neural network (using the Skip-Gram or Continuous Bag of Words architectures) using a word prediction task within a given window size. Once trained, the embeddings are then extracted and used with a range of different language models. Learned embeddings, however, are integral to the transformer you are using and are stored as weights in the first linear layer of the model. These weights, and consequently the learned embedding for each token in the vocabulary, are trained in the same backpropagation steps as the rest of the model parameters. Below is a summary of the training process for learned embeddings.

Step 1: Initialisation

Randomly initialise the weights for each neuron in the linear layer at the beginning of the model, and set the bias terms to 0. This layer is also called the embedding layer, since it is the linear layer that will store the learned embeddings. The weights can be represented as a matrix of size V × d_model , where the word embedding for each word in the vocabulary is stored along the rows. For example, the embedding for the first word in the vocabulary is stored in the first row, the second word is stored in the second row, and so on.

Step 2: Training

At each training step, the Transformer receives an input word and the aim is to predict the next word in the sequence — a task known as Next Token Prediction (NTP). Initially, these predictions will be very poor, and so every weight and bias term in the network will be updated to improve performance against the loss function, including the embeddings. After many training iterations, the learned embeddings should provide a strong vector representation for each word in the vocabulary.

Step 3: Extract the Learned Embeddings

When new input sequences are given to the model, the words are converted into tokens with an associated token ID, which corresponds to the position of the token in the tokenizer’s vocabulary. For example, the word cat may lie at position 349 in the tokenizer’s vocabulary and so will take the ID 349 . Token IDs are used to create one-hot encoded vectors that extract the correct learned embeddings from the weights matrix (that is, V -dimensional vectors where every element is 0 except for the element at the token ID position, which is 1).

Note: PyTorch is a very popular deep learning library in Python that powers some of the most well-known machine learning packages, such as the HuggingFace Transformers library [2]. If you are familiar with PyTorch, you may have encountered the nn.Embedding class, which is often used to form the first layer of transformer networks (the nn denotes that the class belongs to the neural network package). This class returns a regular linear layer that is initialised with the identity function as the activation function and with no bias term. The weights are randomly initialised since they are parameters to be learned by the model during training. This essentially carries out the steps described above in one simple line of code. Remember, the nn.Embeddings layer does not provide pre-trained word embeddings out-of-the-box, but rather initialises a blank canvas of embeddings before training. This is to allow the transformer to learn its own embeddings during the training phase.

1.6 — Transformer Embedding Process

Once the learned embeddings have been trained, the weights in the embedding layer never change. That is, the learned embedding for each word (or more specifically, token) always provides the same starting point for a word’s vector representation. From here, the positional and contextual information will be added to produce a unique representation of the word that is reflective of its usage in the input sequence.

Transformer embeddings are created in a four-step process, which is demonstrated below using the example prompt: Write a poem about a man fishing on a river bank. . Note that the first two steps are the same as the word2vec approach we saw before. Steps 3 and 4 are the further processing that add contextual information to the embeddings.

Step 1) Tokenization:

Tokenization is the process of dividing a longer input sequence into individual words (and parts of words) called tokens. In this case, the sentence will be broken down into:

write , a , poem , about , a , man , fishing , on , a , river , bank

Next, the tokens are associated with their token IDs, which are integer values corresponding to the position of the token in the tokenizer’s vocabulary (see Part 1 of this series for an in-depth look at the tokenization process).

Step 2) Map the Tokens to Learned Embeddings:

Once the input sequence has been converted into a set of token IDs, the tokens are then mapped to their learned embedding vector representations, which were acquired during the transformer’s training. These learned embeddings have the “lookup table” behaviour as we saw in the word2vec example in Part 2 of this series . The mapping takes place by multiplying a one-hot encoded vector created from the token ID with the weights matrix, just as in the word2vec approach. The learned embeddings are denoted V in the image below.

Step 3) Add Positional Information with Positional Encoding:

Positional Encoding is then used to add positional information to the word embeddings. Whereas Recurrent Neural Networks (RNNs) process text sequentially (one word at a time), transformers process all words in parallel. This removes any implicit information about the position of each word in the sentence. For example, the sentences the cat ate the mouse and the mouse ate the cat use the same words but have very different meanings. To preserve the word order, positional encoding vectors are generated and added to the learned embedding for each word. In the image below, the positional encoding vectors are denoted P , and the sums of the learned embeddings and positional encodings are denoted X .

Step 4) Modify the Embeddings using Self-Attention:

The final step is to add contextual information using the self-attention mechanism. This determines which words give context to other words in the input sequence. In the image below, the transformer embeddings are denoted y .

2.1 — The Need for Positional Encoding

Before the self-attention mechanism is applied, positional encoding is used to add information about the order of tokens to the learned embeddings. This compensates for the loss of positional information caused by the parallel processing used by transformers described earlier. There are many feasible approaches for injecting this information, but all methods must adhere to a set of constraints. The functions used to generate positional information must produce values that are:

• Bounded — values should not explode in the positive or negative direction but be constrained (e.g. between 0 and 1, -1 and 1, etc)
• Periodic — the function should produce a repeating pattern that the model can learn to recognise and discern position from
• Predictable — positional information should be generated in such a way that the model can understand the position of words in sequence lengths it was not trained on. For example, even if the model has not seen a sequence length of exactly 412 tokens in its training, the transformer should be able to understand the position of each of the embeddings in the sequence.

These constraints ensure that the positional encoder produces positional information that allows words to attend to (gain context from) any other important word, regardless of their relative positions in the sequence. In theory, with a sufficiently powerful computer, words should be able to gain context from every relevant word in an infinitely long input sequence. The length of a sequence from which a model can derive context is called the context length . In chatbots like ChatGPT, the context includes the current prompt as well as all previous prompts and responses in the conversation (within the context length limit). This limit is typically in the range of a few thousand tokens, with GPT-3 supporting up to 4096 tokens and GPT-4 enterprise edition capping at around 128,000 tokens [3].

2.2 — Positional Encoding in “Attention is All You Need”

The original transformer model was proposed with the following positional encoding functions:

• pos is the position of the word in the input, where pos = 0 corresponds to the first word in the sequence
• i is the index of each embedding dimension, ranging from i=0 (for the first embedding dimension) up to d_model
• d_model is the number of embedding dimensions for each learned embedding vector (and therefore each positional encoding vector). This was previously denoted N in the article on word2vec.

The two proposed functions take arguments of 2i and 2i+1 , which in practice means that the sine function generates positional information for the even-numbered dimensions of each word vector ( i is even), and the cosine function does so for the odd-numbered dimensions ( i is odd). According to the authors of the transformer:

“The positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000·2π. We chose this function because we hypothesised it would allow the model to easily learn to attend by relative positions, since for any fixed offset k , PE_pos+k can be represented as a linear function of PE_pos ”.

The value of the constant in the denominator being 10_000 was found to be suitable after some experimentation, but is a somewhat arbitrary choice by the authors.

2.3 — Other Positional Encoding Approaches

The positional encodings shown above are considered fixed because they are generated by a known function with deterministic (predictable) outputs. This represents the most simple form of positional encoding. It is also possible to use learned positional encodings by randomly initialising some positional encodings and training them with backpropagation. Derivatives of the BERT architecture are examples of models that take this learned encoding approach. More recently, the Rotary Positional Encoding (RoPE) method has gained popularity, finding use in models such as Llama 2 and PaLM, among other positional encoding methods.

2.4 — Implementing a Positional Encoder in Python

Creating a positional encoder class in Python is fairly straightforward. We can start by defining a function that accepts the number of embedding dimensions ( d_model ), the maximum length of the input sequence ( max_length ), and the number of decimal places to round each value in the vectors to ( rounding ). Note that transformers define a maximum input sequence length, and any sequence that has fewer tokens than this limit is appended with padding tokens until the limit is reached. To account for this behaviour in our positional encoder, we accept a max_length argument. In practice, this limit is typically thousands of characters long.

We can also exploit a mathematical trick to save computation. Instead of calculating the denominator for both PE_{pos, 2i} and PE_{pos, 2i} , we can note that the denominator is identical for consecutive pairs of i . For example, the denominators for i=0 and i=1 are the same, as are the denominators for i=2 and i=3 . Hence, we can perform the calculations to determine the denominators once for the even values of i and reuse them for the odd values of i .

2.5 — Visualising the Positional Encoding Matrix

Recall that the positional information generated must be bounded, periodic, and predictable. The outputs of the sinusoidal functions presented earlier can be collected into a matrix, which can then be easily combined with the learned embeddings using element-wise addition. Plotting this matrix gives a nice visualisation of the desired properties. In the plot below, curving bands of negative values (blue) emanate from the left edge of the matrix. These bands form a pattern that the transformer can easily learn to predict.

3.1 — Overview of Attention Mechanisms

Now that we have covered an overview of transformer embeddings and the positional encoding step, we can turn our focus to the self-attention mechanism itself. In short, self-attention modifies the vector representation of words to capture the context of their usage in an input sequence. The “self” in self-attention refers to the fact that the mechanism uses the surrounding words within a single sequence to provide context. As such, self-attention requires all words to be processed in parallel. This is actually one of the main benefits of transformers (especially compared to RNNs) since the models can leverage parallel processing for a significant performance boost. In recent times, there has been some rethinking around this approach, and in the future we may see this core mechanism being replaced [4].

Another form of attention used in transformers is cross-attention. Unlike self-attention, which operates within a single sequence, cross-attention compares each word in an output sequence to each word in an input sequence, crossing between the two embedding matrices. Note the difference here compared to self-attention, which focuses entirely within a single sequence.

3.2 — Visualising How Self-Attention Contextualises Embeddings

The plots below show a simplified set of learned embedding vectors in two dimensions. Words associated with nature and rivers are concentrated in the top right quadrant of the graph, while words associated with money are concentrated in the bottom left. The vector representing the word bank is positioned between the two clusters due to its polysemic nature. The objective of self-attention is to move the learned embedding vectors to regions of vector space that more accurately capture their meaning within the context of the input sequence. In the example input Write a poem about a man fishing on a river bank. , the aim is to move the vector for bank in such a way that captures more of the meaning of nature and rivers, and less of the meaning of money and deposits.

Note: More accurately, the goal of self-attention here is to update the vector for every word in the input, so that all embeddings better represent the context in which they were used. There is nothing special about the word bank here that transformers have some special knowledge of — self-attention is applied across all the words. We will look more at this shortly, but for now, considering solely how bank is affected by self-attention gives a good intuition for what is happening in the attention block. For the purpose of this visualisation, the positional encoding information has not been explicitly shown. The effect of this will be minimal, but note that the self-attention mechanism will technically operate on the sum of the learned embedding plus the positional information and not solely the learned embedding itself.

3.3 — The Self-Attention Algorithm

In the section above, we stated that the goal of self-attention is to move the embedding for each token to a region of vector space that better represents the context of its use in the input sequence. What we didn’t discuss is how this is done. Here we will show a step-by-step example of how the self-attention mechanism modifies the embedding for bank , by adding context from the surrounding tokens.

Step 1) Calculate the Similarity Between Words using the Dot Product:

The context of a token is given by the surrounding tokens in the sentence. Therefore, we can use the embeddings of all the tokens in the input sequence to update the embedding for any word, such as bank . Ideally, words that provide significant context (such as river ) will heavily influence the embedding, while words that provide less context (such as a ) will have minimal effect.

The degree of context one word contributes to another is measured by a similarity score. Tokens with similar learned embeddings are likely to provide more context than those with dissimilar embeddings. The similarity scores are calculated by taking the dot product of the current embedding for one token (its learned embedding plus positional information) with the current embeddings of every other token in the sequence. For clarity, the current embeddings have been termed s elf-attention inputs in this article and are denoted x .

There are several options for measuring the similarity between two vectors, which can be broadly categorised into: distance-based and angle-based metrics. Distance-based metrics characterise the similarity of vectors using the straight-line distance between them. This calculation is relatively simple and can be thought of as applying Pythagoras’s theorem in d_model -dimensional space. While intuitive, this approach is computationally expensive.

For angle-based similarity metrics, the two main candidates are: cosine similarity and dot-product similarity . Both of these characterise similarity using the cosine of the angle between the two vectors, θ. For orthogonal vectors (vectors that are at right angles to each other) cos(θ) = 0 , which represents no similarity. For parallel vectors, cos(θ) = 1 , which represents that the vectors are identical. Solely using the angle between vectors, as is the case with cosine similarity, is not ideal for two reasons. The first is that the magnitude of the vectors is not considered, so distant vectors that happen to be aligned will produce inflated similarity scores. The second is that cosine similarity requires first computing the dot product and then dividing by the product of the vectors’ magnitudes — making cosine similarity a computationally expensive metric. Therefore, the dot product is used to determine similarity. The dot product formula is given below for two vectors x_1 and x_2 .

The diagram below shows the dot product between the self-attention input vector for bank , x_bank, and the matrix of vector representations for every token in the input sequence, X^T . We can also write x_bank as x_11 to reflect its position in the input sequence. The matrix X stores the self-attention inputs for every token in the input sequence as rows. The number of columns in this matrix is given by L_max , the maximum sequence length of the model. In this example, we will assume that the maximum sequence length is equal to the number of words in the input prompt, removing the need for any padding tokens (see Part 4 in this series for more about padding). To compute the dot product directly, we can transpose X and calculate the vector of similarity scores, S_bank using S_bank = x_bank ⋅ X^T . The individual elements of S_bank represent the similarity scores between bank and each token in the input sequence.

Step 2) Scale the Similarity Scores:

The dot product approach lacks any form of normalisation (unlike cosine similarity), which can cause the similarity scores to become very large. This can pose computational challenges, so normalisation of some form becomes necessary. The most common method is to divide each score by √ d_model , resulting in scaled dot-product attention . Scaled dot-product attention is not restricted to self-attention and is also used for cross-attention in transformers.

Step 3) Calculate the Attention Weights using the Softmax Function:

The output of the previous step is the vector S_bank , which contains the similarity scores between bank and every token in the input sequence. These similarity scores are used as weights to construct a transformer embedding for bank from the weighted sum of embeddings for each surrounding token in the prompt. The weights, known as attention weights , are calculated by passing S_bank into the softmax function. The outputs are stored in a vector denoted W_bank . To see more about the softmax function, refer to the previous article on word2vec.

Step 4) Calculate the Transformer Embedding

Finally, the transformer embedding for bank is obtained by taking the weighted sum of write , a , prompt , …, bank . Of course, bank will have the highest similarity score with itself (and therefore the largest attention weight), so the output embedding after this process will remain similar to before. This behaviour is ideal since the initial embedding already occupies a region of vector space that encodes some meaning for bank. The goal is to nudge the embedding towards the words that provide more context. The weights for words that provide little context, such as a and man , are very small. Hence, their influence on the output embedding will be minimal. Words that provide significant context, such as river and fishing , will have higher weights, and therefore pull the output embedding closer to their regions of vector space. The end result is a new embedding, y_bank , that reflects the context of the entire input sequence.

3.4 — Expanding Self-Attention using Matrices

Above, we walked through the steps to calculate the transformer embedding for the singular word bank . The input consisted of the learned embedding vector for bank plus its positional information, which we denoted x_11 or x_bank . The key point here, is that we considered only one vector as the input. If we instead pass in the matrix X (with dimensions L_max × d_model ) to the self-attention block, we can calculate the transformer embedding for every token in the input prompt simultaneously. The output matrix, Y , contains the transformer embedding for every token along the rows of the matrix. This approach is what enables transformers to quickly process text.

3.5 — The Query, Key, and Value Matrices

The above description gives an overview of the core functionality of the self-attention block, but there is one more piece of the puzzle. The simple weighted sum above does not include any trainable parameters, but we can introduce some to the process. Without trainable parameters, the performance of the model may still be good, but by allowing the model to learn more intricate patterns and hidden features from the training data, we observe much stronger model performance.

The self-attention inputs are used three times to calculate the new embeddings, these include the x_bank vector, the X^T matrix in the dot product step, and the X^T matrix in the weighted sum step. These three sites are the perfect candidates to introduce some weights, which are added in the form of matrices (shown in red). When pre-multiplied by their respective inputs (shown in blue), these form the key , query , and value matrices, K , Q , and V (shown in purple). The number of columns in these weight matrices is an architectural choice by the user. Choosing a value for d_q , d_k , and d_v that is less than d_model will result in dimensionality reduction, which can improve model speed. Ultimately, these values are hyperparameters that can be changed based on the specific implementation of the model and the use-case, and are often all set equal to d_model if unsure [5].

3.6 — The Database Analogy

The names for these matrices come from an analogy with databases, which is explained briefly below.

• A query in a database is what you are looking for when performing a search. For example, “show me all the albums in the database that have sold more than 1,000,000 records”. In the self-attention block, we are essentially asking the same question, but phrased as “show me the transformer embedding for this vector (e.g. x_bank )”.
• The keys in the database are the attributes or columns that are being searched against. In the example given earlier, you can think of this as the “Albums Sold” column, which stores the information we are interested in. In self-attention, we are interested in the embeddings for every word in the input prompt, so we can compute a set of attention weights. Therefore, the key matrix is a collection of all the input embeddings.
• The values correspond to the actual data in the database, that is, the actual sale figures in our example (e.g. 2,300,000 copies). For self-attention, this is exactly the same as the input for the key matrix: a collection of all the input embeddings. Hence, the key and value matrices both take in the matrix X as the input.

3.7 — A Note on Multi-Head Attention

Distributing Computation Across Multiple Heads:

The “Attention is All You Need” paper expands self-attention into multi-head attention , which gives even richer representations of the input sequences. This method involves repeating the calculation of attention weights using different key, query, and value matrices which are learned independently within each head . A head is a section of the attention block dedicated to processing a fraction of the input embedding dimensions. For example, an input, x , with 512 dimensions will be divided by the number of heads, h , to create h chunks of size d_k (where d_k = d_model / h ). For a model with 8 heads ( h=8 ), each head will receive 64 dimensions of x ( d_k = 64). Each chunk is processed using the self-attention mechanism in its respective head, and at the end the outputs from all heads are combined using a linear layer to produce a single output with the original 512 dimensions.

The Benefits of Using Multiple Heads:

The core idea is to allow each head to learn different types of relationships between words in the input sequence, and to combine them to create deep text representations. For example, some heads might learn to capture long-term dependencies (relationships between words that are distant in the text), while others might focus on short-term dependencies (words that are close in text).

Building Intuition for Multi-Head Attention:

To build some intuition for the usefulness of multiple attention heads, consider words in a sentence that require a lot of context. For example, in the sentence I ate some of Bob’s chocolate cake , the word ate should attend to I, Bob’s and cake to gain full context. This is a rather simple example, but if you extend this concept to complex sequences spanning thousands of words, hopefully it seems reasonable that distributing the computational load across separate attention mechanisms will be beneficial.

Summary of Multi-Head Attention:

To summarise, multi-head attention involves repeating the self-attention mechanism h times and combining the results to distil the information into rich transformer embeddings. While this step is not strictly necessary, it has been found to produce more impressive results, and so is standard in transformer-based models.

4.1 — Extracting Learned Embeddings and Transformer Embeddings from Transformer Models

Python has many options for working with transformer models, but none are perhaps as well-known as Hugging Face. Hugging Face provides a centralised resource hub for NLP researchers and developers alike, including tools such as:

• transformers : The library at the core of Hugging Face, which provides an interface for using, training, and fine-tuning pre-trained transformer models.
• tokenizers : A library for working with tokenizers for many kinds of transformers, either using pre-built tokenizer models or constructing brand new ones from scratch.
• datasets : A collection of datasets to train models on a variety of tasks, not just restricted to NLP.
• Model Hub: A large repository of cutting-edge models from published papers, community-developed models, and everything in between. These are made freely available and can be easily imported into Python via the transformers API.

The code cell below shows how the transformers library can be used to load a transformer-based model into Python, and how to extract both the learned embeddings for words (without context) and the transformer embeddings (with context). The remainder of this article will break down the steps shown in this cell and describe additional functionalities available when working with embeddings.

4.2 — Import the Transformers Library

The first step to produce transformer embeddings is to choose a model from the Hugging Face transformers library. In this article, we will not use the model for inference but solely to examine the embeddings it produces. This is not a standard use-case, and so we will have to do some extra digging in order to access the embeddings. Since the transformers library is written in PyTorch (referred to as torch in the code), we can import torch to extract data from the inner workings of the models.

4.3 — Choose a Model

For this example, we will use DistilBERT, a smaller version of Google’s BERT model which was released by Hugging Face themselves in October 2019 [6]. According to the Hugging Face documentation [7]:

DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased , runs 60% faster while preserving over 95% of BERT’s performances as measured on the GLUE language understanding benchmark .

We can import DistilBERT and its corresponding tokenizer into Python either directly from the transformers library or using the AutoModel and AutoTokenizer classes. There is very little difference between the two, although AutoModel and AutoTokenizer are often preferred since the model name can be parameterised and stored in a string, which makes it simpler to change the model being used.

After importing DistilBERT and its corresponding tokenizer, we can call the from_pretrained method for each to load in the specific version of the DistilBERT model and tokenizer we want to use. In this case, we have chosen distilbert-base-uncased , where base refers to the size of the model, and uncased indicates that the model was trained on uncased text (all text is converted to lowercase).

4.4 — Create Some Example Sentences

Next, we can create some sentences to give the model some words to embed. The two sentences, s1 and s2 , both contain the word bank but in different contexts. The goal here is to show that the word bank will begin with the same learned embedding in both sentences, then be modified by DistilBERT using self-attention to produce a unique, contextualised embedding for each input sequence.

4.5 — Tokenize an Input Sequence

The tokenizer class can be used to tokenize an input sequence (as shown below) and convert a string into a list of token IDs. Optionally, we can also pass a return_tensors argument to format the token IDs as a PyTorch tensor ( return_tensors=pt ) or as TensorFlow constants ( return_tensors=tf ). Leaving this argument empty will return the token IDs in a Python list. The return value is a dictionary that contains input_ids : the list-like object containing token IDs, and attention_mask which we will ignore for now.

Note: BERT-based models include a [CLS] token at the beginning of each sequence, and a [SEP] token to distinguish between two bodies of text in the input. These are present due to the tasks that BERT was originally trained on and can largely be ignored here. For a discussion on BERT special tokens, model sizes, cased vs uncased , and the attention mask, see Part 4 of this series .

4.6 — Extract the Learned Embeddings from a Model

Each transformer model provides access to its learned embeddings via the embeddings.word_embeddings method. This method accepts a token ID or collection of token IDs and returns the learned embedding(s) as a PyTorch tensor.

4.7 — Extract the Transformer Embeddings from a Model

Converting a context-lacking learned embedding into a context-aware transformer embedding requires a forward pass of the model. Since we are not updating the weights of the model here (i.e. training the model), we can use the torch.no_grad() context manager to save on memory. This allows us to pass the tokens directly into the model and compute the transformer embeddings without any unnecessary calculations. Once the tokens have been passed into the model, a BaseModelOutput is returned, which contains various information about the forward pass. The only data that is of interest here is the activations in the last hidden state, which form the transformer embeddings. These can be accessed using the last_hidden_state attribute, as shown below, which concludes the explanation for the code cell shown at the top of this section.

4.8 — Convert Token IDs to Tokens

It is possible to convert token IDs back into textual tokens, which shows exactly how the tokenizer divided the input sequence. This is useful when longer or rarer words are divided into multiple subwords when using subword tokenizers such as WordPiece (e.g. in BERT-based models) or Byte-Pair Encoding (e.g. in the GPT family of models).

The self-attention mechanism generates rich, context-aware transformer embeddings for text by processing each token in an input sequence simultaneously. These embeddings build on the foundations of static word embeddings (such as word2vec) and enable more capable language models such as BERT and GPT. Further work in this field will continue to improve the capabilities of LLMs and NLP as a whole.

[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is All You Need (2017), Advances in Neural Information Processing Systems 30 (NIPS 2017)

[2] Hugging Face, Transformers (2024), HuggingFace.co

[3] OpenAI, ChatGPT Pricing (2024), OpenAI.com

[4] A. Gu and T. Dao, Mamba: Linear-Time Sequence Modelling with Selective State Spaces (2023), ArXiv abs/2312.00752

[5] J. Alammar, The Illustrated Transformer (2018). GitHub

[6] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019), 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing — NeurIPS 2019

[7] Hugging Face, DistilBERT Documentation (2024) HuggingFace.co

[8] Hugging Face, BERT Documentation (2024) HuggingFace.co

Written by Bradney Smith

Senior Machine Learning Engineer. My work focuses on Natural Language Processing (NLP) and data science communication. Check out my "LLMs from Scratch" series !

Text to speech

CS50: Introduction to Computer Science

An introduction to the intellectual enterprises of computer science and the art of programming.

Associated Schools

Harvard School of Engineering and Applied Sciences

What you'll learn.

A broad and robust understanding of computer science and programming

How to think algorithmically and solve programming problems efficiently

Concepts like abstraction, algorithms, data structures, encapsulation, resource management, security, software engineering, and web development

Familiarity with a number of languages, including C, Python, SQL, and JavaScript plus CSS and HTML

How to engage with a vibrant community of like-minded learners from all levels of experience

How to develop and present a final programming project to your peers

Course description

This is CS50x , Harvard University's introduction to the intellectual enterprises of computer science and the art of programming for majors and non-majors alike, with or without prior programming experience. An entry-level course taught by David J. Malan, CS50x teaches students how to think algorithmically and solve problems efficiently. Topics include abstraction, algorithms, data structures, encapsulation, resource management, security, software engineering, and web development. Languages include C, Python, SQL, and JavaScript plus CSS and HTML. Problem sets inspired by real-world domains of biology, cryptography, finance, forensics, and gaming. The on-campus version of CS50x , CS50, is Harvard's largest course. 

Students who earn a satisfactory score on 9 problem sets (i.e., programming assignments) and a final project are eligible for a certificate. This is a self-paced course–you may take CS50x on your own schedule.

Instructors

David J. Malan

You may also like

CS50's Understanding Technology

This is CS50’s introduction to technology for students who don’t (yet!) consider themselves computer persons.

CS50 for Lawyers

This course is a variant of Harvard University's introduction to computer science, CS50, designed especially for lawyers (and law students).

Using Python for Research

Take your introductory knowledge of Python programming to the next level and learn how to use Python 3 for your research.

Join our list to learn more

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Get early access and see previews of new features.

What is the best way to do automatic attribute assignment in Python, and is it a good idea?

Instead of writing code like this every time I define a class:

I could use this recipe for automatic attribute assignment .

Two questions :

• Are there drawbacks or pitfalls associated with this shortcut?

Is there a better way to achieve similar convenience?

• 4 +1 this is a recurrent pattern in Python. I don't see any drawback in using the autoassign decorator, as it lets you define a list of argument not to assign. –  pberkes Sep 6, 2010 at 16:55
• 1 stackoverflow.com/questions/1389180/… –  pmav99 May 24, 2014 at 12:32

9 Answers 9

There are some things about the autoassign code that bug me (mostly stylistic, but one more serious problem):

autoassign does not assign an 'args' attribute:

autoassign acts like a decorator. But autoassign(*argnames) calls a function which returns a decorator. To achieve this magic, autoassign needs to test the type of its first argument. If given a choice, I prefer functions not test the type of its arguments.

There seems to be a considerable amount of code devoted to setting up sieve , lambdas within lambdas, ifilters, and lots of conditions.

I think there might be a simpler way. (See below).

for _ in itertools.starmap(assigned.setdefault, defaults): pass . I don't think map or starmap was meant to call functions, whose only purpose is their side effects. It could have been written more clearly with the mundane:

Here is an alternative simpler implementation which has the same functionality as autoassign (e.g. can do includes and excludes), and which addresses the above points:

And here is the unit test I used to check its behavior:

PS. Using autoassign or autoargs is compatible with IPython code completion.

• 1 I also found your autoargs incredibly handy. One issue I noticed is that it seems to throw an error if defaults is None , because the call to reversed fails. Am I right in thinking that this should be fixable by adding an if defaults: in front of the line for attr,val in zip(reversed(attrs),reversed(defaults)): , or does that have unintended sideeffects? The unit test passes. –  m01 Feb 12, 2014 at 9:32
• @m01: Sorry for the very belated response. Thanks for the correction! –  unutbu Aug 2, 2017 at 15:38
• @unutbu Just a nit but in 3.10.x inspect.getargspec(func) getargspec is deprecated. you should use inspect.getfullargspec(func) instead –  Frank C. Jan 10, 2023 at 20:46

From Python 3.7+ you can use a Data Class , which achieves what you want and more.

It allows you to define fields for your class, which are attributes automatically assigned.

It would look something like that:

The __init__ method will be automatically created in your class, and it will assign the arguments of instance creation to those attributes (and validate the arguments).

Note that here type hinting is required , that is why I have used int and str in the example. If you don't know the type of your field, you can use Any from the typing module .

• 1 Backport for Python 3.6 : The PEP 557 references a sample implementation ( permalink as of 2019-10-28 ) on GitHub which has a pypi/pip module as well. Therefore it can be installed by pip install dataclasses . –  luckydonald Oct 28, 2019 at 21:19

I don't know if it is necessarily better, but you could do this:

Courtesy Peter Norvig's Python: Infrequently Answered Questions .

• 7 This is a quick solution to the problem, but it has two drawbacks: 1) you have to add much more code if you want to make sure that some arguments are passed to the function; 2) the signature of the function is completely opaque, and it cannot be used to understand how to call the function; one will need to rely on the docstring instead –  pberkes Sep 6, 2010 at 16:57

One drawback: many IDEs parse __init__.py to discover an object's attributes. If you want automatic code completion in your IDE to be more functional, then you may be better off spelling it out the old-fashioned way.

If you have a lot of variables, you could pass one single configuration dict or object.

Similar to the above, though not the same... the following is very short, deals with args and kwargs :

Use like this:

This a simple implementation by judy2k :

In this package you can now find

• @autoargs inspired by answer-3653049
• @autoprops to transform the fields generated by @autoargs into @property , for use in combination with a validation library such as enforce or pycontracts .

Note that this has been validated for python 3.5+

You just can't use *args, but you can store in some instance list (like self.args, don't know)

Your Answer

Reminder: Answers generated by artificial intelligence tools are not allowed on Stack Overflow. Learn more

Sign up or log in

Post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy .

Not the answer you're looking for? Browse other questions tagged python attributes decorator or ask your own question .

• Featured on Meta
• The 2024 Developer Survey Is Live
• The return of Staging Ground to Stack Overflow
• The [tax] tag is being burninated
• Policy: Generative AI (e.g., ChatGPT) is banned

Hot Network Questions

• Book where the populace is split between two sects, that of the “hand” and that of the “mind"
• Was it known in ancient Rome and Greece that boiling water made it safe to drink and if so, what was the theory behind this?
• Why aren't tightly stitched commercial pcbs more common?
• Thought experiment regarding gravity
• Has a country ever by its own volition refused to join the United Nations?
• TikZ - how to draw ticks on circle perpendicularly with ellipses around?
• 1h 10m international > domestic transit at IST (Istanbul) on a single ticket
• Group with a translation invariant ultrafilter
• Have any countries managed to reduce immigration by a significant margin in the absence of economic problems?
• What happens when you target a dead creature with Scrying?
• Is there a phrase like "etymologically related" but for food?
• Why are we abbreviating Player's Handbook to PHB?
• Why is array access not an infix operator?
• siesta 5.0 compilation issues with spack
• What is the difference between Hof and Bauernhof?
• Yosemite national park availability
• For the square wave signal, why does a narrower square wave correspond to more spread in the frequency domain?
• Is it allowed to use patents for new inventions?
• How do Authenticators work?
• I need help in understanding the alternative solution provided to solve this geometry question of calculating area of quadrilateral
• Ubuntu Terminal with alternating colours for each line
• How can I use a transistor to control the segments on this 7-segment display?
• What does "far right tilt" actually mean in the context of the EU in 2024?
• What do humans do uniquely, that computers apparently will not be able to?

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Free Cyber Services #protect2024 Secure Our World Shields Up Report A Cyber Issue

Vulnerability Summary for the Week of June 3, 2024

The CISA Vulnerability Bulletin provides a summary of new vulnerabilities that have been recorded by the  National Institute of Standards and Technology  (NIST)  National Vulnerability Database  (NVD) in the past week. NVD is sponsored by CISA. In some cases, the vulnerabilities in the bulletin may not yet have assigned CVSS scores. Please visit NVD for updated vulnerability entries, which include CVSS scores once they are available.

Vulnerabilities are based on the  Common Vulnerabilities and Exposures  (CVE) vulnerability naming standard and are organized according to severity, determined by the  Common Vulnerability Scoring System  (CVSS) standard. The division of high, medium, and low severities correspond to the following scores:

• High : vulnerabilities with a CVSS base score of 7.0–10.0
• Medium : vulnerabilities with a CVSS base score of 4.0–6.9
• Low : vulnerabilities with a CVSS base score of 0.0–3.9

Entries may include additional information provided by organizations and efforts sponsored by CISA. This information may include identifying information, values, definitions, and related links. Patch information is provided when available. Please note that some of the information in the bulletin is compiled from external, open-source reports and is not a direct result of CISA analysis.  

High Vulnerabilities

Back to top

Medium Vulnerabilities

Low Vulnerabilities

Severity Not Yet Assigned