yield in assignment python

• Python »
• 3.12.3 Documentation »
• The Python Language Reference »
• 6. Expressions
• Theme Auto Light Dark |

6. Expressions ¶

This chapter explains the meaning of the elements of expressions in Python.

Syntax Notes: In this and the following chapters, extended BNF notation will be used to describe syntax, not lexical analysis. When (one alternative of) a syntax rule has the form

and no semantics are given, the semantics of this form of name are the same as for othername .

6.1. Arithmetic conversions ¶

When a description of an arithmetic operator below uses the phrase “the numeric arguments are converted to a common type”, this means that the operator implementation for built-in types works as follows:

If either argument is a complex number, the other is converted to complex;

otherwise, if either argument is a floating point number, the other is converted to floating point;

otherwise, both must be integers and no conversion is necessary.

Some additional rules apply for certain operators (e.g., a string as a left argument to the ‘%’ operator). Extensions must define their own conversion behavior.

6.2. Atoms ¶

Atoms are the most basic elements of expressions. The simplest atoms are identifiers or literals. Forms enclosed in parentheses, brackets or braces are also categorized syntactically as atoms. The syntax for atoms is:

6.2.1. Identifiers (Names) ¶

An identifier occurring as an atom is a name. See section Identifiers and keywords for lexical definition and section Naming and binding for documentation of naming and binding.

When the name is bound to an object, evaluation of the atom yields that object. When a name is not bound, an attempt to evaluate it raises a NameError exception.

Private name mangling: When an identifier that textually occurs in a class definition begins with two or more underscore characters and does not end in two or more underscores, it is considered a private name of that class. Private names are transformed to a longer form before code is generated for them. The transformation inserts the class name, with leading underscores removed and a single underscore inserted, in front of the name. For example, the identifier __spam occurring in a class named Ham will be transformed to _Ham__spam . This transformation is independent of the syntactical context in which the identifier is used. If the transformed name is extremely long (longer than 255 characters), implementation defined truncation may happen. If the class name consists only of underscores, no transformation is done.

6.2.2. Literals ¶

Python supports string and bytes literals and various numeric literals:

Evaluation of a literal yields an object of the given type (string, bytes, integer, floating point number, complex number) with the given value. The value may be approximated in the case of floating point and imaginary (complex) literals. See section Literals for details.

All literals correspond to immutable data types, and hence the object’s identity is less important than its value. Multiple evaluations of literals with the same value (either the same occurrence in the program text or a different occurrence) may obtain the same object or a different object with the same value.

6.2.3. Parenthesized forms ¶

A parenthesized form is an optional expression list enclosed in parentheses:

A parenthesized expression list yields whatever that expression list yields: if the list contains at least one comma, it yields a tuple; otherwise, it yields the single expression that makes up the expression list.

An empty pair of parentheses yields an empty tuple object. Since tuples are immutable, the same rules as for literals apply (i.e., two occurrences of the empty tuple may or may not yield the same object).

Note that tuples are not formed by the parentheses, but rather by use of the comma. The exception is the empty tuple, for which parentheses are required — allowing unparenthesized “nothing” in expressions would cause ambiguities and allow common typos to pass uncaught.

6.2.4. Displays for lists, sets and dictionaries ¶

For constructing a list, a set or a dictionary Python provides special syntax called “displays”, each of them in two flavors:

either the container contents are listed explicitly, or

they are computed via a set of looping and filtering instructions, called a comprehension .

Common syntax elements for comprehensions are:

The comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached.

However, aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don’t “leak” into the enclosing scope.

The iterable expression in the leftmost for clause is evaluated directly in the enclosing scope and then passed as an argument to the implicitly nested scope. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable. For example: [x*y for x in range(10) for y in range(x, x+10)] .

To ensure the comprehension always results in a container of the appropriate type, yield and yield from expressions are prohibited in the implicitly nested scope.

Since Python 3.6, in an async def function, an async for clause may be used to iterate over a asynchronous iterator . A comprehension in an async def function may consist of either a for or async for clause following the leading expression, may contain additional for or async for clauses, and may also use await expressions. If a comprehension contains either async for clauses or await expressions or other asynchronous comprehensions it is called an asynchronous comprehension . An asynchronous comprehension may suspend the execution of the coroutine function in which it appears. See also PEP 530 .

New in version 3.6: Asynchronous comprehensions were introduced.

Changed in version 3.8: yield and yield from prohibited in the implicitly nested scope.

Changed in version 3.11: Asynchronous comprehensions are now allowed inside comprehensions in asynchronous functions. Outer comprehensions implicitly become asynchronous.

6.2.5. List displays ¶

A list display is a possibly empty series of expressions enclosed in square brackets:

A list display yields a new list object, the contents being specified by either a list of expressions or a comprehension. When a comma-separated list of expressions is supplied, its elements are evaluated from left to right and placed into the list object in that order. When a comprehension is supplied, the list is constructed from the elements resulting from the comprehension.

6.2.6. Set displays ¶

A set display is denoted by curly braces and distinguishable from dictionary displays by the lack of colons separating keys and values:

A set display yields a new mutable set object, the contents being specified by either a sequence of expressions or a comprehension. When a comma-separated list of expressions is supplied, its elements are evaluated from left to right and added to the set object. When a comprehension is supplied, the set is constructed from the elements resulting from the comprehension.

An empty set cannot be constructed with {} ; this literal constructs an empty dictionary.

6.2.7. Dictionary displays ¶

A dictionary display is a possibly empty series of dict items (key/value pairs) enclosed in curly braces:

A dictionary display yields a new dictionary object.

If a comma-separated sequence of dict items is given, they are evaluated from left to right to define the entries of the dictionary: each key object is used as a key into the dictionary to store the corresponding value. This means that you can specify the same key multiple times in the dict item list, and the final dictionary’s value for that key will be the last one given.

A double asterisk ** denotes dictionary unpacking . Its operand must be a mapping . Each mapping item is added to the new dictionary. Later values replace values already set by earlier dict items and earlier dictionary unpackings.

New in version 3.5: Unpacking into dictionary displays, originally proposed by PEP 448 .

A dict comprehension, in contrast to list and set comprehensions, needs two expressions separated with a colon followed by the usual “for” and “if” clauses. When the comprehension is run, the resulting key and value elements are inserted in the new dictionary in the order they are produced.

Restrictions on the types of the key values are listed earlier in section The standard type hierarchy . (To summarize, the key type should be hashable , which excludes all mutable objects.) Clashes between duplicate keys are not detected; the last value (textually rightmost in the display) stored for a given key value prevails.

Changed in version 3.8: Prior to Python 3.8, in dict comprehensions, the evaluation order of key and value was not well-defined. In CPython, the value was evaluated before the key. Starting with 3.8, the key is evaluated before the value, as proposed by PEP 572 .

6.2.8. Generator expressions ¶

A generator expression is a compact generator notation in parentheses:

A generator expression yields a new generator object. Its syntax is the same as for comprehensions, except that it is enclosed in parentheses instead of brackets or curly braces.

Variables used in the generator expression are evaluated lazily when the __next__() method is called for the generator object (in the same fashion as normal generators). However, the iterable expression in the leftmost for clause is immediately evaluated, so that an error produced by it will be emitted at the point where the generator expression is defined, rather than at the point where the first value is retrieved. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable. For example: (x*y for x in range(10) for y in range(x, x+10)) .

The parentheses can be omitted on calls with only one argument. See section Calls for details.

To avoid interfering with the expected operation of the generator expression itself, yield and yield from expressions are prohibited in the implicitly defined generator.

If a generator expression contains either async for clauses or await expressions it is called an asynchronous generator expression . An asynchronous generator expression returns a new asynchronous generator object, which is an asynchronous iterator (see Asynchronous Iterators ).

New in version 3.6: Asynchronous generator expressions were introduced.

Changed in version 3.7: Prior to Python 3.7, asynchronous generator expressions could only appear in async def coroutines. Starting with 3.7, any function can use asynchronous generator expressions.

6.2.9. Yield expressions ¶

The yield expression is used when defining a generator function or an asynchronous generator function and thus can only be used in the body of a function definition. Using a yield expression in a function’s body causes that function to be a generator function, and using it in an async def function’s body causes that coroutine function to be an asynchronous generator function. For example:

Due to their side effects on the containing scope, yield expressions are not permitted as part of the implicitly defined scopes used to implement comprehensions and generator expressions.

Changed in version 3.8: Yield expressions prohibited in the implicitly nested scopes used to implement comprehensions and generator expressions.

Generator functions are described below, while asynchronous generator functions are described separately in section Asynchronous generator functions .

When a generator function is called, it returns an iterator known as a generator. That generator then controls the execution of the generator function. The execution starts when one of the generator’s methods is called. At that time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of expression_list to the generator’s caller, or None if expression_list is omitted. By suspended, we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, the internal evaluation stack, and the state of any exception handling. When the execution is resumed by calling one of the generator’s methods, the function can proceed exactly as if the yield expression were just another external call. The value of the yield expression after resuming depends on the method which resumed the execution. If __next__() is used (typically via either a for or the next() builtin) then the result is None . Otherwise, if send() is used, then the result will be the value passed in to that method.

All of this makes generator functions quite similar to coroutines; they yield multiple times, they have more than one entry point and their execution can be suspended. The only difference is that a generator function cannot control where the execution should continue after it yields; the control is always transferred to the generator’s caller.

Yield expressions are allowed anywhere in a try construct. If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator’s close() method will be called, allowing any pending finally clauses to execute.

When yield from <expr> is used, the supplied expression must be an iterable. The values produced by iterating that iterable are passed directly to the caller of the current generator’s methods. Any values passed in with send() and any exceptions passed in with throw() are passed to the underlying iterator if it has the appropriate methods. If this is not the case, then send() will raise AttributeError or TypeError , while throw() will just raise the passed in exception immediately.

When the underlying iterator is complete, the value attribute of the raised StopIteration instance becomes the value of the yield expression. It can be either set explicitly when raising StopIteration , or automatically when the subiterator is a generator (by returning a value from the subgenerator).

Changed in version 3.3: Added yield from <expr> to delegate control flow to a subiterator.

The parentheses may be omitted when the yield expression is the sole expression on the right hand side of an assignment statement.

The proposal for adding generators and the yield statement to Python.

The proposal to enhance the API and syntax of generators, making them usable as simple coroutines.

The proposal to introduce the yield_from syntax, making delegation to subgenerators easy.

The proposal that expanded on PEP 492 by adding generator capabilities to coroutine functions.

6.2.9.1. Generator-iterator methods ¶

This subsection describes the methods of a generator iterator. They can be used to control the execution of a generator function.

Note that calling any of the generator methods below when the generator is already executing raises a ValueError exception.

Starts the execution of a generator function or resumes it at the last executed yield expression. When a generator function is resumed with a __next__() method, the current yield expression always evaluates to None . The execution then continues to the next yield expression, where the generator is suspended again, and the value of the expression_list is returned to __next__() ’s caller. If the generator exits without yielding another value, a StopIteration exception is raised.

This method is normally called implicitly, e.g. by a for loop, or by the built-in next() function.

Resumes the execution and “sends” a value into the generator function. The value argument becomes the result of the current yield expression. The send() method returns the next value yielded by the generator, or raises StopIteration if the generator exits without yielding another value. When send() is called to start the generator, it must be called with None as the argument, because there is no yield expression that could receive the value.

Raises an exception at the point where the generator was paused, and returns the next value yielded by the generator function. If the generator exits without yielding another value, a StopIteration exception is raised. If the generator function does not catch the passed-in exception, or raises a different exception, then that exception propagates to the caller.

In typical use, this is called with a single exception instance similar to the way the raise keyword is used.

For backwards compatibility, however, the second signature is supported, following a convention from older versions of Python. The type argument should be an exception class, and value should be an exception instance. If the value is not provided, the type constructor is called to get an instance. If traceback is provided, it is set on the exception, otherwise any existing __traceback__ attribute stored in value may be cleared.

Changed in version 3.12: The second signature (type[, value[, traceback]]) is deprecated and may be removed in a future version of Python.

Raises a GeneratorExit at the point where the generator function was paused. If the generator function then exits gracefully, is already closed, or raises GeneratorExit (by not catching the exception), close returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit.

6.2.9.2. Examples ¶

Here is a simple example that demonstrates the behavior of generators and generator functions:

For examples using yield from , see PEP 380: Syntax for Delegating to a Subgenerator in “What’s New in Python.”

6.2.9.3. Asynchronous generator functions ¶

The presence of a yield expression in a function or method defined using async def further defines the function as an asynchronous generator function.

When an asynchronous generator function is called, it returns an asynchronous iterator known as an asynchronous generator object. That object then controls the execution of the generator function. An asynchronous generator object is typically used in an async for statement in a coroutine function analogously to how a generator object would be used in a for statement.

Calling one of the asynchronous generator’s methods returns an awaitable object, and the execution starts when this object is awaited on. At that time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of expression_list to the awaiting coroutine. As with a generator, suspension means that all local state is retained, including the current bindings of local variables, the instruction pointer, the internal evaluation stack, and the state of any exception handling. When the execution is resumed by awaiting on the next object returned by the asynchronous generator’s methods, the function can proceed exactly as if the yield expression were just another external call. The value of the yield expression after resuming depends on the method which resumed the execution. If __anext__() is used then the result is None . Otherwise, if asend() is used, then the result will be the value passed in to that method.

If an asynchronous generator happens to exit early by break , the caller task being cancelled, or other exceptions, the generator’s async cleanup code will run and possibly raise exceptions or access context variables in an unexpected context–perhaps after the lifetime of tasks it depends, or during the event loop shutdown when the async-generator garbage collection hook is called. To prevent this, the caller must explicitly close the async generator by calling aclose() method to finalize the generator and ultimately detach it from the event loop.

In an asynchronous generator function, yield expressions are allowed anywhere in a try construct. However, if an asynchronous generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), then a yield expression within a try construct could result in a failure to execute pending finally clauses. In this case, it is the responsibility of the event loop or scheduler running the asynchronous generator to call the asynchronous generator-iterator’s aclose() method and run the resulting coroutine object, thus allowing any pending finally clauses to execute.

To take care of finalization upon event loop termination, an event loop should define a finalizer function which takes an asynchronous generator-iterator and presumably calls aclose() and executes the coroutine. This finalizer may be registered by calling sys.set_asyncgen_hooks() . When first iterated over, an asynchronous generator-iterator will store the registered finalizer to be called upon finalization. For a reference example of a finalizer method see the implementation of asyncio.Loop.shutdown_asyncgens in Lib/asyncio/base_events.py .

The expression yield from <expr> is a syntax error when used in an asynchronous generator function.

6.2.9.4. Asynchronous generator-iterator methods ¶

This subsection describes the methods of an asynchronous generator iterator, which are used to control the execution of a generator function.

Returns an awaitable which when run starts to execute the asynchronous generator or resumes it at the last executed yield expression. When an asynchronous generator function is resumed with an __anext__() method, the current yield expression always evaluates to None in the returned awaitable, which when run will continue to the next yield expression. The value of the expression_list of the yield expression is the value of the StopIteration exception raised by the completing coroutine. If the asynchronous generator exits without yielding another value, the awaitable instead raises a StopAsyncIteration exception, signalling that the asynchronous iteration has completed.

This method is normally called implicitly by a async for loop.

Returns an awaitable which when run resumes the execution of the asynchronous generator. As with the send() method for a generator, this “sends” a value into the asynchronous generator function, and the value argument becomes the result of the current yield expression. The awaitable returned by the asend() method will return the next value yielded by the generator as the value of the raised StopIteration , or raises StopAsyncIteration if the asynchronous generator exits without yielding another value. When asend() is called to start the asynchronous generator, it must be called with None as the argument, because there is no yield expression that could receive the value.

Returns an awaitable that raises an exception of type type at the point where the asynchronous generator was paused, and returns the next value yielded by the generator function as the value of the raised StopIteration exception. If the asynchronous generator exits without yielding another value, a StopAsyncIteration exception is raised by the awaitable. If the generator function does not catch the passed-in exception, or raises a different exception, then when the awaitable is run that exception propagates to the caller of the awaitable.

Returns an awaitable that when run will throw a GeneratorExit into the asynchronous generator function at the point where it was paused. If the asynchronous generator function then exits gracefully, is already closed, or raises GeneratorExit (by not catching the exception), then the returned awaitable will raise a StopIteration exception. Any further awaitables returned by subsequent calls to the asynchronous generator will raise a StopAsyncIteration exception. If the asynchronous generator yields a value, a RuntimeError is raised by the awaitable. If the asynchronous generator raises any other exception, it is propagated to the caller of the awaitable. If the asynchronous generator has already exited due to an exception or normal exit, then further calls to aclose() will return an awaitable that does nothing.

6.3. Primaries ¶

Primaries represent the most tightly bound operations of the language. Their syntax is:

6.3.1. Attribute references ¶

An attribute reference is a primary followed by a period and a name:

The primary must evaluate to an object of a type that supports attribute references, which most objects do. This object is then asked to produce the attribute whose name is the identifier. The type and value produced is determined by the object. Multiple evaluations of the same attribute reference may yield different objects.

This production can be customized by overriding the __getattribute__() method or the __getattr__() method. The __getattribute__() method is called first and either returns a value or raises AttributeError if the attribute is not available.

If an AttributeError is raised and the object has a __getattr__() method, that method is called as a fallback.

6.3.2. Subscriptions ¶

The subscription of an instance of a container class will generally select an element from the container. The subscription of a generic class will generally return a GenericAlias object.

When an object is subscripted, the interpreter will evaluate the primary and the expression list.

The primary must evaluate to an object that supports subscription. An object may support subscription through defining one or both of __getitem__() and __class_getitem__() . When the primary is subscripted, the evaluated result of the expression list will be passed to one of these methods. For more details on when __class_getitem__ is called instead of __getitem__ , see __class_getitem__ versus __getitem__ .

If the expression list contains at least one comma, it will evaluate to a tuple containing the items of the expression list. Otherwise, the expression list will evaluate to the value of the list’s sole member.

For built-in objects, there are two types of objects that support subscription via __getitem__() :

Mappings. If the primary is a mapping , the expression list must evaluate to an object whose value is one of the keys of the mapping, and the subscription selects the value in the mapping that corresponds to that key. An example of a builtin mapping class is the dict class.

Sequences. If the primary is a sequence , the expression list must evaluate to an int or a slice (as discussed in the following section). Examples of builtin sequence classes include the str , list and tuple classes.

The formal syntax makes no special provision for negative indices in sequences . However, built-in sequences all provide a __getitem__() method that interprets negative indices by adding the length of the sequence to the index so that, for example, x[-1] selects the last item of x . The resulting value must be a nonnegative integer less than the number of items in the sequence, and the subscription selects the item whose index is that value (counting from zero). Since the support for negative indices and slicing occurs in the object’s __getitem__() method, subclasses overriding this method will need to explicitly add that support.

A string is a special kind of sequence whose items are characters . A character is not a separate data type but a string of exactly one character.

6.3.3. Slicings ¶

A slicing selects a range of items in a sequence object (e.g., a string, tuple or list). Slicings may be used as expressions or as targets in assignment or del statements. The syntax for a slicing:

There is ambiguity in the formal syntax here: anything that looks like an expression list also looks like a slice list, so any subscription can be interpreted as a slicing. Rather than further complicating the syntax, this is disambiguated by defining that in this case the interpretation as a subscription takes priority over the interpretation as a slicing (this is the case if the slice list contains no proper slice).

The semantics for a slicing are as follows. The primary is indexed (using the same __getitem__() method as normal subscription) with a key that is constructed from the slice list, as follows. If the slice list contains at least one comma, the key is a tuple containing the conversion of the slice items; otherwise, the conversion of the lone slice item is the key. The conversion of a slice item that is an expression is that expression. The conversion of a proper slice is a slice object (see section The standard type hierarchy ) whose start , stop and step attributes are the values of the expressions given as lower bound, upper bound and stride, respectively, substituting None for missing expressions.

6.3.4. Calls ¶

A call calls a callable object (e.g., a function ) with a possibly empty series of arguments :

An optional trailing comma may be present after the positional and keyword arguments but does not affect the semantics.

The primary must evaluate to a callable object (user-defined functions, built-in functions, methods of built-in objects, class objects, methods of class instances, and all objects having a __call__() method are callable). All argument expressions are evaluated before the call is attempted. Please refer to section Function definitions for the syntax of formal parameter lists.

If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots. Next, for each keyword argument, the identifier is used to determine the corresponding slot (if the identifier is the same as the first formal parameter name, the first slot is used, and so on). If the slot is already filled, a TypeError exception is raised. Otherwise, the argument is placed in the slot, filling it (even if the expression is None , it fills the slot). When all arguments have been processed, the slots that are still unfilled are filled with the corresponding default value from the function definition. (Default values are calculated, once, when the function is defined; thus, a mutable object such as a list or dictionary used as default value will be shared by all calls that don’t specify an argument value for the corresponding slot; this should usually be avoided.) If there are any unfilled slots for which no default value is specified, a TypeError exception is raised. Otherwise, the list of filled slots is used as the argument list for the call.

CPython implementation detail: An implementation may provide built-in functions whose positional parameters do not have names, even if they are ‘named’ for the purpose of documentation, and which therefore cannot be supplied by keyword. In CPython, this is the case for functions implemented in C that use PyArg_ParseTuple() to parse their arguments.

If there are more positional arguments than there are formal parameter slots, a TypeError exception is raised, unless a formal parameter using the syntax *identifier is present; in this case, that formal parameter receives a tuple containing the excess positional arguments (or an empty tuple if there were no excess positional arguments).

If any keyword argument does not correspond to a formal parameter name, a TypeError exception is raised, unless a formal parameter using the syntax **identifier is present; in this case, that formal parameter receives a dictionary containing the excess keyword arguments (using the keywords as keys and the argument values as corresponding values), or a (new) empty dictionary if there were no excess keyword arguments.

If the syntax *expression appears in the function call, expression must evaluate to an iterable . Elements from these iterables are treated as if they were additional positional arguments. For the call f(x1, x2, *y, x3, x4) , if y evaluates to a sequence y1 , …, yM , this is equivalent to a call with M+4 positional arguments x1 , x2 , y1 , …, yM , x3 , x4 .

A consequence of this is that although the *expression syntax may appear after explicit keyword arguments, it is processed before the keyword arguments (and any **expression arguments – see below). So:

It is unusual for both keyword arguments and the *expression syntax to be used in the same call, so in practice this confusion does not often arise.

If the syntax **expression appears in the function call, expression must evaluate to a mapping , the contents of which are treated as additional keyword arguments. If a parameter matching a key has already been given a value (by an explicit keyword argument, or from another unpacking), a TypeError exception is raised.

When **expression is used, each key in this mapping must be a string. Each value from the mapping is assigned to the first formal parameter eligible for keyword assignment whose name is equal to the key. A key need not be a Python identifier (e.g. "max-temp °F" is acceptable, although it will not match any formal parameter that could be declared). If there is no match to a formal parameter the key-value pair is collected by the ** parameter, if there is one, or if there is not, a TypeError exception is raised.

Formal parameters using the syntax *identifier or **identifier cannot be used as positional argument slots or as keyword argument names.

Changed in version 3.5: Function calls accept any number of * and ** unpackings, positional arguments may follow iterable unpackings ( * ), and keyword arguments may follow dictionary unpackings ( ** ). Originally proposed by PEP 448 .

A call always returns some value, possibly None , unless it raises an exception. How this value is computed depends on the type of the callable object.

The code block for the function is executed, passing it the argument list. The first thing the code block will do is bind the formal parameters to the arguments; this is described in section Function definitions . When the code block executes a return statement, this specifies the return value of the function call.

The result is up to the interpreter; see Built-in Functions for the descriptions of built-in functions and methods.

A new instance of that class is returned.

The corresponding user-defined function is called, with an argument list that is one longer than the argument list of the call: the instance becomes the first argument.

The class must define a __call__() method; the effect is then the same as if that method was called.

6.4. Await expression ¶

Suspend the execution of coroutine on an awaitable object. Can only be used inside a coroutine function .

New in version 3.5.

6.5. The power operator ¶

The power operator binds more tightly than unary operators on its left; it binds less tightly than unary operators on its right. The syntax is:

Thus, in an unparenthesized sequence of power and unary operators, the operators are evaluated from right to left (this does not constrain the evaluation order for the operands): -1**2 results in -1 .

The power operator has the same semantics as the built-in pow() function, when called with two arguments: it yields its left argument raised to the power of its right argument. The numeric arguments are first converted to a common type, and the result is of that type.

For int operands, the result has the same type as the operands unless the second argument is negative; in that case, all arguments are converted to float and a float result is delivered. For example, 10**2 returns 100 , but 10**-2 returns 0.01 .

Raising 0.0 to a negative power results in a ZeroDivisionError . Raising a negative number to a fractional power results in a complex number. (In earlier versions it raised a ValueError .)

This operation can be customized using the special __pow__() method.

6.6. Unary arithmetic and bitwise operations ¶

All unary arithmetic and bitwise operations have the same priority:

The unary - (minus) operator yields the negation of its numeric argument; the operation can be overridden with the __neg__() special method.

The unary + (plus) operator yields its numeric argument unchanged; the operation can be overridden with the __pos__() special method.

The unary ~ (invert) operator yields the bitwise inversion of its integer argument. The bitwise inversion of x is defined as -(x+1) . It only applies to integral numbers or to custom objects that override the __invert__() special method.

In all three cases, if the argument does not have the proper type, a TypeError exception is raised.

6.7. Binary arithmetic operations ¶

The binary arithmetic operations have the conventional priority levels. Note that some of these operations also apply to certain non-numeric types. Apart from the power operator, there are only two levels, one for multiplicative operators and one for additive operators:

The * (multiplication) operator yields the product of its arguments. The arguments must either both be numbers, or one argument must be an integer and the other must be a sequence. In the former case, the numbers are converted to a common type and then multiplied together. In the latter case, sequence repetition is performed; a negative repetition factor yields an empty sequence.

This operation can be customized using the special __mul__() and __rmul__() methods.

The @ (at) operator is intended to be used for matrix multiplication. No builtin Python types implement this operator.

The / (division) and // (floor division) operators yield the quotient of their arguments. The numeric arguments are first converted to a common type. Division of integers yields a float, while floor division of integers results in an integer; the result is that of mathematical division with the ‘floor’ function applied to the result. Division by zero raises the ZeroDivisionError exception.

This operation can be customized using the special __truediv__() and __floordiv__() methods.

The % (modulo) operator yields the remainder from the division of the first argument by the second. The numeric arguments are first converted to a common type. A zero right argument raises the ZeroDivisionError exception. The arguments may be floating point numbers, e.g., 3.14%0.7 equals 0.34 (since 3.14 equals 4*0.7 + 0.34 .) The modulo operator always yields a result with the same sign as its second operand (or zero); the absolute value of the result is strictly smaller than the absolute value of the second operand [ 1 ] .

The floor division and modulo operators are connected by the following identity: x == (x//y)*y + (x%y) . Floor division and modulo are also connected with the built-in function divmod() : divmod(x, y) == (x//y, x%y) . [ 2 ] .

In addition to performing the modulo operation on numbers, the % operator is also overloaded by string objects to perform old-style string formatting (also known as interpolation). The syntax for string formatting is described in the Python Library Reference, section printf-style String Formatting .

The modulo operation can be customized using the special __mod__() method.

The floor division operator, the modulo operator, and the divmod() function are not defined for complex numbers. Instead, convert to a floating point number using the abs() function if appropriate.

The + (addition) operator yields the sum of its arguments. The arguments must either both be numbers or both be sequences of the same type. In the former case, the numbers are converted to a common type and then added together. In the latter case, the sequences are concatenated.

This operation can be customized using the special __add__() and __radd__() methods.

The - (subtraction) operator yields the difference of its arguments. The numeric arguments are first converted to a common type.

This operation can be customized using the special __sub__() method.

6.8. Shifting operations ¶

The shifting operations have lower priority than the arithmetic operations:

These operators accept integers as arguments. They shift the first argument to the left or right by the number of bits given by the second argument.

This operation can be customized using the special __lshift__() and __rshift__() methods.

A right shift by n bits is defined as floor division by pow(2,n) . A left shift by n bits is defined as multiplication with pow(2,n) .

6.9. Binary bitwise operations ¶

Each of the three bitwise operations has a different priority level:

The & operator yields the bitwise AND of its arguments, which must be integers or one of them must be a custom object overriding __and__() or __rand__() special methods.

The ^ operator yields the bitwise XOR (exclusive OR) of its arguments, which must be integers or one of them must be a custom object overriding __xor__() or __rxor__() special methods.

The | operator yields the bitwise (inclusive) OR of its arguments, which must be integers or one of them must be a custom object overriding __or__() or __ror__() special methods.

6.10. Comparisons ¶

Unlike C, all comparison operations in Python have the same priority, which is lower than that of any arithmetic, shifting or bitwise operation. Also unlike C, expressions like a < b < c have the interpretation that is conventional in mathematics:

Comparisons yield boolean values: True or False . Custom rich comparison methods may return non-boolean values. In this case Python will call bool() on such value in boolean contexts.

Comparisons can be chained arbitrarily, e.g., x < y <= z is equivalent to x < y and y <= z , except that y is evaluated only once (but in both cases z is not evaluated at all when x < y is found to be false).

Formally, if a , b , c , …, y , z are expressions and op1 , op2 , …, opN are comparison operators, then a op1 b op2 c ... y opN z is equivalent to a op1 b and b op2 c and ... y opN z , except that each expression is evaluated at most once.

Note that a op1 b op2 c doesn’t imply any kind of comparison between a and c , so that, e.g., x < y > z is perfectly legal (though perhaps not pretty).

6.10.1. Value comparisons ¶

The operators < , > , == , >= , <= , and != compare the values of two objects. The objects do not need to have the same type.

Chapter Objects, values and types states that objects have a value (in addition to type and identity). The value of an object is a rather abstract notion in Python: For example, there is no canonical access method for an object’s value. Also, there is no requirement that the value of an object should be constructed in a particular way, e.g. comprised of all its data attributes. Comparison operators implement a particular notion of what the value of an object is. One can think of them as defining the value of an object indirectly, by means of their comparison implementation.

Because all types are (direct or indirect) subtypes of object , they inherit the default comparison behavior from object . Types can customize their comparison behavior by implementing rich comparison methods like __lt__() , described in Basic customization .

The default behavior for equality comparison ( == and != ) is based on the identity of the objects. Hence, equality comparison of instances with the same identity results in equality, and equality comparison of instances with different identities results in inequality. A motivation for this default behavior is the desire that all objects should be reflexive (i.e. x is y implies x == y ).

A default order comparison ( < , > , <= , and >= ) is not provided; an attempt raises TypeError . A motivation for this default behavior is the lack of a similar invariant as for equality.

The behavior of the default equality comparison, that instances with different identities are always unequal, may be in contrast to what types will need that have a sensible definition of object value and value-based equality. Such types will need to customize their comparison behavior, and in fact, a number of built-in types have done that.

The following list describes the comparison behavior of the most important built-in types.

Numbers of built-in numeric types ( Numeric Types — int, float, complex ) and of the standard library types fractions.Fraction and decimal.Decimal can be compared within and across their types, with the restriction that complex numbers do not support order comparison. Within the limits of the types involved, they compare mathematically (algorithmically) correct without loss of precision.

The not-a-number values float('NaN') and decimal.Decimal('NaN') are special. Any ordered comparison of a number to a not-a-number value is false. A counter-intuitive implication is that not-a-number values are not equal to themselves. For example, if x = float('NaN') , 3 < x , x < 3 and x == x are all false, while x != x is true. This behavior is compliant with IEEE 754.

None and NotImplemented are singletons. PEP 8 advises that comparisons for singletons should always be done with is or is not , never the equality operators.

Binary sequences (instances of bytes or bytearray ) can be compared within and across their types. They compare lexicographically using the numeric values of their elements.

Strings (instances of str ) compare lexicographically using the numerical Unicode code points (the result of the built-in function ord() ) of their characters. [ 3 ]

Strings and binary sequences cannot be directly compared.

Sequences (instances of tuple , list , or range ) can be compared only within each of their types, with the restriction that ranges do not support order comparison. Equality comparison across these types results in inequality, and ordering comparison across these types raises TypeError .

Sequences compare lexicographically using comparison of corresponding elements. The built-in containers typically assume identical objects are equal to themselves. That lets them bypass equality tests for identical objects to improve performance and to maintain their internal invariants.

Lexicographical comparison between built-in collections works as follows:

For two collections to compare equal, they must be of the same type, have the same length, and each pair of corresponding elements must compare equal (for example, [1,2] == (1,2) is false because the type is not the same).

Collections that support order comparison are ordered the same as their first unequal elements (for example, [1,2,x] <= [1,2,y] has the same value as x <= y ). If a corresponding element does not exist, the shorter collection is ordered first (for example, [1,2] < [1,2,3] is true).

Mappings (instances of dict ) compare equal if and only if they have equal (key, value) pairs. Equality comparison of the keys and values enforces reflexivity.

Order comparisons ( < , > , <= , and >= ) raise TypeError .

Sets (instances of set or frozenset ) can be compared within and across their types.

They define order comparison operators to mean subset and superset tests. Those relations do not define total orderings (for example, the two sets {1,2} and {2,3} are not equal, nor subsets of one another, nor supersets of one another). Accordingly, sets are not appropriate arguments for functions which depend on total ordering (for example, min() , max() , and sorted() produce undefined results given a list of sets as inputs).

Comparison of sets enforces reflexivity of its elements.

Most other built-in types have no comparison methods implemented, so they inherit the default comparison behavior.

User-defined classes that customize their comparison behavior should follow some consistency rules, if possible:

Equality comparison should be reflexive. In other words, identical objects should compare equal:

x is y implies x == y

Comparison should be symmetric. In other words, the following expressions should have the same result:

x == y and y == x x != y and y != x x < y and y > x x <= y and y >= x

Comparison should be transitive. The following (non-exhaustive) examples illustrate that:

x > y and y > z implies x > z x < y and y <= z implies x < z

Inverse comparison should result in the boolean negation. In other words, the following expressions should have the same result:

x == y and not x != y x < y and not x >= y (for total ordering) x > y and not x <= y (for total ordering)

The last two expressions apply to totally ordered collections (e.g. to sequences, but not to sets or mappings). See also the total_ordering() decorator.

The hash() result should be consistent with equality. Objects that are equal should either have the same hash value, or be marked as unhashable.

Python does not enforce these consistency rules. In fact, the not-a-number values are an example for not following these rules.

6.10.2. Membership test operations ¶

The operators in and not in test for membership. x in s evaluates to True if x is a member of s , and False otherwise. x not in s returns the negation of x in s . All built-in sequences and set types support this as well as dictionary, for which in tests whether the dictionary has a given key. For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression x in y is equivalent to any(x is e or x == e for e in y) .

For the string and bytes types, x in y is True if and only if x is a substring of y . An equivalent test is y.find(x) != -1 . Empty strings are always considered to be a substring of any other string, so "" in "abc" will return True .

For user-defined classes which define the __contains__() method, x in y returns True if y.__contains__(x) returns a true value, and False otherwise.

For user-defined classes which do not define __contains__() but do define __iter__() , x in y is True if some value z , for which the expression x is z or x == z is true, is produced while iterating over y . If an exception is raised during the iteration, it is as if in raised that exception.

Lastly, the old-style iteration protocol is tried: if a class defines __getitem__() , x in y is True if and only if there is a non-negative integer index i such that x is y[i] or x == y[i] , and no lower integer index raises the IndexError exception. (If any other exception is raised, it is as if in raised that exception).

The operator not in is defined to have the inverse truth value of in .

6.10.3. Identity comparisons ¶

The operators is and is not test for an object’s identity: x is y is true if and only if x and y are the same object. An Object’s identity is determined using the id() function. x is not y yields the inverse truth value. [ 4 ]

6.11. Boolean operations ¶

In the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: False , None , numeric zero of all types, and empty strings and containers (including strings, tuples, lists, dictionaries, sets and frozensets). All other values are interpreted as true. User-defined objects can customize their truth value by providing a __bool__() method.

The operator not yields True if its argument is false, False otherwise.

The expression x and y first evaluates x ; if x is false, its value is returned; otherwise, y is evaluated and the resulting value is returned.

The expression x or y first evaluates x ; if x is true, its value is returned; otherwise, y is evaluated and the resulting value is returned.

Note that neither and nor or restrict the value and type they return to False and True , but rather return the last evaluated argument. This is sometimes useful, e.g., if s is a string that should be replaced by a default value if it is empty, the expression s or 'foo' yields the desired value. Because not has to create a new value, it returns a boolean value regardless of the type of its argument (for example, not 'foo' produces False rather than '' .)

6.12. Assignment expressions ¶

An assignment expression (sometimes also called a “named expression” or “walrus”) assigns an expression to an identifier , while also returning the value of the expression .

One common use case is when handling matched regular expressions:

Or, when processing a file stream in chunks:

Assignment expressions must be surrounded by parentheses when used as expression statements and when used as sub-expressions in slicing, conditional, lambda, keyword-argument, and comprehension-if expressions and in assert , with , and assignment statements. In all other places where they can be used, parentheses are not required, including in if and while statements.

New in version 3.8: See PEP 572 for more details about assignment expressions.

6.13. Conditional expressions ¶

Conditional expressions (sometimes called a “ternary operator”) have the lowest priority of all Python operations.

The expression x if C else y first evaluates the condition, C rather than x . If C is true, x is evaluated and its value is returned; otherwise, y is evaluated and its value is returned.

See PEP 308 for more details about conditional expressions.

6.14. Lambdas ¶

Lambda expressions (sometimes called lambda forms) are used to create anonymous functions. The expression lambda parameters: expression yields a function object. The unnamed object behaves like a function object defined with:

See section Function definitions for the syntax of parameter lists. Note that functions created with lambda expressions cannot contain statements or annotations.

6.15. Expression lists ¶

Except when part of a list or set display, an expression list containing at least one comma yields a tuple. The length of the tuple is the number of expressions in the list. The expressions are evaluated from left to right.

An asterisk * denotes iterable unpacking . Its operand must be an iterable . The iterable is expanded into a sequence of items, which are included in the new tuple, list, or set, at the site of the unpacking.

New in version 3.5: Iterable unpacking in expression lists, originally proposed by PEP 448 .

A trailing comma is required only to create a one-item tuple, such as 1, ; it is optional in all other cases. A single expression without a trailing comma doesn’t create a tuple, but rather yields the value of that expression. (To create an empty tuple, use an empty pair of parentheses: () .)

6.16. Evaluation order ¶

Python evaluates expressions from left to right. Notice that while evaluating an assignment, the right-hand side is evaluated before the left-hand side.

In the following lines, expressions will be evaluated in the arithmetic order of their suffixes:

6.17. Operator precedence ¶

The following table summarizes the operator precedence in Python, from highest precedence (most binding) to lowest precedence (least binding). Operators in the same box have the same precedence. Unless the syntax is explicitly given, operators are binary. Operators in the same box group left to right (except for exponentiation and conditional expressions, which group from right to left).

Note that comparisons, membership tests, and identity tests, all have the same precedence and have a left-to-right chaining feature as described in the Comparisons section.

Table of Contents

• 6.1. Arithmetic conversions
• 6.2.1. Identifiers (Names)
• 6.2.2. Literals
• 6.2.3. Parenthesized forms
• 6.2.4. Displays for lists, sets and dictionaries
• 6.2.5. List displays
• 6.2.6. Set displays
• 6.2.7. Dictionary displays
• 6.2.8. Generator expressions
• 6.2.9.1. Generator-iterator methods
• 6.2.9.2. Examples
• 6.2.9.3. Asynchronous generator functions
• 6.2.9.4. Asynchronous generator-iterator methods
• 6.3.1. Attribute references
• 6.3.2. Subscriptions
• 6.3.3. Slicings
• 6.3.4. Calls
• 6.4. Await expression
• 6.5. The power operator
• 6.6. Unary arithmetic and bitwise operations
• 6.7. Binary arithmetic operations
• 6.8. Shifting operations
• 6.9. Binary bitwise operations
• 6.10.1. Value comparisons
• 6.10.2. Membership test operations
• 6.10.3. Identity comparisons
• 6.11. Boolean operations
• 6.12. Assignment expressions
• 6.13. Conditional expressions
• 6.14. Lambdas
• 6.15. Expression lists
• 6.16. Evaluation order
• 6.17. Operator precedence

Previous topic

5. The import system

7. Simple statements

• Report a Bug
• Show Source

• Learn Python
• Python Lists
• Python Dictionaries
• Python Strings
• Python Functions
• Learn Pandas & NumPy
• Pandas Tutorials
• Numpy Tutorials
• Learn Data Visualization
• Python Seaborn
• Python Matplotlib

Using Python Generators and yield: A Complete Guide

• February 15, 2023 February 15, 2023

In this tutorial, you’ll learn how to use generators in Python, including how to interpret the yield expression and how to use generator expressions . You’ll learn what the benefits of Python generators are and why they’re often referred to as lazy iteration. Then, you’ll learn how they work and how they’re different from normal functions.

Python generators provide you with the means to create your own iterator functions. These functions allow you to generate complex, memory-intensive operations. These operations will be executed lazily, meaning that you can better manage the memory of your Python program.

By the end of this tutorial, you’ll have learned:

• What Python generators are and how to use the yield expression
• How to use multiple yield keywords in a single generator
• How to use generator expressions to make generators simpler to write
• Some common use cases for Python generators

Table of Contents

Understanding Python Generators

Before diving into what generators are, let’s explore what iterators are. Iterators are objects that can be iterated upon, meaning that they return one action or item at a time . To be considered an iterator, objects need to implement two methods: __iter__() and __next__() . Some common examples of iterators in Python include for loops and list comprehensions .

Generators are a Pythonic implementation of creating iterators, without needing to explicitly implement a class with __iter__() and __next__() methods. Similarly, you don’t need to keep track of the object’s internal state. An important thing to note is that generators iterate over an object lazily, meaning they do not store their contents in memory .

The yield statement’s job is to control the flow of a generator function. The statement goes further to handle the state of the generator function, pausing it until it’s called again, using the next() function.

Creating a Simple Generator

In this section, you’ll learn how to create a basic generator. One of the key syntactical differences between a normal function and a generator function is that the generator function includes a yield statement .

Let’s see how we can create a simple generator function:

Let’s break down what is happening here:

• We define a function, return_n_values() , which takes a single parameter, n
• In the function, we first set the value of num to 0
• We then enter a while loop that evaluates whether the value of num is less than our function argument, n
• While that condition is True , we yield the value of num
• Then, we increment the value of num using the augmented assignment operator

Immediately, there are two very interesting things that happen:

• We use yield instead of return
• A statement follows the yield statement, which isn’t ignored

Let’s see how we can actually use this function:

In the code above, we create a variable values , which is the result of calling our generator function with an argument of 5 passed in. When we print the value of values , a generator object is returned.

So, how do we access the values in our generator object? This is done using the next() function, which calls the internal .__iter__() method. Let’s see how this works in Python:

We can see here that the value of 0 is returned. However, intuitively, we know that the values of 0 through 4 should be returned. Because a Python generator remembers the function’s state, we can call the next() function multiple times . Let’s call it a few more times:

In this case, we’ve yielded all of the values that the while loop will accept. Let’s see what happens when we call the next() function a sixth time:

We can see in the code sample above that when the condition of our while loop is no longer True , Python will raise StopIteration .

In the next section, you’ll learn how to create a Python generator using a for loop.

Creating a Python Generator with a For Loop

In the previous example, you learned how to create and use a simple generator. However, the example above is complicated by the fact that we’re yielding a value and then incrementing it. This can often make generators much more difficult for beginners and novices to understand.

Instead, we can use a for loop, rather than a while loop, for simpler generators . Let’s rewrite our previous generator using a for loop to make the process a little more intuitive:

In the code block above, we used a for loop instead of a while loop. We used the Python range() function to create a range of values from 0 through to the end of the values. This simplifies the generator a little bit, making it more approachable to readers of your code.

Unpacking a Generator with a For Loop

In many cases, you’ll see generators wrapped inside of for loops, in order to exhaust all possible yields. In these cases, the benefit of generators is less about remembering the state (though this is used, of course, internally), and more about using memory wisely.

In the code block above, we used a for loop to loop over each iteration of the generator. This implicitly calls the __next__() method. Note that we’re using the optional end= parameter of the print function, which allows you to overwrite the default newline character .

Creating a Python Generator with Multiple Yield Statements

A very interesting difference between Python functions and generators is that a generator can actually hold more than one yield expressions ! While, theoretically, a function can have more than one return keyword, nothing after the first will execute.

Let’s take a look at an example where we define a generator with more than one yield statement:

In the code block above, our generator has more than one yield statement. When we call the first next() function, it returns only the first yielded value. We can keep calling the next() function until all the yielded values are depleted. At this point, the generator will raise a StopIteration exception.

Understanding the Performance of Python Generators

One of the key things to understand is why you’d want to use a Python generator. Because Python generators evaluate lazily, they use significantly less memory than other objects.

For example, if we created a generator that yielded the first one million numbers, the generator doesn’t actually hold the values. Meanwhile, by using a list comprehension to create a list of the first one million values, the list actually holds the values. Let’s see what this looks like:

In the code block above, we import the sys library which allows us to access the getsizeof() function. We then print the size of both the generator and the list. We can see that the list is over 75,000 times larger.

In the following section, you’ll learn how to simplify creating generators by using generator expressions.

Creating Python Generator Expressions

When you want to create one-off generators, using a function can seem redundant. Similar to list and dictionary comprehensions , Python allows you to create generator expressions. This simplifies the process of creating generators, especially for generators that you only need to use once.

In order to create a generator expression, you wrap the expression in parentheses . Say you wanted to create a generator that yields the numbers from zero through four. Then, you could write (i for i in range(5)) .

In the example above, we used a generator expression to yield values from 0 to 4. We then call the next() function five times to print out the values in the generator.

In the following section, we’ll dive further into the yield statement.

Understanding the Python yield Statement

The Python yield statement can often feel unintuitive to newcomers to generators. What separates the yield statement from the return statement is that rather than ending the process, it simply suspends the current process.

The yield statement will suspend the process and return the yielded value. When the subsequent next() function is called, the process is resumed until the following value is yielded.

What is great about this is that the state of the process is saved. This means that Python will know where to pick up its iteration, allowing it to move forward without a problem.

How to Throw Exceptions in Python Generators Using throw

Python generators have access to a special method, .throw() , which allows them to throw an exception at a specific point of iteration . This can be helpful if you know that an erroneous value may exist in the generator.

Let’s take a look at how we can use the .throw() method in a Python generator:

Let’s break down how we can use the .throw() method to throw an exception in a Python generator:

• We create our generator using a generator expression
• We then use a for loop to loop over each value
• Within the for loop, we use an if statement to check if the value is equal to 3. If it is, we call the .throw() method, which raises an error

In some cases, you may simply want to stop a generator, rather than throwing an exception. This is what you’ll learn in the following section.

How to Stop a Python Generator Using stop

Python allows you to stop iterating over a generator by using the .close() function . This can be very helpful if you’re reading a file using a generator and you only want to read the file until a certain condition is met.

Let’s repeat our previous example, though we’ll stop the generator rather than throwing an exception:

In the code block above we used the .close() method to stop the iteration. While the example above is simple, it can be extended quite a lot. Imagine reading a file using Python – rather than reading the entire file, you may only want to read it until you find a given line.

In this tutorial, you learned how to use generators in Python, including how to interpret the yield expression and how to use generator expressions . You learned what the benefits of Python generators are and why they’re often referred to as lazy iteration. Then, you learned how they work and how they’re different from normal functions.

Additional Resources

To learn more about related topics, check out the resources below:

• Understanding and Using Functions in Python for Data Science
• Python: Return Multiple Values from a Function
• Python Built-In Functions
• Python generators: Official Documentation

Nik Piepenbreier

Nik is the author of datagy.io and has over a decade of experience working with data analytics, data science, and Python. He specializes in teaching developers how to use Python for data science using hands-on tutorials. View Author posts

2 thoughts on “Using Python Generators and yield: A Complete Guide”

Guides like these are highlights in the net. Thanks

Thanks so much, Vik!!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Understanding Python's "yield" Keyword

The yield keyword in Python is used to create generators. A generator is a type of collection that produces items on-the-fly and can only be iterated once. By using generators you can improve your application's performance and consume less memory as compared to normal collections, so it provides a nice boost in performance.

In this article we'll explain how to use the yield keyword in Python and what it does exactly. But first, let's study the difference between a simple list collection and generator, and then we will see how yield can be used to create more complex generators.

• Differences Between a List and Generator

In the following script we will create both a list and a generator and will try to see where they differ. First we'll create a simple list and check its type:

When running this code you should see that the type displayed will be "list".

Now let's iterate over all the items in the squared_list .

The above script will produce following results:

Now let's create a generator and perform the same exact task:

To create a generator, you start exactly as you would with list comprehension, but instead you have to use parentheses instead of square brackets. The above script will display "generator" as the type for squared_gen variable. Now let's iterate over the generator using a for-loop.

The output will be:

The output is the same as that of the list. So what is the difference? One of the main differences lies in the way the list and generators store elements in the memory. Lists store all of the elements in memory at once, whereas generators "create" each item on-the-fly, displays it, and then moves to the next element, discarding the previous element from the memory.

One way to verify this is to check the length of both the list and generator that we just created. The len(squared_list) will return 5 while len(squared_gen) will throw an error that a generator has no length. Also, you can iterate over a list as many times as you want but you can iterate over a generator only once. To iterate again, you must create the generator again.

• Using the Yield Keyword

Now we know the difference between simple collections and generators, let us see how yield can help us define a generator.

In the previous examples, we created a generator implicitly using the list comprehension style. However in more complex scenarios we can instead create functions that return a generator. The yield keyword, unlike the return statement, is used to turn a regular Python function in to a generator. This is used as an alternative to returning an entire list at once. This will be again explained with the help of some simple examples.

Again, let's first see what our function returns if we do not use the yield keyword. Execute the following script:

In this script a function cube_numbers is created that accepts a list of numbers, take their cubes and returns the entire list to the caller. When this function is called, a list of cubes is returned and stored in the cubes variable. You can see from the output that the returned data is in-fact a full list:

Now, instead of returning a list, let's modify the above script so that it returns a generator.

Check out our hands-on, practical guide to learning Git, with best-practices, industry-accepted standards, and included cheat sheet. Stop Googling Git commands and actually learn it!

In the above script, the cube_numbers function returns a generator instead of list of cubed number. It's very simple to create a generator using the yield keyword. Here we do not need the temporary cube_list variable to store cubed number, so even our cube_numbers method is simpler. Also, no return statement is needed, but instead the yield keyword is used to return the cubed number inside of the for-loop.

Now, when cube_number function is called, a generator is returned, which we can verify by running the code:

Even though we called the cube_numbers function, it doesn't actually execute at this point in time, and there are not yet any items stored in memory.

To get the function to execute, and therefore the next item from generator, we use the built-in next method. When you call the next iterator on the generator for the first time, the function is executed until the yield keyword is encountered. Once yield is found the value passed to it is returned to the calling function and the generator function is paused in its current state.

Here is how you get a value from your generator:

The above function will return "1". Now when you call next again on the generator, the cube_numbers function will resume executing from where it stopped previously at yield . The function will continue to execute until it finds yield again. The next function will keep returning cubed value one by one until all the values in the list are iterated.

Once all the values are iterated the next function throws a StopIteration exception. It is important to mention that the cubes generator doesn't store any of these items in memory, rather the cubed values are computed at runtime, returned, and forgotten. The only extra memory used is the state data for the generator itself, which is usually much less than a large list. This makes generators ideal for memory-intensive tasks.

Instead of always having to use the next iterator, you can instead use a "for" loop to iterate over a generators values. When using a "for" loop, behind the scenes the next iterator is called until all the items in the generator are iterated over.

• Optimized Performance

As mentioned earlier, generators are very handy when it comes to memory-intensive tasks since they do not need to store all of the collection items in memory, rather they generate items on the fly and discards it as soon as the iterator moves to the next item.

In the previous examples the performance difference of a simple list and generator was not visible since the list sizes were so small. In this section we'll check out some examples where we can distinguish between the performance of lists and generators.

In the code below we will write a function that returns a list that contains 1 million dummy car objects. We will calculate the memory occupied by the process before and after calling the function (which creates the list).

Take a look at the following code:

Note : You may have to pip install psutil to get this code to work on your machine.

In the machine on which the code was run, following results were obtained (yours may look slightly different):

Before the list was created the process memory was 8 MB , and after the creation of list with 1 million items, the occupied memory jumped to 334 MB . Also, the time it took to create the list was 1.58 seconds.

Now, let's repeat the above process but replace the list with generator. Execute the following script:

Here we have to use the for car in car_list_gen(1000000) loop to ensure that all 1000000 cars are actually generated.

Following results were obtained by executing the above script:

From the output, you can see that by using generators the memory difference is much smaller than before (from 8 MB to 40 MB ) since the generators do not store the items in memory. Furthermore, the time taken to call the generator function was a bit faster as well at 1.37 seconds, which is about 14% faster than the list creation.

Hopefully from this article you have a better understanding of the yield keyword, including how it's used, what it's used for, and why you'd want to use it. Python generators are a great way to improve the performance of your programs and they're very simple to use, but understanding when to use them is the challenge for many novice programmers.

You might also like...

• Hidden Features of Python
• Python Docstrings
• Handling Unix Signals in Python
• The Best Machine Learning Libraries in Python
• Guide to Sending HTTP Requests in Python with urllib3

Improve your dev skills!

Get tutorials, guides, and dev jobs in your inbox.

No spam ever. Unsubscribe at any time. Read our Privacy Policy.

In this article

Building Your First Convolutional Neural Network With Keras

Most resources start with pristine datasets, start at importing and finish at validation. There's much more to know. Why was a class predicted? Where was...

Data Visualization in Python with Matplotlib and Pandas

Data Visualization in Python with Matplotlib and Pandas is a course designed to take absolute beginners to Pandas and Matplotlib, with basic Python knowledge, and...

© 2013- 2024 Stack Abuse. All rights reserved.

Python Enhancement Proposals

• Python »
• PEP Index »

PEP 572 – Assignment Expressions

The importance of real code, exceptional cases, scope of the target, relative precedence of :=, change to evaluation order, differences between assignment expressions and assignment statements, specification changes during implementation, _pydecimal.py, datetime.py, sysconfig.py, simplifying list comprehensions, capturing condition values, changing the scope rules for comprehensions, alternative spellings, special-casing conditional statements, special-casing comprehensions, lowering operator precedence, allowing commas to the right, always requiring parentheses, why not just turn existing assignment into an expression, with assignment expressions, why bother with assignment statements, why not use a sublocal scope and prevent namespace pollution, style guide recommendations, acknowledgements, a numeric example, appendix b: rough code translations for comprehensions, appendix c: no changes to scope semantics.

This is a proposal for creating a way to assign to variables within an expression using the notation NAME := expr .

As part of this change, there is also an update to dictionary comprehension evaluation order to ensure key expressions are executed before value expressions (allowing the key to be bound to a name and then re-used as part of calculating the corresponding value).

During discussion of this PEP, the operator became informally known as “the walrus operator”. The construct’s formal name is “Assignment Expressions” (as per the PEP title), but they may also be referred to as “Named Expressions” (e.g. the CPython reference implementation uses that name internally).

Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts.

Additionally, naming sub-parts of a large expression can assist an interactive debugger, providing useful display hooks and partial results. Without a way to capture sub-expressions inline, this would require refactoring of the original code; with assignment expressions, this merely requires the insertion of a few name := markers. Removing the need to refactor reduces the likelihood that the code be inadvertently changed as part of debugging (a common cause of Heisenbugs), and is easier to dictate to another programmer.

During the development of this PEP many people (supporters and critics both) have had a tendency to focus on toy examples on the one hand, and on overly complex examples on the other.

The danger of toy examples is twofold: they are often too abstract to make anyone go “ooh, that’s compelling”, and they are easily refuted with “I would never write it that way anyway”.

The danger of overly complex examples is that they provide a convenient strawman for critics of the proposal to shoot down (“that’s obfuscated”).

Yet there is some use for both extremely simple and extremely complex examples: they are helpful to clarify the intended semantics. Therefore, there will be some of each below.

However, in order to be compelling , examples should be rooted in real code, i.e. code that was written without any thought of this PEP, as part of a useful application, however large or small. Tim Peters has been extremely helpful by going over his own personal code repository and picking examples of code he had written that (in his view) would have been clearer if rewritten with (sparing) use of assignment expressions. His conclusion: the current proposal would have allowed a modest but clear improvement in quite a few bits of code.

Another use of real code is to observe indirectly how much value programmers place on compactness. Guido van Rossum searched through a Dropbox code base and discovered some evidence that programmers value writing fewer lines over shorter lines.

Case in point: Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code, e.g. instead of writing:

they would write:

Another example illustrates that programmers sometimes do more work to save an extra level of indentation:

This code tries to match pattern2 even if pattern1 has a match (in which case the match on pattern2 is never used). The more efficient rewrite would have been:

Syntax and semantics

In most contexts where arbitrary Python expressions can be used, a named expression can appear. This is of the form NAME := expr where expr is any valid Python expression other than an unparenthesized tuple, and NAME is an identifier.

The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:

There are a few places where assignment expressions are not allowed, in order to avoid ambiguities or user confusion:

This rule is included to simplify the choice for the user between an assignment statement and an assignment expression – there is no syntactic position where both are valid.

Again, this rule is included to avoid two visually similar ways of saying the same thing.

This rule is included to disallow excessively confusing code, and because parsing keyword arguments is complex enough already.

This rule is included to discourage side effects in a position whose exact semantics are already confusing to many users (cf. the common style recommendation against mutable default values), and also to echo the similar prohibition in calls (the previous bullet).

The reasoning here is similar to the two previous cases; this ungrouped assortment of symbols and operators composed of : and = is hard to read correctly.

This allows lambda to always bind less tightly than := ; having a name binding at the top level inside a lambda function is unlikely to be of value, as there is no way to make use of it. In cases where the name will be used more than once, the expression is likely to need parenthesizing anyway, so this prohibition will rarely affect code.

This shows that what looks like an assignment operator in an f-string is not always an assignment operator. The f-string parser uses : to indicate formatting options. To preserve backwards compatibility, assignment operator usage inside of f-strings must be parenthesized. As noted above, this usage of the assignment operator is not recommended.

An assignment expression does not introduce a new scope. In most cases the scope in which the target will be bound is self-explanatory: it is the current scope. If this scope contains a nonlocal or global declaration for the target, the assignment expression honors that. A lambda (being an explicit, if anonymous, function definition) counts as a scope for this purpose.

There is one special case: an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as “comprehensions”) binds the target in the containing scope, honoring a nonlocal or global declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope.

The motivation for this special case is twofold. First, it allows us to conveniently capture a “witness” for an any() expression, or a counterexample for all() , for example:

Second, it allows a compact way of updating mutable state from a comprehension, for example:

However, an assignment expression target name cannot be the same as a for -target name appearing in any comprehension containing the assignment expression. The latter names are local to the comprehension in which they appear, so it would be contradictory for a contained use of the same name to refer to the scope containing the outermost comprehension instead.

For example, [i := i+1 for i in range(5)] is invalid: the for i part establishes that i is local to the comprehension, but the i := part insists that i is not local to the comprehension. The same reason makes these examples invalid too:

While it’s technically possible to assign consistent semantics to these cases, it’s difficult to determine whether those semantics actually make sense in the absence of real use cases. Accordingly, the reference implementation [1] will ensure that such cases raise SyntaxError , rather than executing with implementation defined behaviour.

This restriction applies even if the assignment expression is never executed:

For the comprehension body (the part before the first “for” keyword) and the filter expression (the part after “if” and before any nested “for”), this restriction applies solely to target names that are also used as iteration variables in the comprehension. Lambda expressions appearing in these positions introduce a new explicit function scope, and hence may use assignment expressions with no additional restrictions.

Due to design constraints in the reference implementation (the symbol table analyser cannot easily detect when names are re-used between the leftmost comprehension iterable expression and the rest of the comprehension), named expressions are disallowed entirely as part of comprehension iterable expressions (the part after each “in”, and before any subsequent “if” or “for” keyword):

A further exception applies when an assignment expression occurs in a comprehension whose containing scope is a class scope. If the rules above were to result in the target being assigned in that class’s scope, the assignment expression is expressly invalid. This case also raises SyntaxError :

(The reason for the latter exception is the implicit function scope created for comprehensions – there is currently no runtime mechanism for a function to refer to a variable in the containing class scope, and we do not want to add such a mechanism. If this issue ever gets resolved this special case may be removed from the specification of assignment expressions. Note that the problem already exists for using a variable defined in the class scope from a comprehension.)

See Appendix B for some examples of how the rules for targets in comprehensions translate to equivalent code.

The := operator groups more tightly than a comma in all syntactic positions where it is legal, but less tightly than all other operators, including or , and , not , and conditional expressions ( A if C else B ). As follows from section “Exceptional cases” above, it is never allowed at the same level as = . In case a different grouping is desired, parentheses should be used.

The := operator may be used directly in a positional function call argument; however it is invalid directly in a keyword argument.

Some examples to clarify what’s technically valid or invalid:

Most of the “valid” examples above are not recommended, since human readers of Python source code who are quickly glancing at some code may miss the distinction. But simple cases are not objectionable:

This PEP recommends always putting spaces around := , similar to PEP 8 ’s recommendation for = when used for assignment, whereas the latter disallows spaces around = used for keyword arguments.)

In order to have precisely defined semantics, the proposal requires evaluation order to be well-defined. This is technically not a new requirement, as function calls may already have side effects. Python already has a rule that subexpressions are generally evaluated from left to right. However, assignment expressions make these side effects more visible, and we propose a single change to the current evaluation order:

• In a dict comprehension {X: Y for ...} , Y is currently evaluated before X . We propose to change this so that X is evaluated before Y . (In a dict display like {X: Y} this is already the case, and also in dict((X, Y) for ...) which should clearly be equivalent to the dict comprehension.)

Most importantly, since := is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions.

Conversely, assignment expressions don’t support the advanced features found in assignment statements:

• Multiple targets are not directly supported: x = y = z = 0 # Equivalent: (z := (y := (x := 0)))
• Single assignment targets other than a single NAME are not supported: # No equivalent a [ i ] = x self . rest = []
• Priority around commas is different: x = 1 , 2 # Sets x to (1, 2) ( x := 1 , 2 ) # Sets x to 1
• Iterable packing and unpacking (both regular or extended forms) are not supported: # Equivalent needs extra parentheses loc = x , y # Use (loc := (x, y)) info = name , phone , * rest # Use (info := (name, phone, *rest)) # No equivalent px , py , pz = position name , phone , email , * other_info = contact
• Inline type annotations are not supported: # Closest equivalent is "p: Optional[int]" as a separate declaration p : Optional [ int ] = None
• Augmented assignment is not supported: total += tax # Equivalent: (total := total + tax)

The following changes have been made based on implementation experience and additional review after the PEP was first accepted and before Python 3.8 was released:

• for consistency with other similar exceptions, and to avoid locking in an exception name that is not necessarily going to improve clarity for end users, the originally proposed TargetScopeError subclass of SyntaxError was dropped in favour of just raising SyntaxError directly. [3]
• due to a limitation in CPython’s symbol table analysis process, the reference implementation raises SyntaxError for all uses of named expressions inside comprehension iterable expressions, rather than only raising them when the named expression target conflicts with one of the iteration variables in the comprehension. This could be revisited given sufficiently compelling examples, but the extra complexity needed to implement the more selective restriction doesn’t seem worthwhile for purely hypothetical use cases.

Examples from the Python standard library

env_base is only used on these lines, putting its assignment on the if moves it as the “header” of the block.

• Current: env_base = os . environ . get ( "PYTHONUSERBASE" , None ) if env_base : return env_base
• Improved: if env_base := os . environ . get ( "PYTHONUSERBASE" , None ): return env_base

Avoid nested if and remove one indentation level.

• Current: if self . _is_special : ans = self . _check_nans ( context = context ) if ans : return ans
• Improved: if self . _is_special and ( ans := self . _check_nans ( context = context )): return ans

Code looks more regular and avoid multiple nested if. (See Appendix A for the origin of this example.)

• Current: reductor = dispatch_table . get ( cls ) if reductor : rv = reductor ( x ) else : reductor = getattr ( x , "__reduce_ex__" , None ) if reductor : rv = reductor ( 4 ) else : reductor = getattr ( x , "__reduce__" , None ) if reductor : rv = reductor () else : raise Error ( "un(deep)copyable object of type %s " % cls )
• Improved: if reductor := dispatch_table . get ( cls ): rv = reductor ( x ) elif reductor := getattr ( x , "__reduce_ex__" , None ): rv = reductor ( 4 ) elif reductor := getattr ( x , "__reduce__" , None ): rv = reductor () else : raise Error ( "un(deep)copyable object of type %s " % cls )

tz is only used for s += tz , moving its assignment inside the if helps to show its scope.

• Current: s = _format_time ( self . _hour , self . _minute , self . _second , self . _microsecond , timespec ) tz = self . _tzstr () if tz : s += tz return s
• Improved: s = _format_time ( self . _hour , self . _minute , self . _second , self . _microsecond , timespec ) if tz := self . _tzstr (): s += tz return s

Calling fp.readline() in the while condition and calling .match() on the if lines make the code more compact without making it harder to understand.

• Current: while True : line = fp . readline () if not line : break m = define_rx . match ( line ) if m : n , v = m . group ( 1 , 2 ) try : v = int ( v ) except ValueError : pass vars [ n ] = v else : m = undef_rx . match ( line ) if m : vars [ m . group ( 1 )] = 0
• Improved: while line := fp . readline (): if m := define_rx . match ( line ): n , v = m . group ( 1 , 2 ) try : v = int ( v ) except ValueError : pass vars [ n ] = v elif m := undef_rx . match ( line ): vars [ m . group ( 1 )] = 0

A list comprehension can map and filter efficiently by capturing the condition:

Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:

Note that in both cases the variable y is bound in the containing scope (i.e. at the same level as results or stuff ).

Assignment expressions can be used to good effect in the header of an if or while statement:

Particularly with the while loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value.

An example from the low-level UNIX world:

Rejected alternative proposals

Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above.

A previous version of this PEP proposed subtle changes to the scope rules for comprehensions, to make them more usable in class scope and to unify the scope of the “outermost iterable” and the rest of the comprehension. However, this part of the proposal would have caused backwards incompatibilities, and has been withdrawn so the PEP can focus on assignment expressions.

Broadly the same semantics as the current proposal, but spelled differently.

Since EXPR as NAME already has meaning in import , except and with statements (with different semantics), this would create unnecessary confusion or require special-casing (e.g. to forbid assignment within the headers of these statements).

(Note that with EXPR as VAR does not simply assign the value of EXPR to VAR – it calls EXPR.__enter__() and assigns the result of that to VAR .)

Additional reasons to prefer := over this spelling include:

• In if f(x) as y the assignment target doesn’t jump out at you – it just reads like if f x blah blah and it is too similar visually to if f(x) and y .
• import foo as bar
• except Exc as var
• with ctxmgr() as var

To the contrary, the assignment expression does not belong to the if or while that starts the line, and we intentionally allow assignment expressions in other contexts as well.

• NAME = EXPR
• if NAME := EXPR

reinforces the visual recognition of assignment expressions.

This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow y <- f(x) is not possible in Python, as it would be interpreted as less-than and unary minus.) This syntax has a slight advantage over ‘as’ in that it does not conflict with with , except and import , but otherwise is equivalent. But it is entirely unrelated to Python’s other use of -> (function return type annotations), and compared to := (which dates back to Algol-58) it has a much weaker tradition.

This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable’s scope is encoded into its name, making refactoring harder.

Execution order is inverted (the indented body is performed first, followed by the “header”). This requires a new keyword, unless an existing keyword is repurposed (most likely with: ). See PEP 3150 for prior discussion on this subject (with the proposed keyword being given: ).

This syntax has fewer conflicts than as does (conflicting only with the raise Exc from Exc notation), but is otherwise comparable to it. Instead of paralleling with expr as target: (which can be useful but can also be confusing), this has no parallels, but is evocative.

One of the most popular use-cases is if and while statements. Instead of a more general solution, this proposal enhances the syntax of these two statements to add a means of capturing the compared value:

This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return '' when done), and completely useless in more complicated cases (e.g. where the condition is f(x) < 0 and you want to capture the value of f(x) ). It also has no benefit to list comprehensions.

Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction of possible use-cases, even in if / while statements.

Another common use-case is comprehensions (list/set/dict, and genexps). As above, proposals have been made for comprehension-specific solutions.

This brings the subexpression to a location in between the ‘for’ loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, where reads the most cleanly, but also has the greatest potential for conflict (e.g. SQLAlchemy and numpy have where methods, as does tkinter.dnd.Icon in the standard library).

As above, but reusing the with keyword. Doesn’t read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into “longhand” for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why “with NAME = EXPR:” cannot be used as a statement on its own.

As per option 2, but using as rather than an equals sign. Aligns syntactically with other uses of as for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of with inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax.

Regardless of the spelling chosen, this introduces a stark difference between comprehensions and the equivalent unrolled long-hand form of the loop. It is no longer possible to unwrap the loop into statement form without reworking any name bindings. The only keyword that can be repurposed to this task is with , thus giving it sneakily different semantics in a comprehension than in a statement; alternatively, a new keyword is needed, with all the costs therein.

There are two logical precedences for the := operator. Either it should bind as loosely as possible, as does statement-assignment; or it should bind more tightly than comparison operators. Placing its precedence between the comparison and arithmetic operators (to be precise: just lower than bitwise OR) allows most uses inside while and if conditions to be spelled without parentheses, as it is most likely that you wish to capture the value of something, then perform a comparison on it:

Once find() returns -1, the loop terminates. If := binds as loosely as = does, this would capture the result of the comparison (generally either True or False ), which is less useful.

While this behaviour would be convenient in many situations, it is also harder to explain than “the := operator behaves just like the assignment statement”, and as such, the precedence for := has been made as close as possible to that of = (with the exception that it binds tighter than comma).

Some critics have claimed that the assignment expressions should allow unparenthesized tuples on the right, so that these two would be equivalent:

(With the current version of the proposal, the latter would be equivalent to ((point := x), y) .)

However, adopting this stance would logically lead to the conclusion that when used in a function call, assignment expressions also bind less tight than comma, so we’d have the following confusing equivalence:

The less confusing option is to make := bind more tightly than comma.

It’s been proposed to just always require parentheses around an assignment expression. This would resolve many ambiguities, and indeed parentheses will frequently be needed to extract the desired subexpression. But in the following cases the extra parentheses feel redundant:

Frequently Raised Objections

C and its derivatives define the = operator as an expression, rather than a statement as is Python’s way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between if (x == y) and if (x = y) belies their drastically different semantics. Thus this proposal uses := to clarify the distinction.

The two forms have different flexibilities. The := operator can be used inside a larger expression; the = statement can be augmented to += and its friends, can be chained, and can assign to attributes and subscripts.

Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from for loops or other constructs, and can be solved the same way: del the name once it is no longer needed, or prefix it with an underscore.

(The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2] )

As expression assignments can sometimes be used equivalently to statement assignments, the question of which should be preferred will arise. For the benefit of style guides such as PEP 8 , two recommendations are suggested.

• If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent.
• If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead.

The authors wish to thank Alyssa Coghlan and Steven D’Aprano for their considerable contributions to this proposal, and members of the core-mentorship mailing list for assistance with implementation.

Appendix A: Tim Peters’s findings

Here’s a brief essay Tim Peters wrote on the topic.

I dislike “busy” lines of code, and also dislike putting conceptually unrelated logic on a single line. So, for example, instead of:

instead. So I suspected I’d find few places I’d want to use assignment expressions. I didn’t even consider them for lines already stretching halfway across the screen. In other cases, “unrelated” ruled:

is a vast improvement over the briefer:

The original two statements are doing entirely different conceptual things, and slamming them together is conceptually insane.

In other cases, combining related logic made it harder to understand, such as rewriting:

as the briefer:

The while test there is too subtle, crucially relying on strict left-to-right evaluation in a non-short-circuiting or method-chaining context. My brain isn’t wired that way.

But cases like that were rare. Name binding is very frequent, and “sparse is better than dense” does not mean “almost empty is better than sparse”. For example, I have many functions that return None or 0 to communicate “I have nothing useful to return in this case, but since that’s expected often I’m not going to annoy you with an exception”. This is essentially the same as regular expression search functions returning None when there is no match. So there was lots of code of the form:

I find that clearer, and certainly a bit less typing and pattern-matching reading, as:

It’s also nice to trade away a small amount of horizontal whitespace to get another _line_ of surrounding code on screen. I didn’t give much weight to this at first, but it was so very frequent it added up, and I soon enough became annoyed that I couldn’t actually run the briefer code. That surprised me!

There are other cases where assignment expressions really shine. Rather than pick another from my code, Kirill Balunov gave a lovely example from the standard library’s copy() function in copy.py :

The ever-increasing indentation is semantically misleading: the logic is conceptually flat, “the first test that succeeds wins”:

Using easy assignment expressions allows the visual structure of the code to emphasize the conceptual flatness of the logic; ever-increasing indentation obscured it.

A smaller example from my code delighted me, both allowing to put inherently related logic in a single line, and allowing to remove an annoying “artificial” indentation level:

That if is about as long as I want my lines to get, but remains easy to follow.

So, in all, in most lines binding a name, I wouldn’t use assignment expressions, but because that construct is so very frequent, that leaves many places I would. In most of the latter, I found a small win that adds up due to how often it occurs, and in the rest I found a moderate to major win. I’d certainly use it more often than ternary if , but significantly less often than augmented assignment.

I have another example that quite impressed me at the time.

Where all variables are positive integers, and a is at least as large as the n’th root of x, this algorithm returns the floor of the n’th root of x (and roughly doubling the number of accurate bits per iteration):

It’s not obvious why that works, but is no more obvious in the “loop and a half” form. It’s hard to prove correctness without building on the right insight (the “arithmetic mean - geometric mean inequality”), and knowing some non-trivial things about how nested floor functions behave. That is, the challenges are in the math, not really in the coding.

If you do know all that, then the assignment-expression form is easily read as “while the current guess is too large, get a smaller guess”, where the “too large?” test and the new guess share an expensive sub-expression.

To my eyes, the original form is harder to understand:

This appendix attempts to clarify (though not specify) the rules when a target occurs in a comprehension or in a generator expression. For a number of illustrative examples we show the original code, containing a comprehension, and the translation, where the comprehension has been replaced by an equivalent generator function plus some scaffolding.

Since [x for ...] is equivalent to list(x for ...) these examples all use list comprehensions without loss of generality. And since these examples are meant to clarify edge cases of the rules, they aren’t trying to look like real code.

Note: comprehensions are already implemented via synthesizing nested generator functions like those in this appendix. The new part is adding appropriate declarations to establish the intended scope of assignment expression targets (the same scope they resolve to as if the assignment were performed in the block containing the outermost comprehension). For type inference purposes, these illustrative expansions do not imply that assignment expression targets are always Optional (but they do indicate the target binding scope).

Let’s start with a reminder of what code is generated for a generator expression without assignment expression.

• Original code (EXPR usually references VAR): def f (): a = [ EXPR for VAR in ITERABLE ]
• Translation (let’s not worry about name conflicts): def f (): def genexpr ( iterator ): for VAR in iterator : yield EXPR a = list ( genexpr ( iter ( ITERABLE )))

Let’s add a simple assignment expression.

• Original code: def f (): a = [ TARGET := EXPR for VAR in ITERABLE ]
• Translation: def f (): if False : TARGET = None # Dead code to ensure TARGET is a local variable def genexpr ( iterator ): nonlocal TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Let’s add a global TARGET declaration in f() .

• Original code: def f (): global TARGET a = [ TARGET := EXPR for VAR in ITERABLE ]
• Translation: def f (): global TARGET def genexpr ( iterator ): global TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Or instead let’s add a nonlocal TARGET declaration in f() .

• Original code: def g (): TARGET = ... def f (): nonlocal TARGET a = [ TARGET := EXPR for VAR in ITERABLE ]
• Translation: def g (): TARGET = ... def f (): nonlocal TARGET def genexpr ( iterator ): nonlocal TARGET for VAR in iterator : TARGET = EXPR yield TARGET a = list ( genexpr ( iter ( ITERABLE )))

Finally, let’s nest two comprehensions.

• Original code: def f (): a = [[ TARGET := i for i in range ( 3 )] for j in range ( 2 )] # I.e., a = [[0, 1, 2], [0, 1, 2]] print ( TARGET ) # prints 2
• Translation: def f (): if False : TARGET = None def outer_genexpr ( outer_iterator ): nonlocal TARGET def inner_generator ( inner_iterator ): nonlocal TARGET for i in inner_iterator : TARGET = i yield i for j in outer_iterator : yield list ( inner_generator ( range ( 3 ))) a = list ( outer_genexpr ( range ( 2 ))) print ( TARGET )

Because it has been a point of confusion, note that nothing about Python’s scoping semantics is changed. Function-local scopes continue to be resolved at compile time, and to have indefinite temporal extent at run time (“full closures”). Example:

This document has been placed in the public domain.

Source: https://github.com/python/peps/blob/main/peps/pep-0572.rst

Last modified: 2023-10-11 12:05:51 GMT

• Free Python 3 Tutorial
• Control Flow
• Exception Handling
• Python Programs
• Python Projects
• Python Interview Questions
• Python Database
• Data Science With Python
• Machine Learning with Python
• Grouping the Tests in Pytest
• How to Fix “ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()”
• Python Falcon - WSGI vs ASGI
• Python Pyramid - Environment Setup
• Useful Pycharm Keyboard Shortcuts
• How To Print A Variable's Name In Python
• File Execution in Pytest
• Working with Unicode in Python
• Python Pyramid - Static Assets
• Python Dictionary with For Loop
• How to append elements in Python tuple?
• Use For Loop to Iterate Tuple in Python
• Convert Hex To String Without 0X in Python
• How to uninstall PyCharm?
• Set the First Column and Row as Index in Pandas
• Pandas read_csv: low_memory and dtype options
• Python Pyramid - Authentication
• Pandas: Detect Mixed Data Types and Fix it
• Test Execution Results in XML in Pytest

Python Yield Multiple Values

In Python, yield is a keyword that plays a very crucial role in the creation of a generator. It is an efficient way to work with a sequence of values. It is a powerful and memory-efficient way of dealing with large values. Unlike a return statement which terminates the function after returning a value, yield produces and passes a series of values. In this article, we are going to deal with various scenarios where yield can be found a good fit. We will be dealing with some known questions and how can be they solved with the use of the yield keyword.

Below, we are going to explore various examples of Python Yield Multiple Values. We are going to see different problems with yield keywords.

• Simple List Parsing
• Fibonacci Series
• Finding Factors
• Random Number Generator

Python Yield Multiple Values with Simple List Parsing

In this example, the below code defines a generator function `listIter` that yields each element of a given list. In the main function, a list `[1,2,3,4,5]` is defined, and the generator is created by calling `listIter(l)`. The `for` loop iterates through the generator, printing each element of the list on the same line.

Python Yield Multiple Values with Fibonacci Series

In this example, below code defines a generator function `fibo` to generate the Fibonacci series up to the nth term. It uses variables `a` and `b` as the first two elements, and `c` to calculate the next term. The generator yields each term one by one. In the main function, the generator is called with `n=20`, and the `for` loop prints the first 20 terms of the Fibonacci series.

Python Yield Multiple Values with Finding Factors

In this example, below code defines a generator function `factors` to find and yield the factors of a given number `n`. In the main function, with `n=20`, the generator is called using `list(factors(n))`, and the factors of 20 are printed using a `for` loop.

Python Yield Multiple Values with Random Number Generator

In this example, below code utilizes the `random` module to generate a series of random numbers between 0 and 10 using the `randomNum` generator function. In the main function, with `n=5`, the generator is called using `list(randomNum(n))`, and a `for` loop prints five random numbers on a single line.

In Python, yield keyword in used in the creation of a generator. Unlike returns, it do not terminates a method rather it temporarily suspends the method. We can pass more than one value from a method to the main function or another method with the help of yield keyword. In this article we have covered use cases of yield keyword with some working examples along with their explanations.

Please Login to comment...

Similar reads.

• Geeks Premier League 2023
• Geeks Premier League
• Google Releases ‘Prompting Guide’ With Tips For Gemini In Workspace
• Google Cloud Next 24 | Gmail Voice Input, Gemini for Google Chat, Meet ‘Translate for me,’ & More
• 10 Best Viber Alternatives for Better Communication
• 12 Best Database Management Software in 2024
• 30 OOPs Interview Questions and Answers (2024)

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

Python Tutorial

File handling, python modules, python numpy, python pandas, python matplotlib, python scipy, machine learning, python mysql, python mongodb, python reference, module reference, python how to, python examples, python yield keyword.

❮ Python Keywords

Return three values from a function:

Definition and Usage

The yield keyword is used to return a list of values from a function.

Unlike the return keyword which stops further execution of the function, the yield keyword continues to the end of the function.

When you call a function with yield keyword(s), the return value will be a list of values, one for each yield .

Related Pages

Use the return keyword to return only one value, and stop further execution.

Read more about functions in our Python Functions Tutorial .

COLOR PICKER

Report Error

If you want to report an error, or if you want to make a suggestion, do not hesitate to send us an e-mail:

[email protected]

Top Tutorials

Top references, top examples, get certified.