https://www.programiz.com/python-programming/generator
https://www.programiz.com/python-programming/methods/built-in/next
Python Generators
When a lot of data is present and you cannot load it into the memory at once
Or if a function needs to maintain an internal state everytime it's called
yield:
yield is used in conjunction with generators
Generators are functions
They return lazy iterators.
lazy evaluation, or call-by-need, is an evaluation strategy which delays the evaluation of an expression until its value is needed (non-strict evaluation) and which also avoids repeated evaluations (sharing). The sharing can reduce the running time of certain functions by an exponential factor over other non-strict evaluation strategies, such as call-by-name, which repeatedly evaluate the same function, blindly, regardless whether the function can be memoized.
Iterators are objects that you can loop over like a list
But unlike lists, lazy iterators do not store their contents in memory.
iter() can check if an object is iterable for example objects like, string, list, tuple, dict, set, and frozenset are iterable, an int is not iterable
so if you do iter(x) where x is an int it will return an error, x can be string, list, tuple, dict, set, and frozenset, open files in Python are iterable
itr = iter(['apple'])
Python next()
Syntax: next(iterator, default) :: default is optional - this value is returned if the iterator is exhausted (there is no next item)
next(iter([1, 3, 4, 2]))
we can also pass a a generator to next
next(genfun) # as genfun returns an iterator using yield
Python iterator:
if an object can be iterated you can create an iterator in python using iter()
Generally we do not do this, instead we can use a generator instance which returns an iterator
Iterator:
x = ['apple', 'cat', 'boy', 'dog']
itr = iter(x) # converts list to iterator
print(next(itr)) #prints apple
print(next(itr)) #prints cat
print(next(itr)) #prints boy
print(next(itr)) #prints dog
print(next(itr)) #error - because iterator has exhausted
print(next(itr, 'Iter has exhausted')) #prints Iter has exhausted as we supplied a default value to return when iter is exhausted
so if you want something to loop continuously like over batch
first generate an iter for batch
then when the iter is exhausted make another batch
Back to Generators:
Generators are functions;
Generators is different from a normal function in following ways:
- Generators contain one or more yield
- when called it does not start execution but returns an iterator
- already has iter() and next() implemented
- [important] "once a function yields function is paused and control is transferred to the caller"
- [important] "local variables and their states are remembered between successive calls"
- when the function terminates, StopIteration is raised automatically on further calls
so the return point is yield -
the value of yield is returned and generator's state is paused at that point when the generator is called again, it will continue execution from there
some examples of generators:
def generator1():
x = 1
yield x # <-- resumes from here on second call
x+=1
yield x # <-- resumes from here on third call
x+=1
yield x # <-- resumes from here on fourth call - but since generator instance has ended it will return error
# an instance of this generator can be called upto three times as it has 3 yields after that it will return an error - 'StopIteration'
# but you can have multiple instances
# For example gen1 = generator1(), gen2 = generator2()
# gen1 can be called 3 times, gen2 can be called 3 times
# a generator instance's return value is generally obtained using next
gen1 = generator1()
print(next(gen1)) #prints 1
print(next(gen1)) #prints 2
print(next(gen1)) #prints 3
print(next(gen1)) #error - 'StopIteration'
#but you can create another instance and iterate again
gen1 = generator1()
print(next(gen1)) #prints 1
print(next(gen1)) #prints 2
print(next(gen1)) #prints 3
gen1 = generator1()
print(next(gen1)) #prints 1
print(next(gen1)) #prints 2
print(next(gen1)) #prints 3
you can use yield inside a for statement as well
def generator1():
for i in range(5):
yield i
# an instance of this generator can be called upto 5 times
gen1 = generator1()
print(next(gen1)) #prints 0
print(next(gen1)) #prints 1
print(next(gen1)) #prints 2
print(next(gen1)) #prints 3
print(next(gen1)) #prints 4
print(next(gen1)) #error - 'StopIteration'
def generator1():
for i in range(5):
yield i
# an instance of this generator can be called upto 5 times
gen1 = generator1()
for x in gen1:
print(x) # this will automatically close when StopIteration is encountered