Python Unbuffered: Boost Your I/O Performance

You must have noticed how quickly you are notified of errors in your program. Well, this doesn’t happen suddenly, but on the back end, a lot of storage processing is going on. Here comes the work of buffer. A buffer, in a lucid way, means a temporary storage location, and as the name suggests, unbuffered implies an absence of a buffer. With the help of this blog, you can understand when exactly we require/don’t need a buffer and the uses of PYTHON UNBUFFERED.

Contents

Python Buffered & Unbuffered

A buffer stores your data at a temporary storage location and slowly shares it with you- byte by byte in software terminology. I/O tasks operate with buffer only. With this, there is no need for multiple function calls. The temporary storage releases data itself.

If you want a quick delivery of data/ knowledge about an internal issue with the system, buffering will take time. So, the unbuffered mode works best. Here, there is no buffer, so that you can obtain a quicker output in simple terms.

Buffering decreases the number of function calls, but if the extra buffer is created, it will sometimes result in a replica of existing functions. So, under certain circumstances, it is not desirable.

PYTHONUNBUFFERED as True

PYTHONUNBUFFERED=TRUE and PYTHONUNBUFFERED=1 mean the same thing only. Log messages aid debugging and give the programmer a gist of the system’s internal work. So, when the above keyword is set to TRUE or 1, it helps to display these messages immediately on the stream. As the keyword’s name suggests, in this case, the log messages won’t be a part of the buffer, so the user will get notified quickly when there is a probability of an error. However, if the system stands unbuffered, a slight disadvantage might exist. The programmer can clear buffer himself too. Most importantly, with its help, you can see logs in real-time.

Enable Python Unbuffered

This means you need to disable buffer or buffering to the output stream. You can opt for -u, a command line switch, or do a flush() function call after every step execution. This is made possible by using wrapper functions too.

class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
  ....
# and similar functions
import sys
sys.stdout = Unbuffered(sys.stdout)

Or you can use flush this way, as the flush is an argument of the print function.

print("flush the output", flush=True)

If you wish to use -u directly, use it while naming Python script.

 python -u script.py

Python unbuffered with Jenkins (Ansible)

While working with Ansible Notebook in Jenkins, if you want to check the output of your statements immediately, you need to use the unbuffered mode. This will not store the contents of the notebook in the buffer. This environment variable will be correct for you. You can enable it in this way:

export PYTHONUNBUFFERED=1
ansible-playbook playbook_name.yaml

Python unbuffered in Docker

A docker file need not have buffered input. Hence, we should use PYTHONUNBUFFERED as some value while working on Docker. This works for stdout and stderr streams only but not for stdin. You can check the output on the terminal itself with the help of unbuffered output.

It is also helpful in case of frequent crashes. When you obtain the output directly on the terminal, no information of files (like log files) is lost in case of collision.

The unbuffered mode in Gunicorn

Gunicorn also uses the same syntax to output log statements rather than a buffer.

PYTHONUNBUFFERED=TRUE

python open unbuffered

If you want to open a file in unbuffered mode, specify 0 as an argument. This is because 0 notifies the system not to create a buffer. For this, binary mode is acceptable.

f = open(‘file.txt’, ‘a’,0)  
# 0 means opening in unbuffered mode

python unbuffered read

To read your file in unbuffered mode, use the abovementioned methods. It can be used -u on the command line, or you can use an unbuffered keyword at the beginning of the statement.

python unbuffered write

Opt for binary encoding in case of writing in Python unbuffered mode. That’s because you open this file in binary mode only if you set the buffer to 0, as with no buffer. Hence, utf-8 encoding is a must.

file1.write("write in unbuffered\n".encode("utf-8"))

FAQs

Does flush() work with buffered data only?

No, this command works for both buffered and unbuffered data.

How do I run Python without a buffer?

You can run Python in unbuffered mode. It will transfer the output to the stream directly. The buffer will not be involved.

What is the difference between buffered and unbuffered in Python?

Both of these are opposites of each other. If you want no delay while printing statements, opt for unbuffered mode. Else, store the statements in a buffer with the buffered mode.

Is Unbuffered memory better?

It is only more beneficial when you want to store a file with large data and multiple sequences of bytes. In this case, you wouldn’t want to create an extra buffer to store the same data. So unbuffered mode will be better.

Conclusion

Through this blog, you must know how and when to use Python’s unbuffered mode. The blog also consisted of different ways to achieve unbuffered mode.