Upgrading to Django 3 and Python 3

On this topic I would like to share about my experience of upgrading to Django 3 and Python 3, coming from a deprecated version of Django 1.11 and Python 2.7 environment. In addition, I will discuss how you can setup a virtual environment and use this to isolate the library dependencies in your system.

How To Run Multiple Version Of Python Environment

The first step is to setup a virtual environment that we can use to test and fix the modules that will break while upgrading to Django 3 and Python 3. We can do this by installing Python 3 in user level.

sudo yum install -y wget make libcurl-devel openssl-devel libffi-devel curl gcc 
curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
python /tmp/get-pip.py
pip install virtualenv
wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
tar -xvf Python-3.7.1.tar.xz
mkdir $HOME/.python3
cd Python-3.7.1/
./configure --prefix=$HOME/.python3 --enable-optimizations 
make
make install

After the installation, we can start a new virtual environment with Python 3 interpreter.

virtualenv env --python=$HOME/.python3/bin/python3
source env/bin/activate

Installing Library Dependencies Using Pip Package Manager

Secondly is to install the dependencies in requirements.txt. You can use the pip package manager If you need to create your own requirements.txt file in your system.

pip freeze > requirements.txt

We can also use pip to recursively install all the dependencies, but in my case I end up doing this one library at a time to make sure all dependencies are working properly.

One of the library that I had issue installing was PyCurl, every time I install this library I would encounter an SSL back-end error. Another one is Django 3, this package would complain about the version of SQLite in my system.

Again you can recursively install all the libraries if you prefer to do so by running the command below.

pip install -r requirements.txt

How To Upgrade SQLite3 From Source

In Django 3, the default database is SQLite3.

sudo su root
wget -P /tmp/ https://www.sqlite.org/2019/sqlite-autoconf-3300100.tar.gz
cd /tmp
tar -zxvf sqlite-autoconf-3300100.tar.gz
cd sqlite-autoconf-3300100.tar.gz
./configure
make 
make install
which sqlite3
/usr/bin/sqlite3 --version
/usr/local/bin/sqlite3 --version
mv -v /usr/bin/sqlite3 /usr/bin/sqlite3.7
cp -v /usr/local/bin/sqlite3 /usr/bin/sqlite3
exit

In here, I downloaded a tarball to install a new version of sqlite3 from source. Notice also that we did not remove the old version, we only moved the folder to /usr/bin/sqlite3.7, that way we are able to revert back the changes if needed.

How To Fix: PyCurl SSL Backend Error

“ssl backend (openssl) is different from compile-time ssl backend (none/other)“

In order to fix the SSL back-end error in PyCurl, you will have to specify the proper SSL library that you have in your system.

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl --global-option="--with-openssl"

How To Fix: Django SQLite Version

“django.core.exceptions.improperlyconfigured: sqlite 3.8.3 or later is required”

Install the required version of sqlite3. And to do this follow the instructions in How To Upgrade SQLite3 From Source.

How To Fix: Django Deprecated RequestContext

“TypeError context must be a dict rather than RequestContext”

Passing the instance of RequestContext as parameter to HttpResponse has already been deprecated prior to Django 2. As a solution, replace this with django.shortcuts.render to render the templates and pass a dict value in context parameter instead of a RequestContext.

Django 3

from django.shortcuts import render
from django.views import View

class Index(View):
    template_name = ‘index.html’

   def get(self, request):
     context = {
         ‘form’: form,
     }
     return render(request, 'form_template.html', context={'form': form}, content_type=’text/html’)

How To Fix: Django Deprecated urlresolvers

“ImportError: No module named ‘django.core.urlresolvers’ “

django.core.urlresolvers module has been removed since Django 2.0. Change your imports to use django.urls instead.

Django 3

from django.urls import reverse

How To Fix: Urllib Missing Attributes

Some of the modules in urllib has been moved in a different path. As a result you will need to use instead urllib.parse and urllib.request to import quote, quote_plus, urlencode and urlopen.

“AttributeError: module ‘urllib’ has no attribute ‘quote’ “

“AttributeError: module ‘urllib’ has no attribute ‘urlopen’ “

Python 3

from urllib.parse import quote_plus, quote, urlencode
from urllib.request import urlopen

How To Fix: Python String Data Type

A textual data in Python 3 including a Unicode has now a type str and Binary data is a type byte.

Python 2.7

>>> type(u'Hello World')
<type 'unicode'>
>>> type(r'Hello World')
<type 'str'>
>>> isinstance(u'Hello World', unicode)
True
>>> isinstance(r'Hello World', str)
True

Python 3

>>> type(u'Hello World')
<type 'str'>
>>> type(r'Hello World')
<type 'str'>
>>> isinstance(u'Hello World', str)
True

How To Fix: Python Dict Changed In A Loop

“RuntimeError: dictionary changed size during iteration “

This error happens when you are trying to update the data inside a dict object while in a loop.

Python 2

>>> staff = {'john': 'Dev', 'doe': 'System Admin'}
>>> for i in staff.keys():
...   if i == 'john':
...     staff[i] = 'Senior Dev'
...
>>> staff
{'john': 'Senior Dev', 'doe': 'System Admin'}

Python 3

>>> staff = {'john': 'Dev', 'doe': 'System Admin'}
>>> for i in list(staff):
...   if i == 'john':
...     staff[i] = 'Senior Dev'
...
>>> staff
{'john': 'Senior Dev', 'doe': 'System Admin'}

How To Fix: Python Deprecated iteritems

“‘dict’ object has no attribute iteritems “

Python 2

>>> staff.iteritems()
<dictionary-itemiterator object at 0x7f170226f260>
>>> staff.viewitems()
dict_items([('john', 'Dev'), ('doe', 'System Admin')])

Python 3

>>> staff.iteritems()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'iteritems'
>>> staff.items()
dict_items([('john', 'Dev'), ('doe', 'System Admin')])

Notice that the dict object in Python 3 no longer have dict.iteritems(). Also dict.items() returns an iterator similar to the result of dict.viewitems() in Python 2.7.

How To Fix: Python Deprecated sort

“AttributeError: ‘dict_keys’ objects has no attribute ‘sort'”

Python 2.7

>>> staff_names = staff.keys()
>>> staff_names.sort()
>>> staff_names
['doe', 'john']

Python 3

>>> staff_names = sorted(staff.keys())
>>> staff_names
['doe', 'john']

Notice that the dict_keys object in Python 3 no longer have dict_keys.sort(). To fix this, use another built-in function sorted().

How To Fix: Subprocess Bytes Stream, UnicodeDecodeError

The subprocess module allows you to spawn new processes, connect to input/output/error pipes, and obtain their return codes.

Python 2.7

>>> from subprocess import Popen, STDOUT, PIPE
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT)
>>> type(result.stdout.read())
<type 'str'>

Python 3

In Python 3 the stdin, stdout and stderr has a bytes stream and returns a bytes object by default. Therefore, you cannot make comparison of a regular string into a bytes. However, In subprocess you can set the parameters encoding to unicode or universal_newlines to True to become a text stream for backwards compatibility with Python 2.7.

>>> from subprocess import Popen, STDOUT, PIPE
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT)
>>> type(result.stdout.read())
<class 'bytes'>
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT,encoding='utf-8')
>>> type(result.stdout.read())
<class 'str'>
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT,universal_newlines=True)
>>> type(result.stdout.read())
<class 'str'>

Similarly, if you do not intend to modify the encoding and the universal_newlines parameters you can decode the output instead into a regular text stream. Also when passing additional input parameters make sure that the data is converted into bytes.

>>> from subprocess import Popen, STDOUT, PIPE
>>> import json
>>> staff = json.dumps({'john': 'Dev', 'doe': 'System Admin'})
>>> staff_to_bytes = bytes(staff.encode('utf-8'))
>>> staff_to_bytes
b'{"john": "Dev", "doe": "System Admin"}'
>>> p = Popen(["somescript.sh"], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
>>> (result, error) = p.communicate(input=staff_to_bytes)
>>> if int(p.returncode) != 0:
>>>     result = result.decode('utf-8')
>>>     return result

How To Fix: Text to Bytes

“TypeError: a bytes-like object is required, not ‘str'”

>>> haystack = b'bytes stream'
>>> needle = 'bytes'
>>> needle in haystack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'str'
>>> bytes(needle.encode('utf-8')) in haystack
True

“TypeError: string argument without an encoding”

>>> bytes('to bytes')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string argument without an encoding
>>> bytes('to bytes'.encode('utf-8'))
b'to bytes'

End Of Tutorial

Upgrading to Django 3 and Python 3 can become challenging, but if you try to understand the errors, the solution sometimes is relatively simple and if you are lucky you probably just have to upgrade the dependency libraries or use the updated syntax in Python 3.