Home / Questions / 2164
Accepted
bgarcial
Asked: 2020-01-07 13:48:13 +0800 CST 2020-01-07 13:48:13 +0800 CST

Amazon EC2 error IOError Input/Output every 6 or 12 Hours

  • 772

When putting my project in Amazon EC2, it happened to me that when interacting with my application at one point the error of

errno 5 input/output error django

enter image description here

I found this reported ticket

It appeared on my console that my server needed to be restarted, and when I did, and restarted my app (gunicorn and nginx) everything magically fixed itself.

This error appeared yesterday and today again. The traceback is this:

Environment:

Request Method: GET
Request URL: http://localhost:8000/accounts/profile/

Django Version: 1.9
Python Version: 3.4.3
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'crispy_forms',
 'django_extensions',
 'storages',
 'userprofile']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/core/handlers/base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/core/handlers/base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/utils/decorators.py" in _wrapper
  67.             return bound_func(*args, **kwargs)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/utils/decorators.py" in bound_func
  63.                 return func.__get__(self, type(self))(*args2, **kwargs2)

File "/home/ubuntu/workspace/neurorehabilitation-system/userprofile/mixins.py" in dispatch
  7.         return super(LoginRequiredMixin, self).dispatch(request, *args, **kwargs)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/home/ubuntu/.virtualenvs/nrb_dev/lib/python3.4/site-packages/django/views/generic/base.py" in get
  157.         context = self.get_context_data(**kwargs)

File "/home/ubuntu/workspace/neurorehabilitation-system/userprofile/views.py" in get_context_data
  50.             print (user.is_physiotherapist)

Exception Type: OSError at /accounts/profile/
Exception Value: [Errno 5] Input/output error

Lastly on line 50 a function of mine is referenced get_context_data()which belongs to a class based view that inherits fromTemplateView

I would dare to think that this has nothing to do with the error, given that in my development environment nothing ever happened.

File "/home/ubuntu/workspace/neurorehabilitation-system/userprofile/views.py" in get_context_data
  50.             print (user.is_physiotherapist)

But initially it does match what the ticket I share describes.

When a bug is declared invalid in Django, what does this mean? If I go by logic, one would think that the bug is totally obsolete or that it has no validity... But I don't know what to think.

Is it possible that there is a problem at the level of the EC2 IaaS infrastructure with Django (which I don't think so since we are talking about giant companies) or would it rather be my application?

python

1 Answers

  1. Best Answer

    I have been detailing my code and I have to say that the cause of this input/output problem was there.

    I've made two rookie mistakes :(

    1. I have a sentence print in production

    In the traceback that I have shown above in my question, in my function get_context_data()I have this:

    File "/home/ubuntu/workspace/neurorehabilitation-system/userprofile/views.py" in get_context_data
  50.             print (user.is_physiotherapist)

    This function get_context_data()is executed every time my user logs into the system, it is possible that every time it print (user.is_physiotherapist)is executed, this process tries to write something to the stdout file of my machine in EC2, although it is something that I do not fully understand or dimension

    I've removed this print statement, committed and pushed to my repository, and on my server retrieved those changes and restarted my gunicorn server and everything worked perfectly.

    1. I have DEBUG=True in production

    For my settings I have the following file hierarchy:

    settings/
    base.py --- SIN DEBUG
    development.py --- DEBUG=True
    testing.py --- DEBUG=True
    production.py --- DEBUG=False
    staging.py --- DEBUG=False

    All files ( development.py, testing.py, production.py, staging.py) inherit frombase.py

    But I don't know how to make my EC2 machine, which is my production server, take base.pyand then run production.pyand overwrite the DEBUGaFalse

    In this post , the value of DEBUG( Trueor False) is set according to the hostname of the machine where my Django application runs.

    In my case on my E2 instance, this is the value of my hostname

    (nrb_dev)ubuntu@ip-172-31-27-249:~$ python
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> a=socket.gethostname()
>>> a
'ip-172-31-27-249'
>>> 
>>> if a != 'ip-172-31-27-249':
...     DEBUG = print ('Caleno juiciocito')
... 
>>> DEBUG
True
>>>

    This means that in my file I base.pywould put:

    import socket

if socket.gethostname() == 'ip-172-31-27-249':
    DEBUG = False
else:
    DEBUG = True

    My concern is that I am "hardcoding" the hostname of my production machine, so I am introducing a point that must be manually modified when the production machine to use is not the same

    Is this a good practice even though it works? I would think not, but any contribution in this regard is welcome.

    Because another option I have is to set the value of the environment variable toDJANGO_SETTINGS_MODULE

    In my case, I am using virtualenvwrapperand have two virtual environments:

    nrb_dev for development where I have a hook in like this $VIRTUAL_ENV/bin/postactivate:

    export DJANGO_SETTINGS_MODULE="neurorehabilitation.settings.development"

    In the same way, in my virtual environment nrb_testI have this hook

    export DJANGO_SETTINGS_MODULE="neurorehabilitation.settings.testing"

    This means that on my production machine I simply change this hook to $VIRTUAL_ENV/bin/postactivatelike this:

    export DJANGO_SETTINGS_MODULE="neurorehabilitation.settings.production"

    And when doing the test (I printed the value of DEBUG temporarily in my file settings/production.py) I see that DEBUG is set to False

    (nrb_dev)ubuntu@ip-172-31-27-249:~/workspace/neurorehabilitation-system$ gunicorn -c neurorehabilitation/gunicorn_config.py neurorehabilitation.wsgi 
[2016-01-08 00:26:15 +0000] [6691] [INFO] Starting gunicorn 19.4.5
[2016-01-08 00:26:15 +0000] [6691] [INFO] Listening at: http://127.0.0.1:8000 (6691)
[2016-01-08 00:26:15 +0000] [6691] [INFO] Using worker: sync
[2016-01-08 00:26:15 +0000] [6694] [INFO] Booting worker with pid: 6694
False
^C[2016-01-08 00:26:19 +0000] [6691] [INFO] Handling signal: int

    I think this second option of modifying the DJANGO_SETTINGS_MODULE environment variable on my production machine is better, but any input or consideration in this regard is welcome.

    Additional notes

    If I wanted to keep track of events in my application, I can explore Django's logging functionality

    I need to explore the monitor service to better manage the gunicorn WSGI server. Some interesting resources in this regard are:

    How to install and manage supervisor in Ubuntu and Debian

    Setting up Django with Nginx, Gunicorn, virtualenv, supervisor and PostgreSQL

    • 1