The SIO2 project
  1. The SIO2 project
  2. SIO-1583

Refactor unpackmgr to run asynchronously with sioworkers

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: Near Future
    • Fix Version/s: None
    • Component/s: OIOIOI
    • Labels:
      None

      Description

      In Celery 3.2 it won't be possible for a Celery task (in particular, unpackmgr_job) to block and wait for a result coming from another task (that is, call get(); in this case, unpackmgr_job waits for sioworkers to finish). If Celery 3.2 is going to be used, the unpack manager needs to be rewritten.

        Activity

        Hide
        Gerrit Gerrit added a comment -
        Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 1
        https://gerrit.sio2project.mimuw.edu.pl/2226

        SIO-1344 Upgrade to Django 1.6

        ---
        Update requirements

        Upgrading to Django 1.6.
        Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5.
        Django-registration is no longer being supported nor is it compatible
        with Django 1.6. Upgrading to django-registration-redux, a maintained
        fork of django-registration.

        ---
        Remove django-output-validator

        Django-output-validator is not compatible with Django 1.6 and needs
        to be removed.

        ---
        Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin

        The class inheritance graph has changed in Django 1.6 compared to Django 1.5.
        In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits
        from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited
        from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass
        ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases.

        ---
        Upgrade to grappelli 2.5.5

        ---
        Set STATIC_ROOT to '' during testing

        Empty string ('') was the default value of STATIC_ROOT in Django 1.5.
        In django 1.6, this value changed to None. The test settings file
        relies on this value being equal to ''.

        ---
        Set default to False for BooleanFields without defaults

        In Django 1.5, if the programmer hasn't provided a default value
        for a BooleanField, it was set to False. In Django 1.6 it is set
        to None instead. Because of that, every BooleanField's default
        value must be now explicitly set to False.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false

        ---
        Upgrade to the new transaction APIs

        The transaction management model has changed in Django 1.6,
        and the old APIs were deprecated:
        https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model

        ---
        Add a 'load time zone tables when using MySQL' step into README

        Django 1.6 introduced time zone support for day, month, and week_day
        lookups when UZE_TZ is True. This requires time zone definitions
        in the database. Additional step needs to be taken when using MySQL.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups

        ---
        Change uidb36 parameter to uidb64 in templates

        Before 1.6, Django used base 36 encoding of the User primary key in the password
        reset views and URLs. These views have been modified to take a uidb64 parameter
        instead of uidb36. Templates reversing this view needed to be modified accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk

        ---
        Adjust to a WSGIRequest bugfix

        There was a bug in Django 1.5 where a WSGIRequest instance didn't have
        a resolver_match field if not explicitly set from outside. Now it defaults
        to None. Code relying on previous behavior needed to be fixed.
        https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af

        ---
        Explicitly silence some NoReverseMatch exceptions

        In Django 1.5, when a NoReverseMatch exception was raised from a method
        when rendering a template, it was silenced and the method returned
        "(None)". It is no longer being silenced in Django 1.6, so we must
        do it explicitly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous

        ---
        Expecting 400 status code instead of SuspiciousOperation exception in tests

        Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler,
        a handler400 mechanism and default view are used to return
        a HttpResponseBadRequest, when previously this resulted in
        an InternalServerError. Some tests needed to be changed accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features

        ---
        Fix a translation bug in django/contrib/auth by overriding the translation

        ---
        Change has_auto_field to needs_explicit_pk_field in templates

        In a Django 1.6 update the function has_auto_field has been renamed
        to needs_explicit_pk_field. This caused TabularInline and StackedInline
        to render incorrectly. Some templates needed to be changed accordingly.
        https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5

        ---
        Set the RabbitMQ broker as the default one

        There was is a potential deadlock in the unpackmgr_job Celery task,
        and since upgrading to Django 1.6, when using the Django database
        as a broker, the deadlock actually takes place. A temporary "solution"
        to this problem is to force users to use RabbitMQ as the default broker.
        https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583

        ---
        Update the developer documentation

        ---
        Set session serializer to PickleSerializer

        The default session serializer was changed in Django 1.6 to JSONSerializer
        for security reasons. For backwards compatibility it is set to
        PickleSerializer, the previous default. This needs to be changed in
        near future.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json

        Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Show
        Gerrit Gerrit added a comment - Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 1 https://gerrit.sio2project.mimuw.edu.pl/2226 SIO-1344 Upgrade to Django 1.6 --- Update requirements Upgrading to Django 1.6. Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5. Django-registration is no longer being supported nor is it compatible with Django 1.6. Upgrading to django-registration-redux, a maintained fork of django-registration. --- Remove django-output-validator Django-output-validator is not compatible with Django 1.6 and needs to be removed. --- Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin The class inheritance graph has changed in Django 1.6 compared to Django 1.5. In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases. --- Upgrade to grappelli 2.5.5 --- Set STATIC_ROOT to '' during testing Empty string ('') was the default value of STATIC_ROOT in Django 1.5. In django 1.6, this value changed to None. The test settings file relies on this value being equal to ''. --- Set default to False for BooleanFields without defaults In Django 1.5, if the programmer hasn't provided a default value for a BooleanField, it was set to False. In Django 1.6 it is set to None instead. Because of that, every BooleanField's default value must be now explicitly set to False. https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false --- Upgrade to the new transaction APIs The transaction management model has changed in Django 1.6, and the old APIs were deprecated: https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model --- Add a 'load time zone tables when using MySQL' step into README Django 1.6 introduced time zone support for day, month, and week_day lookups when UZE_TZ is True. This requires time zone definitions in the database. Additional step needs to be taken when using MySQL. https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups --- Change uidb36 parameter to uidb64 in templates Before 1.6, Django used base 36 encoding of the User primary key in the password reset views and URLs. These views have been modified to take a uidb64 parameter instead of uidb36. Templates reversing this view needed to be modified accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk --- Adjust to a WSGIRequest bugfix There was a bug in Django 1.5 where a WSGIRequest instance didn't have a resolver_match field if not explicitly set from outside. Now it defaults to None. Code relying on previous behavior needed to be fixed. https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af --- Explicitly silence some NoReverseMatch exceptions In Django 1.5, when a NoReverseMatch exception was raised from a method when rendering a template, it was silenced and the method returned "(None)". It is no longer being silenced in Django 1.6, so we must do it explicitly. https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous --- Expecting 400 status code instead of SuspiciousOperation exception in tests Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler, a handler400 mechanism and default view are used to return a HttpResponseBadRequest, when previously this resulted in an InternalServerError. Some tests needed to be changed accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features --- Fix a translation bug in django/contrib/auth by overriding the translation --- Change has_auto_field to needs_explicit_pk_field in templates In a Django 1.6 update the function has_auto_field has been renamed to needs_explicit_pk_field. This caused TabularInline and StackedInline to render incorrectly. Some templates needed to be changed accordingly. https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5 --- Set the RabbitMQ broker as the default one There was is a potential deadlock in the unpackmgr_job Celery task, and since upgrading to Django 1.6, when using the Django database as a broker, the deadlock actually takes place. A temporary "solution" to this problem is to force users to use RabbitMQ as the default broker. https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583 --- Update the developer documentation --- Set session serializer to PickleSerializer The default session serializer was changed in Django 1.6 to JSONSerializer for security reasons. For backwards compatibility it is set to PickleSerializer, the previous default. This needs to be changed in near future. https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Hide
        Gerrit Gerrit added a comment -
        Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 2
        https://gerrit.sio2project.mimuw.edu.pl/2226

        SIO-1344 Upgrade to Django 1.6

        ---
        Update requirements

        Upgrading to Django 1.6.
        Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5.
        Django-registration is no longer being supported nor is it compatible
        with Django 1.6. Upgrading to django-registration-redux, a maintained
        fork of django-registration.

        ---
        Remove django-output-validator

        Django-output-validator is not compatible with Django 1.6 and needs
        to be removed.

        ---
        Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin

        The class inheritance graph has changed in Django 1.6 compared to Django 1.5.
        In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits
        from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited
        from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass
        ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases.

        ---
        Upgrade to grappelli 2.5.5

        ---
        Set STATIC_ROOT to '' during testing

        Empty string ('') was the default value of STATIC_ROOT in Django 1.5.
        In django 1.6, this value changed to None. The test settings file
        relies on this value being equal to ''.

        ---
        Set default to False for BooleanFields without defaults

        In Django 1.5, if the programmer hasn't provided a default value
        for a BooleanField, it was set to False. In Django 1.6 it is set
        to None instead. Because of that, every BooleanField's default
        value must be now explicitly set to False.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false

        ---
        Upgrade to the new transaction APIs

        The transaction management model has changed in Django 1.6,
        and the old APIs were deprecated:
        https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model

        ---
        Add a 'load time zone tables when using MySQL' step into README

        Django 1.6 introduced time zone support for day, month, and week_day
        lookups when UZE_TZ is True. This requires time zone definitions
        in the database. Additional step needs to be taken when using MySQL.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups

        ---
        Change uidb36 parameter to uidb64 in templates

        Before 1.6, Django used base 36 encoding of the User primary key in the password
        reset views and URLs. These views have been modified to take a uidb64 parameter
        instead of uidb36. Templates reversing this view needed to be modified accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk

        ---
        Adjust to a WSGIRequest bugfix

        There was a bug in Django 1.5 where a WSGIRequest instance didn't have
        a resolver_match field if not explicitly set from outside. Now it defaults
        to None. Code relying on previous behavior needed to be fixed.
        https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af

        ---
        Explicitly silence some NoReverseMatch exceptions

        In Django 1.5, when a NoReverseMatch exception was raised from a method
        when rendering a template, it was silenced and the method returned
        "(None)". It is no longer being silenced in Django 1.6, so we must
        do it explicitly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous

        ---
        Expecting 400 status code instead of SuspiciousOperation exception in tests

        Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler,
        a handler400 mechanism and default view are used to return
        a HttpResponseBadRequest, when previously this resulted in
        an InternalServerError. Some tests needed to be changed accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features

        ---
        Fix a translation bug in django/contrib/auth by overriding the translation

        ---
        Change has_auto_field to needs_explicit_pk_field in templates

        In a Django 1.6 update the function has_auto_field has been renamed
        to needs_explicit_pk_field. This caused TabularInline and StackedInline
        to render incorrectly. Some templates needed to be changed accordingly.
        https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5

        ---
        Set the RabbitMQ broker as the default one

        There was is a potential deadlock in the unpackmgr_job Celery task,
        and since upgrading to Django 1.6, when using the Django database
        as a broker, the deadlock actually takes place. A temporary "solution"
        to this problem is to force users to use RabbitMQ as the default broker.
        https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583

        ---
        Update the developer documentation

        ---
        Set session serializer to PickleSerializer

        The default session serializer was changed in Django 1.6 to JSONSerializer
        for security reasons. For backwards compatibility it is set to
        PickleSerializer, the previous default. This needs to be changed in
        near future.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json

        Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Show
        Gerrit Gerrit added a comment - Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 2 https://gerrit.sio2project.mimuw.edu.pl/2226 SIO-1344 Upgrade to Django 1.6 --- Update requirements Upgrading to Django 1.6. Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5. Django-registration is no longer being supported nor is it compatible with Django 1.6. Upgrading to django-registration-redux, a maintained fork of django-registration. --- Remove django-output-validator Django-output-validator is not compatible with Django 1.6 and needs to be removed. --- Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin The class inheritance graph has changed in Django 1.6 compared to Django 1.5. In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases. --- Upgrade to grappelli 2.5.5 --- Set STATIC_ROOT to '' during testing Empty string ('') was the default value of STATIC_ROOT in Django 1.5. In django 1.6, this value changed to None. The test settings file relies on this value being equal to ''. --- Set default to False for BooleanFields without defaults In Django 1.5, if the programmer hasn't provided a default value for a BooleanField, it was set to False. In Django 1.6 it is set to None instead. Because of that, every BooleanField's default value must be now explicitly set to False. https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false --- Upgrade to the new transaction APIs The transaction management model has changed in Django 1.6, and the old APIs were deprecated: https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model --- Add a 'load time zone tables when using MySQL' step into README Django 1.6 introduced time zone support for day, month, and week_day lookups when UZE_TZ is True. This requires time zone definitions in the database. Additional step needs to be taken when using MySQL. https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups --- Change uidb36 parameter to uidb64 in templates Before 1.6, Django used base 36 encoding of the User primary key in the password reset views and URLs. These views have been modified to take a uidb64 parameter instead of uidb36. Templates reversing this view needed to be modified accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk --- Adjust to a WSGIRequest bugfix There was a bug in Django 1.5 where a WSGIRequest instance didn't have a resolver_match field if not explicitly set from outside. Now it defaults to None. Code relying on previous behavior needed to be fixed. https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af --- Explicitly silence some NoReverseMatch exceptions In Django 1.5, when a NoReverseMatch exception was raised from a method when rendering a template, it was silenced and the method returned "(None)". It is no longer being silenced in Django 1.6, so we must do it explicitly. https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous --- Expecting 400 status code instead of SuspiciousOperation exception in tests Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler, a handler400 mechanism and default view are used to return a HttpResponseBadRequest, when previously this resulted in an InternalServerError. Some tests needed to be changed accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features --- Fix a translation bug in django/contrib/auth by overriding the translation --- Change has_auto_field to needs_explicit_pk_field in templates In a Django 1.6 update the function has_auto_field has been renamed to needs_explicit_pk_field. This caused TabularInline and StackedInline to render incorrectly. Some templates needed to be changed accordingly. https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5 --- Set the RabbitMQ broker as the default one There was is a potential deadlock in the unpackmgr_job Celery task, and since upgrading to Django 1.6, when using the Django database as a broker, the deadlock actually takes place. A temporary "solution" to this problem is to force users to use RabbitMQ as the default broker. https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583 --- Update the developer documentation --- Set session serializer to PickleSerializer The default session serializer was changed in Django 1.6 to JSONSerializer for security reasons. For backwards compatibility it is set to PickleSerializer, the previous default. This needs to be changed in near future. https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Hide
        Gerrit Gerrit added a comment -
        Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 3
        https://gerrit.sio2project.mimuw.edu.pl/2226

        SIO-1344 Upgrade to Django 1.6

        ---
        Update requirements

        Upgrading to Django 1.6.
        Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5.
        Django-registration is no longer being supported nor is it compatible
        with Django 1.6. Upgrading to django-registration-redux, a maintained
        fork of django-registration.

        ---
        Remove django-output-validator

        Django-output-validator is not compatible with Django 1.6 and needs
        to be removed.

        ---
        Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin

        The class inheritance graph has changed in Django 1.6 compared to Django 1.5.
        In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits
        from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited
        from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass
        ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases.

        ---
        Upgrade to grappelli 2.5.5

        ---
        Set STATIC_ROOT to '' during testing

        Empty string ('') was the default value of STATIC_ROOT in Django 1.5.
        In django 1.6, this value changed to None. The test settings file
        relies on this value being equal to ''.

        ---
        Set default to False for BooleanFields without defaults

        In Django 1.5, if the programmer hasn't provided a default value
        for a BooleanField, it was set to False. In Django 1.6 it is set
        to None instead. Because of that, every BooleanField's default
        value must be now explicitly set to False.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false

        ---
        Upgrade to the new transaction APIs

        The transaction management model has changed in Django 1.6,
        and the old APIs were deprecated:
        https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model

        ---
        Add a 'load time zone tables when using MySQL' step into README

        Django 1.6 introduced time zone support for day, month, and week_day
        lookups when UZE_TZ is True. This requires time zone definitions
        in the database. Additional step needs to be taken when using MySQL.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups

        ---
        Change uidb36 parameter to uidb64 in templates

        Before 1.6, Django used base 36 encoding of the User primary key in the password
        reset views and URLs. These views have been modified to take a uidb64 parameter
        instead of uidb36. Templates reversing this view needed to be modified accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk

        ---
        Adjust to a WSGIRequest bugfix

        There was a bug in Django 1.5 where a WSGIRequest instance didn't have
        a resolver_match field if not explicitly set from outside. Now it defaults
        to None. Code relying on previous behavior needed to be fixed.
        https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af

        ---
        Explicitly silence some NoReverseMatch exceptions

        In Django 1.5, when a NoReverseMatch exception was raised from a method
        when rendering a template, it was silenced and the method returned
        "(None)". It is no longer being silenced in Django 1.6, so we must
        do it explicitly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous

        ---
        Expecting 400 status code instead of SuspiciousOperation exception in tests

        Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler,
        a handler400 mechanism and default view are used to return
        a HttpResponseBadRequest, when previously this resulted in
        an InternalServerError. Some tests needed to be changed accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features

        ---
        Fix a translation bug in django/contrib/auth by overriding the translation

        ---
        Change has_auto_field to needs_explicit_pk_field in templates

        In a Django 1.6 update the function has_auto_field has been renamed
        to needs_explicit_pk_field. This caused TabularInline and StackedInline
        to render incorrectly. Some templates needed to be changed accordingly.
        https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5

        ---
        Set the RabbitMQ broker as the default one

        There was is a potential deadlock in the unpackmgr_job Celery task,
        and since upgrading to Django 1.6, when using the Django database
        as a broker, the deadlock actually takes place. A temporary "solution"
        to this problem is to force users to use RabbitMQ as the default broker.
        https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583

        ---
        Update the developer documentation

        ---
        Set session serializer to PickleSerializer

        The default session serializer was changed in Django 1.6 to JSONSerializer
        for security reasons. For backwards compatibility it is set to
        PickleSerializer, the previous default. This needs to be changed in
        near future.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json

        Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Show
        Gerrit Gerrit added a comment - Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 3 https://gerrit.sio2project.mimuw.edu.pl/2226 SIO-1344 Upgrade to Django 1.6 --- Update requirements Upgrading to Django 1.6. Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5. Django-registration is no longer being supported nor is it compatible with Django 1.6. Upgrading to django-registration-redux, a maintained fork of django-registration. --- Remove django-output-validator Django-output-validator is not compatible with Django 1.6 and needs to be removed. --- Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin The class inheritance graph has changed in Django 1.6 compared to Django 1.5. In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases. --- Upgrade to grappelli 2.5.5 --- Set STATIC_ROOT to '' during testing Empty string ('') was the default value of STATIC_ROOT in Django 1.5. In django 1.6, this value changed to None. The test settings file relies on this value being equal to ''. --- Set default to False for BooleanFields without defaults In Django 1.5, if the programmer hasn't provided a default value for a BooleanField, it was set to False. In Django 1.6 it is set to None instead. Because of that, every BooleanField's default value must be now explicitly set to False. https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false --- Upgrade to the new transaction APIs The transaction management model has changed in Django 1.6, and the old APIs were deprecated: https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model --- Add a 'load time zone tables when using MySQL' step into README Django 1.6 introduced time zone support for day, month, and week_day lookups when UZE_TZ is True. This requires time zone definitions in the database. Additional step needs to be taken when using MySQL. https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups --- Change uidb36 parameter to uidb64 in templates Before 1.6, Django used base 36 encoding of the User primary key in the password reset views and URLs. These views have been modified to take a uidb64 parameter instead of uidb36. Templates reversing this view needed to be modified accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk --- Adjust to a WSGIRequest bugfix There was a bug in Django 1.5 where a WSGIRequest instance didn't have a resolver_match field if not explicitly set from outside. Now it defaults to None. Code relying on previous behavior needed to be fixed. https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af --- Explicitly silence some NoReverseMatch exceptions In Django 1.5, when a NoReverseMatch exception was raised from a method when rendering a template, it was silenced and the method returned "(None)". It is no longer being silenced in Django 1.6, so we must do it explicitly. https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous --- Expecting 400 status code instead of SuspiciousOperation exception in tests Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler, a handler400 mechanism and default view are used to return a HttpResponseBadRequest, when previously this resulted in an InternalServerError. Some tests needed to be changed accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features --- Fix a translation bug in django/contrib/auth by overriding the translation --- Change has_auto_field to needs_explicit_pk_field in templates In a Django 1.6 update the function has_auto_field has been renamed to needs_explicit_pk_field. This caused TabularInline and StackedInline to render incorrectly. Some templates needed to be changed accordingly. https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5 --- Set the RabbitMQ broker as the default one There was is a potential deadlock in the unpackmgr_job Celery task, and since upgrading to Django 1.6, when using the Django database as a broker, the deadlock actually takes place. A temporary "solution" to this problem is to force users to use RabbitMQ as the default broker. https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583 --- Update the developer documentation --- Set session serializer to PickleSerializer The default session serializer was changed in Django 1.6 to JSONSerializer for security reasons. For backwards compatibility it is set to PickleSerializer, the previous default. This needs to be changed in near future. https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Hide
        Gerrit Gerrit added a comment -
        Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 4
        https://gerrit.sio2project.mimuw.edu.pl/2226

        SIO-1344 Upgrade to Django 1.6

        ---
        Update requirements

        Upgrading to Django 1.6.
        Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5.
        Django-registration is no longer being supported nor is it compatible
        with Django 1.6. Upgrading to django-registration-redux, a maintained
        fork of django-registration.

        ---
        Remove django-output-validator

        Django-output-validator is not compatible with Django 1.6 and needs
        to be removed.

        ---
        Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin

        The class inheritance graph has changed in Django 1.6 compared to Django 1.5.
        In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits
        from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited
        from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass
        ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases.

        ---
        Upgrade to grappelli 2.5.5

        ---
        Set STATIC_ROOT to '' during testing

        Empty string ('') was the default value of STATIC_ROOT in Django 1.5.
        In django 1.6, this value changed to None. The test settings file
        relies on this value being equal to ''.

        ---
        Set default to False for BooleanFields without defaults

        In Django 1.5, if the programmer hasn't provided a default value
        for a BooleanField, it was set to False. In Django 1.6 it is set
        to None instead. Because of that, every BooleanField's default
        value must be now explicitly set to False.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false

        ---
        Upgrade to the new transaction APIs

        The transaction management model has changed in Django 1.6,
        and the old APIs were deprecated:
        https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model

        ---
        Change uidb36 parameter to uidb64 in templates

        Before 1.6, Django used base 36 encoding of the User primary key in the password
        reset views and URLs. These views have been modified to take a uidb64 parameter
        instead of uidb36. Templates reversing this view needed to be modified accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk

        ---
        Adjust to a WSGIRequest bugfix

        There was a bug in Django 1.5 where a WSGIRequest instance didn't have
        a resolver_match field if not explicitly set from outside. Now it defaults
        to None. Code relying on previous behavior needed to be fixed.
        https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af

        ---
        Explicitly silence some NoReverseMatch exceptions

        In Django 1.5, when a NoReverseMatch exception was raised from a method
        when rendering a template, it was silenced and the method returned
        "(None)". It is no longer being silenced in Django 1.6, so we must
        do it explicitly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous

        ---
        Expecting 400 status code instead of SuspiciousOperation exception in tests

        Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler,
        a handler400 mechanism and default view are used to return
        a HttpResponseBadRequest, when previously this resulted in
        an InternalServerError. Some tests needed to be changed accordingly.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features

        ---
        Fix a translation bug in django/contrib/auth by overriding the translation

        ---
        Change has_auto_field to needs_explicit_pk_field in templates

        In a Django 1.6 update the function has_auto_field has been renamed
        to needs_explicit_pk_field. This caused TabularInline and StackedInline
        to render incorrectly. Some templates needed to be changed accordingly.
        https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5

        ---
        Set the RabbitMQ broker as the default one

        There was is a potential deadlock in the unpackmgr_job Celery task,
        and since upgrading to Django 1.6, when using the Django database
        as a broker, the deadlock actually takes place. A temporary "solution"
        to this problem is to force users to use RabbitMQ as the default broker.
        https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583

        ---
        Update the developer documentation

        ---
        Set session serializer to PickleSerializer

        The default session serializer was changed in Django 1.6 to JSONSerializer
        for security reasons. For backwards compatibility it is set to
        PickleSerializer, the previous default. This needs to be changed in
        near future.
        https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json

        Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Show
        Gerrit Gerrit added a comment - Change I0c94bb7771090a38999ee89e08dda3208570b217, patchset 4 https://gerrit.sio2project.mimuw.edu.pl/2226 SIO-1344 Upgrade to Django 1.6 --- Update requirements Upgrading to Django 1.6. Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5. Django-registration is no longer being supported nor is it compatible with Django 1.6. Upgrading to django-registration-redux, a maintained fork of django-registration. --- Remove django-output-validator Django-output-validator is not compatible with Django 1.6 and needs to be removed. --- Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin The class inheritance graph has changed in Django 1.6 compared to Django 1.5. In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases. --- Upgrade to grappelli 2.5.5 --- Set STATIC_ROOT to '' during testing Empty string ('') was the default value of STATIC_ROOT in Django 1.5. In django 1.6, this value changed to None. The test settings file relies on this value being equal to ''. --- Set default to False for BooleanFields without defaults In Django 1.5, if the programmer hasn't provided a default value for a BooleanField, it was set to False. In Django 1.6 it is set to None instead. Because of that, every BooleanField's default value must be now explicitly set to False. https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false --- Upgrade to the new transaction APIs The transaction management model has changed in Django 1.6, and the old APIs were deprecated: https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model --- Change uidb36 parameter to uidb64 in templates Before 1.6, Django used base 36 encoding of the User primary key in the password reset views and URLs. These views have been modified to take a uidb64 parameter instead of uidb36. Templates reversing this view needed to be modified accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk --- Adjust to a WSGIRequest bugfix There was a bug in Django 1.5 where a WSGIRequest instance didn't have a resolver_match field if not explicitly set from outside. Now it defaults to None. Code relying on previous behavior needed to be fixed. https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af --- Explicitly silence some NoReverseMatch exceptions In Django 1.5, when a NoReverseMatch exception was raised from a method when rendering a template, it was silenced and the method returned "(None)". It is no longer being silenced in Django 1.6, so we must do it explicitly. https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous --- Expecting 400 status code instead of SuspiciousOperation exception in tests Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler, a handler400 mechanism and default view are used to return a HttpResponseBadRequest, when previously this resulted in an InternalServerError. Some tests needed to be changed accordingly. https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features --- Fix a translation bug in django/contrib/auth by overriding the translation --- Change has_auto_field to needs_explicit_pk_field in templates In a Django 1.6 update the function has_auto_field has been renamed to needs_explicit_pk_field. This caused TabularInline and StackedInline to render incorrectly. Some templates needed to be changed accordingly. https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5 --- Set the RabbitMQ broker as the default one There was is a potential deadlock in the unpackmgr_job Celery task, and since upgrading to Django 1.6, when using the Django database as a broker, the deadlock actually takes place. A temporary "solution" to this problem is to force users to use RabbitMQ as the default broker. https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583 --- Update the developer documentation --- Set session serializer to PickleSerializer The default session serializer was changed in Django 1.6 to JSONSerializer for security reasons. For backwards compatibility it is set to PickleSerializer, the previous default. This needs to be changed in near future. https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217
        Kamil Braun made changes -
        Field Original Value New Value
        Fix Version/s Sprint 2/2014 [ 12004 ]
        Kamil Braun made changes -
        Summary Fix a deadlock in the unpackmgr_job Celery task Refactor unpackmgr to run asynchronously with sioworkers
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Fix Version/s Sprint 2/2014 [ 12004 ]
        Affects Version/s Near Future [ 11001 ]
        Affects Version/s Current Version [ 11000 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Description The following situation leads to a deadlock:
        1. A worker is spawned in which the unpackmgr_job Celery task begins to be processed. The worker pool is now exhausted - no more workers can be spawned.
        2. unpackmgr_job calls sio.workers.run asynchronously. The sio.workers task doesn't start, since it waits for a worker to be available from the pool.
        3. unpackmgr_job calls get() on the async task. It blocks, waiting for the result.
        4. ...
        5. No profit.

        Tested: when using Django 1.6 with the Django database as a Celery broker, the deadlock actually happens. RabbitMQ works, but still, this is a major bug and needs to be fixed.
        In Celery 3.2 it won't be possible for a Celery task (in particular, unpackmgr_job) to block and wait for a result coming from another task (that is, call get(); in this case, unpackmgr_job waits for sioworkers to finish). If Celery 3.2 is going to be used, the unpack manager needs to be rewritten.
        Sebastian Jaszczur made changes -
        Status New [ 10000 ] Resolved [ 5 ]
        Assignee Szymon Acedański [ accek ]
        Resolution Fixed [ 1 ]
        Szymon Acedański made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        New New Resolved Resolved
        373d 20h 41m 1 Sebastian Jaszczur 2015-11-19 11:59
        Resolved Resolved Closed Closed
        1223d 4h 41m 1 Szymon Acedański 2019-03-26 16:40

          People

          • Assignee:
            Szymon Acedański
            Reporter:
            Kamil Braun
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: