Details
-
Type: Improvement
-
Status: Closed
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: Near Future
-
Fix Version/s: None
-
Component/s: OIOIOI
-
Labels:None
Description
In Celery 3.2 it won't be possible for a Celery task (in particular, unpackmgr_job) to block and wait for a result coming from another task (that is, call get(); in this case, unpackmgr_job waits for sioworkers to finish). If Celery 3.2 is going to be used, the unpack manager needs to be rewritten.
Activity
Kamil Braun
made changes -
Field | Original Value | New Value |
---|---|---|
Fix Version/s | Sprint 2/2014 [ 12004 ] |
Kamil Braun
made changes -
Summary | Fix a deadlock in the unpackmgr_job Celery task | Refactor unpackmgr to run asynchronously with sioworkers |
Issue Type | Bug [ 1 ] | Improvement [ 4 ] |
Fix Version/s | Sprint 2/2014 [ 12004 ] | |
Affects Version/s | Near Future [ 11001 ] | |
Affects Version/s | Current Version [ 11000 ] | |
Priority | Major [ 3 ] | Minor [ 4 ] |
Description |
The following situation leads to a deadlock:
1. A worker is spawned in which the unpackmgr_job Celery task begins to be processed. The worker pool is now exhausted - no more workers can be spawned. 2. unpackmgr_job calls sio.workers.run asynchronously. The sio.workers task doesn't start, since it waits for a worker to be available from the pool. 3. unpackmgr_job calls get() on the async task. It blocks, waiting for the result. 4. ... 5. No profit. Tested: when using Django 1.6 with the Django database as a Celery broker, the deadlock actually happens. RabbitMQ works, but still, this is a major bug and needs to be fixed. |
In Celery 3.2 it won't be possible for a Celery task (in particular, unpackmgr_job) to block and wait for a result coming from another task (that is, call get(); in this case, unpackmgr_job waits for sioworkers to finish). If Celery 3.2 is going to be used, the unpack manager needs to be rewritten. |
Sebastian Jaszczur
made changes -
Status | New [ 10000 ] | Resolved [ 5 ] |
Assignee | Szymon Acedański [ accek ] | |
Resolution | Fixed [ 1 ] |
Szymon Acedański
made changes -
Status | Resolved [ 5 ] | Closed [ 6 ] |
Transition | Time In Source Status | Execution Times | Last Executer | Last Execution Date | |||||
---|---|---|---|---|---|---|---|---|---|
|
373d 20h 41m | 1 | Sebastian Jaszczur | 2015-11-19 11:59 | |||||
|
1223d 4h 41m | 1 | Szymon Acedański | 2019-03-26 16:40 |
https://gerrit.sio2project.mimuw.edu.pl/2226
SIO-1344Upgrade to Django 1.6---
Update requirements
Upgrading to Django 1.6.
Grappelli 2.4.4 is not compatible with Django 1.6. Upgrading to 2.5.5.
Django-registration is no longer being supported nor is it compatible
with Django 1.6. Upgrading to django-registration-redux, a maintained
fork of django-registration.
---
Remove django-output-validator
Django-output-validator is not compatible with Django 1.6 and needs
to be removed.
---
Fix metaclass conflict in oioioi/base/admin.py's ModelAdmin
The class inheritance graph has changed in Django 1.6 compared to Django 1.5.
In particular, class BaseModelAdmin in django.conrib.admin.options.py now inherits
from six.with_metaclass(RenameBaseModelAdminMethods), while previously it inherited
from six.with_metaclass(forms.MediaDefiningClass). As a result, ModelAdmin's metaclass
ModelAdminMeta was no longer a subclass of metaclasses of all ModelAdmin's bases.
---
Upgrade to grappelli 2.5.5
---
Set STATIC_ROOT to '' during testing
Empty string ('') was the default value of STATIC_ROOT in Django 1.5.
In django 1.6, this value changed to None. The test settings file
relies on this value being equal to ''.
---
Set default to False for BooleanFields without defaults
In Django 1.5, if the programmer hasn't provided a default value
for a BooleanField, it was set to False. In Django 1.6 it is set
to None instead. Because of that, every BooleanField's default
value must be now explicitly set to False.
https://docs.djangoproject.com/en/1.6/releases/1.6/#booleanfield-no-longer-defaults-to-false
---
Upgrade to the new transaction APIs
The transaction management model has changed in Django 1.6,
and the old APIs were deprecated:
https://docs.djangoproject.com/en/1.6/releases/1.6/#new-transaction-management-model
---
Add a 'load time zone tables when using MySQL' step into README
Django 1.6 introduced time zone support for day, month, and week_day
lookups when UZE_TZ is True. This requires time zone definitions
in the database. Additional step needs to be taken when using MySQL.
https://docs.djangoproject.com/en/1.6/releases/1.6/#time-zone-aware-day-month-and-week-day-lookups
---
Change uidb36 parameter to uidb64 in templates
Before 1.6, Django used base 36 encoding of the User primary key in the password
reset views and URLs. These views have been modified to take a uidb64 parameter
instead of uidb36. Templates reversing this view needed to be modified accordingly.
https://docs.djangoproject.com/en/1.6/releases/1.6/#django-contrib-auth-password-reset-uses-base-64-encoding-of-user-pk
---
Adjust to a WSGIRequest bugfix
There was a bug in Django 1.5 where a WSGIRequest instance didn't have
a resolver_match field if not explicitly set from outside. Now it defaults
to None. Code relying on previous behavior needed to be fixed.
https://github.com/django/django/commit/48ce167d895b7e2a9d00884a4a8679851fa890af
---
Explicitly silence some NoReverseMatch exceptions
In Django 1.5, when a NoReverseMatch exception was raised from a method
when rendering a template, it was silenced and the method returned
"(None)". It is no longer being silenced in Django 1.6, so we must
do it explicitly.
https://docs.djangoproject.com/en/1.6/releases/1.6/#miscellaneous
---
Expecting 400 status code instead of SuspiciousOperation exception in tests
Since Django 1.6, when a SuspiciousOperation reaches the WSGI handler,
a handler400 mechanism and default view are used to return
a HttpResponseBadRequest, when previously this resulted in
an InternalServerError. Some tests needed to be changed accordingly.
https://docs.djangoproject.com/en/1.6/releases/1.6/#minor-features
---
Fix a translation bug in django/contrib/auth by overriding the translation
---
Change has_auto_field to needs_explicit_pk_field in templates
In a Django 1.6 update the function has_auto_field has been renamed
to needs_explicit_pk_field. This caused TabularInline and StackedInline
to render incorrectly. Some templates needed to be changed accordingly.
https://github.com/django/django/commit/3aad955ea8db1592fad0012155eaa25b72e50dc5
---
Set the RabbitMQ broker as the default one
There was is a potential deadlock in the unpackmgr_job Celery task,
and since upgrading to Django 1.6, when using the Django database
as a broker, the deadlock actually takes place. A temporary "solution"
to this problem is to force users to use RabbitMQ as the default broker.
https://jira.sio2project.mimuw.edu.pl/browse/SIO-1583
---
Update the developer documentation
---
Set session serializer to PickleSerializer
The default session serializer was changed in Django 1.6 to JSONSerializer
for security reasons. For backwards compatibility it is set to
PickleSerializer, the previous default. This needs to be changed in
near future.
https://docs.djangoproject.com/en/1.6/releases/1.6/#default-session-serialization-switched-to-json
Change-Id: I0c94bb7771090a38999ee89e08dda3208570b217