Details
-
Type:
Improvement
-
Status:
Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: Current Version
-
Fix Version/s: None
-
Component/s: Evaluation Engine / Workers
-
Labels:None
Description
Motivation:
1) Now we have to predict how much RAM we should reserve for single jobs. If we choose 1G, then we are lowering our possible concurrency. If we choose 512M, then worker could potentially lack memory.
2) We have different configs for judging machines which vary only in concurrency.
What to do:
1) Replace concurrency in worker config with RAM capacity
2) OIOIOI should add to environ information how much RAM single job needs
3) Scheduler should operate on RAM capacities instead of concurrencies.
4) Split this task into smaller ones :)
Adnotation:
Don't forget about taking into account number of CPU cores. Currently used formula is min(# cpu cores * 1.5, (RAM minus a little bit for OS) / (max required mem per problem)). Obviously it is worth to discuss if and how this formula should be changed.
1) Now we have to predict how much RAM we should reserve for single jobs. If we choose 1G, then we are lowering our possible concurrency. If we choose 512M, then worker could potentially lack memory.
2) We have different configs for judging machines which vary only in concurrency.
What to do:
1) Replace concurrency in worker config with RAM capacity
2) OIOIOI should add to environ information how much RAM single job needs
3) Scheduler should operate on RAM capacities instead of concurrencies.
4) Split this task into smaller ones :)
Adnotation:
Don't forget about taking into account number of CPU cores. Currently used formula is min(# cpu cores * 1.5, (RAM minus a little bit for OS) / (max required mem per problem)). Obviously it is worth to discuss if and how this formula should be changed.
Activity
- All
- Comments
- History
- Activity
- Transitions
- Commits
Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 14
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928. It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implementSIO-2070 and SIO-2073.
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
https://gerrit.sio2project.mimuw.edu.pl/2960
This commit is a large part of the implementation of
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
Show
Gerrit Gerrit
added a comment - Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 14
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928 . It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement SIO-2070 and SIO-2073 .
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
Gerrit Gerrit
added a comment - Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 14
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928 . It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement SIO-2070 and SIO-2073 .
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03 Change sioworkers~master~I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89, patchset 8
https://gerrit.sio2project.mimuw.edu.pl/2967
SIO-2073 Setting ram requirements for various worker jobs in prioritizing scheduler based on job type or given limits.
This commit is part ofSIO-1928.
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89
https://gerrit.sio2project.mimuw.edu.pl/2967
This commit is part of
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89
Show
Gerrit Gerrit
added a comment - Change sioworkers~master~I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89, patchset 8
https://gerrit.sio2project.mimuw.edu.pl/2967
SIO-2073 Setting ram requirements for various worker jobs in prioritizing scheduler based on job type or given limits.
This commit is part of SIO-1928 .
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89
Gerrit Gerrit
added a comment - Change sioworkers~master~I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89, patchset 8
https://gerrit.sio2project.mimuw.edu.pl/2967
SIO-2073 Setting ram requirements for various worker jobs in prioritizing scheduler based on job type or given limits.
This commit is part of SIO-1928 .
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89 Change sioworkers~master~I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89, patchset 9
https://gerrit.sio2project.mimuw.edu.pl/2967
SIO-2073 Setting ram requirements for various worker jobs in prioritizing scheduler based on job type or given limits.
This commit is part ofSIO-1928.
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89
https://gerrit.sio2project.mimuw.edu.pl/2967
This commit is part of
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89
Show
Gerrit Gerrit
added a comment - Change sioworkers~master~I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89, patchset 9
https://gerrit.sio2project.mimuw.edu.pl/2967
SIO-2073 Setting ram requirements for various worker jobs in prioritizing scheduler based on job type or given limits.
This commit is part of SIO-1928 .
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89
Gerrit Gerrit
added a comment - Change sioworkers~master~I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89, patchset 9
https://gerrit.sio2project.mimuw.edu.pl/2967
SIO-2073 Setting ram requirements for various worker jobs in prioritizing scheduler based on job type or given limits.
This commit is part of SIO-1928 .
Change-Id: I7f795b81ee95793d89d5a7a3bf29f9bfb205ef89 Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 15
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928. It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implementSIO-2070 and SIO-2073.
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
https://gerrit.sio2project.mimuw.edu.pl/2960
This commit is a large part of the implementation of
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
Show
Gerrit Gerrit
added a comment - Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 15
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928 . It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement SIO-2070 and SIO-2073 .
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
Gerrit Gerrit
added a comment - Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 15
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928 . It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement SIO-2070 and SIO-2073 .
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03 Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 16
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928. It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implementSIO-2070 and SIO-2073.
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
https://gerrit.sio2project.mimuw.edu.pl/2960
This commit is a large part of the implementation of
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
Show
Gerrit Gerrit
added a comment - Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 16
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928 . It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement SIO-2070 and SIO-2073 .
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03
Gerrit Gerrit
added a comment - Change sioworkers~master~I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03, patchset 16
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071 Adapt sioworkersd to respect RAM limits.
This commit is a large part of the implementation of
SIO-1928 . It modifies sioworkersd (mainly the default
PrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement SIO-2070 and SIO-2073 .
Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03 
https://gerrit.sio2project.mimuw.edu.pl/2960
SIO-2071Adapt sioworkersd to respect RAM limits.This commit is a large part of the implementation of
SIO-1928. It modifies sioworkersd (mainly the defaultPrioritizingScheduler) to recognize workers' total
RAM amounts and tasks' required RAM amount. The
core scheduling algorithm is kept the same, but
modified to account for RAM limits, which may have
some impact on the performance.
For these changes to be fully functional, it's also
necessary to implement
SIO-2070andSIO-2073.Change-Id: I625f7e7303bd5f69a96bde1443f8d9c38ef2ef03