by

Weekly Update

First, sorry for the delay with the update. The following is the progress made since the last update post.

Progress:

  • Implemented job and task usage generator. The generator can be configured to use a few distribution models including uniform, normal and pareto.
  • Implemented task terminator which will terminate tasks that has run for a certain duration. The duration can be generated with distribution above.
  • Investigated how the jobs/tasks are terminated. Only a slight increase in chance a task will be evicted if it over use the resource allocated to them.
  • Improved scheduler.
  • Added first fit and next fit scheduling algorithm.
  • Update the config file and corresponding code to allow better customization.

Plan for next week:

  • Add more scheduling algorithms
  • Investigate how jobs/tasks fail and simulate the failure
Advertisement
by

Meeting on 23 September

Date: 23 September 2014
Time: 2.00pm – 3.00pm

Agenda:

  1. Discussed progress so far. Scheduler has been added to the program.
  2. Discussed about conducting experiments.
    • Experiments should be conducted using a single snapshot of the code to be consistent.
    • A concrete design of the experiments should be constructed.
    • Result/performance matrices should be decided.

Planned task for coming weeks:

  • Finalizing the code
  • Deciding matrices that will be collected
  • Planning on the experiements
by

Weekly Update

Progress this week:

  • Used SQLite to store results
  • Implemented a simple scheduler that will allocate task to machines (naive implementation)

Plan for next week:

  • Improve the scheduler
  • Figuring how task terminate
by

Meeting on 21 Jul

Date: 21 July 2014
Time: 2.00pm – 3.00pm

Agenda:

  1. Had a demonstration of the program.
  2. Explained various matrices related to data centre.
  3. Discussed a bit on scheduler and problem related.

Planned task for coming week:

  • Implementing a simple scheduler and solve problem related to it, e.g. how jobs terminate and job usage
by

Weekly Update

Progress this week:

  • Updated some tracer result
  • Used MATLAB to estimate the distribution of the job usage
  • Met with supervisor. He felt that study in usage distribution is beyond the scope of the project. Instead, we should focus on scheduler.

Plan for next week:

  • Implement a simple scheduler
by

Meeting on 10 July

Date: 10 July 2014
Time: 2.30pm – 3.30pm

Agenda:

  1. Discussed the work done in past weeks. The works are mostly on relationship between requests and usage. Supervisor didn’t feel that the work is relevant to the project.
  2. Supervisor expressed his concern that the project is slightly behind schedule. He explained what he had planned. The work done in past few weeks wasn’t on his plan.

Planned task for coming week:

  • Supervisor like to see a demonstration in coming week
by

Weekly Update

Progress this week:

  • Identified factors that interfere with the usage distribution graph. Instead of working at task level, we should work on job level. A job can contains one to thousands of tasks. Tasks belong to same job usually behave similarly. So, these tasks cluster together in the distribution graph and give the graph a local “peak”.
  • Did some study on probability distribution and MATLAB

Plan for next week:

  • Continue working on resource usage estimate based on request
by

Weekly Update

Progress this week:

  • Added a feature to read external configuration file written in XML. Update the code so that several variables are stored externally in the configuration file
  • Added a few more performance reports
  • Still working on finding relationship between request and usage. Tried several methods to remove interference to the distribution, but so far not much success

Plan for next week:

  • Continue working on resource usage estimate based on request
by

Weekly Update

Progress this week:

  • Changed the way statistics are computed. Instead of aggregated over a period of time, the numbers are “sampled”. This will reduce the amount of memory needed to store information during the interval. In turn, the system spends less time on garbage collection and thus improve the overall performance.
  • Added a new package to deal with resource calculation. A lot of code dealing with resources are unified and simplified.
  • Observed  that when tasks is scheduled, there is a 15-window for the system to copy binaries of the task to the machine. During this time, no resource on the machine is used. This window will affect the overall utilisation rate for smaller tasks.

Plan for next week:

  • Have external configuration file to allow customisation
  • Continue working on resource usage estimate based on request
by

Weekly Update

Progress this week:

  • Made a new time system that will trigger events according to timeline. This allows simulation to run chronologically (used to simulate in batch)
  • Removed unnecessary data stored, this improved the efficiency
  • Separated usage from job dispatcher
  • Working on simplifying simulator
  • Working on improving calculation of various statistics

Plan for next week:

  • Get some data on relationship between request and usage
  • Implement a simple conversion from request to usage