Weekly Update

Progress this week:

  • Tried parsing task usage data, however, a few files was corrupted
  • Completed parser and dispatcher for other tables, total process time takes about 15 minutes

Plan for next week:

  • Fix the corrupted files
  • Start working on matrices collectors

Meeting on 28 April

Date: 28 April 2014
Time: 2:00pm – 3:00pm

Agenda:

  1. Discussed the progress since last meeting. The design and structure of the simulator was accepted.
  2. Discussed on how to present the simulation results. More statistical meaningful numbers should be included in the results.
  3. Briefly discussed the future direction of the project, including how to build the scheduler.

Planned tasks for next 2 weeks:

  • Complete the dispatcher
  • Present the simulation results in a more meaningful way

Weekly Update

Progress this week:

  • Noticed that all the table started with timestamp. Used this to read the trace data. Reading all data still take a long time, but we can set it to just read up to a certain timestamp
  • Done more work on the parser and simulator

Plan for next week:

  • Finish up the parser and simulator
  • Preparing for meeting with supervisor

Weekly Update

Progress this week:

  • Expending the parser so that it can read other data including machine events, job events, task events and task attribute
  • Added basics classes for the data center simulator. A dispatcher is made to interpret the data from parser and creating corresponding classes
  • Task events table is 15 GB in size. Reading the file alone takes about 4.5 minutes

Plan for next week:

  • Completing the parser for reading task events table
  • Working on the task usage table (158 GB)

Weekly Update

Progress this week:

  • Decompressed the trace data. The uncompressed size is about 177 GB
  • Made a CSV parser to read the trace data
  • Read part of the data that contains machine attribute. The data file is 1.1 GB. The parser took about a minute to read all the data and used about 600 MB RAM
  • A few method is tried to reduce the memory usage, including use of hash table to map long string to integer. However, these method has overhead and doesn’t improve the memory usage

Plan for next week:

  • Expands the parser to read other tables