performance variation during run time

From: Thomas C. Bishop (bishop_at_latech.edu)
Date: Wed Nov 28 2012 - 18:53:10 CST

Dear NAMD_2.9_Linux-x86_64-ibverbs,

Can you help me figure out why I get the following variations in
performance when running on a 160,000 atom simulation on
180 cores of queenbee.loni.org? On all other systems I use the "hours
remaining" decreases monotonically... as one might expect.
But really jumping from 6hrs remaining to 19hrs remaining is just odd.

Ideas?

Thanks in advance.
Tom

[bishop_at_qb3 log]$ grep -i benchmark ~/work/yeast/chr14/0280419/dyn3.out
Info: Benchmark time: 180 CPUs 0.0484492 s/step 0.280378 days/ns 315.965
MB memory
Info: Benchmark time: 180 CPUs 0.0489626 s/step 0.283348 days/ns 321.41
MB memory
Info: Benchmark time: 180 CPUs 0.0485597 s/step 0.281017 days/ns 321.41
MB memory
[bishop_at_qb3 log]$ grep -i remain ~/work/yeast/chr14/0280419/dyn3.out
TIMING: 5000 CPU: 242.99, 0.048576/step Wall: 243.198, 0.0486176/step,
6.68492 hours remaining, 321.410156 MB of memory in use.
TIMING: 10000 CPU: 486.51, 0.048704/step Wall: 486.844,
0.0487293/step, 6.63261 hours remaining, 321.410156 MB of memory in use.
TIMING: 15000 CPU: 730.33, 0.048764/step Wall: 730.76, 0.0487831/step,
6.57217 hours remaining, 321.410156 MB of memory in use.
TIMING: 20000 CPU: 974.6, 0.048854/step Wall: 975.126, 0.0488732/step,
6.51642 hours remaining, 321.410156 MB of memory in use.
TIMING: 25000 CPU: 1570, 0.11908/step Wall: 1570.64, 0.119104/step,
15.715 hours remaining, 321.410156 MB of memory in use.
TIMING: 30000 CPU: 2091.09, 0.104218/step Wall: 2091.9, 0.104251/step,
13.6105 hours remaining, 321.410156 MB of memory in use.
TIMING: 35000 CPU: 2335.54, 0.04889/step Wall: 2336.47,
0.0489152/step, 6.31821 hours remaining, 321.410156 MB of memory in use.
TIMING: 40000 CPU: 3105.87, 0.154066/step Wall: 3106.93,
0.154091/step, 19.6893 hours remaining, 321.410156 MB of memory in use.
[bishop_at_qb3 log]$

-- 
*******************************
    Thomas C. Bishop
     Tel: 318-257-5209
     Fax: 318-257-3823
    www.latech.edu/~bishop
********************************

This archive was generated by hypermail 2.1.6 : Mon Dec 31 2012 - 23:22:19 CST