agenda and updates for 5/8 #54
Description
Since we skipped a meeting, this is actually the updates for the last 4 weeks. Progress includes:
- Switch from UrbanSim to Orca #50 Switched from simulation framework to "orca" (data pipeline orchestration). No longer have any dependencies on UrbanSim.
- https://github.com/synthicity/activitysim/tree/mode-choice2 - A PR for a fairly complete spec of the tour mode choice model. A few caveats are described in the PR, and the progress report on the wiki is up-to-date with the remaining to-dos.
We spent the last 1.5 weeks working on getting a benchmark for the current approach and fixing obvious performance issues that came up. Right now we're looking at about 2.5 minutes per 10,000 households or about or a little less than 6 cpu/hours for all of the models except CDAP. CDAP is definitely the bottleneck and we'll have new benchmarks on that soon. I don't see any reason why we wouldn't be able to parallelize batches of households as well to get that down more if you throw hardware at the problem.
If you read in all of the skims at the same time it uses about 10GB of memory, but most of those skims only get used once or twice so I'm not sure there's a strong reason to keep them in memory at all times (this is also for a 1454x1454 O-D matrix).
I'll try and follow up on this with a benchmark and profile of the whole model set, and some information on the current state of the performance of CDAP, which is much faster than it was 2 weeks ago but might still be the bottleneck in the simulation.