Adding defaults #4

fscottfoti · 2014-12-09T00:43:45Z

Not planning on pulling yet - just want to leave this open to see the diff

Adding basic data sources and several computed variables

So far so good

there are three basic change here: * addings activitysim/defaults/datasources.py which are similar to urbansim_defaults * an example directory which is similar to bayarea_urbansim (i.e. has data and config dirs right now) * and some sample notebooks for typical workflows We might separate some of this stuff into separate repos in the future but for now I think there is no need.

jiffyclub · 2014-12-09T00:49:47Z

I'd have only one .gitignore, e.g. add a line for example/data/* to the root .gitignore.

fscottfoti · 2014-12-09T03:07:12Z

Problem is in git you can't add empty directories (I've never understood why) so .gitignore is the placeholder that puts the directory there. I will probably end up adding some small data files and can remove it then. Until then I guess it's a bit awkward...

jiffyclub · 2014-12-09T04:39:59Z

I like to put a README in there to explain why there's an empty directory.

right now only using the first 8 rows of the spec, but we're successfully simulating choices This works by taking the UEC spreadsheet and keeping the coefficients almost formatted as-is. I say almost because the filters and alt names had to be tweaked a bit to match Pandas conventions. This commit is working but brings up a few questions that I will post as issues on github in a bit. This is clearly not a final implementation but something to elicit early feedback.

fscottfoti · 2014-12-09T22:10:30Z

A separate issue of this pull request is that it's terribly slow right now. Running on the full household table takes about 200s with only 8 columns (it looks like the spec has 150 columns). @jiffyclub Any ideas here? I will profile this but I'm guessing it's in the calls to "eval".

fscottfoti · 2014-12-09T22:14:25Z

Just leaving a note to myself for todo items that this PR does NOT include for a complete reimplementation of auto ownership

Diagnose performance issues
Make sure we can compute all the columns from the spec (150 columns or so, not including those below)
This includes things like AUTOPEAKRETAIL, etc that are in the spec and need to be taken from the skims
And finally the "auto time savings" variable

jiffyclub · 2014-12-10T06:13:27Z

I added UDST/urbansim@f88f665 to the dcm branch to speed up making the interaction dataset. It still takes 15 seconds, but that's better than 2 minutes. The runtime is dominated now by the doing the draws, e.g. https://github.com/synthicity/activitysim/pull/4/files#diff-ae8114eddeabee70b62a76c66a5b85a5R47.

fscottfoti · 2014-12-10T17:06:59Z

Perfect - I'll try it out. I also just realized the place the fast choice code exists is in mnl.py - i.e. just don't pass returnprobs=True and it makes the choices for you the best way. Totally forgot about that. I'll give it a shot.

fscottfoti · 2014-12-10T18:35:35Z

OK - with the fix you put into interaction, we went from 200s to 100s. I just committed the change to use the choice making from mnl.py and we're down to 43s. Whenever you get a chance, please do another profile of the latest commit and make sure there's not any other easy optimizations.

jiffyclub · 2014-12-10T19:33:43Z

Now there are about 15 seconds going to data processing and 36 sec going to prediction. Within prediction ~17 seconds are going to describe and ~15 sec are going to mnl_interaction_dataset. Within mnl_interaction_dataset most of the time is going to pandas.merge, with a few seconds going to .take.

the big change here is that a lot of the columns could not be expressed as DataFrame.eval calls and so I'm directly making Python "eval" calls if the expression starts with an "@". This is a proposal and not necessarily permanent functionality - something else to talk about. I like the way this works though as it puts quite a bit of power directly in the spreadsheet for the model specs, much the same way it is now. We have the option of never doing a DataFrame.eval call but these calls are a bit easier to express and are apparently much faster. This checkin also has the performance improvements on the choices and converts windows newlines to regular newlines so you can see the spec on github now

fscottfoti · 2014-12-11T01:44:51Z

OK on a server, and with the speedups, the auto ownership model is simulating in 50s. Still a few things to add which will slow this down a bit, but so far so good.

coveralls · 2014-12-11T07:27:39Z

Coverage remained the same when pulling 6692d0d on adding-defaults into 5a68ed1 on master.

…ibility.csv Easier to add the logsums than I thought it would be. Also moved the standard methods into a new module - activitysim.py - will add unit tests and docstrings if we all think this is the right direction to be headed Also added a global param to settings.yaml called household_sample_size to run with a subset of the total households when debugging

coveralls · 2014-12-11T18:30:50Z

Coverage remained the same when pulling 4e87889 on adding-defaults into 5a68ed1 on master.

e-lo · 2014-12-19T06:56:07Z

activitysim/activitysim.py

excel!?! No thanks!

Adding defaults - this PR adds some simple and standard data sources, variables, and models to ActivitySim. I'm going to start merging PRs even though this code is still subject to change somewhat in the future.

fix issue in activitysim.abm.models.util.mode where reset_index level…

cross-border changes after rebase from upstream develop

BayDAG Contribution #4: Joint Tour Destination Performance

* handle dev versions of Larch * test stability * pin multimethod < 2.0 * add availability_expression * starting est docs * Resolve package version conflicts (ActivitySim#923) * limit multimethod version to 2.0 and earlier * add multimethod version to other settings * [makedocs] update installer download link * [makedocs] update branch docs * GitHub Actions updates (ActivitySim#926) * use libmamba solver * add permissions [makedocs] * add write permission for dev docs [makedocs] * conda-solver: classic * trace proto tables if available, otherwise synthetic population (ActivitySim#901) Co-authored-by: Jeffrey Newman <[email protected]> * release instructions (ActivitySim#927) * use libmamba solver * add permissions [makedocs] * add write permission for dev docs [makedocs] * conda-solver: classic * include workflow dispatch option for tests * update release instructions * add installer build to instructions * Pin mamba for now, per conda-incubator/setup-miniconda#392 * conda-remove-defaults * when no unavailability parameters are included * some general estimation docs * Use pandas 2 for docbuild environment (ActivitySim#928) * fix link * allow failure to import larch * workflow * blacken * try some pins * speed up docbuild * use pandas 2 for docs * oops wrong file * restore foundation * Update HOW_TO_RELEASE.md * refactor(shadow_pricing.py): remove a duplicated `default_segment_to_name_dict` (ActivitySim#930) * fix typo * fixing disaggregate accessibility bug in zone sampler * Revert "fixing disaggregate accessibility bug in zone sampler" This reverts commit be5d093. * notes on size terms * clean up docbuild * fix version check * add some doc * tidy * estimation docs * more on alternative avail * model evaluation * add doc on component_model * documentation enhancements * larch6 is now larch>6 * branch docs on workflow_dispatch * missing doc section on model respec --------- Co-authored-by: Yue Shuai <[email protected]> Co-authored-by: David Hensle <[email protected]> Co-authored-by: amarin <[email protected]> Co-authored-by: Ali Etezady <[email protected]> Co-authored-by: Sijia Wang <[email protected]>

* multiprocess initial commit * blacken * parquet format for EDBs * adding pkl, fixing edb concat and write * fixing double naming of coefficient files * blacken * fixing missing cdap coefficients file, write pickle function * combact edb writing, index duplication, parquet datatypes * sorting dest choice bundles * adding coalesce edbs as its own step * CI testing initial commit * infer.py CI testing * estimation sampling for non-mandatory and joint tours * adding survey choice to choices_df in interaction_sample * adding option to delete the mp edb subdirs * changes supporting sandag abm3 estimation mode * running test sandag example through trip dest sample * Estimation Pydantic (#2) * pydantic for estimation settings * allow df as type in config * fix table_info * repair for Pydantic * df is attribute * Estimation settings pydantic update * new compact formatting * handling multiple columns for parquet write * dropping duplicate columns * actually removing duplicate columns * dfs with correct indexes and correct mp sorting * ignore index on sort for mp coalesce edbs * updating estimation checks to allow for non-zero household_sample_size * Re-estimation (#3) * pydantic for estimation settings * allow df as type in config * fix table_info * auto ownership * repair for pydantic * update for ruff * updated for simple models * repair for Pydantic * simple simulate and location choice * df is attribute * scheduling * stop freq * test locations * cdap * nonmand_and_joint_tour_dest_choice * nonmand_tour_freq * fix ci to stop using mamba * test updates * use larch6 from pip * use numba for stop freq * fix for pandas 1.5 * fix stop freq test for numba * Sharrow Cache Dir Setting (#893) * setting necessary filesystem changes from settings file * set for multiprocessing * repair github actions * github action updates (#903) * script to make data * unified script for making data * remove older * bug * doc note * load from parquet if available * add original alt ids to EDB output when using compact * fix MP race * script arg to skip to EDB * clean up CDAP and blacken * refactor model_estimation_table_types change to estimation_table_types, to avoid pydantic namespace clash * repair drop_dupes * blacken * location choice with compact * choice_def for compact * spec changes for simple-simulate * re-estimation demo for auto ownership * clean up status messages * change name to stop pydantic warnings * edit configs * default estimation sample size is same as regular sample size * allow location alts not in cv format * dummy zones for location choice * update scheduling model estimation * various cleanup * stop freq * tidy build script * update 02 school location for larger example * update notebook 04 * editable model re-estimation for location choice * fix test names * update notebooks * cdap print filenames as loading * notebook 07 * tests thru 07 * notebooks 08 09 * build the data first * runnable script * change larch version dependency * keep pandas<2 * notebooks 10 11 * notebook 12 * remove odd print * add matplotlib * notebook 13 14 * test all the notebooks * add xlsxwriter to tests * notebook 15 * CDAP revise model spec demo * notebook 16 * notebook 17 * longer timeout * notebook 18 * notebook 19 * notebook 20 * smaller notebook 15 * configurable est mode setup * notebook 21 * notebook 22 * config sample size in GA * notebook 23 * updates for larch and graphviz * change default to compact * compare model 03 * test updates * rename test targets * repair_av_zq * move doctor up * add another repair * oops --------- Co-authored-by: David Hensle <[email protected]> * Removing estimation.yaml settings that are no longer needed * fixing unit tests, setting parquet edb default * one more missed estimation.yaml * using df.items for pandas 2 compatibility * tidy doc * updating edb file name for NMTF * updating numba and pandas in the conda env files * Improve test stability (#4) * handle dev versions of Larch * test stability * pin multimethod < 2.0 * add availability_expression * starting est docs * Resolve package version conflicts (#923) * limit multimethod version to 2.0 and earlier * add multimethod version to other settings * [makedocs] update installer download link * [makedocs] update branch docs * GitHub Actions updates (#926) * use libmamba solver * add permissions [makedocs] * add write permission for dev docs [makedocs] * conda-solver: classic * trace proto tables if available, otherwise synthetic population (#901) Co-authored-by: Jeffrey Newman <[email protected]> * release instructions (#927) * use libmamba solver * add permissions [makedocs] * add write permission for dev docs [makedocs] * conda-solver: classic * include workflow dispatch option for tests * update release instructions * add installer build to instructions * Pin mamba for now, per conda-incubator/setup-miniconda#392 * conda-remove-defaults * when no unavailability parameters are included * some general estimation docs * Use pandas 2 for docbuild environment (#928) * fix link * allow failure to import larch * workflow * blacken * try some pins * speed up docbuild * use pandas 2 for docs * oops wrong file * restore foundation * Update HOW_TO_RELEASE.md * refactor(shadow_pricing.py): remove a duplicated `default_segment_to_name_dict` (#930) * fix typo * fixing disaggregate accessibility bug in zone sampler * Revert "fixing disaggregate accessibility bug in zone sampler" This reverts commit be5d093. * notes on size terms * clean up docbuild * fix version check * add some doc * tidy * estimation docs * more on alternative avail * model evaluation * add doc on component_model * documentation enhancements * larch6 is now larch>6 * branch docs on workflow_dispatch * missing doc section on model respec --------- Co-authored-by: Yue Shuai <[email protected]> Co-authored-by: David Hensle <[email protected]> Co-authored-by: amarin <[email protected]> Co-authored-by: Ali Etezady <[email protected]> Co-authored-by: Sijia Wang <[email protected]> * handling missing data or availability conditions * add docs on locking size terms * include constants in CDAP * bump larch requirement * require larch 6.0.40 * add xlsxwriter to envs * require larch 6.0.41 * add links * fix typos and formatting * cdap hh and per parquet read match csv * add missing x_validator for mode choice and nonmand tour freq * add tour mode choice edit example * add to docs * union not addition on sets * restore nb kernel * blacken * replacing conda with uv in estimation tests * add requests to github-action dependencies * running with created virtual env instead * Fix estimation notebook tests (#8) * Update scheduling.py --------- Co-authored-by: Jeffrey Newman <[email protected]> Co-authored-by: Yue Shuai <[email protected]> Co-authored-by: amarin <[email protected]> Co-authored-by: Ali Etezady <[email protected]> Co-authored-by: Sijia Wang <[email protected]>

fscottfoti added 2 commits December 8, 2014 12:58

all basic variables implemented for auto ownership

09cfe10

jiffyclub added this to the Automobile Ownership milestone Dec 9, 2014

jiffyclub assigned fscottfoti Dec 9, 2014

fscottfoti mentioned this pull request Dec 9, 2014

Design decisions around low-level vs high-level objects, yaml vs. csv serialization, etc... #5

Closed

performance improvements

6692d0d

fscottfoti added 3 commits December 11, 2014 09:48

add the activitysim.py module too

d72c2c1

pep8

4e87889

e-lo reviewed Dec 19, 2014
View reviewed changes

activitysim/activitysim.py

Copy link

Collaborator

e-lo Dec 19, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excel!?! No thanks!

fscottfoti merged commit 79593f8 into master Jan 5, 2015

fscottfoti deleted the adding-defaults branch January 5, 2015 21:34

toliwaga added a commit that referenced this pull request Sep 26, 2017

Merge pull request #4 from RSGInc/asim-core

86b0bf6

fix issue in activitysim.abm.models.util.mode where reset_index level…

bstabler pushed a commit that referenced this pull request Oct 20, 2021

Merge pull request #4 from SANDAG/xborder

13b3967

cross-border changes after rebase from upstream develop

jpn-- added a commit that referenced this pull request Mar 19, 2024

Merge pull request #770 from SANDAG/BayDAG_cont4_jtd_just_joint

ff61744

BayDAG Contribution #4: Joint Tour Destination Performance

Adding defaults #4

Adding defaults #4

Uh oh!

Conversation

fscottfoti commented Dec 9, 2014

Uh oh!

jiffyclub commented Dec 9, 2014

Uh oh!

fscottfoti commented Dec 9, 2014

Uh oh!

jiffyclub commented Dec 9, 2014

Uh oh!

fscottfoti commented Dec 9, 2014

Uh oh!

fscottfoti commented Dec 9, 2014

Uh oh!

jiffyclub commented Dec 10, 2014

Uh oh!

fscottfoti commented Dec 10, 2014

Uh oh!

fscottfoti commented Dec 10, 2014

Uh oh!

jiffyclub commented Dec 10, 2014

Uh oh!

fscottfoti commented Dec 11, 2014

Uh oh!

coveralls commented Dec 11, 2014

Uh oh!

coveralls commented Dec 11, 2014

Uh oh!

e-lo Dec 19, 2014

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants