2025-07-24 Engineering Team

[jpn--]

@ActivitySim/engineering

Agenda

• Status of open PR's

• Discuss Pandas 2 assign_in_place() fix with categoricals activitysim#948:

  • just merge it or think harder about categorical ordering?

• Technical:

  • meeting technology: have bot join meetings and take notes?
  • meeting notes editing
    • Idea 1: grant all members higher level access to this meeting-notes repo, we edit notes in issue comments
    • Idea 2: notes from meetings are committed to the repo itself, not the issue; anyone can edit in the normal manner for GitHub
    • Idea 3: meeting notes go to Zephyr's google docs account, we edit and store them there. Side effect: meeting notes are only available to authorized users, not the general public. Maybe this is a feature instead of a bug?

[jpn--]

Engineering Team Meeting Notes

Meeting Overview

Participants: Jeffrey Newman (Driftless), David Hensle (RSG), Stefan Coe (PSRC), Joe Castiglione (SFCTA), Andrew Kay (CamSys), Sijia Wang (WSP), Bo Wen (TransLink), Joe Flood (SANDAG)

Agenda Recap

• Review and status of open pull requests
• Discussion of technical issues and bug fixes
• Future development planning and technology exploration

Meeting Summary

Pull Request Reviews and Status

The meeting began with Jeffrey Newman reviewing the status of several open pull requests:

• Hard-coded vehicle table bug fix: Under review by David, awaiting test results before merging
• Check YAML spec issues: Andrew identified failing test models that need collaboration with Jeffrey to resolve
• Expression profiling: Multiple team members (Stefan, Bo) experiencing different technical issues with file paths and directory access
• Telecommute status model: Needs review from David, with Joe emphasizing this should be merged as a standard deliverable

Technical Discussion: Categorical Variables

A significant portion of the meeting focused on handling categorical variables when merging data from different models. The issue arose from SANDAG's airport passenger model having different mode categories than their resident activity model. Sijia Wang implemented a solution to union categoricals when new categories are introduced, but this raised questions about category ordering and consistency across multiprocessing scenarios.

Jeffrey Newman expressed concerns about maintaining consistent categorical ordering across different processes, suggesting sorting categories to ensure consistency. However, this would affect user-expected orderings (e.g., transportation modes appearing in logical rather than alphabetical order). The team ultimately decided to merge the current fix as-is, treating it as the minimal viable solution for the reported bug.

Future Development Planning

The discussion expanded to broader architectural questions:

Data Model Implementation: Joe Castiglione pushed for understanding impediments to implementing a data model in ActivitySim 1.x. Jeffrey explained that while defining a data model is straightforward, enforcing it within the existing codebase would be a significant engineering effort.

Technology Exploration: Jeffrey proposed exploring four different technological approaches for ActivitySim 2.0:

1. Polars: As a pandas replacement with better performance
2. JAX: Google's machine learning framework offering GPU acceleration and automatic differentiation for parameter calibration
3. Rust|C|C++ Compiled Extension: Lower-level programming for performance improvements
4. Awkward: Non-rectangular arrays, allows us to hew closer to the "households have people, people have tours" idea.

Each approach has different risk/reward profiles and would require prototyping to evaluate feasibility.

Prioritization Discussion

David Hensle raised concerns about prioritization, suggesting automatic calibration functionality might be more pressing for users than data model implementation or technology platform changes. The team acknowledged the need for careful prioritization of Phase 11 work.

Action Items

1. Jeffrey Newman & Andrew Kay: Meet to resolve Check YAML spec test failures
2. David Hensle:
   • Review telecommute status model pull request
   • Update UV PR test failures
3. Jeffrey Newman: Investigate and fix expression profiling bugs reported by Stefan and Bo
4. All team members: Write brief issue descriptions (1-2 paragraphs) for Phase 11 prototyping priorities, including:
   • Automatic calibration functionality
   • Data model prototyping for single component
   • Technology platform exploration (JAX, Polars, etc.)
   • Include order-of-magnitude cost estimates ($10K, $50K, $100K)
5. Product team: Review and prioritize proposed Phase 11 initiatives
6. Sijia Wang: Will miss next two engineering meetings

Next Steps

The team will continue reviewing pull requests and focus on completing current development work while preparing for Phase 11 planning discussions with the product and community team.