Development ¶

Contributing to Network Wrangler ¶

Setup ¶

Recommended Tools ¶

GitHub desktop to manage access to the main repository.
Git to conduct required version control.
MiniConda to manage your Python environments.
VSCode to edit and test code.
Some type of terminal application (note, this comes with Mac/Ubuntu).

Setup Virtual Environment ¶

Create and/or activate the virtual environment where you want to install Network Wrangler.

Creating and activating a virtual environment using conda

conda config --add channels conda-forge
conda create python=3.11 -n wrangler-dev #if you don't already have a virtual environment
conda activate wrangler-dev

Clone ¶

To effectively work on Network Wrangler locally, install it from a clone by either:

Use the GitHub user interface by clicking on the green button “clone or download” in the main network wrangler repository page.
Use the command prompt in a terminal to navigate to the directory that you would like to store your network wrangler clone and then using a git command to clone it.

Clone network wrangler

cd path to where you want to put wrangler
git clone https://github.com/wsp-sag/network_wrangler

Install in Develop Mode ¶

Install Network Wrangler in “develop” mode using the -e flag so that changes to your code will be reflected when you are using and testing network wrangler:

Install Network Wrangler from Clone

cd network_wrangler
pip install -e .

Install development dependencies

pip install -r requirements.tests.txt
pip install -r requirements.docs.txt

IDE Settings ¶

VSCode ¶

Select conda env as Python interpreter:

1	- `cmd-shift-P`: Python: Select Interpreter

If you are using VS Code, here are some recommended extensions and settings to leverage the IDE capabilities:

Extension	Purpose
Microsoft Python	Pytest integration, code-completion
Astral Ruff	Linting and formatting
Microsoft Jupyter	Edit and run python notebooks
Microsoft Data Wrangler	Review and edit data in pandas dataframes
David Anson markdownlint	Lint markdown
Github Pull Requests	Manage github issues and PRs
Dvir Yitzchaki parquet-viewer	Review parquet data as json
Random Fractals Inc. Geo Data Viewer	Render geojson data

Leveraging these extensions to their full potential may take some configuration. Here are some examples. YMMV.

settings.json for VS Code

{
    "[python]": {
    "editor.formatOnSave": true,
    "editor.defaultFormatter": "charliermarsh.ruff",
    "editor.codeActionsOnSave": {
    "source.fixAll": "explicit",
    "source.organizeImports": "explicit"
    }
},
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.cwd": "",
    "python.testing.autoTestDiscoverOnSaveEnabled": true,
    "python.defaultInterpreterPath": "/usr/bin/env python",
    "python.testing.pytestPath": "/opt/miniconda3/envs/wrangler-dev/bin/pytest",
}

Code > Settings > Settings

For tests to run in conda environment, add path to it. To find it, you can run conda info --envs

@id:python.condaPath: opt/miniconda3/envs/wrangler-dev

Development Workflow ¶

Create an issue for any features/bugs that you are working on.
Create a branch to work on a new issue (or checkout an existing one where the issue is being worked on).
Develop comprehensive tests in the /tests folder.
Modify code including inline documentation such that it passes all tests (not just your new ones)
Lint code using pre-commit run --all-files
Fill out information in the pull request template
Submit all pull requests to the develop branch.
Core developer will review your pull request and suggest changes.
After requested changes are complete, core developer will sign off on pull-request merge.

Tip

Keep pull requests small and focused. One issue is best.

Tip

Don’t forget to update any associated documentation as well!

Documentation ¶

Documentation is stored in the /docs folder and created by mkdocs using the material-for-mkdocs theme.

Build and locally serve documentation

mkdocs serve

Documentation is deployed using the mike package and Github Actions configured in .github/workflows/ for each “ref” (i.e. branch) in the network_wrangler repository.

Making sure your code works ¶

Linting and Type Checking ¶

Before even running the tests, its a good idea to lint and check the types of the code using pre-commit:

Example

pre-commit run --all-files

Your code must pass the pre-commit tests as a part of continuous integration, so you might as well fix anything now if it arises.

Adding Tests ¶

..to come

Running Tests ¶

Tests and test data reside in the /tests directory:

Example

pytest

Your code must pass the these tests as a part of continuous integration, so you might as well fix anything now if it arises.

Profiling Performance ¶

When you run the tests, their performance is profiled using pytest-profiling and profiles for tests are stored in .prof directory. If you want to explore what is taking time in a particular test, you can do so using products like snakviz

Explore performance of a test

snakeviz .prof/<test_name>.prof

We also benchmark some specific tests (test_benchmarks.py) that we want to compare when reviewing pull requests. If you want to review how you are doing on these benchmarks you can save the benchmarks when you run pytestand compare these numbers to another branch.

Compare benchmarks between branches

pytest --benchmark-save=branch_1
git checkout branch_2
pytest --benchmark-save=branch_2
pytest-benchmark compare branch_1 branch_2

Evaluate Code Maintainability ¶

Using Radon

Maintainability Index is a software metric which measures how maintainable (easy to support and change) the source code is. The maintainability index is calculated as a factored formula consisting of SLOC (Source Lines Of Code), Cyclomatic Complexity and Halstead volume.

radon mi ../network_wrangler -s

Cyclomatic Complexity corresponds to the number of decisions a block of code contains plus 1. This number (also called McCabe number) is equal to the number of linearly independent paths through the code. This number can be used as a guide when testing conditional logic in blocks.

radon cc ../network_wrangler --average

Continuous Integration ¶

Continuous Integration is managed by Github Actions in .github/workflows.
All tests other than those with the decorator @pytest.mark.skipci will be run.

Project Governance ¶

The project is currently governed by representatives of its two major organizational contributors:

Metropolitan Council (MN)
Metropolitan Transportation Commission (California)

Code of Conduct ¶

Contributors to the Network Wrangler Project are expected to read and follow the CODE_OF_CONDUCT for the project.

Contributors ¶

Lisa Z. - initial Network Wrangler implementation at SFCTA
Billy C. - initial Network Wrangler implementation at SFCTA
Elizabeh S.
Sijia W.
David O.
Ashish K.
Yue S.

Note

There are likely more contributors - feel free to add your name if we missed it!

Release History ¶

Changelog ¶

Notable changes and version history.

Version	Date	Comment
v1.0-beta-2	20204-10-15	Bug fixes in scenario loading, projectcard API and compatibility of transit net with roadway deletions. Some additional performance improvements.
v1.0-beta-1	20204-10-9	Feature-complete for 1.0
v1.0-alpha-2	2024-10-8	Testing for Met Council
v1.0-alpha-1	2024-07-17	Full refactor
v0.2.0-alpha	2020-09-16	-
v0.1.0-alpha	2020-09-09	-
0.0.2	2020-02-05	-