For Developers

Welcome to the Developers page! This section provides resources and instructions for developers who want to contribute to PT3S.

Contact: For additional information regarding PT3S, this documentation, and contribution inquiries, please contact jablonski@3sconsult.de. If you need feedback on a specific issue with PT3S, please include the PT3S version you are using and, if available, the Jupyter Notebook you are working with.
GitHub Repository: You can find the source code and contribute to the project on our GitHub page.
PyPI Project: The PT3S package is also available on PyPI.

Note

If you are working with a Spyder Console or in a similar environment, remember to place a ‘!’ before each command. This tells the environment to run the command as a shell command, not as a Python command. For instance, write !git clone instead of git clone.

Setting Up Git on Your Computer

Follow these steps to install and configure Git:

Download Git: Visit the official Git website and download the version that is compatible with your operating system.
Install Git: Launch the downloaded installer and follow the setup wizard to complete the installation.
Configure Your GitHub Username: Open your terminal or command prompt and enter the following command, replacing “Your Name” with your actual GitHub username:
```
git config --global user.name "Your Name"
```
Configure Your GitHub Email: Similarly, set your GitHub email using the following command, replacing your.email@example.com with your actual email:
```
git config --global user.email "your.email@example.com"
```

Note

If you’re pushing your first commit to GitHub, you might be prompted to authenticate your GitHub account. This usually involves entering your GitHub account credentials in a browser dialogue window that pops up. This is a standard security measure to ensure you have the necessary permissions to push to the repository.

Working with GitHub

Note

In general before running any command that includes origin, use the git fetch origin command. To ensure the latest version of the origin is used in the following command.

Cloning the GitHub Repository

To clone a GitHub repository to your local folder, follow these steps:

Navigate to the Parent Directory of Your Project: Use the cd command followed by the path to the parent directory of your project (This is the directory that should contain your project folder).
```
cd "C:\Users\User\3S"
```
Clone the GitHub Repository: Use the git clone command followed by the URL of the repository.
```
git clone https://github.com/3SConsult/PT3S
```

Now, you are advised to following the steps of Installing PT3S in Editable Mode.

General GitHub Version Control Procedure

These instructions lay out the different steps of the GitHub procedure around contributing to PT3S. Especially due to the GitHub repository currentley sitting on only one branch (master), following these basic rules is crucial. As soon as PT3S has a higher amount of frequent contributors, a more suitable system with multiple branches will be implemented.

Note

Before following each step for the first time, read their instructions fully including notes like this one. If an unexpected problem occurs, you can search the Collection of Useful Git Commands for a solution.

Follow these steps every time you contribute to PT3S:

Get the Latest Version from GitHub: Get the Latest Version from GitHub
Edit PT3S: Now you can edit the entire PT3S project locally. Please ensure, that nobody else is working on the project simultaneously in the same sourcefiles, because this could cause problems, when trying to commit.
Commit Your Changes to the GitHub Repository: Commit Your Changes to the GitHub Repository

Get the Latest Version from GitHub

To fetch the latest changes from the origin and merge them into your current branch, follow these steps:

Navigate to project directory: Use the cd command followed by the path to the directory of your project (This directory should contain an invisible .git folder).
```
cd "C:\Users\User\3S\PT3S"
```
Fetch the latest changes from the origin: Use the git fetch origin command.
```
git fetch origin
```
A: Pull the latest changes from the origin: Use the git pull command.
```
git pull origin master
```
For a more detailed updating process, follow steps 3B instead.

B: Merge the fetched changes into your current branch: Use the git merge origin/master command.
```
git merge origin/master
```

Note

If you made local changes to files that were also edited by a remote commit, make a local copy of your project directory and use git reset --hard origin/master. Afterwards you can paste you local changes back in. Just make sure that the remote changes to these files were not important or manually include them in your files.

git fetch origin
git reset --hard origin/master

Commit Your Changes to the GitHub Repository

To commit your changes to the GitHub repository, follow these steps:

Navigate to project directory: Use the cd command followed by the path to the directory of your project (This directory should contain an invisible .git folder).
```
cd "C:\Users\User\3S\PT3S"
```
Fetch the the origin: Use the git fetch origin command.
```
git fetch origin
```

Add files to the staging area: Use the git add command followed by the name of the file. Use git add . to add all files.
```
git add .
```
Create a new commit with a descriptive message: Use the git commit -m "commit_message" command.
```
git commit -m "commit_message"
```
Push your commit to the GitHub Repository: Use the git push origin master command.
```
git push origin master
```

Collection of Useful Git Commands

To discard all local changes and set your local HEAD to the master, use:

git fetch origin
git reset --hard origin/master

To discard all remote changes and force push local HEAD to the master, use:

git push origin master --force

To uncommit commited but not yet pushed changes of the previous commit without changing local files (move HEAD pointer back by one commit), use:

git reset --soft HEAD~1

To revert all changes caused by a commit, use:

git revert commitID

To load a branch locally, use:

git checkout <branchname>

GitHub Workflows

Our GitHub repository uses workflows to facilitate certain processes by automating tasks. Workflows are defined using YAML files and are stored in the .github/workflows directory of our repository.

Current Workflows

All of our workflows can be triggered using Manually Triggering Workflows.

We currently use the following workflows:

Name	Triggers (Apart from manually triggering)	Tasks
Publish to PyPI	Commit tagged with ‘v*’	Builds and uploads package to PyPI
Automatic Copying of HTML-Files	Push to PT3S/sphinx_docs/_build/html/	Copies HTML files from master PT3S/sphinx_docs/_build/html to gh-pages PT3S/docs
Automatic Deletion of Example Data		Deletes example data in all PT3S/Examples/WDExampleX/B1/V0/BZ1 except .xml and .mx1

Manually Triggering Workflows

Follow these steps to manually trigger a workflow via GitHub:

Navigate to the Actions tab of the PT3S GitHub repository.
Select the workflow you want to trigger from the list on the left.
Click the Run workflow button.
Optionally, provide any required inputs and click Run workflow again to start the workflow.

Change Local Reference to New Remote

To update your local repository to point to the new remote (after transferring a GitHub repo), follow these steps:

Open your terminal and navigate to your local repository.
```
cd C:\Users\User\3S\PT3S
```
Verify the current remote URL:
```
git remote -v
```
Update the remote URL to the new repository location:
```
git remote set-url origin <new-repo-URL>
```
Replace <new-repo-URL> with the URL of the new repository.
Verify the change:
```
git remote -v
```
Fetch Remote: Fetch the repository
```
git fetch origin
```

Working with PyPI

Release a New Version

Before uploading a new release to PyPI, follow these steps:

Document the Release: Describe new additions or fixes, that are included in this release, to the PT3S/sphinx_docs/releases.rst file.

90.14.20.0.dev1
---------------
- readDxAndMx:
    **Fix:**
        - m is constructed (instead of reading m-pickle) if SIR 3S' dbFile is newer than m-pickle; in previous releases m-pickle was read even if dbFile is newer
    **New:**
        - INFO: if SIR 3S' dbFile is newer than SIR 3S' mxFile; in this case the results are maybe dated or (worse) incompatible to the model

90.14.19.0.dev1
---------------
**New:**

- SIR 3S db3 and mx files used in Examples are now included in the package.

On the Releases page you can view how this rst code is transformed into html.

Change Release Number: Change the release numbers in the files: PT3S/PT3S/conf.py, (PT3S/setup.py), PT3S/sphinx_docs/conf.py, PT3S/pyproject.toml (setup.py is not used anymore but kept as a backup for now)
Run Doctests: Follow the steps of Running Doctests. And make sure they are executed successfully.
Generate the Documentation: Follow the steps in Generating the Documentation.

Upload a New Version to PyPI

Follow one of the two methods below to upload a new version of your project to PyPI.

Manual Upload via Command Line

Update the Version Number: Make sure you have documented your changes and changed the release number in all necessary files according to Release a New Version.
Navigate to Your Project Directory: Use the cd command to move into your project folder.
```
cd "C:\Users\User\3S\PT3S"
```
Clean Old Distributions: Remove any existing files in the dist directory to avoid uploading outdated builds.
```
rm -rf dist/
```
Build the Package: Use the build module to generate both source and wheel distributions.
```
python -m build
```
Upload the Distribution: Generate an API token on PyPI (under Account Settings > API Tokens), then upload your package using twine:
```
python -m twine upload -u __token__ -p <YOUR TOKEN> dist/* --verbose
```
Note

For security, avoid hard-coding your token. Instead, store it in an environment variable or use a secret manager.

Automated Upload via GitHub Actions

The PT3S Repository includes a GitHub Actions workflow for publishing to PyPI. To use it, follow these steps:

Update the Version Number: Make sure you have documented your changes and changed the release number in all necessary files according to Release a New Version.

Commit and Push Your Changes: Commit Your Changes to the GitHub Repository

git add .

git commit -m "v90.14.XX.0.dev1"

git push origin master

Tag the Release: Create a Git tag that matches the pattern expected by the GitHub Actions workflow (e.g. v90.14.47.0.dev1). The last authored commit will be tagged:
```
git tag v90.14.XX.0.dev1
```
```
git push origin v90.14.XX.0.dev1
```
This will automatically trigger the GitHub Actions workflow to build and upload your package to PyPI.
Test the Deployment: Follow the steps in Running Tests

Installing PT3S in Editable Mode

After Cloning the GitHub Repository, you can install the package in editable mode. Here are the steps:

Navigate to the Directory of the Cloned Repository: Use the cd command followed by the path to the directory of your project.
```
cd "C:\Users\User\3S\PT3S"
```
Install the Package in Editable Mode: Use the pip install -e . command to install the package in editable mode.
```
pip install -e .
```

Now, your package is installed in editable mode. This means that you can make changes to the source code of the package and those changes will take effect immediately without needing to reinstall the package.

By installing PT3S in editable mode, a PT3S.egg-link file is created in the C:\Users\User\AppData\Local\anaconda3\Lib\site-packages directory. This file is a link to your project directory and allows Python to import the package as if it were installed normally. If you no longer need the package to be in editable mode, you can simply reinstall PT3S using pip. You can also reinstall an older version this way to test it.

Generating the Documentation

The PT3S documentation is edited in PT3S/sphinx_docs and files hosting the documentation are located in PT3S/docs.

If you want to edit the documentation yourself, you have to install sphinx related python packages.

pip install nbsphinx sphinx_copybutton sphinx-rtd-theme

Before generating the documentation for the first time, follow the steps of Installing PT3S in Editable Mode.

To generate documentation, follow these steps:

Edit the documentation: Make your changes in the PT3S/sphinx_docs directory.
Navigate to the PT3S/sphinx_docs directory: Use the cd command.
```
cd "C:\Users\User\3S\PT3S\sphinx_docs"
```
Make an HTML build: Use python3 -m sphinx.cmd.build -b html . /_build/html (for python env) or .\make.bat html (for conda env).
```
python3 -m sphinx.cmd.build -b html . /_build/html
```
```
.\make.bat html
```

Use Build File: Alternatively, instead of using the .\make.bat html command, you can simply open the PT3S/sphinx_docs/make_html_docs.py file and run it to generate the documentation. This method will not print any Sphinx debugging output and will save time. This alternative is recommended when making many iterative improvements to the documentation.
Commit the changes. Commit all files from PT3S/sphinx_docs to GitHub (Commit Your Changes to the GitHub Repository).

The new documentation can be found at https://3sconsult.github.io/PT3S/index.html

Note

The created files in PT3S/sphinx/docs/_build/html on the master branch are moved to PT3S/docs on the gh-pages branch by one of our GitHub Workflows and then hosted via GitHubPages. It might take a couple of minutes until the changes are visible on the website.

Running Doctests

Follow these tests to run all doctests included in this documentation:

Navigate to sphinx_docs directory: Open your terminal or command prompt and navigate to the directory sphinx_docs.
```
cd "C:\Users\User\3S\PT3S\sphinx_docs"
```
Make a doctest build: Use the .\make.bat doctest command.
```
.\make.bat doctest
```

You will get a console output and a output.txt file in the sphinx_docs_builddoctest directory.

If you want the newly added or edited tests included into the hosted documentation follow the steps of Generating the Documentation. Running the tests beforehand is only necessary if the tests are inclueded outside of .rst files.

Testing the Deployment with Docker

To ensure that the examples provided on the Examples page run smoothly on devices of users not involved in the development process, we test them using nbval inside a Docker container. This container simulates a Windows environment, including SIR 3S, the latest release of PT3S with its dependencies, the example data, and the example notebooks.

Environment Versions

This list provides information about the versions of various tools used throughout this project regarding development, creation of documentation, use of examples, etc. It is recommended to use the same versions of these tools, especially if you are contributing. These versions are used in the Docker testing.

Tool	Version
Python	3.11.8
Anaconda (Not used in Docker)	24.11.0
Sphinx-build	5.0.2

Initial Test Setup Process

Note

Not all files mentioned below are publicly available.

To set up all necessary files and programs to run tests on Notebooks, follow these steps:

Setup Docker: Download and install Docker Desktop. There might be some issues that need fixing in Windows settings. Help from the technical team is advised.
Enable Windows Containers: Right-click on the Docker Desktop icon in your taskbar and click “Switch to Windows Containers”.
Get Docker Files: Copy T:/interne_Projekte/PT3S/docker and T:/interne_Projekte/PT3S/dockerNotebooks to C:/Users/User/3S.
Copy SirCalc: The C:/Users/User/3S/docker/SIR 3S directory is empty and needs a working copy of SirCalc. The easiest way to achieve this is to copy all files from your local C:/3S/SIR 3S to C:/Users/User/3S/docker/SIR 3S and then delete unnecessary files. This prevents them from being included in the container, which would make the build process even longer.
Create v: Create a folder at C:/Users/User/3S/dockerNotebooks. The dockerNotebooks folder on your local machine is used as a volume for the Docker container. Therefore, all changes made to the notebooks inside the container are applied to these files. You can also save additional notebooks to this folder to add them into the container for testing.
Start Docker Engine: Open Docker Desktop and start the engine.
Navigate to Docker Folder: Open your terminal or command prompt and navigate to the directory containing your Dockerfile.
```
cd C:/Users/User/3S/docker
```
Build the Docker image: Run the following command in a cmd with the name you want to give to your Docker image (e.g., pt3stestpotsdam). This process can take around half an hour. So make sure everything is set up properly.
```
docker build -t pt3stestpotsdam .
```

Running Tests

These tests are run on Environment Versions.

Follow these steps to run tests on the Example Notebooks currently hosted at Examples:

Start Docker Engine: Open Docker Desktop and start the engine.
Navigate to your project directory: Open your terminal or command prompt and navigate to the directory containing your Dockerfile.
```
cd "C:/Users/User/3S/docker"
```
Run the Docker container: Run the following command with the name of your Docker image.

Note

The port must differ from a local JupyterLab you might be running.
```
docker run -it --rm -v C:\Users\User\3S\dockerNotebooks:C:\3S\notebooks -p 8889:8888 pt3stestpotsdam
```
The container should now be running, downloading the Example Notebooks and upgrading PT3S to its newest version automatically. The dockerNotebooks folder on your local machine is used as a volume for the Docker container. Therefore, all changes made to the notebooks inside the container are applied to these files. You can also save additional notebooks to this folder to add them into the container for testing (rerun necessary).

You now have access to a cmd running in the container environment. The -it option starts the container in interactive mode, and the –rm option removes the container after it exits.
Start Tests: Run the following command inside the container cmd. You should now be provided with the test results in the cmd.
```
pytest --nbval
```
With config file (currentley not useful):
```
pytest --nbval --nbval-sanitize-with sanitize.cfg
```
Open new Container CMD: Run the following command in a local cmd. The container_id can be found on Docker Desktop.
```
docker exec -it container_id cmd
```
Start JupyterLab: Run the following command in the new container cmd.
```
python -m jupyter lab --ip=0.0.0.0 --allow-root
```
Open in local Browser: Due to there not being a browser installed inside the docker container, JupyterLab will not open automatically. Click on one of the links provided in the cmd output or click on the host of the running container under the container tab in Docker Desktop. You might need to enter a token. This can be found in the cmd output as well. Now you can edit the notebooks inside the docker container. Saved changes are applied to your local files in the dockerNotebooks folder.
Test manually: To test one specific or all examples, run the following commands.
```
pytest --nbval ExampleX.ipynb
```
```
pytest --nbval
```