(Python) packaging


module: Python file (.py) that contains definitions and statements.

package: a collection of modules in the same directory


Package directory must contain __init__.py for Python to "see" it*

* ... Not true with namespace packages

compphys package structure:



                            compphys/
                            |-- __init__.py
                            |-- constants.py
                            |-- physics.py
                            |-- more/
                            |   |-- __init__.py
                            |   |-- morephysics.py
                            |   |-- evenmorephysics.py
                            |-- assets/
                            |   |-- data.txt
                            |   |-- orphan.py
                            |-- tests/
                            |   |-- test_physics.py
                            |   |-- test_morephysics.py
                            ...
                        

compphys contents:


  • __init__.py: tells Python this is a package; typically empty. Executed before any other modules imported.
  • Two modules: constants.py and physics.py
  • more is a submodule
  • raw is a subdirectory; not a submodule since it doesn't have __init__.py (orphan.py is unreachable!)
  • tests: subdirectory with tests

Importing submodules


Use attribute access operator: .


                            import compphys.constants
                            import compphys.more.morephysics

                            two_pi = 2 * compphys.constants.pi
                        

These are absolute imports

Explicit relative imports


From physics.py:


                            from . import constants
                            from .constants import pi, h
                            from .more import morephysics
                        

From evenmorephysics.py:


                            from . import morephysics
                            from .. import constants
                            from ..constants import pi, h
                        

Important: while easy to create new functions and modules, don't reinvent the wheel!


Rely on the Python standard library, NumPy, SciPy, etc. as much as you need.


                            from time import sleep
                            print("This is my file to demonstrate best practices.")

                            def process_data(data):
                                print("Beginning data processing...")
                                modified_data = data + " that has been modified"
                                sleep(3)
                                print("Data processing finished.")
                                return modified_data

                            def read_data_from_web():
                                print("Reading data from the Web")
                                data = "Data from the web"
                                return data

                            def write_data_to_database(data):
                                print("Writing data to a database")
                                print(data)

                            def main():
                                data = read_data_from_web()
                                modified_data = process_data(data)
                                write_data_to_database(modified_data)

                            if __name__ == "__main__":
                                main()
                        

Source: https://realpython.com/python-main-function/

Package management options


  • Source-based: link to code. Good for dynamic languages, less for compiled.
  • Binary: popular for both dynamic and compiled. Create packages for various architectures/systems.
  • Virtualization: package both software and environment.

Python: pip and PyPI


  • pip can be used to install Python packages from source or from PyPI
  • We will tell pip how to install your package by creating a setup.py file for the setuptools package to use.
  • setup.py goes at the same level as your package's source code directory.

                        setup.py
                        compphys/
                        |--  __init__.py
                        |-- _version.py
                        |-- constants.py
                        |-- physics.py
                        |-- more/
                        |   |-- __init__.py
                        |   |-- morephysics.py
                        |   |-- evenmorephysics.py
                        |-- assets/
                        |   |-- data.txt
                        |   |-- orphan.py
                        |-- tests/
                        |   |-- test_physics.py
                        |   |-- test_morephysics.py
                        LICENSE
                        README.md
                        CHANGELOG.md
                        CONTRIBUTING.md 
                        CODE_OF_CONDUCT.md
                        

setup.py imports and calls setup() function, which can both install a package locally and make/upload to PyPI


                            from setuptools import setup
                            
                            setup(
                                name='compphys',
                                version='0.1.0',
                                description='Effective Computation in Physics',
                                author='Anthony Scopatz and Kathryn D. Huff',
                                author_email='koolkatz@gmail.com',
                                url='http://physics.codes',
                                classifiers=[
                                    'License :: OSI Approved :: BSD License',
                                    'Intended Audience :: Developers',
                                    'Intended Audience :: Science/Research',
                                    'Natural Language :: English',
                                    'Programming Language :: Python :: 3',
                                ],
                                license='BSD-3-Clause',
                                python_requires='>=3',
                                zip_safe=False,
                                packages=['compphys', 'compphys.more', 'compphys.tests'],
                                # or find automatically:
                                # packages=find_packages(),
                                package_dir={
                                    'compphys': 'compphys',
                                    'compphys.more': 'compphys/more',
                                    'compphys.tests': 'compphys/tests',
                                    },
                                #
                                include_package_data=True,
                                # or you can specify explicitly:
                                package_data={
                                    'compphys': ['assets/*.txt']
                                    },
                                )
                        

Better:

  • Have setup.py copy in README, version information from _version.py file in your package, and include dependency information.
  • Add a setup.cfg file with some additional configuration options
  • Add a CHANGELOG file to describe how your code changes with each version.
  • Add a MANIFEST.in file to tell PyPI to bring other files.

                        from setuptools import setup
                        from codecs import open
                        from os import path
                        import sys

                        here = path.abspath(path.dirname(__file__))
                        
                        version = {}								
                        with open(path.join(here, 'compphys', '_version.py')) as version_file:
                            exec(version_file.read(), version)
                        __version__ = version['__version__']

                        with open(path.join(here, 'README.md')) as readme_file:
                            readme = readme_file.read()

                        with open(path.join(here, 'CHANGELOG.md')) as changelog_file:
                            changelog = changelog_file.read()
                        
                        long_description = readme + '\n\n' + changelog

                        install_requires = [
                            'numpy',
                        ]

                        tests_require = [
                            'pytest',
                            'pytest-cov',
                        ]

                        needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
                        setup_requires = ['pytest-runner'] if needs_pytest else []

                        setup(
                            name='compphys',
                            version=__version__,
                            description='Effective Computation in Physics',
                            long_description=long_description,
                            author='Anthony Scopatz and Kathryn D. Huff',
                            author_email='koolkatz@gmail.com',
                            url='http://physics.codes',
                            classifiers=[
                                'License :: OSI Approved :: BSD License',
                                'Intended Audience :: Developers',
                                'Intended Audience :: Science/Research',
                                'Natural Language :: English',
                                'Programming Language :: Python :: 3',
                            ],
                            license='BSD-3-Clause',
                            install_requires=install_requires,
                            tests_require=tests_require,
                            python_requires='>=3',
                            setup_requires=setup_requires,
                            zip_safe=False,
                            packages=['compphys', 'compphys.more', 'compphys.tests'],
                            package_dir={
                                'compphys': 'compphys',
                                'compphys.more': 'compphys/more',
                                'compphys.tests': 'compphys/tests',
                                },
                            include_package_data=True,
                        )
                        

Contents of _version.py:



                        __version_info__ = (0, 1, 0, 'a0')
                        __version__ = '.'.join(map(str, __version_info__[:3]))
                        if len(__version_info__) == 4:
                            __version__ += __version_info__[-1]
                        

Contents of CHANGELOG.md:



                        # Changelog
                        All notable changes to this project will be documented in this file.

                        The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
                        and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

                        ## [Unreleased]

                        ## [0.1.0] - 2015-10-06
                        ### Added
                        - Answer "Should you ever rewrite a change log?".

                        ### Changed
                        - Improve argument against commit logs.
                        - Start following [SemVer](http://semver.org) properly.

                        ## [0.0.8] - 2015-02-17
                        ### Changed
                        - Update year to match in every README example.
                        - Reluctantly stop making fun of Brits only, since most of the world
                            writes dates in a strange way.

                        ### Fixed
                        - Fix typos in recent README changes.
                        - Update outdated unreleased diff link.

                        ## [0.0.7] - 2015-02-16
                        ### Added
                        - Link, and make it obvious that date format is ISO 8601.

                        ### Changed
                        - Clarified the section on "Is there a standard change log format?".

                        ### Fixed
                        - Fix Markdown links to tag comparison URL with footnote-style links.

                        ## [0.0.6] - 2014-12-12
                        ### Added
                        - README section on "yanked" releases.

                        ## [0.0.5] - 2014-08-09
                        ### Added
                        - Markdown links to version tags on release headings.
                        - Unreleased section to gather unreleased changes and encourage note
                        keeping prior to releases.

                        ## [0.0.4] - 2014-08-09
                        ### Added
                        - Better explanation of the difference between the file ("CHANGELOG")
                        and its function "the change log".

                        ### Changed
                        - Refer to a "change log" instead of a "CHANGELOG" throughout the site
                        to differentiate between the file and the purpose of the file — the
                        logging of changes.

                        ### Removed
                        - Remove empty sections from CHANGELOG, they occupy too much space and
                        create too much noise in the file. People will have to assume that the
                        missing sections were intentionally left out because they contained no
                        notable changes.

                        ## [0.0.3] - 2014-08-09
                        ### Added
                        - "Why should I care?" section mentioning The Changelog podcast.

                        ## [0.0.2] - 2014-07-10
                        ### Added
                        - Explanation of the recommended reverse chronological release ordering.

                        ## 0.0.1 - 2014-05-31
                        ### Added
                        - This CHANGELOG file to hopefully serve as an evolving example of a
                            standardized open source project CHANGELOG.
                        - CNAME file to enable GitHub Pages custom domain
                        - README now contains answers to common questions about CHANGELOGs
                        - Good examples and basic guidelines, including proper date formatting.
                        - Counter-examples: "What makes unicorns cry?"

                        [Unreleased]: https://github.com/olivierlacan/keep-a-changelog/compare/v1.0.0...HEAD
                        [0.1.0]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.8...v0.1.0
                        [0.0.8]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.7...v0.0.8
                        [0.0.7]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.6...v0.0.7
                        [0.0.6]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.5...v0.0.6
                        [0.0.5]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.4...v0.0.5
                        [0.0.4]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.3...v0.0.4
                        [0.0.3]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.2...v0.0.3
                        [0.0.2]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.1...v0.0.2
                        
  1. Install the package from source: python setup.py install
  2. Or, install using pip: pip install <path>
  3. Can also install in development mode: pip install -e <path>

Semantic Versioning: Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

To start: set initial development release at 0.1.0 and increment minor version for subsequent releases.