Building custom libraries for your code base becomes essential as soon as you have multiple Repos of code that interact with one another. Otherwise you will end up duplicating code, causing horrible maintainability problems down the road. For our python stack we wanted to have a common code library, stored in a private GitHub Repo, that could be easily installed using pip.
Why Blog This?
I spent a significant amount of time looking for an all inclusive, straightforward tutorial on how to accomplish this and could not find one. Once I finally pieced enough together and got things to work, I figured I would write up our solution, and hopefully save someone else some time.
First off, you will need to have your local ssh keys in your GitHub account, this will allow pip access via your ssh credentials. To learn how to do this, follow the instructions here.
Setting up the Library
You should organize your library folder structure as so:
$ tree . ├── library_name │ ├── __init__.py │ ├── another_file.py │ └── some_module_group │ ├── __init__.py │ └── some_file.py └── setup.py
Once we pip install this library, you will be able to use it like so:
from library_name.some_module_group.some_file import SomeClass from library_name.another_file import some_function # ... whatever code using these imported items
The next important bit is organizing your setup.py file. An absolute minimalistic file looks like:
import os from setuptools import setup setup( name="SomeLibrary", version="0.0.1", packages=["library_name"], # can have multiple "libraries using this install_requires=['library_dependency==2.8.2', 'another_one==1.1.0'] )
There are a whole bunch more handy parameters to the setup function, like author name, description, license, url, etc., and you can read about them here. I’ve only listed the ones I needed to get started.
Once you have your library set up you should git tag a version (this will be very useful later) and push it to GitHub (for our example, to a Repo we named SomeLibrary).
$ git add library_name/ setup.py $ git commit -m "Initial library setup for pip install." $ git tag -a 0.0.1 -m "First version available for pip install." $ git push && git push --tags
Using the Library
Now in a project that wants to use this code we can:
$ pip install -e git+ssh://email@example.com/you/SomeLibrary.firstname.lastname@example.org#egg=SomeLibrary
Now you can use the libraries modules like any other pip install (as shown above). Importantly, for us, this also works in requirements.txt files for pip install -r commands and shows up in a pip freeze to reflect that.
This didn’t take a huge amount of time to figure out, but I did have to jump around a bunch to get to it. Hopefully this saves you some time.
GitHub, Libraries, pip, Python