diff options
Diffstat (limited to 'content/blog/2022-07-01-git-server.md')
| -rw-r--r-- | content/blog/2022-07-01-git-server.md | 655 |
1 files changed, 655 insertions, 0 deletions
diff --git a/content/blog/2022-07-01-git-server.md b/content/blog/2022-07-01-git-server.md new file mode 100644 index 0000000..61c4049 --- /dev/null +++ b/content/blog/2022-07-01-git-server.md @@ -0,0 +1,655 @@ ++++ +date = 2022-07-01 +title = "Self-Hosting a Personal Git Server" +description = "" +draft = false ++++ + +# My Approach to Self-Hosting Git + +I have often tried to self-host my Git repositories, but have always +fallen short when I tried to find a suitable web interface to show on +the front-end. + +After a few years, I have finally found a combination of methods that +allow me to easily self-host my projects, view them on the web, and +access them from anywhere. + +Before I dive into the details, I want to state a high-level summary of +my self-hosted Git approach: + +- This method uses the `ssh://` (read & write) and + `git://` (read-only) protocols for push and pull access. + - For the `git://` protocol, I create a + `git-daemon-export-ok` file in any repository that I + want to be cloneable by anyone. + - The web interface I am using (`cgit`) allows simple + HTTP cloning by default. I do not disable this setting as I want + beginners to be able to clone one of my repositories even if + they don\'t know the proper method. +- I am not enabling Smart HTTPS for any repositories. Updates to + repositories must be pushed via SSH. +- Beyond the actual repository management, I am using + `cgit` for the front-end web interface. + - If you use the `scan-path=<path>` configuration in + the `cgitrc` configuration file to automatically find + repositories, you can\'t exclude a repository from + `cgit` if it\'s stored within the path that + `cgit` reads. To host private repositories, you\'d + need to set up another directory that `cgit` can\'t + read. + +# Assumptions + +For the purposes of this walkthrough, I am assuming you have a URL +(`git.example.com`) or IP address +(`207.84.26.991`) addressed to the server that you will be +using to host your git repositories. + +# Adding a Git User + +In order to use the SSH method associated with git, we will need to add +a user named `git`. If you have used the SSH method for other +git hosting sites, you are probably used to the following syntax: + +```sh +git clone [user@]server:project.git +``` + +The syntax above is an `scp`-like syntax for using SSH on the +`git` user on the server to access your repository. + +Let\'s delete any remnants of an old `git` user, if any, and +create the new user account: + +```sh +sudo deluser --remove-home git +sudo adduser git +``` + +## Import Your SSH Keys to the Git User + +Once the `git` user is created, you will need to copy your +public SSH key on your local development machine to the `git` +user on the server. + +If you don\'t have an SSH key yet, create one with this command: + +```sh +ssh-keygen +``` + +Once you create the key pair, the public should be saved to +`~/.ssh/id_rsa.pub`. + +If your server still has password-based authentication available, you +can copy it over to your user\'s home directory like this: + +```sh +ssh-copy-id git@server +``` + +Otherwise, copy it over to any user that you can access. + +```sh +scp ~/.ssh/id_rsa.pub your_user@your_server: +``` + +Once on the server, you will need to copy the contents into the +`git` user\'s `authorized_keys` file: + +```sh +cat id_rsa.pub > /home/git/.ssh/authorized_keys +``` + +## (Optional) Disable Password-Based SSH + +If you want to lock down your server and ensure that no one can +authenticate in via SSH with a password, you will need to edit your SSH +configuration. + +```sh +sudo nano /etc/ssh/sshd_config +``` + +Within this file, find the following settings and set them to the values +I am showing below: + +``` conf +PermitRootLogin no +PasswordAuthentication no +AuthenticationMethods publickey +``` + +You may have other Authentication Methods required in your personal +set-up, so the key here is just to ensure that +`AuthenticationMethods` does not allow passwords. + +### Setting up the Base Directory + +Now that we have set up a `git` user to handle all transport +methods, we need to set up the directory that we will be using as our +base of all repositories. + +In my case, I am using `/git` as my source folder. To create +this folder and assign it to the user we created, execute the following +commands: + +```sh +sudo mkdir /git +sudo chown -R git:git /git +``` + +### Creating a Test Repository + +On your server, switch over to the `git` user in order to +start managing git files. + +```sh +su git +``` + +Once logged-in as the `git` user, go to your base directory +and create a test repository. + +```sh +cd /git +mkdir test.git && cd test.git +git init --bare +``` + +If you want to make this repo viewable/cloneable to the public via the +`git://` protocol, you need to create a +`git-daemon-export-ok` file inside the repository. + +```sh +touch git-daemon-export-ok +``` + +# Change the Login Shell for `git` + +To make sure that the `git` user is only used for git +operations and nothing else, you need to change the user\'s login shell. +To do this, simply use the `chsh` command: + +```sh +sudo chsh git +``` + +The interactive prompt will ask which shell you want the +`git` user to use. You must use the following value: + +```sh +/usr/bin/git-shell +``` + +Once done, no one will be able to SSH to the `git` user or +execute commands other than the standard git commands. + +# Opening the Firewall + +Don\'t forget to open up ports on the device firewall and network +firewall if you want to access these repositories publicly. If you\'re +using default ports, forward ports `22` (ssh) and +`9418` (git) from your router to your server\'s IP address. + +If your server also has a firewall, ensure that the firewall allows the +same ports that are forwarded from the router. For example, if you use +`ufw`: + +```sh +sudo ufw allow 22 +sudo ufw allow 9418 +``` + +## Non-Standard SSH Ports + +If you use a non-standard port for SSH, such as `9876`, you +will need to create an SSH configuration file on your local development +machine in order to connect to your server\'s git repositories. + +To do this, you\'ll need to define your custom port on your client +machine in your `~/.ssh/config` file: + +```sh +nano ~/.ssh/config +``` + +``` conf +Host git.example.com + # HostName can be a URL or an IP address + HostName git.example.com + Port 9876 + User git +``` + +## Testing SSH + +There are two main syntaxes you can use to manage git over SSH: + +- `git clone [user@]server:project.git` +- `git clone ssh://[user@]server/project.git` + +I prefer the first, which is an `scp`-like syntax. To test +it, try to clone the test repository you set up on the server: + +```sh +git clone git@git.example.com:/git/test.git +``` + +# Enabling Read-Only Access + +If you want people to be able to clone any repository where you\'ve +placed a `git-daemon-export-ok` file, you will need to start +the git daemon. + +To do this on a system with `systemd`, create a service file: + +```sh +sudo nano /etc/systemd/system/git-daemon.service +``` + +Inside the `git-daemon.service` file, paste the following: + +``` conf +[Unit] +Description=Start Git Daemon + +[Service] +ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/git/ /git/ + +Restart=always +RestartSec=500ms + +StandardOutput=syslog +StandardError=syslog +SyslogIdentifier=git-daemon + +User=git +Group=git + +[Install] +WantedBy=multi-user.target +``` + +Once created, enable and start the service: + +```sh +sudo systemctl enable git-daemon.service +sudo systemctl start git-daemon.service +``` + +To clone read-only via the `git://` protocol, you can use the +following syntax: + +```sh +git clone git://git.example.com/test.git +``` + +# Migrating Repositories + +At this point, we have a working git server that works with both SSH and +read-only access. + +For each of the repositories I had hosted a different provider, I +executed the following commands in order to place a copy on my server as +my new source of truth: + +Server: + +```sh +su git +mkdir /git/<REPOSITORY_NAME>.git && cd /git/<REPOSITORY_NAME>.git +git init --bare + +# If you want to make this repo viewable/cloneable to the public +touch git-daemon-export-ok +``` + +Client: + +```sh +git clone git@<PREVIOUS_HOST>:<REPOSITORY_NAME> +git remote set-url origin git@git.EXAMPLE.COM:/git/<REPOSITORY_NAME>.git +git push +``` + +# Optional Web View: `cgit` + +If you want a web viewer for your repositories, you can use various +tools, such as `gitweb`, `cgit`, or +`klaus`. I chose `cgit` due to its simple +interface and fairly easy set-up (compared to others). Not to mention +that the [Linux kernel uses `cgit`](https://git.kernel.org/). + +## Docker Compose + +Instead of using my previous method of using a `docker run` +command, I\'ve updated this section to use `docker-compose` +instead for an easier installation and simpler management and +configuration. + +In order to use Docker Compose, you will set up a +`docker-compose.yml` file to automatically connect resources +like the repositories, `cgitrc`, and various files or folders +to the `cgit` container you\'re creating: + +```sh +mkdir ~/cgit && cd ~/cgit +nano docker-compose.yml +``` + +``` conf +# docker-compose.yml +version: '3' + +services: + cgit: + image: invokr/cgit + volumes: + - /git:/git + - ./cgitrc:/etc/cgitrc + - ./logo.png:/var/www/htdocs/cgit/logo.png + - ./favicon.png:/var/www/htdocs/cgit/favicon.png + - ./filters:/var/www/htdocs/cgit/filters + ports: + - "8763:80" + restart: always +``` + +Then, just start the container: + +```sh +sudo docker-compose up -d +``` + +Once it\'s finished installing, you can access the site at +`<SERVER_IP>:8763` or use a reverse-proxy service to forward +`cgit` to a URL, such as `git.example.com`. See +the next section for more details on reverse proxying a URL to a local +port. + +## Nginx Reverse Proxy + +I am using Nginx as my reverse proxy so that the `cgit` +Docker container can use `git.example.com` as its URL. To do +so, I simply created the following configuration file: + +```sh +sudo nano /etc/nginx/sites-available/git.example.com +``` + +``` conf +server { + listen 80; + server_name git.example.com; + + if ($host = git.example.com) { + return 301 https://$host$request_uri; + } + + return 404; +} + +server { + server_name git.example.com; + listen 443 ssl http2; + + location / { + # The final `/` is important. + proxy_pass http://localhost:8763/; + add_header X-Frame-Options SAMEORIGIN; + add_header X-XSS-Protection "1; mode=block"; + proxy_redirect off; + proxy_buffering off; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Port $server_port; + } + + # INCLUDE ANY SSL CERTS HERE + include /etc/letsencrypt/options-ssl-nginx.conf; + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; +} +``` + +Once created, symlink it and restart the web server. + +```sh +sudo ln -s /etc/nginx/sites-available/git.example.com /etc/nginx/sites-enabled/ +sudo systemctl restart nginx.service +``` + +As we can see below, my site at `git.example.com` is +available and running: + +## Settings Up Git Details + +Once you have `cgit` running, you can add some small details, +such as repository owners and descriptions by editing the following +files within each repository. + +Alternatively, you can use the `cgitrc` file to edit these +details if you only care to edit them for the purpose of seeing them on +your website. + +The `description` file within the repository on your server +will display the description online. + +```sh +cd /git/example.git +nano description +``` + +You can add a `[gitweb]` block to the `config` +file in order to display the owner of the repository. + +```sh +cd /git/example.git +nano config +``` + +``` conf +[gitweb] + owner = "YourName" +``` + +Note that you can ignore the configuration within each repository and +simply set up this information in the `cgitrc` file, if you +want to do it that way. + +## Editing `cgit` + +In order to edit certain items within `cgit`, you need to +edit the `cgitrc` file. + +```sh +nano ~/cgit/cgitrc +``` + +Below is an example configuration for `cgitrc`. You can find +all the configuration options within the \[configuration manual\] +(<https://git.zx2c4.com/cgit/plain/cgitrc.5.txt>). + +``` conf +css=/cgit.css +logo=/logo.png +favicon=/favicon.png +robots=noindex, nofollow + +enable-index-links=1 +enable-commit-graph=1 +enable-blame=1 +enable-log-filecount=1 +enable-log-linecount=1 +enable-git-config=1 + +clone-url=git://git.example.com/$CGIT_REPO_URL ssh://git@git.example.com:/git/$CGIT_REPO_URL + +root-title=My Git Website +root-desc=My personal git repositories. + +# Allow download of tar.gz, tar.bz2 and zip-files +snapshots=tar.gz tar.bz2 zip + +## +## List of common mimetypes +## +mimetype.gif=image/gif +mimetype.html=text/html +mimetype.jpg=image/jpeg +mimetype.jpeg=image/jpeg +mimetype.pdf=application/pdf +mimetype.png=image/png +mimetype.svg=image/svg+xml + +# Highlight source code +# source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.sh +source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py + +# Format markdown, restructuredtext, manpages, text files, and html files +# through the right converters +about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh + +## +## Search for these files in the root of the default branch of repositories +## for coming up with the about page: +## +readme=:README.md +readme=:readme.md +readme=:README.mkd +readme=:readme.mkd +readme=:README.rst +readme=:readme.rst +readme=:README.html +readme=:readme.html +readme=:README.htm +readme=:readme.htm +readme=:README.txt +readme=:readme.txt +readme=:README +readme=:readme + +# Repositories + +# Uncomment the following line to scan a path instead of adding repositories manually +# scan-path=/git + +## Test Section +section=git/test-section + +repo.url=test.git +repo.path=/git/test.git +repo.readme=:README.md +repo.owner=John Doe +repo.desc=An example repository! +``` + +## Final Fixes: Syntax Highlighting & README Rendering + +After completing my initial install and playing around with it for a few +days, I noticed two issues: + +1. Syntax highlighting did not work when viewing the source code within + a file. +2. The `about` tab within a repository was not rendered to + HTML. + +The following process fixes these issues. To start, let\'s go to the +`cgit` directory where we were editing our configuration file +earlier. + +```sh +cd ~/cgit +``` + +In here, create two folders that will hold our syntax files: + +```sh +mkdir filters && mkdir filters/html-converters && cd filters +``` + +Next, download the default filters: + +```sh +curl https://git.zx2c4.com/cgit/plain/filters/about-formatting.sh > about-formatting.sh +chmod 755 about-formatting.sh +curl https://git.zx2c4.com/cgit/plain/filters/syntax-highlighting.py > syntax-highlighting.py +chmod 755 syntax-highlighting.py +``` + +Finally, download the HTML conversion files you need. The example below +downloads the Markdown converter: + +```sh +cd html-converters +curl https://git.zx2c4.com/cgit/plain/filters/html-converters/md2html > md2html +chmod 755 md2html +``` + +If you need other filters or html-converters found within [the cgit +project files](https://git.zx2c4.com/cgit/tree/filters), repeat the +`curl` and `chmod` process above for whichever +files you need. + +However, formatting will not work quite yet since the Docker cgit +container we\'re using doesn\'t have the formatting package installed. +You can install this easily by install Python 3+ and the +`pygments` package: + +```sh +# Enter the container's command line +sudo docker exec -it cgit bash +``` + +```sh +# Install the necessary packages and then exit +yum update -y && \ +yum upgrade -y && \ +yum install python3 python3-pip -y && \ +pip3 install markdown pygments && \ +exit +``` + +**You will need to enter the cgit docker container and re-run these +`yum` commands every time you kill and restart the +container!** + +If not done already, we need to add the following variables to our +`cgitrc` file in order for `cgit` to know where +our filtering files are: + +``` conf +# Highlight source code with python pygments-based highlighter +source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py + +# Format markdown, restructuredtext, manpages, text files, and html files +# through the right converters +about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh +``` + +Now you should see that syntax highlighting and README rendering to the +`about` tab is fixed. + +## Theming + +I won\'t go into much detail in this section, but you can fully theme +your installation of `cgit` since you have access to the +`cgit.css` file in your web root. This is another file you +can add as a volume to the `docker-compose.yml` file if you +want to edit this without entering the container\'s command line. + +### :warning: Remember to Back Up Your Data! + +The last thing to note is that running services on your own equipment +means that you\'re assuming a level of risk that exists regarding data +loss, catastrophes, etc. In order to reduce the impact of any such +occurrence, I suggest backing up your data regularly. + +Backups can be automated via `cron`, by hooking your base +directory up to a cloud provider, or even setting up hooks to push all +repository info to git mirrors on other git hosts. Whatever the method, +make sure that your data doesn\'t vanish in the event that your drives +or servers fail. |