diff options
Diffstat (limited to 'blog/git-server/index.org')
| -rw-r--r-- | blog/git-server/index.org | 617 |
1 files changed, 617 insertions, 0 deletions
diff --git a/blog/git-server/index.org b/blog/git-server/index.org new file mode 100644 index 0000000..c716484 --- /dev/null +++ b/blog/git-server/index.org @@ -0,0 +1,617 @@ +#+title: Self-Hosting a Personal Git Server +#+date: 2022-07-01 +#+description: A guide to self-hosting a Git server on your own server. +#+filetags: :selfhosting: + +* My Approach to Self-Hosting Git +I have often tried to self-host my Git repositories, but have always +fallen short when I tried to find a suitable web interface to show on +the front-end. + +After a few years, I have finally found a combination of methods that +allow me to easily self-host my projects, view them on the web, and +access them from anywhere. + +Before I dive into the details, I want to state a high-level summary of +my self-hosted Git approach: + +- This method uses the =ssh://= (read & write) and =git://= (read-only) + protocols for push and pull access. + - For the =git://= protocol, I create a =git-daemon-export-ok= file in + any repository that I want to be cloneable by anyone. + - The web interface I am using (=cgit=) allows simple HTTP cloning by + default. I do not disable this setting as I want beginners to be + able to clone one of my repositories even if they don't know the + proper method. +- I am not enabling Smart HTTPS for any repositories. Updates to + repositories must be pushed via SSH. +- Beyond the actual repository management, I am using =cgit= for the + front-end web interface. + - If you use the =scan-path=<path>= configuration in the =cgitrc= + configuration file to automatically find repositories, you can't + exclude a repository from =cgit= if it's stored within the path that + =cgit= reads. To host private repositories, you'd need to set up + another directory that =cgit= can't read. + +* Assumptions +For the purposes of this walkthrough, I am assuming you have a URL +(=git.example.com=) or IP address (=207.84.26.991=) addressed to the +server that you will be using to host your git repositories. + +* Adding a Git User +In order to use the SSH method associated with git, we will need to add +a user named =git=. If you have used the SSH method for other git +hosting sites, you are probably used to the following syntax: + +#+begin_src sh +git clone [user@]server:project.git +#+end_src + +The syntax above is an =scp=-like syntax for using SSH on the =git= user +on the server to access your repository. + +Let's delete any remnants of an old =git= user, if any, and create the +new user account: + +#+begin_src sh +sudo deluser --remove-home git +sudo adduser git +#+end_src + +** Import Your SSH Keys to the Git User +Once the =git= user is created, you will need to copy your public SSH +key on your local development machine to the =git= user on the server. + +If you don't have an SSH key yet, create one with this command: + +#+begin_src sh +ssh-keygen +#+end_src + +Once you create the key pair, the public should be saved to +=~/.ssh/id_rsa.pub=. + +If your server still has password-based authentication available, you +can copy it over to your user's home directory like this: + +#+begin_src sh +ssh-copy-id git@server +#+end_src + +Otherwise, copy it over to any user that you can access. + +#+begin_src sh +scp ~/.ssh/id_rsa.pub your_user@your_server: +#+end_src + +Once on the server, you will need to copy the contents into the =git= +user's =authorized_keys= file: + +#+begin_src sh +cat id_rsa.pub > /home/git/.ssh/authorized_keys +#+end_src + +** (Optional) Disable Password-Based SSH +If you want to lock down your server and ensure that no one can +authenticate in via SSH with a password, you will need to edit your SSH +configuration. + +#+begin_src sh +sudo nano /etc/ssh/sshd_config +#+end_src + +Within this file, find the following settings and set them to the values +I am showing below: + +#+begin_src conf +PermitRootLogin no +PasswordAuthentication no +AuthenticationMethods publickey +#+end_src + +You may have other Authentication Methods required in your personal +set-up, so the key here is just to ensure that =AuthenticationMethods= +does not allow passwords. + +*** Setting up the Base Directory +Now that we have set up a =git= user to handle all transport methods, we +need to set up the directory that we will be using as our base of all +repositories. + +In my case, I am using =/git= as my source folder. To create this folder +and assign it to the user we created, execute the following commands: + +#+begin_src sh +sudo mkdir /git +sudo chown -R git:git /git +#+end_src + +*** Creating a Test Repository +On your server, switch over to the =git= user in order to start managing +git files. + +#+begin_src sh +su git +#+end_src + +Once logged-in as the =git= user, go to your base directory and create a +test repository. + +#+begin_src sh +cd /git +mkdir test.git && cd test.git +git init --bare +#+end_src + +If you want to make this repo viewable/cloneable to the public via the +=git://= protocol, you need to create a =git-daemon-export-ok= file +inside the repository. + +#+begin_src sh +touch git-daemon-export-ok +#+end_src + +* Change the Login Shell for =git= +To make sure that the =git= user is only used for git operations and +nothing else, you need to change the user's login shell. To do this, +simply use the =chsh= command: + +#+begin_src sh +sudo chsh git +#+end_src + +The interactive prompt will ask which shell you want the =git= user to +use. You must use the following value: + +#+begin_src sh +/usr/bin/git-shell +#+end_src + +Once done, no one will be able to SSH to the =git= user or execute +commands other than the standard git commands. + +* Opening the Firewall +Don't forget to open up ports on the device firewall and network +firewall if you want to access these repositories publicly. If you're +using default ports, forward ports =22= (ssh) and =9418= (git) from your +router to your server's IP address. + +If your server also has a firewall, ensure that the firewall allows the +same ports that are forwarded from the router. For example, if you use +=ufw=: + +#+begin_src sh +sudo ufw allow 22 +sudo ufw allow 9418 +#+end_src + +** Non-Standard SSH Ports +If you use a non-standard port for SSH, such as =9876=, you will need to +create an SSH configuration file on your local development machine in +order to connect to your server's git repositories. + +To do this, you'll need to define your custom port on your client +machine in your =~/.ssh/config= file: + +#+begin_src sh +nano ~/.ssh/config +#+end_src + +#+begin_src conf +Host git.example.com + # HostName can be a URL or an IP address + HostName git.example.com + Port 9876 + User git +#+end_src + +** Testing SSH +There are two main syntaxes you can use to manage git over SSH: + +- =git clone [user@]server:project.git= +- =git clone ssh://[user@]server/project.git= + +I prefer the first, which is an =scp=-like syntax. To test it, try to +clone the test repository you set up on the server: + +#+begin_src sh +git clone git@git.example.com:/git/test.git +#+end_src + +* Enabling Read-Only Access +If you want people to be able to clone any repository where you've +placed a =git-daemon-export-ok= file, you will need to start the git +daemon. + +To do this on a system with =systemd=, create a service file: + +#+begin_src sh +sudo nano /etc/systemd/system/git-daemon.service +#+end_src + +Inside the =git-daemon.service= file, paste the following: + +#+begin_src conf +[Unit] +Description=Start Git Daemon + +[Service] +ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/git/ /git/ + +Restart=always +RestartSec=500ms + +StandardOutput=syslog +StandardError=syslog +SyslogIdentifier=git-daemon + +User=git +Group=git + +[Install] +WantedBy=multi-user.target +#+end_src + +Once created, enable and start the service: + +#+begin_src sh +sudo systemctl enable git-daemon.service +sudo systemctl start git-daemon.service +#+end_src + +To clone read-only via the =git://= protocol, you can use the following +syntax: + +#+begin_src sh +git clone git://git.example.com/test.git +#+end_src + +* Migrating Repositories +At this point, we have a working git server that works with both SSH and +read-only access. + +For each of the repositories I had hosted a different provider, I +executed the following commands in order to place a copy on my server as +my new source of truth: + +Server: + +#+begin_src sh +su git +mkdir /git/<REPOSITORY_NAME>.git && cd /git/<REPOSITORY_NAME>.git +git init --bare + +# If you want to make this repo viewable/cloneable to the public +touch git-daemon-export-ok +#+end_src + +Client: + +#+begin_src sh +git clone git@<PREVIOUS_HOST>:<REPOSITORY_NAME> +git remote set-url origin git@git.EXAMPLE.COM:/git/<REPOSITORY_NAME>.git +git push +#+end_src + +* Optional Web View: =cgit= +If you want a web viewer for your repositories, you can use various +tools, such as =gitweb=, =cgit=, or =klaus=. I chose =cgit= due to its +simple interface and fairly easy set-up (compared to others). Not to +mention that the [[https://git.kernel.org/][Linux kernel uses =cgit=]]. + +** Docker Compose +Instead of using my previous method of using a =docker run= command, +I've updated this section to use =docker-compose= instead for an easier +installation and simpler management and configuration. + +In order to use Docker Compose, you will set up a =docker-compose.yml= +file to automatically connect resources like the repositories, =cgitrc=, +and various files or folders to the =cgit= container you're creating: + +#+begin_src sh +mkdir ~/cgit && cd ~/cgit +nano docker-compose.yml +#+end_src + +#+begin_src conf +# docker-compose.yml +version: '3' + +services: + cgit: + image: invokr/cgit + volumes: + - /git:/git + - ./cgitrc:/etc/cgitrc + - ./logo.png:/var/www/htdocs/cgit/logo.png + - ./favicon.png:/var/www/htdocs/cgit/favicon.png + - ./filters:/var/www/htdocs/cgit/filters + ports: + - "8763:80" + restart: always +#+end_src + +Then, just start the container: + +#+begin_src sh +sudo docker-compose up -d +#+end_src + +Once it's finished installing, you can access the site at +=<SERVER_IP>:8763= or use a reverse-proxy service to forward =cgit= to a +URL, such as =git.example.com=. See the next section for more details on +reverse proxying a URL to a local port. + +** Nginx Reverse Proxy +I am using Nginx as my reverse proxy so that the =cgit= Docker container +can use =git.example.com= as its URL. To do so, I simply created the +following configuration file: + +#+begin_src sh +sudo nano /etc/nginx/sites-available/git.example.com +#+end_src + +#+begin_src conf +server { + listen 80; + server_name git.example.com; + + if ($host = git.example.com) { + return 301 https://$host$request_uri; + } + + return 404; +} + +server { + server_name git.example.com; + listen 443 ssl http2; + + location / { + # The final `/` is important. + proxy_pass http://localhost:8763/; + add_header X-Frame-Options SAMEORIGIN; + add_header X-XSS-Protection "1; mode=block"; + proxy_redirect off; + proxy_buffering off; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Port $server_port; + } + + # INCLUDE ANY SSL CERTS HERE + include /etc/letsencrypt/options-ssl-nginx.conf; + ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; +} +#+end_src + +Once created, symlink it and restart the web server. + +#+begin_src sh +sudo ln -s /etc/nginx/sites-available/git.example.com /etc/nginx/sites-enabled/ +sudo systemctl restart nginx.service +#+end_src + +As we can see below, my site at =git.example.com= is available and +running: + +** Settings Up Git Details +Once you have =cgit= running, you can add some small details, such as +repository owners and descriptions by editing the following files within +each repository. + +Alternatively, you can use the =cgitrc= file to edit these details if +you only care to edit them for the purpose of seeing them on your +website. + +The =description= file within the repository on your server will display +the description online. + +#+begin_src sh +cd /git/example.git +nano description +#+end_src + +You can add a =[gitweb]= block to the =config= file in order to display +the owner of the repository. + +#+begin_src sh +cd /git/example.git +nano config +#+end_src + +#+begin_src conf +[gitweb] + owner = "YourName" +#+end_src + +Note that you can ignore the configuration within each repository and +simply set up this information in the =cgitrc= file, if you want to do +it that way. + +** Editing =cgit= +In order to edit certain items within =cgit=, you need to edit the +=cgitrc= file. + +#+begin_src sh +nano ~/cgit/cgitrc +#+end_src + +Below is an example configuration for =cgitrc=. You can find all the +configuration options within the [configuration manual] +([[https://git.zx2c4.com/cgit/plain/cgitrc.5.txt]]). + +#+begin_src conf +css=/cgit.css +logo=/logo.png +favicon=/favicon.png +robots=noindex, nofollow + +enable-index-links=1 +enable-commit-graph=1 +enable-blame=1 +enable-log-filecount=1 +enable-log-linecount=1 +enable-git-config=1 + +clone-url=git://git.example.com/$CGIT_REPO_URL ssh://git@git.example.com:/git/$CGIT_REPO_URL + +root-title=My Git Website +root-desc=My personal git repositories. + +# Allow download of tar.gz, tar.bz2 and zip-files +snapshots=tar.gz tar.bz2 zip + +## +## List of common mimetypes +## +mimetype.gif=image/gif +mimetype.html=text/html +mimetype.jpg=image/jpeg +mimetype.jpeg=image/jpeg +mimetype.pdf=application/pdf +mimetype.png=image/png +mimetype.svg=image/svg+xml + +# Highlight source code +# source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.sh +source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py + +# Format markdown, restructuredtext, manpages, text files, and html files +# through the right converters +about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh + +## +## Search for these files in the root of the default branch of repositories +## for coming up with the about page: +## +readme=:README.md +readme=:readme.md +readme=:README.mkd +readme=:readme.mkd +readme=:README.rst +readme=:readme.rst +readme=:README.html +readme=:readme.html +readme=:README.htm +readme=:readme.htm +readme=:README.txt +readme=:readme.txt +readme=:README +readme=:readme + +# Repositories + +# Uncomment the following line to scan a path instead of adding repositories manually +# scan-path=/git + +## Test Section +section=git/test-section + +repo.url=test.git +repo.path=/git/test.git +repo.readme=:README.md +repo.owner=John Doe +repo.desc=An example repository! +#+end_src + +** Final Fixes: Syntax Highlighting & README Rendering +After completing my initial install and playing around with it for a few +days, I noticed two issues: + +1. Syntax highlighting did not work when viewing the source code within + a file. +2. The =about= tab within a repository was not rendered to HTML. + +The following process fixes these issues. To start, let's go to the +=cgit= directory where we were editing our configuration file earlier. + +#+begin_src sh +cd ~/cgit +#+end_src + +In here, create two folders that will hold our syntax files: + +#+begin_src sh +mkdir filters && mkdir filters/html-converters && cd filters +#+end_src + +Next, download the default filters: + +#+begin_src sh +curl https://git.zx2c4.com/cgit/plain/filters/about-formatting.sh > about-formatting.sh +chmod 755 about-formatting.sh +curl https://git.zx2c4.com/cgit/plain/filters/syntax-highlighting.py > syntax-highlighting.py +chmod 755 syntax-highlighting.py +#+end_src + +Finally, download the HTML conversion files you need. The example below +downloads the Markdown converter: + +#+begin_src sh +cd html-converters +curl https://git.zx2c4.com/cgit/plain/filters/html-converters/md2html > md2html +chmod 755 md2html +#+end_src + +If you need other filters or html-converters found within +[[https://git.zx2c4.com/cgit/tree/filters][the cgit project files]], +repeat the =curl= and =chmod= process above for whichever files you +need. + +However, formatting will not work quite yet since the Docker cgit +container we're using doesn't have the formatting package installed. You +can install this easily by install Python 3+ and the =pygments= package: + +#+begin_src sh +# Enter the container's command line +sudo docker exec -it cgit bash +#+end_src + +#+begin_src sh +# Install the necessary packages and then exit +yum update -y && \ +yum upgrade -y && \ +yum install python3 python3-pip -y && \ +pip3 install markdown pygments && \ +exit +#+end_src + +*You will need to enter the cgit docker container and re-run these =yum= +commands every time you kill and restart the container!* + +If not done already, we need to add the following variables to our +=cgitrc= file in order for =cgit= to know where our filtering files are: + +#+begin_src conf +# Highlight source code with python pygments-based highlighter +source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py + +# Format markdown, restructuredtext, manpages, text files, and html files +# through the right converters +about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh +#+end_src + +Now you should see that syntax highlighting and README rendering to the +=about= tab is fixed. + +** Theming +I won't go into much detail in this section, but you can fully theme +your installation of =cgit= since you have access to the =cgit.css= file +in your web root. This is another file you can add as a volume to the +=docker-compose.yml= file if you want to edit this without entering the +container's command line. + +*** :warning: Remember to Back Up Your Data! +The last thing to note is that running services on your own equipment +means that you're assuming a level of risk that exists regarding data +loss, catastrophes, etc. In order to reduce the impact of any such +occurrence, I suggest backing up your data regularly. + +Backups can be automated via =cron=, by hooking your base directory up +to a cloud provider, or even setting up hooks to push all repository +info to git mirrors on other git hosts. Whatever the method, make sure +that your data doesn't vanish in the event that your drives or servers +fail. |