diff options
Diffstat (limited to 'blog/2022-07-01-git-server.org')
| -rw-r--r-- | blog/2022-07-01-git-server.org | 678 |
1 files changed, 0 insertions, 678 deletions
diff --git a/blog/2022-07-01-git-server.org b/blog/2022-07-01-git-server.org deleted file mode 100644 index 049603e..0000000 --- a/blog/2022-07-01-git-server.org +++ /dev/null @@ -1,678 +0,0 @@ -#+title: Self-Hosting a Personal Git Server -#+date: 2022-07-01 - -** My Approach to Self-Hosting Git -:PROPERTIES: -:CUSTOM_ID: my-approach-to-self-hosting-git -:END: -I have often tried to self-host my Git repositories, but have always -fallen short when I tried to find a suitable web interface to show on -the front-end. - -After a few years, I have finally found a combination of methods that -allow me to easily self-host my projects, view them on the web, and -access them from anywhere. - -Before I dive into the details, I want to state a high-level summary of -my self-hosted Git approach: - -- This method uses the =ssh://= (read & write) and =git://= (read-only) - protocols for push and pull access. - - For the =git://= protocol, I create a =git-daemon-export-ok= file in - any repository that I want to be cloneable by anyone. - - The web interface I am using (=cgit=) allows simple HTTP cloning by - default. I do not disable this setting as I want beginners to be - able to clone one of my repositories even if they don't know the - proper method. -- I am not enabling Smart HTTPS for any repositories. Updates to - repositories must be pushed via SSH. -- Beyond the actual repository management, I am using =cgit= for the - front-end web interface. - - If you use the =scan-path=<path>= configuration in the =cgitrc= - configuration file to automatically find repositories, you can't - exclude a repository from =cgit= if it's stored within the path that - =cgit= reads. To host private repositories, you'd need to set up - another directory that =cgit= can't read. - -** Assumptions -:PROPERTIES: -:CUSTOM_ID: assumptions -:END: -For the purposes of this walkthrough, I am assuming you have a URL -(=git.example.com=) or IP address (=207.84.26.991=) addressed to the -server that you will be using to host your git repositories. - -** Adding a Git User -:PROPERTIES: -:CUSTOM_ID: adding-a-git-user -:END: -In order to use the SSH method associated with git, we will need to add -a user named =git=. If you have used the SSH method for other git -hosting sites, you are probably used to the following syntax: - -#+begin_src sh -git clone [user@]server:project.git -#+end_src - -The syntax above is an =scp=-like syntax for using SSH on the =git= user -on the server to access your repository. - -Let's delete any remnants of an old =git= user, if any, and create the -new user account: - -#+begin_src sh -sudo deluser --remove-home git -sudo adduser git -#+end_src - -*** Import Your SSH Keys to the Git User -:PROPERTIES: -:CUSTOM_ID: import-your-ssh-keys-to-the-git-user -:END: -Once the =git= user is created, you will need to copy your public SSH -key on your local development machine to the =git= user on the server. - -If you don't have an SSH key yet, create one with this command: - -#+begin_src sh -ssh-keygen -#+end_src - -Once you create the key pair, the public should be saved to -=~/.ssh/id_rsa.pub=. - -If your server still has password-based authentication available, you -can copy it over to your user's home directory like this: - -#+begin_src sh -ssh-copy-id git@server -#+end_src - -Otherwise, copy it over to any user that you can access. - -#+begin_src sh -scp ~/.ssh/id_rsa.pub your_user@your_server: -#+end_src - -Once on the server, you will need to copy the contents into the =git= -user's =authorized_keys= file: - -#+begin_src sh -cat id_rsa.pub > /home/git/.ssh/authorized_keys -#+end_src - -*** (Optional) Disable Password-Based SSH -:PROPERTIES: -:CUSTOM_ID: optional-disable-password-based-ssh -:END: -If you want to lock down your server and ensure that no one can -authenticate in via SSH with a password, you will need to edit your SSH -configuration. - -#+begin_src sh -sudo nano /etc/ssh/sshd_config -#+end_src - -Within this file, find the following settings and set them to the values -I am showing below: - -#+begin_src conf -PermitRootLogin no -PasswordAuthentication no -AuthenticationMethods publickey -#+end_src - -You may have other Authentication Methods required in your personal -set-up, so the key here is just to ensure that =AuthenticationMethods= -does not allow passwords. - -** Setting up the Base Directory -:PROPERTIES: -:CUSTOM_ID: setting-up-the-base-directory -:END: -Now that we have set up a =git= user to handle all transport methods, we -need to set up the directory that we will be using as our base of all -repositories. - -In my case, I am using =/git= as my source folder. To create this folder -and assign it to the user we created, execute the following commands: - -#+begin_src sh -sudo mkdir /git -sudo chown -R git:git /git -#+end_src - -** Creating a Test Repository -:PROPERTIES: -:CUSTOM_ID: creating-a-test-repository -:END: -On your server, switch over to the =git= user in order to start managing -git files. - -#+begin_src sh -su git -#+end_src - -Once logged-in as the =git= user, go to your base directory and create a -test repository. - -#+begin_src sh -cd /git -mkdir test.git && cd test.git -git init --bare -#+end_src - -If you want to make this repo viewable/cloneable to the public via the -=git://= protocol, you need to create a =git-daemon-export-ok= file -inside the repository. - -#+begin_src sh -touch git-daemon-export-ok -#+end_src - -** Change the Login Shell for =git= -:PROPERTIES: -:CUSTOM_ID: change-the-login-shell-for-git -:END: -To make sure that the =git= user is only used for git operations and -nothing else, you need to change the user's login shell. To do this, -simply use the =chsh= command: - -#+begin_src sh -sudo chsh git -#+end_src - -The interactive prompt will ask which shell you want the =git= user to -use. You must use the following value: - -#+begin_src sh -/usr/bin/git-shell -#+end_src - -Once done, no one will be able to SSH to the =git= user or execute -commands other than the standard git commands. - -** Opening the Firewall -:PROPERTIES: -:CUSTOM_ID: opening-the-firewall -:END: -Don't forget to open up ports on the device firewall and network -firewall if you want to access these repositories publicly. If you're -using default ports, forward ports =22= (ssh) and =9418= (git) from your -router to your server's IP address. - -If your server also has a firewall, ensure that the firewall allows the -same ports that are forwarded from the router. For example, if you use -=ufw=: - -#+begin_src sh -sudo ufw allow 22 -sudo ufw allow 9418 -#+end_src - -*** Non-Standard SSH Ports -:PROPERTIES: -:CUSTOM_ID: non-standard-ssh-ports -:END: -If you use a non-standard port for SSH, such as =9876=, you will need to -create an SSH configuration file on your local development machine in -order to connect to your server's git repositories. - -To do this, you'll need to define your custom port on your client -machine in your =~/.ssh/config= file: - -#+begin_src sh -nano ~/.ssh/config -#+end_src - -#+begin_src conf -Host git.example.com - # HostName can be a URL or an IP address - HostName git.example.com - Port 9876 - User git -#+end_src - -*** Testing SSH -:PROPERTIES: -:CUSTOM_ID: testing-ssh -:END: -There are two main syntaxes you can use to manage git over SSH: - -- =git clone [user@]server:project.git= -- =git clone ssh://[user@]server/project.git= - -I prefer the first, which is an =scp=-like syntax. To test it, try to -clone the test repository you set up on the server: - -#+begin_src sh -git clone git@git.example.com:/git/test.git -#+end_src - -** Enabling Read-Only Access -:PROPERTIES: -:CUSTOM_ID: enabling-read-only-access -:END: -If you want people to be able to clone any repository where you've -placed a =git-daemon-export-ok= file, you will need to start the git -daemon. - -To do this on a system with =systemd=, create a service file: - -#+begin_src sh -sudo nano /etc/systemd/system/git-daemon.service -#+end_src - -Inside the =git-daemon.service= file, paste the following: - -#+begin_src conf -[Unit] -Description=Start Git Daemon - -[Service] -ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/git/ /git/ - -Restart=always -RestartSec=500ms - -StandardOutput=syslog -StandardError=syslog -SyslogIdentifier=git-daemon - -User=git -Group=git - -[Install] -WantedBy=multi-user.target -#+end_src - -Once created, enable and start the service: - -#+begin_src sh -sudo systemctl enable git-daemon.service -sudo systemctl start git-daemon.service -#+end_src - -To clone read-only via the =git://= protocol, you can use the following -syntax: - -#+begin_src sh -git clone git://git.example.com/test.git -#+end_src - -** Migrating Repositories -:PROPERTIES: -:CUSTOM_ID: migrating-repositories -:END: -At this point, we have a working git server that works with both SSH and -read-only access. - -For each of the repositories I had hosted a different provider, I -executed the following commands in order to place a copy on my server as -my new source of truth: - -Server: - -#+begin_src sh -su git -mkdir /git/<REPOSITORY_NAME>.git && cd /git/<REPOSITORY_NAME>.git -git init --bare - -# If you want to make this repo viewable/cloneable to the public -touch git-daemon-export-ok -#+end_src - -Client: - -#+begin_src sh -git clone git@<PREVIOUS_HOST>:<REPOSITORY_NAME> -git remote set-url origin git@git.EXAMPLE.COM:/git/<REPOSITORY_NAME>.git -git push -#+end_src - -** Optional Web View: =cgit= -:PROPERTIES: -:CUSTOM_ID: optional-web-view-cgit -:END: -If you want a web viewer for your repositories, you can use various -tools, such as =gitweb=, =cgit=, or =klaus=. I chose =cgit= due to its -simple interface and fairly easy set-up (compared to others). Not to -mention that the [[https://git.kernel.org/][Linux kernel uses =cgit=]]. - -*** Docker Compose -:PROPERTIES: -:CUSTOM_ID: docker-compose -:END: -Instead of using my previous method of using a =docker run= command, -I've updated this section to use =docker-compose= instead for an easier -installation and simpler management and configuration. - -In order to use Docker Compose, you will set up a =docker-compose.yml= -file to automatically connect resources like the repositories, =cgitrc=, -and various files or folders to the =cgit= container you're creating: - -#+begin_src sh -mkdir ~/cgit && cd ~/cgit -nano docker-compose.yml -#+end_src - -#+begin_src conf -# docker-compose.yml -version: '3' - -services: - cgit: - image: invokr/cgit - volumes: - - /git:/git - - ./cgitrc:/etc/cgitrc - - ./logo.png:/var/www/htdocs/cgit/logo.png - - ./favicon.png:/var/www/htdocs/cgit/favicon.png - - ./filters:/var/www/htdocs/cgit/filters - ports: - - "8763:80" - restart: always -#+end_src - -Then, just start the container: - -#+begin_src sh -sudo docker-compose up -d -#+end_src - -Once it's finished installing, you can access the site at -=<SERVER_IP>:8763= or use a reverse-proxy service to forward =cgit= to a -URL, such as =git.example.com=. See the next section for more details on -reverse proxying a URL to a local port. - -*** Nginx Reverse Proxy -:PROPERTIES: -:CUSTOM_ID: nginx-reverse-proxy -:END: -I am using Nginx as my reverse proxy so that the =cgit= Docker container -can use =git.example.com= as its URL. To do so, I simply created the -following configuration file: - -#+begin_src sh -sudo nano /etc/nginx/sites-available/git.example.com -#+end_src - -#+begin_src conf -server { - listen 80; - server_name git.example.com; - - if ($host = git.example.com) { - return 301 https://$host$request_uri; - } - - return 404; -} - -server { - server_name git.example.com; - listen 443 ssl http2; - - location / { - # The final `/` is important. - proxy_pass http://localhost:8763/; - add_header X-Frame-Options SAMEORIGIN; - add_header X-XSS-Protection "1; mode=block"; - proxy_redirect off; - proxy_buffering off; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Port $server_port; - } - - # INCLUDE ANY SSL CERTS HERE - include /etc/letsencrypt/options-ssl-nginx.conf; - ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; -} -#+end_src - -Once created, symlink it and restart the web server. - -#+begin_src sh -sudo ln -s /etc/nginx/sites-available/git.example.com /etc/nginx/sites-enabled/ -sudo systemctl restart nginx.service -#+end_src - -As we can see below, my site at =git.example.com= is available and -running: - -*** Settings Up Git Details -:PROPERTIES: -:CUSTOM_ID: settings-up-git-details -:END: -Once you have =cgit= running, you can add some small details, such as -repository owners and descriptions by editing the following files within -each repository. - -Alternatively, you can use the =cgitrc= file to edit these details if -you only care to edit them for the purpose of seeing them on your -website. - -The =description= file within the repository on your server will display -the description online. - -#+begin_src sh -cd /git/example.git -nano description -#+end_src - -You can add a =[gitweb]= block to the =config= file in order to display -the owner of the repository. - -#+begin_src sh -cd /git/example.git -nano config -#+end_src - -#+begin_src conf -[gitweb] - owner = "YourName" -#+end_src - -Note that you can ignore the configuration within each repository and -simply set up this information in the =cgitrc= file, if you want to do -it that way. - -*** Editing =cgit= -:PROPERTIES: -:CUSTOM_ID: editing-cgit -:END: -In order to edit certain items within =cgit=, you need to edit the -=cgitrc= file. - -#+begin_src sh -nano ~/cgit/cgitrc -#+end_src - -Below is an example configuration for =cgitrc=. You can find all the -configuration options within the [configuration manual] -(https://git.zx2c4.com/cgit/plain/cgitrc.5.txt). - -#+begin_src conf -css=/cgit.css -logo=/logo.png -favicon=/favicon.png -robots=noindex, nofollow - -enable-index-links=1 -enable-commit-graph=1 -enable-blame=1 -enable-log-filecount=1 -enable-log-linecount=1 -enable-git-config=1 - -clone-url=git://git.example.com/$CGIT_REPO_URL ssh://git@git.example.com:/git/$CGIT_REPO_URL - -root-title=My Git Website -root-desc=My personal git repositories. - -# Allow download of tar.gz, tar.bz2 and zip-files -snapshots=tar.gz tar.bz2 zip - -## -## List of common mimetypes -## -mimetype.gif=image/gif -mimetype.html=text/html -mimetype.jpg=image/jpeg -mimetype.jpeg=image/jpeg -mimetype.pdf=application/pdf -mimetype.png=image/png -mimetype.svg=image/svg+xml - -# Highlight source code -# source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.sh -source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py - -# Format markdown, restructuredtext, manpages, text files, and html files -# through the right converters -about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh - -## -## Search for these files in the root of the default branch of repositories -## for coming up with the about page: -## -readme=:README.md -readme=:readme.md -readme=:README.mkd -readme=:readme.mkd -readme=:README.rst -readme=:readme.rst -readme=:README.html -readme=:readme.html -readme=:README.htm -readme=:readme.htm -readme=:README.txt -readme=:readme.txt -readme=:README -readme=:readme - -# Repositories - -# Uncomment the following line to scan a path instead of adding repositories manually -# scan-path=/git - -## Test Section -section=git/test-section - -repo.url=test.git -repo.path=/git/test.git -repo.readme=:README.md -repo.owner=John Doe -repo.desc=An example repository! -#+end_src - -*** Final Fixes: Syntax Highlighting & README Rendering -:PROPERTIES: -:CUSTOM_ID: final-fixes-syntax-highlighting-readme-rendering -:END: -After completing my initial install and playing around with it for a few -days, I noticed two issues: - -1. Syntax highlighting did not work when viewing the source code within - a file. -2. The =about= tab within a repository was not rendered to HTML. - -The following process fixes these issues. To start, let's go to the -=cgit= directory where we were editing our configuration file earlier. - -#+begin_src sh -cd ~/cgit -#+end_src - -In here, create two folders that will hold our syntax files: - -#+begin_src sh -mkdir filters && mkdir filters/html-converters && cd filters -#+end_src - -Next, download the default filters: - -#+begin_src sh -curl https://git.zx2c4.com/cgit/plain/filters/about-formatting.sh > about-formatting.sh -chmod 755 about-formatting.sh -curl https://git.zx2c4.com/cgit/plain/filters/syntax-highlighting.py > syntax-highlighting.py -chmod 755 syntax-highlighting.py -#+end_src - -Finally, download the HTML conversion files you need. The example below -downloads the Markdown converter: - -#+begin_src sh -cd html-converters -curl https://git.zx2c4.com/cgit/plain/filters/html-converters/md2html > md2html -chmod 755 md2html -#+end_src - -If you need other filters or html-converters found within -[[https://git.zx2c4.com/cgit/tree/filters][the cgit project files]], -repeat the =curl= and =chmod= process above for whichever files you -need. - -However, formatting will not work quite yet since the Docker cgit -container we're using doesn't have the formatting package installed. You -can install this easily by install Python 3+ and the =pygments= package: - -#+begin_src sh -# Enter the container's command line -sudo docker exec -it cgit bash -#+end_src - -#+begin_src sh -# Install the necessary packages and then exit -yum update -y && \ -yum upgrade -y && \ -yum install python3 python3-pip -y && \ -pip3 install markdown pygments && \ -exit -#+end_src - -*You will need to enter the cgit docker container and re-run these =yum= -commands every time you kill and restart the container!* - -If not done already, we need to add the following variables to our -=cgitrc= file in order for =cgit= to know where our filtering files are: - -#+begin_src conf -# Highlight source code with python pygments-based highlighter -source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py - -# Format markdown, restructuredtext, manpages, text files, and html files -# through the right converters -about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh -#+end_src - -Now you should see that syntax highlighting and README rendering to the -=about= tab is fixed. - -*** Theming -:PROPERTIES: -:CUSTOM_ID: theming -:END: -I won't go into much detail in this section, but you can fully theme -your installation of =cgit= since you have access to the =cgit.css= file -in your web root. This is another file you can add as a volume to the -=docker-compose.yml= file if you want to edit this without entering the -container's command line. - -** :warning: Remember to Back Up Your Data! -:PROPERTIES: -:CUSTOM_ID: warning-remember-to-back-up-your-data -:END: -The last thing to note is that running services on your own equipment -means that you're assuming a level of risk that exists regarding data -loss, catastrophes, etc. In order to reduce the impact of any such -occurrence, I suggest backing up your data regularly. - -Backups can be automated via =cron=, by hooking your base directory up -to a cloud provider, or even setting up hooks to push all repository -info to git mirrors on other git hosts. Whatever the method, make sure -that your data doesn't vanish in the event that your drives or servers -fail. |