Backup Brain: Self-Hosted Bookmarking & More

Overview

The following is a step-by-step guide to downloading and manually installing Backup Brain on a computer or server you control. I strongly recommend you just use Docker instead. It’s going to be much less of a pain in your butt. Note that I say this as someone who hates Docker.

I’ve attempted to document everything you need to do, and none of it is hard, but it does require some basic familiarity with the command line, and how to clone a repo with Git.

If you’ve never opened a terminal and installed open source code from a Git repository this project probably isn’t ready for you… yet. 🙇‍♀️

Making installation easy for non-geeks to install is a high priority.

Before You Start

This document contains step-by-step instructions for macOS and Linux users. It should work on Windows too. If you tackle that, please file a ticket with some tips I can pass along.

This document will help you to…

install Ruby >= v3.0 if necessary Backup Brain is a Ruby on Rails app that needs a modern Ruby.
install Go >= v1.20.0 if necessary We use the reader tool to generate our archives.
install MongoDB if necessary That’s where we’ll store all your data.
Install Meilisearch A search engine to help us find our things.
Download & configure Backup Brain itself
Setup a cron job to backup your database regularly

Download

Backup Brain’s source code is hosted on GitHub, so the first step is to clone the repo using Git.

The official repo is at https://github.com/masukomi/backup_brain

After cloning I recommend you checkout the latest release to be sure you get a nice stable release. Check the Releases listed in the right sidebar of the GitHub page. Then check out the latest one.

git checkout v1.2.0

Backup Brain follows strict Semantic Versioning. Any “breaking change” will result in a new major version number.

Setup

Unless stated otherwise, you should run the specified commands in the terminal, from within the directory you cloned Backup Brain into.

All instructions assume you’re going to be running this on the same computer you’re using as your daily driver, but there’s no reason you can’t install it on an old laptop, Raspberry Pi, or whatever.

Homebrew (for mac folks)

There’s a good chance you’ve already got it installed, but if not go to brew.sh and follow the instructions. It’s the most popular package manager for developer stuff on macOS.

Ruby

We’re using Ruby 3.x. macOS ships with an ancient version of Ruby, and most other OSs don’t tend to try and keep this up to date either.

If you’re unsure what ruby version you have, run

ruby --version

If you’re not a Ruby developer, this section discusses how to get that. Ruby devs can skip to the next section.

If you’re already an ASDF user though, go ahead and use that instead of rbenv to configure Ruby. There’s a .tool-versions file for you.

Everyone else: I’m going to recommend you use rbenv.

Once you’ve set up rbenv, your terminal environment will read the .ruby-version file and switch to the appropriate version of Ruby every time you cd into the Backup Brain directory.

rbenv Installation

macOS

brew install rbenv ruby-build

Debian flavored things

sudo apt install rbenv

rbenv Setup

rbenv init

It’ll be wired up the next time you open a new terminal window, so open a new window cd into the Backup Brain home directory.

First, make sure the version mentioned in this README isn’t out of date.

cat .ruby-version

Now, install whatever version number that said. Like this:

rbenv install 3.2.0

Now cd to another directory (cd ../ is fine), then cd back to your backup_brain directory, and run ruby --version it should output a version that matches whatever was in the .ruby-version file.

If we bump the required version of Ruby in the future rbenv will let you know when you cd into it. Then you just run rbenv install <whatever version it tells you>. You probably saw that message when you cd’d into this directory after installing rbenv.

Go (programming language)

We currently use a tool written in Go to generate our page archives. This dependency will be removed, but for the moment you’ll need to have Go installed in order to compile that tool.

If you’ve already got Go installed, you’ll need to make sure you’re running v1.22 or higher by running

go version

macOS w/ homebrew

If you’re unsure if you installed it with homebrew you can check by running

brew ls --versions golang

If it gives you something like go 1.22.4 then it was. If it’s just blank, then it wasn’t.

To install it, run

brew install golang

To upgrade it run

brew upgrade golang

Raspberry Pi OS & Linux

Run the following to install the latest version of go.

apt-get install golang

everyone else

Please follow the instructions on the Go.dev web site for the latest installation instructions for your OS.

MongoDB

Install MongoDB Community Edition locally, or set up a MongoDB cloud instance. The free tier should be more than enough.

Installation

macOS Homebrew installation

install it
```
brew tap mongodb/brew
brew install mongodb-community
```
start it running in the background
```
brew services start mongodb/brew/mongodb-community
```
Official macOS MongoDB Homebrew Tutorial

Raspberry Pi OS / RaspianOS

Note that you can only install modern versions of MongoDB on a Raspberry Pi 5 or later. Newer versions don’t compile on the older chips.

Follow the instructions on the

Linux (including Raspberry Pi OS)

Go to the Community Edition Installation Instructions and choose your OS, distribution, and if you want to use a package manager or download a .tgz file.

Once you’ve got it running start the service. On Debian / Ubuntu this would be
```
sudo service mongod start
```

Configuration

The default .env.sample file was configured for Docker.

If you’re running MongoDB locally, then edit .env and change bb_mongodb:2707 to localhost:2707 or whatever domain your MongoDB instance can be found at.

Meilisearch

Our full text search is powered by Meilisearch. Official installation instructions offer a variety of installation options, but here are the 2 that’ll work for most of you.

Installation

macOS homebrew installation
```
# install it
brew install meilisearch
```
Before we start the service we need to secure it with a master key. So do not start the service with brew services start meilisearch yet.

curl
```
curl -L https://install.meilisearch.com | sh
```
Then copy the resulting meilisearch executable to a directory in your $PATH **

Setup

Run it once with no arguments & retrieve the master-key it generates for you.

meilisearch

look for lines like these (DO NOT USE THE KEY BELOW)

We generated a new secure master key for you (you can safely use this token):

>> --master-key PIYLd4loyTeAce4-Mr7rW5zLz3eLxk3RlQK-uQJdOjs <<

Restart Meilisearch with the argument above to use this new and secure master key.

Copy that master key. It’s just the text after master-key & before << without the surrounding spaces. Then stop the server with ^c (control + c), and do not restart it.

We need to put your master key in the config file for Backup Brain so that we can provide it to Meilisearch if it isn’t already running when Backup Brain starts up.

Copy the .env.sample file to .env You’ll probably have to do this from the command line because most systems hide “dot files”. Run this to copy it.

cp .env.sample .env

Now, open that new .env file in your favorite text editor. E.g. run code .env in the terminal to open it up in VSCode. It should look like this. Paste in the master-key you got when you launched meilisearch directly after MEILI_MASTER_KEY=. Save the file and close it out.

If you already have the search or admin keys for your Meilisearch go ahead and add them now. Otherwise, don’t worry about it.

HOST_NAME=localhost
HOST_USES_SSH=false
PORT=3334
DELAYED_JOB_WORKERS=2
SEARCH_ENABLED=true
#example: MEILI_MASTER_KEY=AbC123aBc456AbC123aBc456AbC123aBc456AbC123a
MEILI_MASTER_KEY=
#MEILISEARCH_SEARCH_KEY=
#MEILISEARCH_ADMIN_KEY=
MEILISEARCH_URL=http://127.0.0.1:7700
MEILISEARCH_TIMEOUT=10
MEILISEARCH_MAX_RETRIES=2
I_INSTALLED_READER=false

Next, change the MEILISEARCH_URL from http://bb_meilisearch:7700 to http://localhost:7700 if you’re running it locally, or the appropriate URL if you’ve set it up elsewhere.

Web Interface

Meilisearch ships with a limited web interface.

By default it’s available at http://127.0.0.1:7700/ This is mostly useful as a sanity check that everything is running & that the data you expected to be there, is actually there.

You will need your MEILI_MASTER_KEY to access the data.

Reader

🙇‍♀️ apologies for this geekery. There was drama. I sincerely hope to remove this requirement shortly.

reader is a command line tool written in Go. It takes a file, or URL, extracts the core content of the page, and converts it to Markdown.

For the moment, I’m using it to generate our archives.

It requires version 1.22 or higher to compile. Once you’ve installed Go, clone the reader repository. It doesn’t matter where as long as it’s not under Backup Brain. We’ll be deleting the cloned directory when we’re done.

cd into your cloned directory
run go build

This should produce an executable named reader in the current directory.

Test the newly created executable by running ./reader -o --image-mode none https://example.com/ You should see a short bit of text explaining that this is an Example domain.
Assuming it worked, copy this file over to the bin/ directory of your Backup Brain installation.
Edit your .env file and change I_INSTALLED_READER=false to I_INSTALLED_READER=true
Delete the directory where you cloned reader. We don’t need it anymore.

Ruby On Rails

This presumes you’ve got Ruby 3.x or newer installed (see above).

install libraries in console: run bundle install 👋 See notes below regarding some of the odd output you may encounter.
The config/mongoid.yml controls how Rails talks to MongoDB. I’ve given you a default file that should “just work” if you’ve got a MongoDB installation running on the same computer. If you’ve installed MongoDD elsewhere you’ll need to edit the host section in this file to point to your installation.

Bundle Install Output

There are three things of note in the output of bundle install which may be confusing or concerning to non-ruby devs. All of these should be ignored.

A complaint from Bundler about the lockfile being created with a different version.
DEVISE output about reviewing a changelog and upgrade guide.
“When you HTTParty, you must party hard!”

Delayed Job

Delayed Job handles tasks in the background for us. Before we can use it we need to configure a place in the database to store queued Jobs for it to work on.

bundle exec rails runner 'Delayed::Backend::Mongoid::Job.create_indexes'

The serve script will take care of starting the appropriate number of background worker processes & killing them when it’s closed.

⚠ This will kill ALL delayed job workers, even ones from other tasks. If you know how to make it kill only the ones started by serve please file an issue and let me know.

Troubleshooting

If you get the following error:
```
No primary_preferred server is available in cluster
```
It probably means MongoDB isn’t actually running. Run one of the following commands to find out:
```
# linux
ps -AH S | grep mongo | grep -v grep
# macOS / BSD
ps -awx | grep mongod | grep -v grep
```
If there’s no output it’s not running. If it is running, then your .env file probably has the wrong MONGODB_URL See the MongoDB Configuration

Starting the Server

Run ./serve in the root of the project. It’ll make sure meilisearch is running and start it if it isn’t, and then start the Rails server. It assumes MongoDB is already running the background.

Note: the serve script reads the port number from your .env. This defaults to 3334 so as not to conflict with the commonly used port 3000. Feel free to change this in your .env if you need to.

It will output a green line like the following near the end of the output. The address it shows is based on your configuration. This is where you’ll find your Backup Brain going forwards, unless you change the HOST_NAME or PORT

VISIT YOUR BACKUP BRAIN AT http://localhost:3334

Running it in the background

The problem with just running ./serve is that closing the terminal window will shut down your Backup Brain. You probably don’t want that.

Once you know everything’s working run the following to start your Backup Brain and have it run in the background.

Background for the moment

./serve & disown

Background after restart too

systemd
- Daemonizing Meilisearch
  
  You can install meilisearch with a package manager, which should make it just a matter of telling systemd to run it, but then it’s harder to set the MEILI_MASTER_KEY so, we’re doing it the “hard” way.
  
  First, edit scripts/services/meilisearch.toml and replace YOUR_MASTER_KEY_VALUE with your master key.
  
  Next
```
# make a directory for its data
sudo mkdir -p /var/lib/meilisearch/data
# change its permissions
sudo chmod -R 750 /var/lib/meilisearch
# put the systemd Unit for meilisearch somewhere systemd can find it
sudo cp scripts/services/meilisearch.service /etc/systemd/system/
# copy over the config file to where the unit expects it
sudo cp scripts/services/meilisearch.toml /etc/meilisearch.toml
sudo systemctl daemon-reload
sudo systemctl enable meilisearch
sudo systemctl start meilisearch
```
  Now, check if it’s running with:
```
systemctl status meilisearch
```
  The second line should start with Active: active (running)
- Daemonizing Backup Brain
  
  Note: this systemd unit requires the mongod service to be running as well.
  
  There’s a “Unit” file at scripts/services/backup_brain.service
  
  First, edit that and change the following lines.
  
  User, and Group should be your username and group (usually the same as your username) on the server. The ExecStart and ExecReload should end with the full “absolute” path to the serve script at the root of your Backup Brain
```
User=YOUR_USERNAME
Group=USER_GROUP
ExecStart=/bin/bash -lc /path/to/backup_brain/serve
ExecReload=/bin/bash -lc /path/to/backup_brain/serve
```
  Note: The scripts/bash_helpers/migrations.sh file assumes you’re using rbenv for your ruby version management. If you’ve chosen a different way to manage ruby versions, then change the PATH_TO_BUNDLE variable at the top of that file.
  
  Next, copy your modified backup_brain.service somewhere where systemd can find it.
```
sudo cp scripts/services/backup_brain.service /etc/systemd/system/
```
  Next, we’ll reload the list of things systemd knows about, enable it, and start it.
```
sudo systemctl daemon-reload
sudo systemctl enable backup_brain
sudo systemctl start backup_brain
```
  Now, check if it’s running with:
```
systemctl status meilisearch
```
  The second line should start with Active: active (running)
  
  You can always stop it by saying…
```
sudo systectl stop backup_brain
```

Stopping the server

After starting with ./serve

If just started the server by running ./serve & didn’t have it run in the background, the best way to stop it is to hit ^c That’s control (not command) + c That will also shut down the background jobs.

If you’re running it in the background then run ./bin/shutdown That script will shutdown the server and any background jobs it can find.

NOTE: both of these methods will shut down all “Delayed Job” processes they find.

After starting with systemctl

systemctl stop backup_brain

You Did It!

THAT’S IT. Everything’s installed. Sorry it was so much manual geekery. It’s a high priority of mine to improve this in future releases.

Now it’s time to actually use what you’ve set up.

Run the ./serve command (or ./serve & disown) and go to the url shown in the output.

Be sure to check out the “Extra niceties” that’ll make your usage a little bit nicer.

Adding and importing bookmarks is covered on Your First Bookmarks

Extra niceties

Browser Extensions

See the Browser Extensions & Helpers page for details & links.

Domain Name

Publicly Accessible

If you’d like to be able to share your bookmarks with others, or access your Backup Brain on your phone when away from home, you’ll need to set up remote access. I recommend you play with Backup Brain locally first, and make sure it’s something you want to keep using.

This is a multi-step process that I’ve documented on the Setting Up Remote Access page.

Local Only

You’ll probably be running this locally, but you can still have a pretty name. Simply add a new line to your /etc/hosts file. You’ll have to do this as root. For example sudo code /etc/hosts to open it in VSCode and then add the following line and save.
```
127.0.0.1   backupbrain
```
Then replace HOST_NAME=localhost in your .env file (see above).

After you boot the app you’ll be able to visit http://backupbrain:3334 instead of http://localhost:3334

You can use anything you want in /etc/hosts as long as it’s only one word. bb would be a nice and short domain name.

Browser Shortcodes & Custom Search

Firefox
- Shortcode
  1. start up your instance, navigate to it, and create a bookmark.
  2. open Bookmarks > Manage Bookmarks
  3. find, and click on your new bookmark
  4. give it a new “keyword” in the “keyword” field. I used @bb
    1. hit tab to move to another field and cause the change to be saved.
  5. Close the bookmarks manager
  6. Type your shortcut in the address bar & hit enter. It should change to the address of your site, and then load the site.
- Custom Search
  1. Visit your Backup Brain in Firefox.
  2. Right-click in the search field, and choose “Add a Keyword for this Search…”
  3. Give it a name, and a keyword that’s different from any others. I used @bbs (Backup Brain Search).
  4. Click “Save”.
  Now you can search your Backup Brain by typing your new search keyword followed by a space and the words you want to search for. It should take you to the search results for those words in your Backup Brain.

Chrome

Chrome doesn’t offer bookmark shortcuts, but you can create a “Custom Search Engine”
1. Right-click in the address bar and choose “Manage Search Engines & Site Search”
2. Scroll down to “Site Search”
3. Click “Add”
4. Give it a name
5. Give it a shortcut. @bb for example.
6. Change MY_BACKUP_BRAIN_DOMAIN_AND_PORT in the following URL to match the domain & port you use to access your Backup Brain and enter that in the “URL” field. https://MY_BACKUP_BRAIN_DOMAIN_AND_PORT/bookmarks/search?query=%s&sort=match
7. Click “Add”.
Now you can search your Backup Brain by typing @bb and the words you want to search for.

Backing Up Data

I don’t have your data, so I can’t back it up for you.

In the scripts directory you’ll find an export and import script. I’d recommend setting up a cron job to run the export script a couple times a day. It’ll create a mongo_exports directory, and save a file for each data type. Running the import script on a fresh database will load all of them from the last backup.

⚠️ Running the import script on an existing database will delete and replace the existing data.

Make sure these files are being backed up to another computer in case yours dies ☠, goes 💥, or catches fire 🔥.

NOTE: these scripts use the MONGODB_URI and DATABASE_NAME environment variables. If you don’t set those, then it’ll assume you haven’t changed the database name, and that it’s running on localhost. If your MongoDB installation is on a different server from your Backup Brain then you’ll need to set the MONGODB_URI according to the connection string documentation. This can be set in the environment or you can just hardcode it into the scripts.

Example Backup using Cron

Here’s an example cronjob that will
- cd into the directory where I installed Backup Brain on my computer
- run the export script
- send its output to log/cron.log (overwriting output from the last run)
```
0 * * * * /bin/bash -l -c 'cd ~/workspace/backup_brain && ./scripts/export.sh > ~/workspace/backup_brain/log/cron.log 2>&1'
```
Change the ~/workspace/backup_brain to match the location you’ve installed yours. The backups will be stored in a mongo_exports directory in the root of your Backup Brain install. Make sure that your automated backup tool is backing up this directory.

If you want them to be stored elsewhere, just replace the 2 instances of mongo_exports in the scripts/export.sh script with the path from Backup Brain’s root to where you want things stored. Be sure to change the single instance of it in scripts/import.sh so that it’ll know where to look if you need to re-import your data.

See this post about using cron if you’re unfamiliar. Note that cron’s default editor is Vim, but that page includes instructions on how to edit your crontab (list of cron jobs) without using Vim.

Remote Access

If you’re hosting this locally and want to access bookmarks when away from home, or share them with friends, you’ll need to set up a “tunnel”. Check out the page on Setting Up Remote Access

Improved Meilisearch Security

This is highly recommended if your Meilisearch instance is on a different machine. Otherwise it probably doesn’t matter.

If you only provide the master key when configuring Meilisearch, the mongodb_meilisearch gem will use it to make one call to retrieve the default search & administration keys which it will use for all subsequent calls. However, if you provide those API keys in the configuration it will never use the master key.

The easiest way to retrieve those keys is to first launch the Rails console from the root of your Backup Brain.

bundle exec rails console

Make sure Meilisearch is running, then execute the following

Search::Client.instance.get_default_keys("your master key here")

That will return a Hash that looks like this:

{:search=>"your search api key",
 :admin=>"your admin api key"}

Type exit to leave the Rails console.

In your .env file, add the keys that were in the hash above. There should be no spaces and no quotation marks.

# put the values from the console here:
MEILISEARCH_SEARCH_KEY=
MEILISEARCH_ADMIN_KEY=

It is important that you leave the MEILI_MASTER_KEY in place so that the serve script can launch Meilisearch with it. This is the only place where the master key is used by Backup Brain.

Contributing

Before contributing please read the about page.

I’m always happy to accept contributions from good people. Just fork the repo, and make a Pull Request. See the contributing.md file for more information.

License

This software is distributed under the AGPL + the Commons Clause license condition. See the LICENSE.md file for details.

The basic gist is that you can do all the normal open source things with it, except you can’t sell it.

Getting Started Without Docker

Overview

Before You Start

Download

Setup

Homebrew (for mac folks)

Ruby

rbenv Installation

rbenv Setup

Go (programming language)

macOS w/ homebrew

Raspberry Pi OS & Linux

everyone else

MongoDB

Installation

Configuration

Meilisearch

Installation

Setup

Web Interface

Reader

Ruby On Rails

Bundle Install Output

Delayed Job

Starting the Server

Running it in the background

Background for the moment

Background after restart too

Stopping the server

After starting with ./serve

After starting with systemctl

You Did It!

Extra niceties

Browser Extensions

Domain Name

Browser Shortcodes & Custom Search

Backing Up Data

Remote Access

Improved Meilisearch Security

Contributing

License