Getting Started Without Docker

Table of Contents

Overview

The following is a step-by-step guide to downloading and manually installing Backup Brain on a computer or server you control. I strongly recommend you just use Docker instead. It’s going to be much less of a pain in your butt. Note that I say this as someone who hates Docker.

I’ve attempted to document everything you need to do, and none of it is hard, but it does require some basic familiarity with the command line, and how to clone a repo with Git.

If you’ve never opened a terminal and installed open source code from a Git repository this project probably isn’t ready for you… yet. 🙇‍♀️

Making installation easy for non-geeks to install is a high priority.

Before You Start

This document contains step-by-step instructions for macOS and Linux users. It should work on Windows too. If you tackle that, please file a ticket with some tips I can pass along.

This document will help you to…

  • install Ruby >= v3.0 if necessary Backup Brain is a Ruby on Rails app that needs a modern Ruby.
  • install Go >= v1.20.0 if necessary We use the reader tool to generate our archives.
  • install MongoDB if necessary That’s where we’ll store all your data.
  • Install Meilisearch A search engine to help us find our things.
  • Download & configure Backup Brain itself
  • Setup a cron job to backup your database regularly

Download

Backup Brain’s source code is hosted on GitHub, so the first step is to clone the repo using Git.

The official repo is at https://github.com/masukomi/backup_brain

After cloning I recommend you checkout the latest release to be sure you get a nice stable release. Check the Releases listed in the right sidebar of the GitHub page. Then check out the latest one.

git checkout v1.2.0

Backup Brain follows strict Semantic Versioning. Any “breaking change” will result in a new major version number.

Setup

Unless stated otherwise, you should run the specified commands in the terminal, from within the directory you cloned Backup Brain into.

All instructions assume you’re going to be running this on the same computer you’re using as your daily driver, but there’s no reason you can’t install it on an old laptop, Raspberry Pi, or whatever.

Homebrew (for mac folks)

There’s a good chance you’ve already got it installed, but if not go to brew.sh and follow the instructions. It’s the most popular package manager for developer stuff on macOS.

Ruby

We’re using Ruby 3.x. macOS ships with an ancient version of Ruby, and most other OSs don’t tend to try and keep this up to date either.

If you’re unsure what ruby version you have, run

ruby --version

If you’re not a Ruby developer, this section discusses how to get that. Ruby devs can skip to the next section.

If you’re already an ASDF user though, go ahead and use that instead of rbenv to configure Ruby. There’s a .tool-versions file for you.

Everyone else: I’m going to recommend you use rbenv.

Once you’ve set up rbenv, your terminal environment will read the .ruby-version file and switch to the appropriate version of Ruby every time you cd into the Backup Brain directory.

rbenv Installation

macOS

brew install rbenv ruby-build

Debian flavored things

sudo apt install rbenv

rbenv Setup

rbenv init

It’ll be wired up the next time you open a new terminal window, so open a new window cd into the Backup Brain home directory.

First, make sure the version mentioned in this README isn’t out of date.

cat .ruby-version

Now, install whatever version number that said. Like this:

rbenv install 3.2.0

Now cd to another directory (cd ../ is fine), then cd back to your backup_brain directory, and run ruby --version it should output a version that matches whatever was in the .ruby-version file.

If we bump the required version of Ruby in the future rbenv will let you know when you cd into it. Then you just run rbenv install <whatever version it tells you>. You probably saw that message when you cd’d into this directory after installing rbenv.

Go (programming language)

We currently use a tool written in Go to generate our page archives. This dependency will be removed, but for the moment you’ll need to have Go installed in order to compile that tool.

If you’ve already got Go installed, you’ll need to make sure you’re running v1.22 or higher by running

go version

macOS w/ homebrew

If you’re unsure if you installed it with homebrew you can check by running

brew ls --versions golang

If it gives you something like go 1.22.4 then it was. If it’s just blank, then it wasn’t.

To install it, run

brew install golang

To upgrade it run

brew upgrade golang

everyone else

Please follow the instructions on the Go.dev web site for the latest installation instructions for your OS.

MongoDB

Install MongoDB Community Edition locally, or set up a MongoDB cloud instance. The free tier should be more than enough.

Installation

  • macOS Homebrew installation

    install it

    brew tap mongodb/brew
brew install mongodb-community

    start it running in the background

    brew services start mongodb/brew/mongodb-community

    Official macOS MongoDB Homebrew Tutorial

  • everything else

    First a warning. The “MongoDB Community Server Download” page is misleading. The instructions on it (when viewed on a mac at least) are for installing MongoDB Atlas which is their hosted cloud database. It has a free tier, but they’re hoping you’ll pay for more.

    Backup Brain will work with MongoDB Atlas, but you shouldn’t need to install anything locally to use that.

    The download page will attempt to guess what operating system you’re running and clicking “Download” should give you a pre-compiled MongoDB for your computer. There’s also a list of the current releases where you can download it manually. This list includes .rpm and .deb files for Red Hat & Debian Linux users & .tgz files for everyone else.

    Here are the installation instructions for those files.

Configuration

The default config/mongoid.yml is configured for Docker.

Edit config/mongoid.yml and change bb_mongodb:2707 to localhost:2707 or whatever domain your MongoDB instance can be found at.

Meilisearch

Our full text search is powered by Meilisearch. Official installation instructions offer a variety of installation options, but here are the 2 that’ll work for most of you.

Installation

  • macOS homebrew installation

    # install it
brew install meilisearch

    Before we start the service we need to secure it with a master key. So do not start the service with brew services start meilisearch yet.

  • curl

    curl -L https://install.meilisearch.com | sh

    Then copy the resulting meilisearch executable to a directory in your $PATH **

Setup

Run it once with no arguments & retrieve the master-key it generates for you.

meilisearch

look for lines like this

We generated a new secure master key for you (you can safely use this token):

>> --master-key PIYLd4loyTeAce4-Mr7rW5zLz3eLxk3RlQK-uQJdOjs <<

Restart Meilisearch with the argument above to use this new and secure master key.

Copy that master key. It’s just the text after master-key & before << without the surrounding spaces. Then stop the server with ^c (control + c), and do not restart it.

We need to put your master key in the config file for Backup Brain so that we can provide it to Meilisearch if it isn’t already running when Backup Brain starts up.

Copy the .env.sample file to .env You’ll probably have to do this from the command line because most systems hide “dot files”. Run this to copy it.

cp .env.sample .env

Now, open that new .env file in your favorite text editor. E.g. run code .env in the terminal to open it up in VSCode. It should look like this. Paste in the master-key you got when you launched meilisearch directly after MEILI_MASTER_KEY=. Save the file and close it out.

If you already have the search or admin keys for your Meilisearch go ahead and add them now. Otherwise, don’t worry about it.

HOST_NAME=localhost
HOST_USES_SSH=false
PORT=3334
DELAYED_JOB_WORKERS=2
SEARCH_ENABLED=true
#example: MEILI_MASTER_KEY=AbC123aBc456AbC123aBc456AbC123aBc456AbC123a
MEILI_MASTER_KEY=
#MEILISEARCH_SEARCH_KEY=
#MEILISEARCH_ADMIN_KEY=
MEILISEARCH_URL=http://127.0.0.1:7700
MEILISEARCH_TIMEOUT=10
MEILISEARCH_MAX_RETRIES=2
I_INSTALLED_READER=false

Web Interface

Meilisearch ships with a limited web interface.

By default it’s available at http://127.0.0.1:7700/ This is mostly useful as a sanity check that everything is running & that the data you expected to be there, is actually there.

You will need your MEILI_MASTER_KEY to access the data.

Reader

🙇‍♀️ apologies for this geekery. There was drama. I sincerely hope to remove this requirement shortly.

reader is a command line tool written in Go. It takes a file, or URL, extracts the core content of the page, and converts it to Markdown.

For the moment, I’m using it to generate our archives.

It requires version 1.22 or higher to compile. Once you’ve installed Go, clone the reader repository. It doesn’t matter where as long as it’s not under Backup Brain. We’ll be deleting the cloned directory when we’re done.

  • cd into your cloned directory
  • run go build

This should produce an executable named reader in the current directory.

  • Test the newly created executable by running ./reader -o --image-mode none https://example.com/ You should see a short bit of text explaining that this is an Example domain.
  • Assuming it worked, copy this file over to the bin/ directory of your Backup Brain installation.
  • Edit your .env file and change I_INSTALLED_READER=false to I_INSTALLED_READER=true
  • Delete the directory where you cloned reader. We don’t need it anymore.

Ruby On Rails

This presumes you’ve got Ruby 3.x or newer installed (see above).

  1. install libraries in console: run bundle install 👋 See notes below regarding some of the odd output you may encounter.
  2. The config/mongoid.yml controls how Rails talks to MongoDB. I’ve given you a default file that should “just work” if you’ve got a MongoDB installation running on the same computer. If you’ve installed MongoDD elsewhere you’ll need to edit the host section in this file to point to your installation.

Bundle Install Output

There are three things of note in the output of bundle install which may be confusing or concerning to non-ruby devs. All of these should be ignored.

  1. A complaint from Bundler about the lockfile being created with a different version.
  2. DEVISE output about reviewing a changelog and upgrade guide.
  3. “When you HTTParty, you must party hard!”

Delayed Job

Delayed Job handles tasks in the background for us. Before we can use it we need to configure a place in the database to store queued Jobs for it to work on.

bundle exec rails runner 'Delayed::Backend::Mongoid::Job.create_indexes'

The serve script will take care of starting the appropriate number of background worker processes & killing them when it’s closed.

⚠ This will kill ALL delayed job workers, even ones from other tasks. If you know how to make it kill only the ones started by serve please file an issue and let me know.

Starting the Server

Run ./serve in the root of the project. It’ll make sure meilisearch is running and start it if it isn’t, and then start the Rails server. It assumes MongoDB is already running the background.

Note: the serve script reads the port number from your .env. This defaults to 3334 so as not to conflict with the commonly used port 3000. Feel free to change this in your .env if you need to.

It will output a green line like the following near the end of the output. The address it shows is based on your configuration. This is where you’ll find your Backup Brain going forwards, unless you change the HOST_NAME or PORT

VISIT YOUR BACKUP BRAIN AT http://localhost:3334

Running it in the background

The problem with just running ./serve is that closing the terminal window will shut down your Backup Brain. You probably don’t want that.

Once you know everything’s working run the following to start your Backup Brain and have it run in the background.

./serve & disown

Stopping the server

If just started the server by running ./serve & didn’t have it run in the background, the best way to stop it is to hit ^c That’s control (not command) + c That will also shut down the background jobs.

If you’re running it in the background then run ./bin/shutdown That script will shutdown the server and any background jobs it can find.

NOTE: both of these methods will shut down all “Delayed Job” processes they find.

You Did It!

THAT’S IT. Everything’s installed. Sorry it was so much manual geekery. It’s a high priority of mine to improve this in future releases.

Now it’s time to actually use what you’ve set up.

Run the ./serve command (or ./serve & disown) and go to the url shown in the output.

Be sure to check out the “Extra niceties” that’ll make your usage a little bit nicer.

Adding and importing bookmarks is covered on Your First Bookmarks

Extra niceties

Browser Extensions

See the Browser Extensions & Helpers page for details & links.

Domain Name

  • Publicly Accessible

    If you’d like to be able to share your bookmarks with others, or access your Backup Brain on your phone when away from home, you’ll need to set up remote access. I recommend you play with Backup Brain locally first, and make sure it’s something you want to keep using.

    This is a multi-step process that I’ve documented on the Setting Up Remote Access page.

  • Local Only

    You’ll probably be running this locally, but you can still have a pretty name. Simply add a new line to your /etc/hosts file. You’ll have to do this as root. For example sudo code /etc/hosts to open it in VSCode and then add the following line and save.

    127.0.0.1   backupbrain

    Then replace HOST_NAME=localhost in your .env file (see above).

    After you boot the app you’ll be able to visit http://backupbrain:3334 instead of http://localhost:3334

    You can use anything you want in /etc/hosts as long as it’s only one word. bb would be a nice and short domain name.

  • Firefox

    • Shortcode

      1. start up your instance, navigate to it, and create a bookmark.
      2. open Bookmarks > Manage Bookmarks
      3. find, and click on your new bookmark
      4. give it a new “keyword” in the “keyword” field. I used @bb
        1. hit tab to move to another field and cause the change to be saved.
      5. Close the bookmarks manager
      6. Type your shortcut in the address bar & hit enter. It should change to the address of your site, and then load the site.

    • Custom Search

      1. Visit your Backup Brain in Firefox.
      2. Right-click in the search field, and choose “Add a Keyword for this Search…”
      3. Give it a name, and a keyword that’s different from any others. I used @bbs (Backup Brain Search).
      4. Click “Save”.

      Now you can search your Backup Brain by typing your new search keyword followed by a space and the words you want to search for. It should take you to the search results for those words in your Backup Brain.

  • Chrome

    Chrome doesn’t offer bookmark shortcuts, but you can create a “Custom Search Engine”

    1. Right-click in the address bar and choose “Manage Search Engines & Site Search”
    2. Scroll down to “Site Search”
    3. Click “Add”
    4. Give it a name
    5. Give it a shortcut. @bb for example.
    6. Change MY_BACKUP_BRAIN_DOMAIN_AND_PORT in the following URL to match the domain & port you use to access your Backup Brain and enter that in the “URL” field. https://MY_BACKUP_BRAIN_DOMAIN_AND_PORT/bookmarks/search?query=%s&sort=match
    7. Click “Add”.

    Now you can search your Backup Brain by typing @bb and the words you want to search for.

Backing Up Data

I don’t have your data, so I can’t back it up for you.

In the scripts directory you’ll find an export and import script. I’d recommend setting up a cron job to run the export script a couple times a day. It’ll create a mongo_exports directory, and save a file for each data type. Running the import script on a fresh database will load all of them from the last backup.

⚠️ Running the import script on an existing database will delete and replace the existing data.

Make sure these files are being backed up to another computer in case yours dies ☠, goes 💥, or catches fire 🔥.

NOTE: these scripts use the MONGODB_URI and DATABASE_NAME environment variables. If you don’t set those, then it’ll assume you haven’t changed the database name, and that it’s running on localhost. If your MongoDB installation is on a different server from your Backup Brain then you’ll need to set the MONGODB_URI according to the connection string documentation. This can be set in the environment or you can just hardcode it into the scripts.

  • Example Backup using Cron

    Here’s an example cronjob that will

    • cd into the directory where I installed Backup Brain on my computer
    • run the export script
    • send its output to log/cron.log (overwriting output from the last run)
    0 * * * * /bin/bash -l -c 'cd ~/workspace/backup_brain && ./scripts/export.sh > ~/workspace/backup_brain/log/cron.log 2>&1'

    Change the ~/workspace/backup_brain to match the location you’ve installed yours. The backups will be stored in a mongo_exports directory in the root of your Backup Brain install. Make sure that your automated backup tool is backing up this directory.

    If you want them to be stored elsewhere, just replace the 2 instances of mongo_exports in the scripts/export.sh script with the path from Backup Brain’s root to where you want things stored. Be sure to change the single instance of it in scripts/import.sh so that it’ll know where to look if you need to re-import your data.

    See this post about using cron if you’re unfamiliar. Note that cron’s default editor is Vim, but that page includes instructions on how to edit your crontab (list of cron jobs) without using Vim.

Remote Access

If you’re hosting this locally and want to access bookmarks when away from home, or share them with friends, you’ll need to set up a “tunnel”. Check out the page on Setting Up Remote Access

Improved Meilisearch Security

This is highly recommended if your Meilisearch instance is on a different machine. Otherwise it probably doesn’t matter.

If you only provide the master key when configuring Meilisearch, the mongodb_meilisearch gem will use it to make one call to retrieve the default search & administration keys which it will use for all subsequent calls. However, if you provide those API keys in the configuration it will never use the master key.

The easiest way to retrieve those keys is to first launch the Rails console from the root of your Backup Brain.

bundle exec rails console

Then run the following

Search::Client.instance.get_default_keys("your master key here")

That will return a Hash that looks like this:

{:search=>"your search api key",
 :admin=>"your admin api key"}

Type exit to leave the Rails console.

Uncomment these lines in your .env file (remove the #) and fill in the keys that were in the hash above. There should be no spaces and no quotation marks.

#MEILISEARCH_SEARCH_KEY=
#MEILISEARCH_ADMIN_KEY=

It is important that you leave the MEILI_MASTER_KEY in place so that the serve script can launch Meilisearch with it. This is the only place where the master key is used by Backup Brain.

Contributing

Before contributing please read the about page.

I’m always happy to accept contributions from good people. Just fork the repo, and make a Pull Request. See the contributing.md file for more information.

License

This software is distributed under the AGPL + the Commons Clause license condition. See the LICENSE.md file for details.

The basic gist is that you can do all the normal open source things with it, except you can’t sell it.
Mastodon