If you found this post, you probably use the Topicbox listserv platform. My company used Topicbox but I couldn’t find any specific articles on how to convert from Topicbox to Discourse so I hope this will help you. Discourse is an open source discussion platform that can be used as a mailing list, discussion forum, long-form chat room, and more. It has a really compelling feature set and it’s very economical to run. In contrast, Topicbox calls itself a group email product for teams. I’m sure it’s a great fit for many organizations, but we outgrew it and we wanted something we could integrate with our association management system.
This article will not cover every minute detail of the process. It’s meant to supplement existing resources that will be referenced below. Also, I’m not a software developer and I have written this for colleagues who are like me — someone with tech skills which includes familiarity with hosted server technology, hosted email services, and DNS. It’s also recommended that you know some basic Linux utilities, like how to change directories (cd); how to copy files (cp); how to make directories (mkdir); and how to edit text files (nano or vi).
I’m going to assume you have already either set up a hosted instance of Discourse with them directly (see this link), or you have chosen to set up your own Linux server, either in the cloud or on premise. You should be familiar with Discourse before continuing.
Summary of Converting mbox files from Topicbox into Discourse
- Set up a temporary hosted instance of Discourse as a droplet on Digital Ocean (or another Linux cloud environment)
- Ensure that your production version of Discourse has already be configured to your needs, which includes categories that will match with the groups you have defined in Topicbox
- Send a request to Topicbox support for an export of your groups into mbox files
- Switch your production version of Discourse into read-only mode
- Create a backup of your current Discourse site
- Restore that backup onto your temporary Discourse instance
- Import the mbox files
- Create a backup of your temporary Discourse instance
- Upload that backup to your production Discourse
- If you are a hosted Discourse customer, send a support request so that they can restore your backup. If you run your own production instance on Digital Ocean (or another cloud service), you can restore the backup yourself.
Step 1 – Set up a Temporary Instance of Discourse in the Cloud
This Discourse instance will be for importing your messages from Topicbox. I strongly recommend doing all of that into a second instance of Discourse (aka, a temporary instance) — not your production version! So step #1 is for setting up a temporary instance of Discourse. The process for doing this is nicely documented on the github site here. I personally went with Digital Ocean as my host. They are top-notch, and they also have a how to tutorial here. So you might find that helpful to at least get Ubuntu set up along with Docker. But I recommend following the github link for detailed instructions on installing Discourse.
Note: don’t skip the step on Setting Up Email. Even though this is a temporary instance of Discourse, you won’t be able to complete setup without having a properly configured email service to go with this instance of Discourse.
Tip to Save Time
During the initial launch of Discourse setup, and after registering your admin account, the setup wizard will offer to guide you through the basic configuration of Discourse (13 steps). You can safely click on “Maybe Later” because you’ll be restoring a backup from your production instance of Discourse.
Step 2 – Review Your Production Setup of Discourse
If you’re moving from Topicbox to Discourse, you likely have several groups defined in Topicbox. And if you were like me, those groups were probably private. In other words, users had to be group members of a particular group in order to be permitted to message the group. Assuming that is the case for you, you’ll probably want to create categories in Discourse that match with your group names in Topicbox. I recommend doing so before importing your Topicbox messages.
Note: we used private groups in Topicbox. To mimic a similar setup in Discourse, we created both groups and categories with the same names as our Topicbox groups. And then in Discourse, we set the security of each category to only be accessible by the group we created with the same name. And next, we added our users to the Discourse groups which in turn gave them access to the categories with those same names. Confusing? Hopefully not! But I’ll add that we also changed the labeling in Discourse so that “categories” uses the term “communities” in our configuration.
Step 3 – Get Your Topicbox Emails Exported to MBOX Files
Unfortunately, you can’t export your Topicbox messages yourself. You have to send a request to Topicbox support and request an export of your messages into mbox files. They will give you a ZIP file, and inside there will be a separate mbox file for each group. That request to the support team can take a while. In my case, it took about a week before support gave me a link to download my export of Topicbox messages.
Step 4 – Switch your production version of Discourse into read-only mode
If your first import will be a test, this step can be skipped. You can safely initiate a backup of your production instance of Discourse without putting it into read-only mode first. But when you’re ready to do the final migration, it’s a good idea to put Discourse into read-only mode so that users can’t post new messages while you’re working on your migration.
To do so, use your browser to login to Discourse and go to Admin –> Backups. You’ll find the button to turn on read-only mode on that screen.
Step 5 – Perform a Backup of Discourse
Click the Backup button to initiate a backup job on your Discourse system. Depending on the amount of data in your production instance of Discourse, this can take several minutes or more. Once done, you’ll receive an email with the link to download your backup, which will be in a “tar.gz” format. You’ll need this for the next step.
Step 6 – Restore the Backup into your Temporary Discourse Instance
Use your browser to sign into your temporary instance of Discourse and restore the backup that was made from the previous step. To do so, visit Admin –> Backups and use the Upload button to upload the “tar.gz” file from the previous step. After doing so, you’ll notice that the “restore” button isn’t available and there’s usually a note that says, Restore is disabled in the site settings. Click that link so that you can check-mark the box to allow a restore. Then return to the backup screen and click to restore the already-uploaded backup file.
Step 7 – Import your mbox files
This will be the hardest part of the process. If you’re not familiar with basic Linux commands, this phase can feel rather daunting. Gerhard Schlager has written a guide on migrating a mailing list to Discourse. That guide is appropriate for Topicbox and you’ll want to use it along with some additional notes that follow. I’ll reference Gerhard’s section numbers in the following lists.
- 1.1 – Review Discourse Settings (after installing Discourse). If you follow the import guide from Gerhard, he suggests configuring a few settings within Discourse before starting the import. Be sure to review section 1.1 in this regard. One setting I did choose was to disable download_remote_images_to_local so that Discourse wouldn’t download images embedded in posts.
- Connect to your Ubuntu server via ssh. You’ll need to log in to your Ubuntu server with ssh for the next steps. I personally recommend taking the time to generate “keys” so you can connect more easily going forward. If you’re on a Windows PC, this is a good article on the process. For Mac users, here’s a guide.
TIP: I used the built-in ssh command in Windows 10, and it worked well. It’s called OpenSSH Client and it’s considered an optional feature in Windows 10 you can install. But beware the common frustration of copy/paste. If you have copied a long password or some other line of text to paste into your ssh session, Ctrl+V won’t work. A single right-click of the mouse in the ssh (terminal) window will usually ‘paste’, or you can go into the command prompt properties screen and enable Ctrl+Shift+C/V as your keyboard friendly way to copy/paste.
- 1.2 – Prepare the Docker container. Follow the directions of the import guide in section 1.2, entitled preparing the docker container. The “regular import” was appropriate. Disregard the Google Groups import section. Once completed, you should have edited your “import.yml” file and added a line for importing mbox files. Next, you’ll run the commands that build the container for importing. To do so:
/var/discourse/launcher stop app
/var/discourse/launcher rebuild import- 1.4 – Configure the importer. Section 1.4 of the import guide is entitled configuring the importer. In my instance, I was happy with the defaults and I did not have to make any changes to the “settings.yml” file that was mentioned. It was good as-is.
- 1.5 – Prepare files – Install FileZilla. You’ll need to transfer the mbox files to the temporary Discourse server. My preference was to use FileZilla, which provides a graphical interface to transfer files. Here’s a DigitalOcean guide for setting up FileZilla. You’ll need the SSH keys from the previous step for this.
- 1.5 – Prepare files (continued) – Use FileZilla to tranfer your mbox files. When you have ssh and FileZilla working, you’ll need to transfer the mbox files to your temporary Discourse server. I recommend unzipping (extracting) the ZIP file (that Topicbox support sends to you) before transferring to your Discourse server. You can wait and unzip the file on Linux, but that often requires having to first install the unzip tool on Linux. I transferred my files to my home directory in Ubuntu.
- Create directories for each mbox file. For each mbox file you want to import, you’ll need to create a separate directory for each file. See section “1.5” of the import guide for more information. For example, let’s say my previous Topicbox groups were all sports related. I would need to create directories similar to these:
mkdir /var/discourse/shared/standalone/import/data/soccer
mkdir /var/discourse/shared/standalone/import/data/golf
mkdir /var/discourse/shared/standalone/import/data/football- Copy each mbox file to the appropriate folder. Next, you’ll need to copy your mbox files into the corresponding directories. As an example, if I had used FileZilla to transfer my mbox files to my home directory, I would copy them to the corresponding folders like so:
cp ~/soccer.mbox /var/discourse/shared/standalone/import/data/soccer
cp ~/golf.mbox /var/discourse/shared/standalone/import/data/golf
cp ~/football.mbox /var/discourse/shared/standalone/import/data/football- Use the Ruby console to set each categories’ import directory source. I’m not familiar with the Ruby on Rails development environment, but I was still able to follow Gerhard’s instructions that he gave a user on how to do this. Here’s what Gerhard wrote but I’ve changed it to go with my sports-related example above.
./launcher enter app
rails c
# Use the category ID shown in the URL, for example
# it's 16 when the category's path looks like this: /c/soccer/16
category = Category.find(16)
# Use the directory name where the mbox files are stored. For example,
# when the files are stored in import/data/foo, you should use "foo" as directory name.
category.custom_fields["import_id"] = "soccer"
category.save!Here are a few notes for the above. I relied on a lot of copy/paste of the above lines into my ssh (terminal) window. Nobody wants to type all of that from scratch! And when you enter the line that sets the “category” value, e.g., category = Category.find(16), you’ll see a long list of settings and values, and the last line on your screen will be “:” which is similar to viewing text files with the “more” command. Just press ‘q’ to quit the viewer and you’ll be back to a Ruby console prompt so that you can enter the next two lines that set the “import_id” and then saves your change.
Repeat those 3 category commands for each Topicbox group you are importing.
When you are done, enter the word exit to quit out of the Ruby console. If you enter the word exit again, you’ll be back to a Linux prompt.
- 1.6 – Execute the import script. You can now start the import script. To do so:
/var/discourse/launcher enter import
import_mbox.sh # inside the Docker containerOnce the import script starts to run, you can watch its progress and refer to the notes and suggestions listed in the import guide (section 1.6).
Errors during import. During my import runs, I received several errors on specific emails within the mbox files. Since I didn’t care if a few emails didn’t transfer, I ignored those errors. For example I saw errors like, “Failed to map post for xxxxxx” or “Badly formed IFD”. And a few times, my import kicked out and stopped running. But I just re-ran import_mbox.sh and it picked up where it left off and continued on. I was also able to login to the Discourse website and review the count of users being added and empty topic headings being added. That was cool.
- 1.7 – Finishing the import and Starting Discourse. As long as the import concluded with something similar to “Done (00h 32min 52sec)”, you can follow the guide which says to stop the import container and start discourse.
exit #enter this inside the Docker container
/var/discourse/launcher stop import
/var/discourse/launcher start app #start DiscourseGerhard’s notes indicate that after starting Discourse, a process called Sidekiq will being post-processing all the imported posts, which can take a considerable amount of time. The guide says that you can watch the progress via http://discourse.example.com/sidekiq (substitute your URL but append ‘/sidekiq’ to the end). When all is well, you can proceed to the next major step.
Step 8 – Create a backup of your temporary Discourse instance
After reviewing your temporary instance of Discourse, if all is well, the next step is to run a backup and download the resulting ‘gz’ file.
Step 9 & 10 – Upload that backup file to your production Discourse and Restore
Use your browser to log in to your production instance of Discourse, and upload the ‘gz’ backup file via the Admin –> Backup screen. After the file has uploaded, you are ready to restore it. For hosted customers, you have to contact team@discourse.org and request the restoration. If you are self-hosted, you should have the ability to restore the backup yourself from that same screen.
Wrapping Things Up with Topicbox and Users
Once I had my messages imported into Discourse, I was ready to notify my users and begin retiring the former Topicbox groups. But one thing I noticed is that my total count of users was different between Topicbox and Discourse. The import process creates users in Discourse, but that’s only true of users of Topicbox who posted a message. Lurkers who never post an email won’t be added as Discourse users because there’s no email message from them to scan by the import job. So you’ll need to consider them. My choice was to download the member lists of each group into Excel, and then copy/paste their email addresses into the appropriate group in Discourse.
I also wanted to prevent users from continuing to post messages to Topicbox. So my preference was to retire each group. That option is available under group settings, at the bottom of the screen. After retiring a group, the old messages are still intact and users can still browse them. But if they attempt to message the group, they will get an undeliverable bounce-back.
Good luck!
A note about starting over: after I ran some test imports, I was ready to start over and run a final migration with more recent mbox files. The article written by Gerhard has steps that will delete everything so you can start the import over. However, I noticed I had to frequently run those commands with “sudo”. So I have listed that below.
#Starting an import over entirely (everything is deleted)
cd /var/discourse
./launcher stop app
./launcher stop import
sudo rm -r ./shared/standalone/!(import)
rm ./shared/standalone/import/data/index.db
./launcher rebuild import
# Follow the steps mentioned earlier about using the Ruby console to set
# your category values. Do that before running the import script!
./launcher enter import
import_mbox.sh # inside the Docker container