GoToSocial and Digital Ocean with either S3 or Volume Storage

What type of hardware are you using: VPS bought online
What YunoHost version are you running: 12.0.11
What app is this about: GoToSocial

Describe your issue

1 Core
1 gb RAM
24 gb storage

I hope to futureproof my GTS instance’s storage.

Installed Yunohost and GoToSocial, very happy with the result. I went ahead and moved my follow list over, and GTS populated the timeline. At fresh installation, VPS storage space started at 60% used. After about 3 days, storage space used has slowly climbed to 78%. From the GoToSocial Documentation, this set up should last for a few years for someone with a few hundred follows. Admittedly, I probably have closer to 1000 so I’m not sure how much time I have.

If I get to that bridge, besides pricing, would I just [1] increase my droplet size and features, [2] look at S3 in Digital Ocean Spaces, or [3] add Volume Storage.

For [1], looks like I wouldn’t be doing much else other than selecting more cores/ram/storage, and that’s it.

For [2], S3, GTS Documentation provides for a config file that looks fairly safe for me. If I do end up taking this route, at some point I imagine I’d have to move or copy over the GTS storage-local-base-path to wherever my bucket ends up. Is there a command to stop GTS so that this copy procedure can be completed?

For [3], I imagine it’s close to [2], and I still have to change the storage-local-base-path to wherever the Volume Storage ends up.

Obviously, I’m hoping I can continue at this VPS size. I suppose if the time comes, [1] may be the gateway for me for more self-hosting. [2] I don’t really get yet. I imagine I will once I continue reading more about it. At this time it seems like a feature intended to increase the availability of a server’s assets. [3] more or less sounds like connecting another hard drive to a computer.

Thank you everyone, for your time.

I am grateful to Yunohost and all of you for making it possible for even me to self-host.

Share relevant logs or error messages

n/a

Just by way of update here, looks like my VPS is now at 86.5% disk usage and there’s no sign I’ve plateaued. I must admit though, that my GTS instance is running very smooth at 1 core and 1 gb ram otherwise.

I looked over the /var/www/gotosocial/config.yaml and saw the below warning at the top of the file.

I don’t have a github account, but I suppose I should consider one if I need to make an issue.

In the meantime, I think I’ll modestly increase my VPS size to 35 GB to see if that will be enough.

:warning: IMPORTANT WARNING: do NOT edit this config file by hand,

else your modifications will be ERASED at each update of the GTS package!

Instead, use the config panel in your web admin interface:

Applications :arrow_right: GoToSocial :arrow_right: Config panel :arrow_right: do your config edit :arrow_right: Save

Most of the relevant settings are available in the config panel, if you’re

missing one, open an issue: GitHub · Where software is built

hi and welcome to the yunohost community!

i added the S3 settings to the config panel

you can test it by updating the package:
yunohost app upgrade gotosocial -u https://github.com/YunoHost-Apps/gotosocial_ynh/tree/s3-config-panel --force

then by configuring your S3 bucket in the config panel, under Applications :arrow_right: GoToSocial :arrow_right: Config panel :arrow_right: Storage settings :arrow_right: do your config edit :arrow_right: Save

you can also do your configuration by using the cli: yunohost app config set gotosocial main.storage

you will have to follow the official docs for migrating your data to the S3 bucket: Storage - GoToSocial Documentation
note: the storage base path is /home/yunohost.app/gotosocial

please tell me if everything is alright or not, so i could push the new config panel to the main branch

2 Likes

@OniriCorpe wow, thank you very much for your reply, and for adding settings to the config panel.

I did take a more conservative approach by increasing the storage space modestly. I do see that GTS continues to use more space so I’m going to keep an eye on it before I decide to take the step up to a S3 bucket. I’m mainly hesitant because I want to take more time to learn about this bucket idea.

Thanks to your help, setting up the DigitalOcean bucket, and editing the GTS config sounds simple.

I grabbed my gotosocial user id and group id. In my case it was 991:991

cat /etc/passwd

I set up the DigitalOcean Space and obtained my Access Key Name, Access Key ID, and Secret.

I then followed this procedure to set up a folder to mount my DigitalOcean space within my droplet with s3fs.

I tested out my mounted space and it seems to work as intended.

So now I’m wondering what the best way to do this is. Should I just rsync the contents of /home/yunohost.app/gotosocial into my new spaces mounted directory?

Alternatively, I found a brief introduction to s3cmd where there’s some kind of sync feature. I’m not immediately clear how I could use it. I’m inclined to use the rsync idea.

@OniriCorpe Happy to report that all appears to be working excellently.

I’ll come back and try to write up the steps I did to help out later. I probably didn’t take all the brightest steps, and will need correction in the future.

2 Likes

I just wanted to document my steps so that others can follow. I’m writing with the intent of guiding people who are right at my skill level - dangerous with a little bit of knowledge.

Digital Ocean

For me, I went with Digital Ocean. Amazon and other service providers will have their own service to enable their users to interact with their S3. You’ll have to review their HOW-TOs, but I imagine they would have similar features.

In DO’s case, you can consider installing s3fs and s3cmd. For s3fs, this command enables you to mount a bucket (or space in DO lingo) to a directory. You can move files into the space as needed once you’re done. You’ll also configure your directory to enable the user and group permissions in favour of GoToSocial so GTS can access the files properly.

My scenario is I set up GTS and ran it for a few days before realizing that I would need additional space. So I’ll

[1] mount my space to a directory of my choice,
[2] backup my original Storage Base Path contents into a folder in my space with rsync,
[3] rename the original Storage Base Path to something else like GoToSocial-backup,
[4] mount my space again, but this time with the same Storage Base Path

NOTE: This is what I did, but just know that once GTS is configured to use the S3, it’ll just directly dump the contents of the Storage Base Path files into the bucket. You can just copy the contents of the Storage Base Path directly into the S3 bucket, configure GTS to use the S3, and call it a day. As I never used an S3 before, I back things up and keep stuff around in the event I break something.

As you’ll see, I ended up making a directory within my bucket and copied over my original files into it. In hindsight, you probably don’t have to do that.

When you go into your DO web admin space, you’ll be given the option to set up your Spaces Object Storage. You’ll be asked to associate the Space Bucket with an existing Datacenter, which is also associated with your existing droplets. Just select the one where your GTS server is.

I did not enable a CDN at this time.

You will also create a space bucket name, preferably short and nondescript.

You will also select the project.

Once you create your bucket, there will be options to generate an Access Key. For a learner like myself, I chose an all access type key, but select the one that makes sense to your use.

Copy and set aside your Origin Endpoint, Access Key ID, Security Key, S3 Endpoint.

SPECIAL NOTE: You will only be presented your Security Key once. If you lose it, you’ll have to generate a new one.

Now refer to this link to install and configure s3fs:

If you need to figure out the user ID and group ID for GoToSocial, simply

cat /etc/passwd

Go ahead and set aside the IDs for later.

NOTE: I don’t think I needed s3cmd to actually complete my task, but it’s a good learning opportunity and you can explore this command later with the link below:

[1] mount my space to a directory of my choice:

You can make a mount point anywhere, and the purpose is to back up the existing GoToSocial Storage Base Path. If you’re starting from scratch and already decided you needed to set up an S3 upfront, you don’t need to worry about backing up and existing directory.

For example, we’ll call this folder “gts”, and you can change the [region] accordingly. Now you’ll also change your UID and GID to match your GoToSocial (don’t just punch in 666)

s3fs bucketname:/path/to/folder/ gts/ -o url=https://[region].digitaloceanspaces.com -o use_cache=/tmp -o allow_other -o use_path_request_style -o uid=666 -o gid=666

You can test this out by making a test folder in your new mounted directory. Then you’ll load up the space on your web panel in Digital Ocean to see if everything matches up.

Congrats, you’ve just used s3fs.

[2] backup my original Storage Base Path contents into a folder in my space with rsync:

For copy and paste jobs that I care about, I use rsync.

apt-get update
apt-get install rsync

rsync -avh /home/yunohost.app/gotosocial /path/to/mounted/space

So with rsync, given the small VPS, this can take hours depending on how large GTS has grown. For reference I copied about 15 GB worth of stuff which took pretty much 4-5 hours.

NOTE: for rsync the path to the original files “/home/yunohost.app/gotosocial” will copy the folder and its contents to the target destination.

Alternatively, ending the original file path with a forward slash “/home/yunohost.app/gotosocial/“ will tell rsync to just copy the contents of the folder to the target destination.

So figure out what you want to do. In my case I did put the contents into a new folder within the bucket.

[3] rename the original Storage Base Path to something else like GoToSocial-backup:

Change directory to /home/yunohost.app/gotosocial

cd /home/yunohost.app/gotosocial
mv /home/yunohost.app/gotosocial /home/yunohost.app/gotosocial-back

You’ll end up with your original Storage Base Path renamed as “gotosocial-back”.

[4] mount my space again, but this time with the same Storage Base Path:

I created the same folder “gotosocial”, but this time mounted it and directed the path to my files that were copied over to my bucket earlier with rsync.

s3fs bucketname:/path/to/folder/ gotosocial/ -o url=https://[region].digitaloceanspaces.com -o use_cache=/tmp -o allow_other -o use_path_request_style -o uid=666 -o gid=666

Now, you can head over to the Yunohost Config Panel for GTS, and simply enter your DO space details using the information you set aside earlier when you made your space. Once you save and let Yunohost start up the service, GTS should directly access the bucket you made to start loading files in.

2 Likes

You may run into an issue with updates to GoToSocial and a permission issue with your Storage Base Path.

Yunohost may not play well with the idea of a s3 mounted to a folder so you just need to turn that off before you run the update to GoToSocial, then turn it back on afterward.