Digital Ocean 'Spaces' mounting and scripts
If you are using DO (Digital Ocean) Spaces feature then you are probably thinking of scripting or even mounting this "share" on your pc or even on Droplet itself.
First, I would like to mention that I recently switch this blog to Digital Ocean and pointed my domain registrar to DO DNS service which is accessible to you as soon as you open your DO account and you can use this DNS completely free. For account creation you need 5$ which later you can use to pay 1 VM (virtual machine) for 1 month. In your account later, you can create your referral links so anyone who registers their account using the link will get 10$ on their account and later if they spend 25$ you get also 25$ on your account. So in light of that, here is mine and you can use it to make your DO account and get your 10$ :)
In short, DO has one of the best interfaces for managing your VMs or, as they call them, Droplets. True, I haven't seen much of them, but I'm sure this is one of the best. You can ofcourse take snapshot of your Droplet for 1$/month. I believe in background are KVM or Xen hypervisors. But this post isn't about all the features on DO so I won't go into much details cause DO documentation is great on every subject you need. Now to mention storage side...
Besides local space on your Droplet you can create block device and mount them to you Droplet. Block device is called Volumes on DO, and basic price is 10$/month for 100GB. You can even go as low as 1$/month for 10G but then losing the point of storage space. Minimum local space on Droplet itself is 25G.
Next thing is Spaces. Spaces are an object storage implementation. Basically you can transfer you data using HTTP API. If you need more details here is great documentation on object and block storage . And now this post is about 2 tools for using Spaces with your linux machine. One is s3cmd command line utility and the other is s3fs for mounting it.
s3cmd  is available on Debian repos but it's on version 1.6.1. So we will manually install it. If you find s4cmd on package list don't bother cause it doesn't work with DO Spaces s3 implementation.
First download latest version using wget:
Then unpack it for example into /opt/s3cmd and create symlink in /usr/local/bin:
sudo ln -s /opt/s3cmd/s3cmd /usr/local/bin/s3cmd
Now on your DO account create your API access keys if you havent already done so. Here is the guide if needed .
After just run
s3cmd --configure for initial configuration. It will create .s3cfg file in your home path. Most values stay at default except few of them. Here it is in short:
[default] access_key = [your_access_key] gpg_command = /usr/bin/gpg #or whereve it is on your system # you can user ```date | md5sum``` to generate gpg gpg_passphrase = 15165dd2e3b97af3ebd00e329730e9bc # you can get endpoint address on your Spaces interfaces host_base = nyc3.digitaloceanspaces.com # just add endpoint address after %(bucket)s host_bucket = %(bucket)s.nyc3.digitaloceanspaces.com secret_key = [your_secret_key] use_https = True ...
Now you can list all your Spaces with:
s3cmd ls 2018-08-07 14:21 s3://space1
Or list all content on your Space:
s3cmd ls s3://space1 DIR s3://space1/test/ DIR s3://space1/test2/ 2018-08-08 13:42 1537768 s3://space1/MegaCli-8.07.07-1.noarch.rpm 2018-08-08 10:15 12749576 s3://space1/teamviewer_13.1.3026_amd64.deb 2018-08-08 11:42 1073741824 s3://space1/testfile
For command reference please use
s3cmd -h any you will get all the commands available for uploading, downloading etc.
This is a great tool to work with your Spaces on DO from command line cause it's easy to use in your scripts for example. Soon as I rewrite my backup script I will share it with you.
If you wanna mount your DO Spaces, you can do it using s3fs .
Clone git repo:
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
And then build it all:
./autogen.sh ./configure --prefix=/opt --with-openssl sudo make sudo make install
I installed it with prefix /opt so just link it's executable to get it into your path:
ln -s /opt/s3fs/bin/s3fs /usr/local/bin/
Now create in your home
.passwd-s3fs file and input access key and secret key:
echo [your_access_key]:[your_secret_key] > ~/.passwd-s3fs chmod 600 ~/.passwd-s3fs
Then mount it through command line:
sudo s3fs space1 /mnt/space1 -ourl=https://nyc3.digitaloceanspaces.com -o passwd_file=~/.passwd-s3fs -o allow_other
Or using fstab:
s3fs#space1 /mnt/space1 fuse _netdev,allow_other,passwd_file=/home/user1/.passwd-s3fs,url=https://nyc3.digitaloceanspaces.com/ 0 0
That's it. But do keep in mind that object storage is not ideal for every day r/w tasks. But, for example, it's good for backup. Not to repeat, as it is said in this article on object vs block storage [1:1]:
...on a filesystem, you can easily append a single line to the end of a log file. On an object storage system, you'd need to retrieve the object, add the new line, and write the entire object back...