Hackviking He killed Chuck Norris, he ruled dancing so he took up a new hobby…

4Feb/150

Amazon EC2 Linux – Add additional volumes

EBS Mappings

Adding additional storage to your Amazon EC2 instance have several advantages. You can select the right storage type for the use. Why use a fast SSD backed volume for storing nightly backups instead of magnetic storage, that ar slower but come at a much lower price.

First you need to provision storage and assign it to your instance. Amazon provides a good guide on how to add additional volumes to your instances. There are several advantages to using several different volumes. As I wrote in my guide to move mysql storage you will not risk running the boot disk full witch will make the system halt. Other advantages include the selection of storage fit for your purpose and price range, as mentioned above. External volumes can also easily be migrated between instances if and when you get a need for that. It is also easier when you need to extend your storage space. Instead of making a snapshot of the entire instance and then launching a new one with a bigger drive you can attach new storage and migrate the data. This approach will make the downtime much shorter.

When selecting the correct storage for you solution there are a few things to keep in mind. When it comes to EBS it comes in three basic flavors. All with there benefits and disadvantages, it is there for important to make an educated decision.

Storage Types

Magnetic - This is the most affordable EBS storage for larger volumes with out specific performance needs. That sad it is not slow but a database solution would benefit from better I/O performance. If you want even more TB for your money you should look at S3 which holds larger amounts of data for a smaller cost. How ever you can not mount S3 storage straight to you instance. If you have even larger amounts of data Glacier is a service were you can put your data into archives. It works a lot like S3 but with much slower response times. It is not for average access use but will keep you archived data safe for a small cost.

General Purpose SSD - As the name states this is SSD backed storage. The larger the drive is the better I/O performance it will have. This disk is used for boot volumes as standard on the smaller instances. For most implementations that are small to midsize this is enough for running database files and other performance critical operations. When you need higher performance you need to take a look at provisioned IOPS and instance storage.

Provisioned IOPS SSD - This is for performance intensive applications like database servers. Even though Amazon recommends this for all database implementations General Purpose SSD is enough for most website implementations. My recommendation is to start out on General Purpose SSD and upgrade if needed. The biggest difference is that you will be guaranteed stable IOPS performance at you level of payment. The General Purpose SSD is specified as it could fluctuate in performance but over the last year I haven't seen any problems in IOPS performance on these volumes.

There is also something called Amazon EC2 Instance Store which is storage directly attached to the virtual environment while EBS can be compared with a traditional SAN. This storage is only available to the current instances and can not be detached or transferred between instances. It is however the storage available with the highest performance. It can be used for swap, log files, cache and other implementation where performance is key. When the instance is stopped the data is lost. It is also not redundant storage in the same way EBS is, so you have to keep this data safe and only keep temporary data on it. The instance storage is also not available for all instance types and sizes.

You can also read more about the Amazon EBS Storage types here.

When you have selected the correct storage for your implementation you have to create a file system on it and mount it to you instance before use. It is also a very good idea to "preheat" the storage which will increase the performance. Before all of the EBS volume is accessed the performance is decreased. More information about EBS Volume Types and there characteristics.

Attach a new volume

Follow the guide for attaching new volumes to your instance. Then use lsblk to list the available devices/volumes attached to your instance:

lsblk
 xvda 202:0 0 8G 0 disk
 └─xvda1 202:1 0 8G 0 part /
 xvdb 202:4608 0 10G 0 disk

Here we can see that we have a new 10GB volume of EBS storage attached to our instance. As you can see the volume present it self as xvdb. If you run the ls command in the /dev folder you will see that it also is mapped to sdb. Even though either one can be used Amazon recommends that you use the sdb, sdc, sdd and so on names for interaction with the storage. This is true for both paravirtual and HVM instances. You can read more about Amazon EC2 Block Device Mapping Concepts.

Prepare the storage for use

Preheat the volume by filling it with random data, this will initialize all the storage and therefor give you higher performance. This will take a while depending on the size of the new volume. If you run it twice you can see the performance increase.

sudo dd if=/dev/zero of=/dev/sdb bs=1M

The EBS storage is "raw" disk and doesn't contain any file system. Therefor we create an ext4 file system on the volume.

sudo mkfs -t ext4 /dev/sdb

Then you have to create a directory to mount it in. You can go for a /mnt or /var or what ever you like. On several instances I have mounted new volumes directly into the file system location for www storage or database storage, so I don't have to make any changes on the configuration for the applications. When the directory is in place just mount the volume to that directory.

mkdir /var/datavol
sudo mount /dev/sdb /var/datavol

Make persistent after reboot

To make the instance auto mount the volume on reboot we have to add a line in fstab. This can be done with a text editor like vi but i prefer to just add the line with sed.

sed -i '$ a\/dev/sds /var/www ext4 defaults 0 2' /etc/fstab

If you make an error in fstab it can render the instance unbootable. To test you fstab entries before you reboot just run mount -a, if you don't get any errors the system will be safe to reboot.

Troubleshooting

As I wrote above you should always check your fstab before reboot and correct any errors. If you do the steps above in the wrong order you might end up with the error below. That means that you missed to make, or destroyed, the file system. Just run the step for making a file system again and you should be all set to go!

mount: /dev/xvds is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/xvds,
missing codepage or helper program, or other error

Posted by Kristofer Källsbo

Comments (0) Trackbacks (0)

No comments yet.


Leave a Reply

No trackbacks yet.