Hackviking He killed Chuck Norris, he ruled dancing so he took up a new hobby…

18May/1616

WD MyPassport Wireless with BitTorrent Sync

SD-card backups in the field as well as automatic upload to your NAS whenever you have an internet connection? Yes it can be done with Western Digital MyPassport Wireless and BitTorrent Sync!

In the past I have always uploaded the footage from my GoPro and compact camera via my laptop in the hotel rooms (or where ever I can find an internet connection). I wanted a more streamlined process and an easy way to empty an SD-card in the field. The MyPassport Wireless takes care of the first problem out of the box. With a built in SD-card reader you can move all content on an SD-card to the built in hard drive. Just set it up to do an automatic copy as soon as an SD-card is present in the reader and it will dump everything to disc.

I also wanted to make an "of site" backup whenever possible. I have been setting up a geo-location backup built on BitTorrent Sync so I already have those servers in place. Since the MyPassport Wireless is ARM based, like a Raspberry Pi, it's pretty straight forward to setup BtSync on this device. It also has built in support for accessing wifi hotspots and connect to the Internet. This can be done via the mobile app. So this guide will give you the following functionality:

  • Dump SD-cards to a hard drive just by carrying the small WD MyPassport Wireless
  • When ever it's conected to the Internet it will start to sync all the new data to your BitTorrent Sync servers.

I'm not sure what the WD warranty would say about this so you do this on your own risk. There is no package manager installed on the MyPassport Wireless so it all have to be done manually. I'm making no claims that this is the best way to do it but I have been testing it out for a while now and it works great so far. Continue reading...

10Mar/160

Raspberry Pi: BitTorrent sync alternative Syncthing

Btsync is a great tool for syncing data and backups to several different locations. By leveraging the torrent technology you can increase the download/upload speeds by adding more nodes to the swarm. Installing Btsync on a built in system like the Raspberry Pi or Odroid-C1 is a challenge since it consumes a lot of system resources, read more about installing BitTorrent Sync on Raspberry Pi. Reading up on the subject a lot of people did suggest that Syncthing takes up less system resources then btsync so let's install it and try it out.

Pre requirements

Same as for the btsync install I need to access my NAS root NFS share to get all the data that I will sync. So first I setup a mount point and do a persistent mount of that share.

sudo mkdir /mnt/stroot
sudo chmod 777 /mnt/stroot

Then let's edit /etc/fstab so run sudo nano /etc/fstab and add a line representing your share.

192.168.0.5:/nfs /mnt/stroot nfs rsize=8192,wsize=8192,timeo=14,nolock,intr 0 0

This will give me the root for all the shares so I don't have to map every share individually but are able to access all of them as subfolders.

To test this out we can run sudo mount -a. Then reboot the system and check that the mount is persistent. If you have issues with the persistence and are running DHCP please see my Raspbian:fstab doesn't mount NFS on boot article for reference.

Install

Download the Syncthing repository key and add it.

wget -O - https://syncthing.net/release-key.txt | sudo apt-key add -

Add the repository

echo "deb http://apt.syncthing.net/ syncthing release" | sudo tee /etc/apt/sources.list.d/syncthing.list

Update the repository with sudo apt-get update and then install with

sudo apt-get install syncthing

Then try to run it with syncthing, as soon as you see it start properly you can stop it with CTRL + C. To set it up to run as a service we need first reconfigure the bind address.

nano /home/pi/.config/syncthing/config.xml

Look for the gui option, it should look like this:

<gui enabled="true" tls="false">
<address>127.0.0.1:8384</address>
<apikey>2Yy41r5k5oocz6rDIWVsKeP4wUx9uknD</apikey>
<theme>dark</theme>
</gui>

Here you can change all the settings in regard to the UI. Change the bind address to 0.0.0.0 so it will respond to all ip addresses. If your running on a fixed IP and not DHCP you can go ahead and bind it to a specific address if you like. You can also change the tls setting to true, enabling https for the UI. This will however result in a security warning when you browse to the UI due to the self signed certificate.

If you now run syncthing from the command line it will bind to the IP and you will be able to navigate to the UI.

Autostart Syncthing as a "service"

First we need to create a init.d script for Syncthing.

sudo nano /etc/init.d/syncthing

Copy and paste the following into it:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          Syncthing
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $local_fs $remote_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Syncthing
# Description:       Syncthing is for backups
### END INIT INFO
 
 
# Documentation available at
# http://refspecs.linuxfoundation.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptfunc.html
# Debian provides some extra functions though
. /lib/lsb/init-functions
 
 
DAEMON_NAME="syncthing"
DAEMON_USER=pi
DAEMON_PATH="/usr/bin/syncthing"
DAEMON_OPTS=""
DAEMON_PWD="${PWD}"
DAEMON_DESC=$(get_lsb_header_val $0 "Short-Description")
DAEMON_PID="/var/run/${DAEMON_NAME}.pid"
DAEMON_NICE=0
DAEMON_LOG='/var/log/syncthing'
 
[ -r "/etc/default/${DAEMON_NAME}" ] &amp;&amp; . "/etc/default/${DAEMON_NAME}"
 
do_start() {
  local result
 
	pidofproc -p "${DAEMON_PID}" "${DAEMON_PATH}" &gt; /dev/null
	if [ $? -eq 0 ]; then
		log_warning_msg "${DAEMON_NAME} is already started"
		result=0
	else
		log_daemon_msg "Starting ${DAEMON_DESC}" "${DAEMON_NAME}"
		touch "${DAEMON_LOG}"
		chown $DAEMON_USER "${DAEMON_LOG}"
		chmod u+rw "${DAEMON_LOG}"
		if [ -z "${DAEMON_USER}" ]; then
			start-stop-daemon --start --quiet --oknodo --background \
				--nicelevel $DAEMON_NICE \
				--chdir "${DAEMON_PWD}" \
				--pidfile "${DAEMON_PID}" --make-pidfile \
				--exec "${DAEMON_PATH}" -- $DAEMON_OPTS
			result=$?
		else
			start-stop-daemon --start --quiet --oknodo --background \
				--nicelevel $DAEMON_NICE \
				--chdir "${DAEMON_PWD}" \
				--pidfile "${DAEMON_PID}" --make-pidfile \
				--chuid "${DAEMON_USER}" \
				--exec "${DAEMON_PATH}" -- $DAEMON_OPTS
			result=$?
		fi
		log_end_msg $result
	fi
	return $result
}
 
do_stop() {
	local result
 
	pidofproc -p "${DAEMON_PID}" "${DAEMON_PATH}" &gt; /dev/null
	if [ $? -ne 0 ]; then
		log_warning_msg "${DAEMON_NAME} is not started"
		result=0
	else
		log_daemon_msg "Stopping ${DAEMON_DESC}" "${DAEMON_NAME}"
		killproc -p "${DAEMON_PID}" "${DAEMON_PATH}"
		result=$?
		log_end_msg $result
		rm "${DAEMON_PID}"
	fi
	return $result
}
 
do_restart() {
	local result
	do_stop
	result=$?
	if [ $result = 0 ]; then
		do_start
		result=$?
	fi
	return $result
}
 
do_status() {
	local result
	status_of_proc -p "${DAEMON_PID}" "${DAEMON_PATH}" "${DAEMON_NAME}"
	result=$?
	return $result
}
 
do_usage() {
	echo $"Usage: $0 {start | stop | restart | status}"
	exit 1
}
 
case "$1" in
start)   do_start;   exit $? ;;
stop)    do_stop;    exit $? ;;
restart) do_restart; exit $? ;;
status)  do_status;  exit $? ;;
*)       do_usage;   exit  1 ;;
esac

Make the init.d script executable and make it start on boot.

sudo chmod +x /etc/init.d/syncthing
sudo update-rc.d syncthing defaults

Now you can start the Syncthing service like any other service.

sudo service syncthing start

Conclusion

The reason I looked into Syncthing in the first place was the memory consumption of btsync. After running a few tests I can see that it's the same with Syncthing. By the time I'm writing this my btsync instance running on my Odroid-C1 has index almost 4Tb of data and is running smoothly. Syncthing is opensource, which I like a lot, but it's a much more immature product then btsync. The windows client is just an .exe that will run a local web server that you connect to. Btsync seems more mature and robust from my initial tests. Syncthing will probably develop into something better but at this time I will go with btsync instead. Btsync is also developed by the torrent technology creators which make me believe in it even more. Install guide for btsync...

Why don't you install both of them and try them out and let me know what you think in the comments?

8Mar/160

Raspberry Pi: BitTorrent Sync

BitTorrent is not only pirated movies and tv-shows it's an actual technology for moving large amounts of data between several nodes at once. There are several applications for this sync technology. Large companies like Facebook use it to sync new releases among all their production servers for example. If three or more nodes are syncing the same content it will increase the sync speed since it's a non centralized technology. All the clients will help to seed the data they already have which will make the sync go faster and faster the more nodes the data propagates to.

Btsync will make use off all the memory available in your Raspberry Pi so I recommend running this on it's own. I have tested this out on both a Raspberry Pi 2 B and the older Raspberry Pi B and they more or less run the same. It will take some time to index all your data but when testing it on a regular windows pc it was pretty slow as well. Continue reading...