Hackviking He killed Chuck Norris, he ruled dancing so he took up a new hobby…

10Apr/160

Pi: BtSync satellite – spin down hard drive

My BitTorrent satellite has finally synced my 6tb of data over the local network. The initial sync took several days but so far it seems to pretty quick picking up new files and syncing them. Before I move it to my office I want to make sure I get some peace and quite in the office. I need it to spin down the hard drive when not syncing data. I had the same issue with the BitTorrent Sync server in my apartment always spinning up my NAS but this was actually a bit different.

Since this node uses a USB-disk instead of the network shares on a NAS it can actually do some basic stuff, like indexing, without spinning up the drive. I don't know if it's due to the utilization of Truecrypt or if it's built in but there is some cache which allows the btsync daemon to list the files on disk without the disk spinning up. So I don't have to reconfigure the indexing intervall like I had to on the node uses the NAS. That is communicating over the network to the NFS shares of the NAS and it will spin up it's disk every time someone access it. So there I had to reset the sync intervall to 12 hours. But for my backup solution that will be just fine.

The second thing I was sure I had to change was my script for the LCD display. Since it's reads a JSON file with user credentials from the encrypted disk every 45 seconds I thought it would spin up the drive. No it also ended up cached somewhere and everything is working great at the moment. Have tested throwing new files in there and it synconices just fine! The disk spins up, writes the data and goes back to sleep again after 30 minutes.

To achieve this we need to use hdparm, if your on a Raspberry you need sudo before these commands:

apt-get install hdparm

Then we can run it from the command line:

hdparm - S120 /dev/sda1
/dev/sda:
setting standby to 120 (10 minutes)

To make it persistant after reboot just nano /etc/hdparm.conf, and add this at the end of the file:

/dev/sda1 {
spindown_time = 120
}

So this is the last step before I can move it to my office and really test out the GEO-location backup. Here is a list of the other posts about this:

6Apr/160

Pi: Python script for BtSync status LCD

Adding and LCD display to a Pi project can make it so much easier to use. Displaying current IP address and status of some task that you only have to interact with when something went wrong. In this example we have the 20x4 (20 characters x 4 lines) LCD status display of my BtSync Satellite that I built a while back. Since this box is going to sit on a DHCP network I wanted to display the IP-address so I know what to SSH against. I also wanted to display some status metrics about disk mounts, services and application specific performance counters.

In this scenario the box is running BtSync to keep an offsite encrypted backup off my NAS. For security reasons I have to SSH to the box after a power cycle or failure to enter the encryption key for the disk. That's why I want it to display it's current IP-address on the display. I also want to see the current status of the encryption mount, BtSync service and the upload/downloads going on. That way I know when I have to SSH into the box to sort something out.

So what does this script actually do? It runs an infinit loop until you kill the process. Every 45 seconds it checks the stuff that doesn't need updating all that often and every 3 seconds it checks the current status of the BtSync operations.

Every 45 seconds:

  • Check the current IP-address
  • Check if the Truecrypt volume is actually mounted
  • Check if the BtSync daemon is running

Every 3 seconds:

  • Checks number of files synced
  • Checks number of files to be synced
  • Checks the current download speed
  • Checks the current upload speed

Pre-Requirements

First you need to wire up the LCD, it differs a bit from model to model but there are ton of descriptions on pinouts if you Google your specific model. Then go ahead and run raspi-config or what ever equivalent your brand of Pi uses. Go under Advanced and enable I2C. Then we download some tools that we need:

sudo apt-get install i2c-tools python-dev libxml2-dev libxslt1-dev zlib1g-dev python-smbus

This will install all the things you need to communicate over the GPIO header to your LCD and also libraries needed for the features in the script. Then you can go ahead and download the script:

 wget -O https://raw.githubusercontent.com/kallsbo/PiBtSyncLCD/master/lcd_info.py

Configuration

There are a few configs you can do in the script, just use nano to edit the script file.

# Configuration - LCD
LCD_BUS = 2 # The bus that the LCD is connected to. (Raspberry Pi usually 1, Banana Pi usually 2 - can be checked with i2cdetect)
LCD_I2C_ADDR = 0x27 # I2C device address of the LCD, use i2cdetect to find your displays address
LCD_WIDTH = 20 # Number of characters that each line can handle
LCD_BACKLIGHT = 0x08 # On
#LCD_BACKLIGHT = 0x00 # Off

# Enviorment config
NETWORK_NIC = "eth0" # Network card used
TRUECRYPT_MOUNT_PATH = "/mnt/tc_disk" # path where the truecrypt disk is mounted
BTSYNC_SRV_NAME = "btsync" # name of the btsync service
BTSYNC_URL = "https://localhost:8888/gui/" # Web GUI address for btsync
BTSYNC_CRED_FILE = "/mnt/tc_disk/btsync_cred.json" # JSON file with btsync credentials

Script functions

If we first look at the main method it is simple enough. We run the lcd_init() function to initialize the LCD. All the LCD functions was forked from a script written by Matt Hawkins @ Raspberry Pi Spy. Then we set a simple update counter that keeps track of if the 45 second mark has been hit and if we should check the IP, mount and daemon status. It's initially set to 16 so it will run the first loop and the counter is reset. Then it pluses one for every 3 second run so whenever it's larger then 15 the 45 seconds has elapsed.

get_ip_address() - Simple function that takes the adapter name (eth0) as a parameter and then grabs the current IP-address of that adapter.

is_trucrypt_mounted() - Uses the os.path.ismount() function to check if the mount point is actually utilized by the Truecrypt drive.

get_btsync_cred() - Checks for the json file on the encrypted volume containing the UI username and password for BitTorrent Sync. I used this approach to keep the credentials safe. This function is executed every 45 seconds to make sure that the script get's the credentials when the disk get's mounted.

get_btsync_token() - Sends the initial request to the BitTorrent Sync UI (api) to get the token needed for all the requests to the API. This will also run every 45 seconds to make sure the token never times out and to counter any recycles of the web service.

Every three seconds the script checks if it has the credentials and token needed for the requests and if so runs the get_btsync_info().

get_btsync_info() - This function takes two parameters LLforSpeed and LLforFiles which stands for LCD Line. This value is used to display the information on the LCD panel row you like. It simply builds an url with the GLOBAL credentials and token and get the same json that the UI uses. Then parses it and get the total file count for downloaded files as well for files that are in the download queue. It also grabs the current upload and download speed and converts it to Mb/s and displays it on the LCD.

Credentials JSON file

This is just a plain JSON file containing the credentials. You can modify the script to hard code the credentials in the script but that will impact the security of the script. Here is an example of the credential files:

{
"BTSYNC_USR": "btuser",
"BTSYNC_PSW": ":wDHz56L.blDgM,3Jm"
}

Cred and final thoughts

This is a simple setup for keeping track of your BitTorrent Sync daemon. It can be modified to just display the current info about btsync and not care about Truecrypt and the other extras I implemented for the "satellite" build.

I want to give cred to, as mentioned before, Matt Hawkins for the LCD example scripts that my LCD code is based upon. Also want to thank all bloggers and forum users for the posts I have read to be able to do this. This was my first time ever to use the GPIO header on the Pi for anything else then pre-built stuff like touch displays.

Any questions or suggestions? Please comment! And please follow me on a social media of your choice for updates...

3Apr/160

Pi: Geo-location backup with BtSync

Building a geo-location backup for your NAS is a good idea! To spread the risk over two or more locations increases your backup value a lot. Most people confuse redundancy and backup. If you only have a USB-disk backup of your NAS it only protects you against hardware failure. If there is a fire or a break in you will still lose your data. A lot of people take a USB-disk to a second location, like their office, to mitigate this problem. But to be honest how often will that backup be done if you have to remember to bring the disk back and forth? We want automatic backups to our offsite location, in this case my office. So we are going to build a BitTorrent Sync "satellite"
Continue reading...

1Feb/143

Python: Remove querystring from URL

Needed to clean an URL from it's querystring in Python. Found a lot of examples telling me to use urlparse and then put all the bits and pieces back together. An easier and more efficient  way is this:

url = 'http://www.hackviking.com/?var=value'
url = url[:url.find('?')]

url now reads 'http://www.hackviking.com/'

30Oct/130

Google Picasa API Python: Developers Guide

Before you ask... Yes I struggled with the title of this post, but give me a break it's 4:30 am here! 🙂

I'm currently developing in python for Google App Engine and are using the Picasa API. I found this great guide from Google with exampels: https://developers.google.com/picasa-web/docs/1.0/developers_guide_python It's far from complete but the examples pushes me in the right direction and then some testing, trial and error gives me a chance to learn even more! So i thought I would share one of the things that got me in to trouble this morning.

I was trying to add a new album to the feed and got this error:

(404, 'Not Found', 'Unknown user.')

Even though the user is authenticated via oauth. But then i realized that this was the same issue I head before, the users e-mail isn't in the mix. I needed it for layout purposes before, to display profiles and so on, you can read about that here: Python: Get user info after oauth All the request I have done before, getting feeds of albums, pictures and so on, has been uri input from me in the code. This function just looks like this:

gd_client.InsertAlbum(title='Test', summary='a test album')

Where the gd_client is an instance of gdata.photos.service.PhotosService() with the oauth token all-ready specified using the code example from the official Google guide and spiced with the functions from my previous mentioned post. But still it doesn't work! So I realized that even though it is authorized it has no idea who the user is trying to insert the new album. According to the feed documentation you should do a post like this:

POST https://picasaweb.google.com/data/feed/api/user/userID

<entry xmlns='http://www.w3.org/2005/Atom'
xmlns:media='http://search.yahoo.com/mrss/'
xmlns:gphoto='http://schemas.google.com/photos/2007'>
<title type='text'>Trip To Italy</title>
<summary type='text'>This was the recent trip I took to Italy.</summary>
<gphoto:location>Italy</gphoto:location>
<gphoto:access>public</gphoto:access>
<gphoto:timestamp>1152255600000</gphoto:timestamp>
<media:group>
<media:keywords>italy, vacation</media:keywords>
</media:group>
<category scheme='http://schemas.google.com/g/2005#kind'
term='http://schemas.google.com/photos/2007#album'></category>
</entry>

So here is what stuff goes wrong. Looking what the gd_client is actually doing is posting to https://picasaweb.google.com/data/feed/api/user/. There is a really simple soloution for this, if we return to the oauth example from the guide:

gd_client = gdata.photos.service.PhotosService()
gd_client.auth_token = authsub_token
gd_client.UpgradeToSessionToken()

This is a cut out where you get the "use once" token back and upgrade it to a session token. Here you can do two things to make the gd_client work with the oauth user as a default all the time, either:

gd_client = gdata.photos.service.PhotoService(email='default')

or

gd_client.email = 'default'

Then it all will work just fine!

21Oct/135

Python: Get user info after oauth

Building webapps with Google APIs are pretty straight forward for the most part. Specially when using the Google API Client Libraries! But if you want to accomplish something that are outside of all the nice tutorials that Google supply us with it get's trickier! Being a C# programmer for many years I have noticed that there is loads of examples, tutorials and how-to write ups on the web for any and (almost) all problems I might run into. With the Google APIs and Google App Engine, not as much. I don't know why it should be pretty many people in that community willing to share?

Background

I'm currently developing a web application running on Google App Engine that uses the Picasa Web API. User get's redirected to the oauth site to approve access and I get a token back, easy enough! Getting lists of albums, pictures and movies is easy but getting user info? I'm fairly new to both python and Google App Engine so I browsed around for a long while without finding any solution to what I wanted to do! So before going to bed, defeated, I published a question on stackoverflow: http://stackoverflow.com/questions/19473250/how-to-get-user-email-after-oauth-with-google-api-python-client

Usually you get pretty quick answers from your peers but this time, nothing! Again... I'm I the only one doing stuff like this? Doubt it!
So I needed to solve the issue my self, getting dirty!

Break it down...

So my problem was that I wanted the users profile information like email, name and so on. I started by reading the protocol definition for the Picasa Web API, found here: https://developers.google.com/picasa-web/docs/2.0/reference Soon I realized that what I needed wasn't even sent back to me. So I needed to request the information!

After some Googleing for the issue I found a couple of suggestions that I should use Google Contacts API and get the author field from my own contacts, not a pretty solution!

Went over to the OAuth 2.0 Playground to check out what feeds would help me get the user info. In the list on the playground I didn't find much! Some of the other services provided the information with there feeds but I wanted something that just replied the information that I wanted. So I went to the source, Google APIs Explorer. I found a couple of scopes that would fit my needs:

  • https://www.googleapis.com/auth/userinfo.email
  • https://www.googleapis.com/auth/userinfo.profile

Checking out the scopes

So over to the Oauth playground and tried them out to see what info I could get out of them.

Scope: https://www.googleapis.com/auth/userinfo.email

Request URL: https://www.googleapis.com/userinfo/v2/me

Response:

{
"id": "113303504361682404626",
"email": "puh.kallsbo@gmail.com",
"verified_email": true
}

More or less exactly what I was looking for, needed the e-mail to get the gravatar. But why stop here let's see what else is available...

Scope: https://www.googleapis.com/auth/userinfo.profile

Request URL: https://www.googleapis.com/userinfo/v2/me

Response:

{
"id": "113303504361682404626",
"name": "Kristofer Källsbo",
"given_name": "Kristofer",
"family_name": "Källsbo",
"link": "https://profiles.google.com/puh.kallsbo",
"picture": "https://lh3.googleusercontent.com/-n6hDipRgm5k/AAAAAAAAAAI/AAAAAAAAVYY/XZd9zlnMmGk/photo.jpg",
"gender": "male",
"locale": "en"
}

So this more or less gave me all the user info but not the e-mail. So combining this with the one above would be a good idea!

Getting down to business (python)

So now I knew where to find the information. But how do a I access it as well as the Picasa Web API? So if we start from this exampel: https://developers.google.com/picasa-web/docs/1.0/developers_guide_python?csw=1#AuthSub

def GetAuthSubUrl():
next = 'http://www.example.com/welcome.pyc'
scope = 'https://picasaweb.google.com/data/'
secure = False
session = True
gd_client = gdata.photos.service.PhotosService()
return gd_client.GenerateAuthSubURL(next, scope, secure, session);
authSubUrl = GetAuthSubUrl();
print '&lt;a href="%s"&gt;Login to your Google account&lt;/a&gt;' % authSubUrl

So we need to add an additional scope for the user to authorize. Instead of passing a string as the scope variable we can pass a list of strings like this:

scope = ['https://picasaweb.google.com/data/', 'https://www.googleapis.com/auth/userinfo.email']

That will generate a token that is valid for both scopes! Following the example about the Picasa Web API we go ahead and retrieve a list of the users albums, but then what?

#Setup a basic gdata client
userClient = gdata.client.GDClient()
#Create a gauth sub token object from the session token we allready have in the gd_client
oauthToken = gdata.gauth.AuthSubToken(gd_client.GetAuthSubToken())
#Get the request, this userinfo is in json
response = userClient.request('GET','https://www.googleapis.com/userinfo/v2/me', oauthToken)
#Parse the webresponse
jobj = json.loads(response.read())
#Display
self.response.write('Your email is: {0}'.format(jobj['email']))

We start a gdata client and use that to retrieve the json. Line by line we actually doing the following:

userClient = gdata.client.GDClient() - Just creating an instanse of the Gdata client.

oauthToken = gdata.gauth.AuthSubToken(gd_client.GetAuthSubToken()) - Retrieving the OAuth token all ready used in the gd_client for the Picasa Web Albums. To be able to use it in the Gdata client we have to pass it as an object of type AuthSubToken or similar.

response = userClient.request('GET','https://www.googleapis.com/userinfo/v2/me', oauthToken) - Will return a standard python httplib HTTPResponse with the data returned from the server.

jobj = json.loads(response.read()) - The .read() function returns the body of the HTTPResponse object that we then can parse into a json object.

self.response.write('Your email is: {0}'.format(jobj['email'])) - Finally we have the users e-mail to do what we need to display profile info and so on...

Conclusion

Not all the documentation is great when it comes to Google APIs and Google App Engine. I have been programming sense the age of 8 so I usually find my way through it but I realize that this isn't a beginners language or field. On the other hand all the technical info is there like protocol specifications and stuff like that. Then there is some great tools to try the API out like OAuth playground. I hope this helps someone and a complete demo project for Google App Engine is available on Google Code, link below.

Links