Hackviking He killed Chuck Norris, he ruled dancing so he took up a new hobby…

18Aug/160

Google Compute Engine: Monitor disk usage with Stackdriver

Setting up monitoring of your cloud servers are not only useful to get alerts when a server goes down or run out of disk. It's also very good to have historical data over performance metrics when troubleshooting issues. In this post I cover the basic setup and issues you can run into using Stackdriver for monitoring your Google Compute Engine servers. There are several Stackdriver features already plugged into the Google Cloud, we will focus on monitoring. Keeping track of our virtual infrastructure and use. Out of the box, just by enabling monitoring in the Google Cloud console, it will collect 5 basic metrics for all your instances.

  • CPU Usage
  • Disk Read I/O
  • Disk Write I/O
  • Network Inbound Traffic
  • Network Outbound Traffic

To get to a basic monitoring level we need at least memory and disk usage. Then we can start to look into more metrics in regards to applications and their performance.

Monitoring agent

To be able to collect these metrics we need to install the Stackdriver agent on the servers. Google have install instructions for the agent in their documentation where you can get a basic understanding of the whole setup process. It's actually very straight forward as long as you didn't change any of the "Cloud API access scopes" when you created the VM. If so make sure that your VM have at least "Write only" for the "Cloud Monitor API". Then you can just download the agent from the link in the install instructions and you will see additional metrics come in. The additional metrics are:

  • Memory usage
  • Open TCP connections
  • Page File Usage
  • Volume Usage (disk usage)

Issues with Volume Usage (disk usage)

So far I have installed the agent on 20 windows machines and all of them report all additional metrics except for the the disk usage.  In high through put and data intensive solutions this is one of the most important metrics. After ours of trouble shooting and browsing of documentation and forums I realized that I'm not the only one having the problem but no one had a good solution for it.  I then noticed that the download link for windows in the install instructions was named stackdriverInstaller-GCM-17.exe while the latest I could find directly from Stackdriver was stackdriverInstaller-39.exe. This leads me to believe that the versioned linked from the install instructions are outdated.

The GCM branded one is an automatic install, no input needed. The one downloaded from Stackdriver needs the API key to install. I couldn't find a good download page on the Stackdriver homepage but after Googling found a Support Center entry linking to their repo. At the same time this entry is from April 21st 2015 it seems to be outdated. I did however try different version numbers on the download link and 39 seems to be the latest one. Anyhow it's much never then 17 at least, but as stated in the Google install instructions their is no way to check the current version of the Stackdriver agent currently installed on windows.

Enough about that! This install requires you to input your Stackdriver API key. If you open up the Stackdriver web-ui via the link in the Google Cloud Console and go under "Account settings" and "Agent" you will find it there. Account settings are found under the project dropdown next to the Stackdriver logo in the top left corner of the UI. Just copy the "Key" and past it into the install wizard.

Dashboards and filtering

Now you can create dashboards with different metric charts to get a good overview of your system. The charts can be filtered on resource name via regex. So far I have not been able to filter out specific drive letters. In the view, as well as the underlying JSON, the data is in the format of {instance name}(C:) for example. So a regex like ^.*\(C:\) should match all C:\ drives but it doesn't work. It's not a big issue but there is a few improvements that I hope will come shortly. We have to remember that at this point Stackdriver functionality comes to Google Cloud as beta and does not have any SLA at all.

15Aug/160

Visual studio: Dump data to file from immediate window

Friday afternoon I started a test run against an API to sort some file issues out. Saving the result in a List<string> while running to dump it out to a text file when done. Due to bad formatting of the filename I got an unhandled exception. Quick test code doesn't always include try-catch statements where they should as you know. Since the the information was in the list I went ahead and dumped it out to a new text file via the immediate window.

You can more or less run any code via the immediate window as long as everything you need are in the current context. I wrote a short example on how to do this if you ever run into the same issue. Immediate-window-demo on Github.

10Aug/168

Unable to delete file: System cannot find the file specified

Running hybrid systems spanning from windows to different flavors of Linux sometimes present you with interesting behavior. One that I have faced every now and then is files that you can't delete due to special characters in the filename. They do show up in the file explorer but when you try to delete them you get "Item not find" or similar error. Seen a lot of different solutions online with third party software and other complex solutions but there is two simple "built-in" ways to deal with this in windows.

dir /x method

Open up a cmd window and navigate to the folder in question. Run a simple dir /x command and it will list the files with the non-8dot3 short names. Then you can just go del {non-8.3-filename} and you will get rid of the file.

rd /s "\\?\c:\temp" method

Not all files generate the non-8dot3 name for some reason, don't ask me why - didn't dig that deep. For this there is a solution as well. In this scenario make sure that the files you want to get rid of are the only one/ones in the directory and run rd /s "\\?\C:\folder\containing\problem\file". This command will remove all the files and the directory as well.

9Aug/162

Front-end Demos on Github

I really like both JSFiddle and Plunker but they come with limitations. To counteract cross site scripting issues and other security concerns they sandbox the code with iFrames and similar methods. That is just fine when you do simple examples of front-end implementations. I actually managed to implement a connection to Google oAuth ina JSFiddle but it was hard and requires several re-loads of the page before the user actually get anywhere. Plunker suffer from similar limitations but is much better for larger demos since you can split the code into several files. I also like their editor better.

For more complex demos that implement oAuth, requiring post-backs or other more advanced features I'm now using the Github.io pages. Simply put the service allows static pages, which is great for JavaScript demos, to be served straight from a Github repository!

First you setup a repository named {githubusername}.github.io and check in whatever HTML, CSS and JavaScript content you'd like to run. This will be accessible via the {githubusername}.github.io web address. It has support for SSL (HTTPS) as well as custom domain names. If you use a custom domain name you will not be able to use SSL at this time. I have seen workarounds with CDN solutions like CloudFlare but I haven't tested it my self yet.

For any other repository you create you can commit files to a branch named "gh-pages" and they will be served at {githubusername}.github.io/{repositoryname}. In my case I put a "front page" in my {githubusername}.github.io repository linking other demos but you could actually build a complete blog in that space if you like. So far I have only scratched the surface of what can be done with this but there are a lot of information out there. At the bottom of this post there is a link to some more information to get you started.

The reason I ended up with this solution was due to JSFiddle/Plunker implementation complexity for my latest demo. When I moved to the US from Sweden my phone took care of my phone numbers missing country code and allowed me to dial them as +46. When I used Skype to dial them it just got the original number entered in the phone without country code. One big difference between the dial screen in Skype and the phone is that you can't edit the phone number in Skype. You have to delete the whole thing and type it in with country code and then the number. Since I sync my phone with Google Contacts I figured I'll use their API and a E164 (country code + phone number) javascript library to update all contacts in my address book.

Since the code use for my self was a bit of a one of, I now type in new numbers with country code, I thought I'll make a functioning demo out of the code. If people want to use it to correct their own address book they can. At the same time it's a complete Google oAuth, API implementation demo written in AngularJS.

E164 formatter demo: https://kallsbo.github.io/gcontactse164/
G
ithub demos: https://kallsbo.github.io

More info on Github pages: https://pages.github.com/

In the menu above you will find links to my demos on JSFiddle, Plunker and Github.

14Jun/160

Control Kodi with your TV-remote without CEC

HDMI-CEC is a wonderful thing when you have it! It enables your TV to relay control signals over the HDMI cable which gives you the ability to control your Kodi mediaplayer, or similar device, with your TV remote. Unfortunately not all TV's support this and that leaves you with a few options with disadvantages.

The smart phone apps available for controlling Kodi are good, no question about it. But every time I put on my TV I need to find my smartphone or tablet and make sure I'm on the correct wifi. Call me old fashioned but I like the TV as a semi stand alone device if you know what I mean.

Secondary remotes come in a number of different types. There are the regular IR based once that require additional hardware, if you don't have IR reception on your device. Since all my Kodi boxes are Raspberry Pi based I need additional hardware. There is also stand alone remotes connected to wifi ro Bluetooth which are pretty good but expensive. As soon as your dog, child or other semi destructive member of your household get their hands on it will be broken or gone.

As I mentioned before you need additional hardware on the Raspberry Pi to support a remote control unless it's communicating over the HDMI port. I also want a cheap solution that I can easily replace if damaged or lost. And if possible I don't want a second remote control for my TV setup. Here comes Flirc and saves the day!

Flirc is a programmable USB IR-receiver that can be used with any remote control! It even has a profile for Kodi in it's setup application. It's usually recommended for Kodi users in combination with a standard Apple TV remote. Even if I like the Apple TV remote it's still a second remote. There are a number of unused buttons on any TV remote that can be programmed into the Flirc. In my case I realized that the up, down, right and left keys on my TV remote, that are crucial for Kodi operations, where unused while not in the menu system of my TV. If I tried to use them without the menu open there where no response from the TV at all.

The setup of the Flirc is really easy! Just connect it to your computer and download the software, select the profile and then program the buttons you want. Then just unplug it and connect it to your Kodi box. It is also very sensitive for IR-signals, all my Raspberry Pi's are strapped to the back of my TV furniture but thanks to my white walls the IR-signal bounces in there and are picked up without any problem.

It's currently priced under $23 on Amazon and really worth it. It can be used with any remote and you can replace your TV set and just reprogram it if needed.

 

13Jun/161

WinSCP: SFTP – FTP over SSH

It was a while since I did a tool post and I realize that many people doesn't know about WinSCP or even SFTP, FTP over SSH. I use it all the time to quickly transfer files to Linux based boxes like Raspberry Pi or my Amazon Web Services VPS machines. As long as you have SSH access you can use WinSCP to transfer files. You can set it up to use sudo and make every part of the file system writable but I wouldn't recommend it, it's easy to make a mistake that destroys your system - especially if your working with remote systems. By default WinSCP, or other SFTP clients, end up in the logged in users home directory. If you then need the files anywhere else on the system you can use an SSH client, like Putty, to move the files to the correct location later.

winscp

12Jun/162

Raspbian Jessie: Set a static IP-address

For many of my projects on the Raspberry Pi a static, or fixed, IP-address has been needed. Here is a quick tutorial on how to set it up. This is aimed for SSH users who have no GUI on there Pi. Before you configured this by editing the network interfaces config file but not any more. Raspbian Jessie comes with dhcpcd5 by default and you can uninstall it it's just easier to append to it's configuration. Start by opening it's configuration.

sudo nano /etc/dhcpcd.conf

dhcpcd.conf

At the end of this line you can add a static IP-address configuration. Here is an example:

#static ip
interface eth0
static ip_address=192.168.0.3/24
static routers=192.168.0.1
static domain_name_servers=8.8.8.8 8.8.4.4

First we specify the interface eth0 then all options follows with the prefix static. Ip address is specified with subnet, /24 is the equivalent of a subnet mask of 255.255.255.0. We also specify the networks default gateway for all traffic that will leave the network. In most cases this is your router for home built projects. We also need some DNS servers so we can use FQDN instead of just ip addresses when we communicate. In this example I have used the two Google DNS servers.

3Jun/160

HifiBerry Dac on Raspberry Pi OpenElec Kodi

Have been experiencing static noise from the analog output on my Raspberry Pi for some time. Tested several different power sources and it came to a point where I wasn't sure if the noise has always been present or not. In the end I got a HifiBerry Dac+ and some decent cables and the issue went away. The installation was really easy and well documented in the HifiBerry Knowledge base. In the video above you can see the difference, I very much recommend this!

20May/160

BtSync: Refuses to connect to any peers

Have a few ARM based nodes running BitTorrent Sync (btsync) and needed to re-install one of them. Trying to remove it I ended up with my main node (owner) for all my folders to stop connecting to peers or accepting incoming connections. Took me a while to figure out a solution and I couldn't find much about it on the forums or when I googled so I thought I'll share this quick story.

Background

This applies, in my case at least, to the distribution installed via apt-get from YeaSoft. After reading a forum thread about how to remove old and abandoned peers I decided to set the peer_expiration_days setting to 0 to clean the old peer out. So I used the dpkg-reconfigure btsync command and set it to 0. The old peers where cleaned out so I went back to revert the config back to it's original. In the "wizard" it stated that leaving it blank would render the default value of 7 days. So I removed the 0 and saved the configuration. This might have been a mistake on my part but the configuration tool actually seems broken in this distribution.

Error

After doing that I could not set the value via dpkg-reconfigure btsync to anything else and no peers could connect or where contacted. Right after recycling the daemon they showed up for a few seconds and then disconnected. Since I'm running the free, unlicensed version, I can't switch owner of the folders so I had to get this online again. Changing config files didn't matter since they were, as stated in them, overwritten every time the daemon started again.

Solution

Finally I downloaded the latest version from the getsync.com website and unpacked it in the temp folder. Looking at the command line used in the /etc/init.d/btsync script I could find what config file it used. So I started the latest version, which have support for "power user options" in the UI, with the same config file parameter. Went in to the UI and changed the peer_expiration_days back to it's original value, there even is a reset value link. Then shot down the process and started the original daemon with init.d and order where restored.

18May/1616

WD MyPassport Wireless with BitTorrent Sync

SD-card backups in the field as well as automatic upload to your NAS whenever you have an internet connection? Yes it can be done with Western Digital MyPassport Wireless and BitTorrent Sync!

In the past I have always uploaded the footage from my GoPro and compact camera via my laptop in the hotel rooms (or where ever I can find an internet connection). I wanted a more streamlined process and an easy way to empty an SD-card in the field. The MyPassport Wireless takes care of the first problem out of the box. With a built in SD-card reader you can move all content on an SD-card to the built in hard drive. Just set it up to do an automatic copy as soon as an SD-card is present in the reader and it will dump everything to disc.

I also wanted to make an "of site" backup whenever possible. I have been setting up a geo-location backup built on BitTorrent Sync so I already have those servers in place. Since the MyPassport Wireless is ARM based, like a Raspberry Pi, it's pretty straight forward to setup BtSync on this device. It also has built in support for accessing wifi hotspots and connect to the Internet. This can be done via the mobile app. So this guide will give you the following functionality:

  • Dump SD-cards to a hard drive just by carrying the small WD MyPassport Wireless
  • When ever it's conected to the Internet it will start to sync all the new data to your BitTorrent Sync servers.

I'm not sure what the WD warranty would say about this so you do this on your own risk. There is no package manager installed on the MyPassport Wireless so it all have to be done manually. I'm making no claims that this is the best way to do it but I have been testing it out for a while now and it works great so far. Continue reading...