Hackviking He killed Chuck Norris, he ruled dancing so he took up a new hobby…

18Aug/160

Google Compute Engine: Monitor disk usage with Stackdriver

Setting up monitoring of your cloud servers are not only useful to get alerts when a server goes down or run out of disk. It's also very good to have historical data over performance metrics when troubleshooting issues. In this post I cover the basic setup and issues you can run into using Stackdriver for monitoring your Google Compute Engine servers. There are several Stackdriver features already plugged into the Google Cloud, we will focus on monitoring. Keeping track of our virtual infrastructure and use. Out of the box, just by enabling monitoring in the Google Cloud console, it will collect 5 basic metrics for all your instances.

  • CPU Usage
  • Disk Read I/O
  • Disk Write I/O
  • Network Inbound Traffic
  • Network Outbound Traffic

To get to a basic monitoring level we need at least memory and disk usage. Then we can start to look into more metrics in regards to applications and their performance.

Monitoring agent

To be able to collect these metrics we need to install the Stackdriver agent on the servers. Google have install instructions for the agent in their documentation where you can get a basic understanding of the whole setup process. It's actually very straight forward as long as you didn't change any of the "Cloud API access scopes" when you created the VM. If so make sure that your VM have at least "Write only" for the "Cloud Monitor API". Then you can just download the agent from the link in the install instructions and you will see additional metrics come in. The additional metrics are:

  • Memory usage
  • Open TCP connections
  • Page File Usage
  • Volume Usage (disk usage)

Issues with Volume Usage (disk usage)

So far I have installed the agent on 20 windows machines and all of them report all additional metrics except for the the disk usage.  In high through put and data intensive solutions this is one of the most important metrics. After ours of trouble shooting and browsing of documentation and forums I realized that I'm not the only one having the problem but no one had a good solution for it.  I then noticed that the download link for windows in the install instructions was named stackdriverInstaller-GCM-17.exe while the latest I could find directly from Stackdriver was stackdriverInstaller-39.exe. This leads me to believe that the versioned linked from the install instructions are outdated.

The GCM branded one is an automatic install, no input needed. The one downloaded from Stackdriver needs the API key to install. I couldn't find a good download page on the Stackdriver homepage but after Googling found a Support Center entry linking to their repo. At the same time this entry is from April 21st 2015 it seems to be outdated. I did however try different version numbers on the download link and 39 seems to be the latest one. Anyhow it's much never then 17 at least, but as stated in the Google install instructions their is no way to check the current version of the Stackdriver agent currently installed on windows.

Enough about that! This install requires you to input your Stackdriver API key. If you open up the Stackdriver web-ui via the link in the Google Cloud Console and go under "Account settings" and "Agent" you will find it there. Account settings are found under the project dropdown next to the Stackdriver logo in the top left corner of the UI. Just copy the "Key" and past it into the install wizard.

Dashboards and filtering

Now you can create dashboards with different metric charts to get a good overview of your system. The charts can be filtered on resource name via regex. So far I have not been able to filter out specific drive letters. In the view, as well as the underlying JSON, the data is in the format of {instance name}(C:) for example. So a regex like ^.*\(C:\) should match all C:\ drives but it doesn't work. It's not a big issue but there is a few improvements that I hope will come shortly. We have to remember that at this point Stackdriver functionality comes to Google Cloud as beta and does not have any SLA at all.

9Aug/162

Front-end Demos on Github

I really like both JSFiddle and Plunker but they come with limitations. To counteract cross site scripting issues and other security concerns they sandbox the code with iFrames and similar methods. That is just fine when you do simple examples of front-end implementations. I actually managed to implement a connection to Google oAuth ina JSFiddle but it was hard and requires several re-loads of the page before the user actually get anywhere. Plunker suffer from similar limitations but is much better for larger demos since you can split the code into several files. I also like their editor better.

For more complex demos that implement oAuth, requiring post-backs or other more advanced features I'm now using the Github.io pages. Simply put the service allows static pages, which is great for JavaScript demos, to be served straight from a Github repository!

First you setup a repository named {githubusername}.github.io and check in whatever HTML, CSS and JavaScript content you'd like to run. This will be accessible via the {githubusername}.github.io web address. It has support for SSL (HTTPS) as well as custom domain names. If you use a custom domain name you will not be able to use SSL at this time. I have seen workarounds with CDN solutions like CloudFlare but I haven't tested it my self yet.

For any other repository you create you can commit files to a branch named "gh-pages" and they will be served at {githubusername}.github.io/{repositoryname}. In my case I put a "front page" in my {githubusername}.github.io repository linking other demos but you could actually build a complete blog in that space if you like. So far I have only scratched the surface of what can be done with this but there are a lot of information out there. At the bottom of this post there is a link to some more information to get you started.

The reason I ended up with this solution was due to JSFiddle/Plunker implementation complexity for my latest demo. When I moved to the US from Sweden my phone took care of my phone numbers missing country code and allowed me to dial them as +46. When I used Skype to dial them it just got the original number entered in the phone without country code. One big difference between the dial screen in Skype and the phone is that you can't edit the phone number in Skype. You have to delete the whole thing and type it in with country code and then the number. Since I sync my phone with Google Contacts I figured I'll use their API and a E164 (country code + phone number) javascript library to update all contacts in my address book.

Since the code use for my self was a bit of a one of, I now type in new numbers with country code, I thought I'll make a functioning demo out of the code. If people want to use it to correct their own address book they can. At the same time it's a complete Google oAuth, API implementation demo written in AngularJS.

E164 formatter demo: https://kallsbo.github.io/gcontactse164/
G
ithub demos: https://kallsbo.github.io

More info on Github pages: https://pages.github.com/

In the menu above you will find links to my demos on JSFiddle, Plunker and Github.

12Feb/160

Google Speech API – returns no result

Google Speech API takes a flac audio file and converts it to text. This API is aimed at Chromium and Android developers first and for most so the documentation for other plattforms isn't that good. This article contains C# code but the principal is the same for other languages. I haven't looked at this before today but got a question from a developer that was stuck only receiving {"result":[]} in return from the API even though the HTTP response was 200 OK. I was pretty impressed with the whole thing so here is what I found today and a simple example to get you started.

Get access

To get access to the API you need to obtain an API key. Simply logging into Google Cloud Console, creating a new project and adding the API will not work. First you need to join the Chromium-dev group on Google groups with the same Google account used in the cloud console.

Then you can head over to Google Cloud Console and:

  1. Create a new project.
  2. Go into API and search for Speech API and add it.
  3. Create credentials for the API, select Server Key.
  4. Put in your public IP and click Create.
  5. Save your API key generated somewhere.

Note: The API only allows you 50 requests a day!

Test is out

You can use the same flac file I used or get/create your own. Just make sure its 16-bit PCM and mono. That was the issue causing the {"result":[]} result.

I used a simple C# windows application with one button running this code:

string api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
string path = @"C:\temp\good-morning-google.flac";

byte[] bytes = System.IO.File.ReadAllBytes(path);

WebClient client = new WebClient();
client.Headers.Add("Content-Type", "audio/x-flac; rate=44100");
byte[] result = client.UploadData(string.Format(
				"https://www.google.com/speech-api/v2/recognize?client=chromium&lang=en-us&key={0}", api_key), "POST", bytes);

string s = client.Encoding.GetString(result);

Then you should get this result back:

{"result":[{"alternative":[{"transcript":"good morning Google how are you feeling today","confidence":0.987629}],"final":true}],"result_index":0}

Important about the flac file

For this to work you need to have a 16-bit PCM mono flac file. The issue that the developer I helped with this ran into was that his file was 32-bit float and stereo. If your not sure download Audacity and check/convert your file.

audacity google speech api

19Oct/150

Implementing Google OAuth in JSFiddle

The implementation of Google OAuth for use with there API's are simple with there API Client Library for JavaScript, if you are developing a normal webpage. If you want the functionality in a JSFiddle it's a little more complex for several reasons. I have put together a demo on JSFiddle that shows how it can be accomplished!

Background

I wanted to be able to include Google Oauth for API access in demos I create on JSFiddle. I did a few tests with the API Client Library for JavaScript provided by Google. Ran into several issues with cross-site limitations and sandboxing. Since JSFiddle runs in i sandboxed iframes it limits access to navigation, URL reading and other stuff that the client.js relies on to authenticate the user and get the token back. I found that if the user is already authenticated with Google and you set the immediate parameter to true it will work. That's probably good enough for several demos but it's not a clean solution so I spent a few hours and came up with a solution. If the user isn't already authenticated two new tabs will open up and nothing more will happen.

Pre-requirements

You need to setup a project in the Google Cloud Platform Console and get the clientID. You also need to setup Authorized redirect URIs that matches your JSFiddle. Due to the iframe nature of JSFiddle you cant use the jsfiddle.net address, since the result frame being served from the fiddle.jshell.net domain. For this demo I setup these two redirect URIs:

One address for the case of direct access to the fiddle (someone fining it via Google for example) and one if they navigate to it from my dashboard.

To get this to work I implemented the OAuth flow from the ground up. The demo uses jquery, because it's easier, but it can be done in pure JavaScript if you want to. I'm using the standard Oauth 2 flow were you send the user to Google for login with your client_id, redirect_uri and scope as query string parameters. The user authenticates with Google and get's redirected back to the provided redirect_uri with access_token, token_type and expires_in parameters as query string parameters. Google OAuth supports redirecting the user back to a http address but that's no good from a security standpoint! JSFiddle does support https on all it's fiddles so let's make use of it!

Different then a standard JSFiddle

There are a few things that are different from a regular JSFiddle implementation, to create a safe demo that handles the iframe limitations.

  • Check for HTTPS
    This is just for making the demo safe. When Google returns the user to the site with the token as query string parameter it is possible to intercept if you don't use HTTPS. A simple implementation of location.protocol == 'https:' checks if we are secure or not. This isn't needed since you can send the user from a HTTP site to Google and get the user back on a HTTPS redirect. This is just put there to make a point of the HTTPS implementation.
  • The use of fiddle.jshell.net instead of jsfiddle.net
    Since the iframe with the result, where all the javascript is executed, is served from fiddle.jshell.net all the content needs to be served from there. If not the script cant access the parent frame URL containing the access token returned from Google.

Points of interest in the code

  • Check for HTTPS
    As above this isn't really needed since I can set the redirect_uri to whatever address I want as long as it's specified in the cloud console.
  • Create the authentication link
    For this to work properly it needs to be a target="_blank" link. Navigation inside the iframe is blocked outside the jsfiddle domains. This is the reason why the demo needs to open a new tab to be able to return with the access token. The parameters at the top will set the basic information for building the link:

    var oAuthEndPoint = "https://accounts.google.com/o/oauth2/auth";
    var oAuthClientID = "702841265150-iaravt3jmu0c67k892pivj9kgkb8dbco.apps.googleusercontent.com";
    var oAuthScope = "https://www.googleapis.com/auth/userinfo.profile";
    

    Then we build the redirect uri from the current JSFiddle address. I implemented it this way to make it easy to fork it to create other demos based on this implementation. I use the a element for easy manipulation of the domain and path. Replacing the protocol and domain with https://fiddle.jshell.net:

    var a = document.createElement('a');
    a.href = document.referrer;
    var redirect_uri = ['https://fiddle.jshell.net', a.pathname].join('');
    a = '';
    

    document.referrer is used to get the actual address of the fiddle from the parent frame, even if the domains doesn't match at this point. An easy way to get around the same origin limitations of the iframe.

  • User clicks the authentication link/button
    The user is sent to Google to authenticate and allow the app access to the user information. The only scope requested in this demo is basic profile information.
  • User is returned to JSFiddle
    When the user is returned the fiddle will load all over again. So each time it loads it will check if HTTPS is used, if so check for the URL parameter access_token. This is why we need to use the fiddle.jshell.net domain, it will make all the iframes of the fiddle to load from the same domain giving our script access to it's parent. We need that access to grab the URL and extract the parameters. This function grabs different parameters from the URL of the parent frame:

    function GetURLParameter(sParam) {
        var sPageURL = window.parent.location.hash.substring(1);
        var sURLVariables = sPageURL.split('&');
        
        for (var i = 0; i < sURLVariables.length; i++) {
            var sParameterName = sURLVariables[i].split('=');
            if (sParameterName[0] == sParam) {
                return sParameterName[1];
            }
        }
    }
    
  • Use the token for API access
    As soon as we have the token we can use that for accessing the API with a simple ajax request via this simple function:

    function LoadUserInfo(access_token, expires_in) {
        $.ajax({
            url: 'https://www.googleapis.com/userinfo/v2/me',
            type: 'GET',
            dataType: 'json',
            headers: {
                'Authorization': 'Bearer ' + access_token
            },
            success: function (data) {
                // Hide login
                $("#login").hide();
    
                // Populate demo, img and name
                $("#user_pic").attr("src", data.picture);
                $("#user_name").attr("href", data.link);
                $("#user_name").text(data.name);
    
                // Show raw data
                for (var property in data) {
                    if (data.hasOwnProperty(property)) {
                        $("#raw_data").append("<b>" + property + "</b>" + data[property] + "<br/>");
                    }
                }
    
                // Display demo
                $("#demo").show();
            },
            error: function () {
                $('#demo').html('<p>An error has occurred</p>');
            }
        });
    }
    

Limitations

So there are a few limitations with this approach. You get the token that is valid for 3600 seconds (one hour) and implementing code for renewing that with out losing state is not possible. So if the user have started creating something and the token expires renewing it will end up in a reload of the fiddle. The other, smaller limitation, is that there always will be a second tab opened to authenticate the user.

You also need to set the code as base in the fiddle at all times, redirects to a version number like /3 will make the redirect back from Google to fail!

Any thoughts, questions or suggestions? Please share in the comments below!

Complete JSFiddle demo

29Sep/140

SEO tracking

Before you start doing any search engine optimization (SEO) work on your site you should implement tracking on your site. If you don't track events on your site you will not be able to target your SEO efforts and follow up on the results. If you haven't read my SEO Basics post you should take some time and do that. In that post I talk about tracking in two different ways, both tracking your visitors (or traffic) as well as "tracking" your site content behavior via Google Webmaster Tools. In this post I will explain more about the tracking part of SEO.

Continue reading...

22Sep/140

SEO impacted by structured data

Structured data helps Google Bot and other web crawlers to understand and interpret your data. By tagging your data with structured data tags like MicroFormats h-entry Google can index all relevant information. The tags also affects your search appearance, i.e. what your site looks like in the Google search results. You can also be sure that this will effect your page rank!

Continue reading...

20Aug/140

SEO Basics

To be honest I had limited knowledge about SEO (Search Engine Optimization). Haven't really spent much time or reflected about it at all. But then I started looking through my Google Analytics info for this site and realized that most of my traffic isn't from referrers, like links or articles I wrote on other sites, but from Google Search. That put it in an whole other perspective for me, and also I started receiving a lot of traffic, upgraded my servers, and realized that some Google Adsense would be in place to pay for the servers. So my thought was to do a series of articles about improving SEO for this site.

Where to start?

If you start searching for SEO on Google you get a lot of information but also a lot of BS. The golden rule here is if anyone promises a specific ranking they are not for real. You can do all you can about SEO but at the same time your competitors might have done a better job.  Start by reading the Official Google Optimization Starter Guide.

You should also, if you haven't already, implement Google Analytics on your site. It's an easy way to track you visitors and views and find room for improvement as well as follow your sites progress.

Google Webmaster Tools is also a very good idea to be able to check how the Google Bot looks at your site. You ranking will be damaged if the Google Bot can't read and understand your site.

More information

Google provides information and recommendations without telling us exactly how the ranking works. The internet is overflowing of people claiming to know something you don't but as I sad before if it sounds to good to be true it probably is! Instead go to the source of the information, Official Google Webmaster Blog.

30Jun/140

Migrate WordPress to Google App Engine

Wordpress on Google App Engine

Why not run WordPress on Google App Engine? You will get performance and stability while only paying for the resources you actually use. Reading the official Google tutorial "Running WordPress in App Engine" it gives you a fare idea what you are in for. But if you want to migrate a currently running site then you have to do some tweaking. So here is a run down on how to do it!

Continue reading...

15Oct/130

Picasa Web Downloader Source

Had a few downloads of the Picasa Web Downloader that I put up yesterday. Also had a few request of the source code, people seem to be nervous about putting there login information into an application that isn't open source.

So now it is open source available at: https://code.google.com/p/picasa-web-downloader/
Compiled and readu to run:


14Oct/130

Picasa Web Downloader

I store all my memories on Picasa! Even spent a lot of time to scan all old pictures and upload them. Trusting a provider to keep your data safe from data loss is something that doesn't make me sleep well at night. Even though I trust the guys at Google to take care of the data I want a backup. Before I used Backupify.com but they don't provide this service anymore. The Picasa application it self is really bad! Maybe if your an old iTunes user you can figure out how it works? All kidding aside... It actually didn't manage to download all my pictures and got some strange errors. So I looked around the internet for a third party application for downloading all my content but I couldn't find anything useful!

I did how ever find a great application named Picasa Web Sync (http://www.geekytidbits.com/picasawebsync/) written by Brady Holt. Great peace of software for syncing up to Picasa but not the way I want to do it! So i started to check out the API for developing my own application, I took a quick look at the Google API Client Libraries. Not that surprising the java and python libraries was awesome and complete. The .Net C# however is not that good! There are things missing and stuff that doesn't work, spelling mistakes and so on. Google usually delivers above and beyond but in this case not so much.

How ever I started to code and used what I could from the client libraries, mostly to parse the feeds from the API. Then I realized there was a load of videos in my albums so I started to try to download them. Nothing in the documentation, that I could find anyway. But I found a way to do it and now I have a complete offline backup of my Picasa Web Albums, on a USB drive... Encrypted of course!

So the first working version of this console application is available for download! Written in c# and source will be posted soon for those who like that.