The Hiltmon

On walkabout in life and technology

DuckDuckGo

Note: This is not an advertisement. I love what the folks at DuckDuckGo are doing and want to spread the word.

DuckDuckGo is my search engine of choice (and is the page that opens in all new Safari tabs for me). If you are not using it as your default search engine, I’d advise you to start now.

I started using DuckDuckGo well over two years ago. My initial impressions back then was that it was slow, the results were not nearly as good as Google’s and the name was stupid. None of those first impressions turned out to be true.

Why DuckDuckGo?

  • They do not track you. It always feels creepy using Google Search because they know so much about me. In a post-Snowdon world, we know we’re being tracked. One fewer tracking source a good start.
  • It’s really fast.
  • The results are the results you are looking for and nothing else. No Ads. No Promoted Links. No “personalized search results”. And none of those scumbag aggregation sites that serve copied content and ads.
  • No page breaks, just scroll for more.
  • Their instant answers are great and getting better.
  • They are adding new search features every day. Just recently they added live MLB scores.
  • Bang commands allow us to search specific sites. Just start with a !. Brilliant. Which means, at worst case, you can get to an encrypted private Google Search by starting with g! if the DuckDuckGo search does not find what you are looking for.

Making it the Default

In OS X, in Safari Preferences:

In iOS, In Settings / Safari:

I strongly recommend you switch to DuckDuckGo for your search needs now. And if you are web smart, feel free to give them feedback or get involved in making it better.

Follow the author as @hiltmon on Twitter.

A Simple Markdown Spotlight Importer

I noticed recently that Spotlight on Yosemite was no longer indexing my Markdown files. Since all my notes are in Markdown format, and Spotlight is how I find my notes, this was a big problem. Reinstalling my current set of Markdown editors did not help.

This did. Huge thanks to Gereon Sommer for the idea in Mac OS X Spotlight Enhancement.

The idea is to use a built-in importer, in this case RichText, to do all the work for you. Note that this will probably need to be repeated on each Operating System install or upgrade.

Edit /System/Library/Spotlight/RichText.mdimporter/Contents/Info.plist and add <string>net.daringfireball.markdown</string> under LSItemContentTypes:

...
<key>LSItemContentTypes</key>
<array>
    <string>public.rtf</string>
    <string>public.html</string>
    <string>public.xml</string>
    <string>public.plain-text</string>
    <string>com.apple.traditional-mac-plain-text</string>
    <string>com.apple.rtfd</string>
    <string>com.apple.webarchive</string>
    <string>org.oasis-open.opendocument.text</string>
    <string>org.openxmlformats.wordprocessingml.document</string>
    <string>net.daringfireball.markdown</string>
</array>
...     

You will need an admin password to save the changes.

Restart the Rich Text importer

mdimport -r /System/Library/Spotlight/RichText.mdimporter

And force a Spotlight re-index

sudo mdutil -E /

To test this I created a new Markdown file in BBEdit and added the word wagga to it. After a few minutes of re-indexing, Spotlight found this file in a wagga search.

Good to go.

Follow the author as @hiltmon on Twitter.

New Google Analytics for Status Board Server Edition

Last week, Google finally deprecated their non-Oauth APIs, which means that the Google Analytics for Status Board code I have been publishing stopped working. Fortunately for us, Github user erebusnz updated the PHP API to work with OAuth2 and we can access Google again. I have included it in a new version of the Server Edition Package.

Quick Install Instructions

  1. Download the statusboard.zip file.
  2. Expand it in the root of your web server. It creates a statusboard folder.
  3. New: Register your application with Google Developers Console and get your Service Account email and P12 certificate file.
  4. New: Determine your profile ID (its the number after the ‘p’ in the URL generated for your analytics)
  5. New:Replace the 3 instances of <—-> in each PHP file with your service account, P12 file name and profile ID.
  6. The URL to use in Status Board is http://<your-domain>/statusboard/analytics_<file>.php, replacing <your-domain> with your server domain name, and <file> with one of views (graph), hourly (graph) or pages (table).

You should get something like this (Yes, my follower count is still tiny. Yes, I live in New York, but I still use Celsius for weather. And awesome Inbox Zero!):

Details

There are four files in the archive:

  • analytics_views.php to present a graph of Page views, Visitors and New Visits for the past week for a site.
  • analytics_hourly.php to present the same data over the last 24 hours.
  • analytics_pages.php to show the top pages today.
  • gapi.class.php is required to access Google Analytics

Registering your API Access

You need to create a new Project in the Google Developers Console in order to permit your web application to access Google’s data.

  1. Click Create Project

  1. Give the Project a Name, such as “My-GAPI-Project”
  2. Wait while Google does its thing and moves you to the Project Dashboard
  3. Click on “APIs & Auth”
  4. Click “APIs” and search for “Analytics”

  1. Click on “Analytics API” then click “Enable API”

  1. Click on “Credentials” to create a service account

  1. Click on “Create new Client ID”
  2. Choose “Service account” and click “Create Client ID”
  3. Wait for it…
  4. Google will download a JSON file to you. So nice of them.
  5. Click “OK, got it”

Fill in the Details

You will now create, copy and paste the three elements needed for the Analytics PHP files to access Googles Analytics API.

  1. Copy the “Email address” which ends with “@developer.gserviceaccount.com” and replace the “<Email address @developer.gserviceaccount.com>” in each analytics PHP file.
  2. Click “Generate new P12 Key”. Google will send you a “.p12” file.
  3. Copy this file into the SAME folder as the analytics PHP files.
  4. Replace the “<P-File Name.p12>” in each Analytics PHP file with the execat name of the P12 file (including extension, no path needed)
  5. Go to Google Analytics and bring up one of the reports for the site you want to track.
  6. Copy and paste the URL into a text editor, we’re going spelunking for the profile ID. E.g. httpz://www.google.com/analytics/web/?hl=en#realtime/rt-overview/a1234567w1234567p9999999/"
  7. Your profile ID is the set of digits after the ‘p’ at the end of the URL (in this case a fake 9999999). Paste that into the “<Use Yours>” in each analytics PHP file.

The top of each file should look something like this (codes are fake):

1
2
3
4
5
6
7
8
define('ga_profile_id','9999999');

require 'gapi.class.php';

date_default_timezone_set('America/New_York');

$ga = new gapi('729847129341293-938825he75l8mdabcxsh72hdqkdmpfobq273@developer.gserviceaccount.com', 
  'My-GAPI-Project-384idsucy44f.p12');

Accessing from Status Board

For the ‘hourly’ “and ‘views’ graphs, create a new Status Board chart panel and set the URL to ”<your domain>/status_board/analytics_hourly.php" and “<your domain>/status_board/analytics_views.php” respectively. For the Top pages list, create a new Status Board table and set the URL to “<your domain>/status_board/analytics_pages.php”.

Enjoy.

Follow the author as @hiltmon on Twitter.

That Last Effin 20%

I love creating new products. I love shipping new products more. That last effin 20% of work to create a new product that takes 80% of all the development time kills me. I really want to ship, but it has to be right.

For those who do not know the 80/20 rule, it’s a variant of the misnamed Pareto Principal wherein Vilfredo Pareto created a mathematical formula in 1906 to describe the unequal distribution of wealth in his country, observing that 20% of the people owned 80% of the wealth. This was then applied to other areas and seemed to hold true.

In reality, Tom Cargill had it right where the right measure was 180% of the time:

The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.

Tom Cargill Bell Labs

Over the past few months I have been buried in the last effin 20% (or remaining 90% of time depending on who you follow) of a major product development. Designing it was quite easy. Building the core architecture was great. Getting the primary functions up and working was quick. Getting it fast, right and accurate is the last effin 20% and it has taken months.

Today, the product passed its first certification, which means I’m almost ready to ship. Today is a good day. The effin 20% is almost over.

Follow the author as @hiltmon on Twitter.

Rust Hits 1.0

Rust is a project I have been observing for a while now, an attempt to create a new, fast, safe, systems programming language to replace C++, but with better memory safety and a bullet-proof concurrency model. Last week, Rust hit 1.0, and I think I may start testing it out for highly threaded, small and fast calculation tools that I would normally write and fight in C++. If any of you are using it, please let me know what for in the comments.

And before anyone points me at Google’s excellent Go language, I am already looking into it to speed up, parallelize and replace a ton of Ruby and Python code on the server side.

Follow the author as @hiltmon on Twitter.

The Perfect Keyboard(s) (for Me)

I’m not a keyboard snob. But I love using my current keyboards now.

Background: I was very happy with the Apple “chicklet” style keyboards and even wrote about it in Not the Clicky Keyboard. There I declared that clicky keyboards were not for me.

I was wrong. But I did not know it at the time.

A while back, my boss went on a buying bender and purchased a whole bunch of different keyboards, including the DAS and some Microsoft ones. I tried them out too. Neither of us liked any of them, so he returned them all. As a joke, I ordered the CODE keyboard to show that I could buy “better” keyboards than him, and fully expected to return it.

The CODE Keyboard - I interrupted writing this post to take the picture.

The CODE is a keyboard designed by Jeff Attwood (@codinghorror) and assembled by WASD keyboards. It’s a big, black, heavy, back-lit, mechanical keyboard running the heavy and clicky Cherry MX Clear Switches.

Since I do not use a numeric keypad and like the mouse closer, I managed to luckily purchase the 87-key version with the rare Cherry MX Clears. I unpacked it and the initial thought was it looked like Battlestar Galactica compared to my sleek Apple keyboard.

I plugged it in and started to use it.

And no-one could be more surprised than I by the experience.

The keyboard layout was felt perfect for me. Better than the Apple one. Somehow my fingers found the keys I needed (and I am not by any measure a proper touch typist). The keyboard was the right size for my hands and it forced me to place my hands in a better position, which led to fewer aches at the end of the day. And best of all, there is no need to actually thump the keys down, they take pretty much in the middle. And I do like the clacky sound.

The only negative is that the pressure is a little high for programming. When I am writing, like now, my fingers tend to stay over the letter keys and the stiffer keys make me feel like I am working on some kind of modern mechanical typewriter. But when coding, my fingers move a lot out to the punctuation and numeric keys and I found the tension when pushing with an extended finger a little wearing.

So I ordered another WASD keyboard. I got the exact same 87-key layout, but with Mac keycaps and the lighter Cherry MX Brown switches with dampening O-Rings. The only other differences with the CODE is the Caps-Lock light is red and there is no back-light.

The second WASD looks the same, but the keys are reasonably softer than the CODE and they take even higher up. As a result, I can use lighter presses to type on it which makes coding for me easier.

So now I have two of them. The CODE resides at home where I do a lot more writing and I love the heavier presses and back-light at night. The regular WASD resides at work where I program all day.

There are many who will correctly point out that $150 (each) on keyboards is quite expensive and that there are many cheaper options out there. Sure, if that works, go for it. But I believe that anything you use or touch daily should be the best for you and cost should not be a factor. Then again, I am using a $100 mouse, $1,000 screen and $2,600 computer so the WASD is not that expensive in the scheme of things. And I use them all day and all night, every day.

I therefore highly recommend the WASD keyboards (CODE or regular, 102 or 87 key) but not for snobby reasons. I just prefer the layout, size and feel. I seem to make fewer mistakes and enjoy the typing experience more because my hands seem to be in the right place. I have not found that experience on any other keyboards. They work best for me.

Follow the author as @hiltmon on Twitter.

One-Key Format Presets in Scrivener

I am writing a lot of business documents in Scrivener, and even though it’s the wrong way to do it, I use Scrivener’s format presets (think paragraph styles) to quickly format headings, bulleted lists and paragraphs in each sub-document. This way I minimize post-processing (and the business format is intentionally boring so I do not need any fancy layouts).

But it’s a hassle to have to take my hands off the keyboard to click through the Format / Apply Formatting / Apply Preset menu tree or hit the tiny format preset dropdown ¶a to set the current paragraph style. What I want is a single key-combination to apply a preset style.

Here’s how I made it happen.

Create your Format Presets

The first thing you need is to create the Format Presets you want to use.

To create a preset:

  • Select the paragraph you want to format.
  • Set the font, size, etc using the toolbar
  • With the paragraph still selected, choose Format / Formatting / New Preset from Selection from the menus and give it a name. Make sure this name is unique to the application, this is key.
  • Click OK when done.

You should now see the new preset with your name on the preset drop down and in the Format / Formatting / Apply Preset menu.

To manually apply this preset to another paragraph, select the new paragraph and then click the new preset off the menu.

Create the Keyboard Shortcuts

Open System Preferences, choose Keyboard and select the Shortcuts tab. Click on App Shortcuts on the left.

To add a new Scrivener Shortcut:

  • Click the Plus Button
  • Select the Application - Scrivener
  • Type in the name of the format preset. Note: The name must match the format preset exactly, one character difference and this will not work.
  • Hit the keyboard shortcut you wish to use
  • Click Add to save

To see if this worked, click on the Format / Formatting / Apply Preset menu in Scrivener and you should see your new keyboard shortcuts on the menu next to their names.

All Done

So now you can focus on writing in Scrivener and format paragraphs without leaving the keyboard.

Follow the author as @hiltmon on Twitter.

Merge Asana Projects Into OmniFocus (Updated)

A long long time ago in a galaxy far far away, I wrote a script to import Asana tasks into Omnifocus. If you ran it again, it would update the tasks in OmniFocus. See Merge Asana Projects Into OmniFocus.

I recently updated the script to work a smidge better with Asana and Omnifocus 2. The main changes are:

  • Date formats can be changed (for those of us not in the USA)
  • The Non-Project tasks now load

Setup

First, follow the instructions from the original post to set up Omnifocus and get your API keys.

Download the updated script from https://gist.github.com/hiltmon/d1f79e95dd11252ce6ca.

Open it in a text editor and insert Your API Key and Profile Name in the slots provided. You can also change the date format to suit.

class MergeAsanaIntoOmnifocus

  API_KEY = '<YOUR KEY HERE>'
  ASIGNEE_NAME = '<YOUR PROFILE NAME HERE>'
  SUBTASKS = true
  DATE_FORMAT = '%A %B %d, %Y at %H:%M:%S' # (Try http://strftimer.com to build your own)

Make sure the prerequisites are in-place from the old post and then run it

 $ ruby merge_asana_into_omnifocus.rb

Warning: This is just my script, it may or may not work for you. Backup your OmniFocus before running it the first time. And use at your own peril. Inelegant, alpha-level, probably quite buggy, my first AppleScripts, you get the drift.

But if you do use it, let me know in the comments or via Twitter. And if you have any ideas for changes or enhancements, let me know.

Follow the author as @hiltmon on Twitter.

Seeing the Bigger Picture

It is quite astounding to me that an astonishing amount of work is done and code written without any knowledge, view or understanding of the bigger picture. And yet no-one seems to have taken the time to consider just how remarkably ridiculous this common situation is. How can people be expected to perform at their best and create the best products when they have no clue as to what the big picture or goal is?

And yet that seems to be the norm.

I started thinking more about this topic yesterday when discussing the work our Quantitative Analysts were doing with them. They were keenly focussed on figuring out the best and most correct way to calculate the results asked of them, but were struggling to understand what the results would mean and how they would be used. Without that understanding, they were unsure whether they were even thinking about the problem in the right way. What they needed was context, the bigger picture. Once they understood how their numbers and results would be used in the business and how their code would be used on a calculation server, they were far clearer on what they needed to do and how to architect it better for the business. And a whole lot more.

I really do not understand how or why the big picture should remain such a mystery, but being the nerd that I am, here’s my weak attempt to codify the reasons for I think folks hide the big picture:

  • They don’t know: The person setting the task also has no idea about the bigger picture, commonly in a dystopian bureaucracy gone mad or larger organization (Can you tell the difference?)
  • They don’t care: The person setting the task is just trying to get paid and cannot wait until the 5PM go-home alarm, otherwise known as “It’s just a job” or “I work in the public service” syndrome.
  • They have no time to explain it: The person setting the task is under pressure to get it done, has a hundred other things to do and is handing off a piece to get it off their list, pretty much like every overworked Business Analyst out there.
  • You don’t need to know: The person setting the task does not work for an intelligence agency and likes to play power games, or “I’m the boss, just do it.”
  • You should already know: The person setting the task honestly assumes that the big picture is clear and has been communicated in the past, the home version being “I’m having a fight with my partner and they keep saying I should know why” scenario.
  • It does not matter if you know: The person setting the task is a control-freak douchebag, believes themselves to be a superior being and is just acting like a middle ages noble.
  • They don’t know that you need to know: The person setting the task genuinely does not know that sharing the big picture is a good thing. Point them at this post and they will receive a good, healthy dose of wonderful karma points.

From first-hand observations, the impacts of not knowing the big picture are many:

  • Duplicated or unnecessary processes, workflows and code because the folks involved had no idea that it was so. If they knew, they would find better things to do.
  • Inconsistent deliverables, products and results, where different groups producing the same things come up with different answers. Look at any large firm with multiple accounting groups and try to get a single picture of how the business looks.
  • Signs of disconnected, fractious, people fighting against each-other over small fiefdoms when the whole purpose of an organization is to work together to achieve the bigger picture.
  • Frustrations between groups and departments where the “other” is always getting things you asked them to do wrong, a refrain we programmers are so used to hearing.
  • Outright conflicts where one team’s work-product negates another team’s work product, and neither knows about the other.

Whereas, in the situations where the big picture and context is communicated first, I see:

  • Better architected business, workflows, processes and systems as everyone involved understands and is working towards the same big picture.
  • Requests, deliverables and systems better tuned to meet the needs, not the stated wants, reducing rework and frustrations.
  • Combines the intelligence and domain knowledge of the requester, programmer and worker/user to find the right, optimal and useable solution, not a quick and dirty one decided by a manager.
  • Gets the work done better as all involved have “bought in” and are motivated to succeed as a team.

Next time you set a task for someone else, start with the big picture and explain to them how the task fits in, how, where and by whom the deliverables will be used, and how it will make things better for others. Then, and only then, explain the work needed to be done. You will find that they will understand the task better, deliver better solutions, enjoy what they do more, and most importantly, give feedback insight into ways of doing it better.

Follow the author as @hiltmon on Twitter.

Eliminating Google Analytics Referral Spam

It seems that the Google Analytics use of a public tracking code was a mistake. Anyone can use your tracking code on their site, which adds to your traffic, or they can create ghost visits and spam referrals by hammering Google Analytics with your code while never visiting your site. Oh, and the codes are easy to guess, so they do not even need to know about your site.

In most cases, this is perfectly harmless (IMHO) as only Google Analytics data is affected and your reports look funny. Your ad stats, if not Google, remain the same as they are independent.

The only reason I am talking about this is because I was up early this morning and noticed that a page with the title of “Luck” was trending in my analytics1.

Except for one problem.

There is no post entitled “Luck”.

I never wrote one.

Ever!

Which led me down the rabbit hole.

You can spot these fake referrals pretty easily. Go to Reporting / Acquisition / Referrals on the Google Analytics site. Then, next to Primary Dimension, select Other / Behavior / Hostname:

My site is hiltmon.com, who the blazes is that?

Whatever you do, please do not visit that URL.

Michael Sullivan (@AnalyticsEdge) from Analytics Edge wrote two excellent posts recently on what these are and how to deal with this referral spam:

Go ahead, read them.

I have updated my Google Analytics filters and am waiting to see if the changes work. I have also created the segment as directed and that seems to work great, but only for some reports.

Follow the author as @hiltmon on Twitter.


  1. Still running Status Board by the seriously handsome folks at Panic on the original iPad.