Hiltmon

On walkabout in life and technology

Reminder: Update Your Tools

At the start of every new year, I spend a the time to update all my tools and recompile all my products with them. At the cost of a few hours (or at worst, days) work, I get rid of most the maintenance hassles I faced before I used to do this and gain the benefits of all the new features and performance from updated tools.

I understand that, for all of us, updating our tools is surprisingly hard. Things that used to work now fail, incompatibilities have to be addressed and it takes time, perceived as unproductive time, to do. And then there is the risk that updating production servers will cause downtime for no valid reason. It’s easy to feel that we should wait for the next release, or when we have some mythical down-time, or when we’re forced to do so as the versions of our tools become obsolete. And it’s hard to justify updating tools when there’s real work to be done and no current perceived benefit in changing tools.

On the other hand, updating our tools now means we get the performance and feature benefits of these new tools now. We get the incompatibilities from last year out of the way, meaning that the products we make with these tools can move forward more easily (we’re less likely to get stuck with a version dependency). We get to use newer and better versions of our libraries. Our servers get updated, run better and require less maintenance. We get to learn and practice the updated tools ideas and technologies. And the start of the year is the best time to do this as it’s one of the slowest business times.

For me, that’s actually quite a lot of work, but so worth it. Amongst other things, I have:

  • Upgraded to Ruby 2.1.0 as it’s the latest and fastest.
  • Moved our Python code to 3.3.
  • Moved all our Rails applications to 4.0.2 as that is current.
  • Updated to the latest SASS, Rake, Sinatra, Node.JS and other tools.
  • Updated all my Homebrew installs (brew upgrade). This entailed a non-trivial upgrade of PostgreSQL from 9.1 to 9.2 (I had to export and import all my databases)
  • Updated all my CentOS servers and their tools to the new PostgreSQL, Ruby and Rails.
  • Moved all our Windows C++ and C# applications to Visual Studio 2010 from 2008 (we run one version behind on all Microsoft products - been burned before)
  • Recompiled all our C++ projects using the latest clang and gcc on each platform.
  • Updated all Objective-C applications to the latest Xcode and recompiled with ARC and the latest libraries.

What did I get out of all of this work? On the surface, nothing much. The same old applications doing the same old things. But each and every one of them is now ready to take advantage of new tools, new technologies and new features in 2014. And most seem to run just a little better and a little faster and use a tad less memory than before. And best of all I have peace of mind, I know there are no legacy dependencies that are holding me back.

There is no good time to do this, so do it now. Get it over with for the year. Update your tools now. Or you never will.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

Python: It's a Trap

TL;DR: Python is the best language for quantitative scripting (because of its libraries). But it’s a trap. Almost all programmers and libraries require Python 2 when Python 3 is out and way better. Choosing to use an old language for new project traps us into supporting old languages, libraries and platforms. And that’s not smart. The intent of this post is to point out this odd stagnation and encourage the migration and adoption of Python 3, or, as it has happened in the past, we’ll have to look elsewhere for a solution.

Over the past few weeks, I have been looking at switching to the Python programming language for quantitative analytics development at work. On the surface, Python looks perfect for this - it has the best quantitative libraries (e.g. numpy, scipy and pandas), it is the language taught in finance schools and it is a beautiful, fast and reliable scripting language to work with.

It's a Trap!

Admiral Ackbar

In my research I felt there was something wrong with the picture I was seeing, something on the periphery in the corner of my eye. And then I saw it: The stats on Python 2.7 usage and compatibility vs Python 3.3 usage and compatibility make no sense.

As a non-pythonista, my expectation was that after five years, Python 3.x would be the most used and most compatible platform. After 5 years, all libraries should have moved over, all new code should be written in it, all new technologies compatible with it. The reality is that less than 2%1 of all work is done in Python 3 even after 5 years! Python 2 is still 98% of the game. WTF?

That’s where the trap lies.

An old language stagnates, increases support burden and cost, and slows growth. People sticking with old languages are forced to use old libraries, old tools and run on old platforms. The reason languages update is to make them better, faster and to eliminate bad language design decisions. Picking an old platform for a new project traps you into long term support and maintenance of that old platform. That’s the trap.

All this has happened before. All this will happen again.

Pythia

We’ve been there before. Back in the early 2000’s, I was a Perl programmer. Perl was the scripting language at the time, the one with the best libraries (see CPAN), the one with the latest tools and the best compatibility. But as the naughts progressed, Perl started to stagnate as a language. First, they started talking about a non-compatible Perl 6 that never happened, and then the community left for Ruby and Python because they were happening. Perl, it’s ecosystem and its community stagnated. 10 years later, Perl 5 is still with us. But I believed then that sticking with Perl beyond the middle 2000’s would have been a bad idea. And it was for those who stayed.

Back then, the Python 2 community and ecosystem was dynamic and growing, as was the Ruby one. Back then, I chose Ruby over Python by a hair, and it was a great decision for me. Since I started with Ruby, it has gone through three revisions (1.8, 1.9 and now 2.0) and the libraries and community have grown with it. Even when an incompatible version, 1.9, came out, the speed at which the community adopted it was tremendous. Ruby 2.1 came out a few days ago and already way more than 2% of the community is using it.

Other languages too have gone through changes as significant as the Python 3 move over the same period. C#, one of the most popular corporate programming platforms, went from 2.0 to 3.0 to 3.5 to 4.0, all incompatible revisions, yet most new C# code requires 4.0. C++ went to C++11 over this time and the most popular C++ compilers happily compile it.

On the other hand, other languages too have remained weirdly behind, just like Python 2. Java went to 7 but no-one seems to write for it. Oh, they all install Java 7 (in order to get rid of those infernal update dialogs) but still run 6 in production.

It is possible that the reason some languages stay behind and others grow is because of the way the change happened. Ruby, C# and C++ (Clang) all have compilers that not only point out the incompatibilities with old code, but have ways to fix them easily. Maybe the Java compiler and the Python one do not.

Or maybe the reason they stay behind is because they are too well embedded in corporate infrastructure and too much code has been written in them to make it viable to change. That sure explains Java and Python, but does not explain C#.

Or maybe it’s because these languages are taught in schools and the education system is slow to adopt the new version. Or maybe the community around these languages is more conservative, with an “it ain’t broke, don’t fix it” kind of attitude. Or maybe it’s just too popular.

I do not know why people have stuck with Python 2.

I do know that Python 3 has been out for five years and is about to go through its fourth revision. I do know that it improves the way the language works in a myriad of ways, removes some really bad stuff, is stable, mature and runs brilliantly. And yet it has not been adopted.

Since I do not know why everyone is still on Python 2, I cannot offer the right solution. Only some ideas:

  • If the reason people still use Python 2 is the community, then the community should start 2014 by installing Python 3.3 and using that. Put pressure on the library developers to get their products up to speed.
  • If the reason is because your distribution installs an old version, pressure your vendor to put the new version in as default.
  • If the reason is legacy code, spend the time now to fix and upgrade. Or you will get deeper and deeper into the trap.
  • And if the reason is the language provider, then python.org should stop supporting the Python 2 stream. It works for other languages and it will force all users to upgrade.

I still think Python is currently the best language for quantitive analytics and development of ideas, but it is also a trap. The trap being that we really need to use the old version to do what we need to be done. It’s like requiring users to be running Windows XP when Windows 8 is out and mature. This trap is holding everyone back.

If the trap remains, and it seems to be the case, then me choosing Python 2 as our development and production platform for new projects is nuts. I can, and probably will choose Python 3.3 for now as numpy and pandas should work, but I worry that if I wish to use other libraries, they will remain incompatible. And rewrite in C++ for production to avoid the trap.

Unfortunately, no real alternative exists for quantitative scripting. R is way too slow, and few know MATLAB. Java and C++ are just too heavy and require too much effort.

Or, if someone could make a numpy and pandas for Ruby (or even Go or Javascript), I’d use that in a New York minute.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.


  1. “Looking at download statistics for the Python Package Index, we can see that Python 3 represents under 2% of package downloads. Worse still, almost no code is written for Python 3.” in About Python 3

Can't Innovate Anymore, My Ass

Over the past few days I have seen so many references to the Quartz post 2013 was a lost year for tech that it has become a meme. “Innovation is dead”, “2013 proves it”, “RIP Innovation”, it screams.

Rubbish!

2013 was a great year for innovation in the tech space.

in·no·vate /ˈinəˌvāt/ verb
1. make changes in something established, esp. by introducing new methods, ideas, or products.

Remember, innovation is not about creating something completely new (that’s invention), it is about making changes to established products that significantly improve their abilities or experience. And we got that in spades in 2013.

Innovative Hardware

Apple continued it’s well known strategy of continuous innovation across its product line. The new iPad Air is the iPad they have been trying to make since they released the original one: thin, light, fast, with mind bending battery life. The design and concept of the new Mac Pro is insane, a small portable powerhouse, the first true computer built around Open CL. And the iPhone went 64-bit, so we now have a portable computer in our hands more powerful than all the computers on all the Space Shuttles combined.

Intel leaped forward with its Haswell line of CPUs. This innovative design provided us for the first time in 2013 with the ability to run for more than 10 hours on our laptops without recharging, and without penalizing speed or core count.

Even Microsoft innovated, it took the seriously limited Surface tablet concept of 2012 and made one that may look the same on the outside, but actually works this time, the Surface Pro 2. And their new Xbox One platform wipes the floor with their previous platform as evidenced by first day sales.

Even beleaguered RIM, er Blackberry Ltd, innovated by finally releasing the QNX based Blackberry 10, the next generation Blackberry. And it really is a good one.

Google let its innovation leak out of the labs with Google Glass on the developer program. No innovation, my ass, even SNL had a sketch on it.

And lets not forget the sensational Pebble Watch or the magnificent innovative Leap Motion.

Innovative Software

Since I use Macs and iOS all day at work and at home, I am really only exposed to innovations in Apple ecosystem software, so I’ll point some out here. Would love to know what innovations happened in other platform software too.

Apple release not one but two innovative Operating Systems this year, Mavericks and iOS 7. Mavericks may be the best release of OS X yet, with its innovative power saving techniques, interactive notifications, ability to send maps to your iPhone and they finally figured out how to handle multiple monitors on Macs. iOS 7 is also a remarkably innovative release, not because of the new look or its innovative ability to make folks queasy, but because it is the first 64-bit mobile operating system, and it uses the antennae in such a way as to maximize connection and minimize battery.

And in apps, just to name the first few that come to mind:

  • Marked 2 came out. It used to be a visualizer for Markdown documents, now it’s a publishing ecosystem.
  • Acorn 4 turned the old Acorn on its head and is the best Photoshop alternative out there.
  • Kaleidoscope 2 was released with innovative image diffing. Who did that before?
  • Napkin changed the way we annotate screenshots with its innovative zoom bubbles.
  • Omnigraffle 6 proved that you can innovate on perfection.
  • Ulysses III’s innovations turned the text editor on its head.

And lets not forget all those developers who took the opportunity when updating their applications for Mavericks or iOS 7 to add innovative features instead of just re-skinning.

“Can’t innovate anymore, my ass”.

When Apple’s Phil Schiller said that on stage at WWDC, he was taking a poke at the tech press and the memes that scream innovation is dead. The only place innovation is truly dead is where they have published its obituary.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

Writers: Thank You

I want to say thank you to a bunch of writers, to people who do what they do because they love to do it, from an anonymous one that reads and appreciates what they do every day.

They may never see this, but, in no particular order, a huge thank you from me to you all.

Tech and Opinions

Matt Gemmell at Matt Gemmell is owed free drinks at my house forever. He may be well known for his tech, tweets and speech making, but the depth, emotion and soulfulness of his recent writing has touched us more deeply than he knows.

Brett Terpstra on his blog at BrettTerpstra.com shares some brilliant, mad-scientist scripts and automation ideas. I use his Slogger and Markdown products all the time, and am always surprised by the creativity, breadth and depth of his ideas.

John Gruber at Daring Fireball is the original and still the best. He has an opinion, but more importantly, he justifies his opinion with clearly written thought and argument. And he never claims to be right. May I write as well as him one day.

Federico Viticci and friends review products at MacStories and its clear that they love what they do. Instead of just reviewing products, they use them, push them hard and render clear opinions on them.

I came late to Shawn Blanc’s blog at Shawn Blanc. And boy is it good. Shawn walks a personal journey with tech, and helps us all connect with it.

Marco Arment writes at Marco.org and pulls no punches. He has an intelligent and opinionated style, but never lets his personality override the facts that he ensures are totally, nerdily correct. Aside: Is nerdily a word? It is now.

Stephen Hackett writes at 512 Pixels and seems to point out all the cool stuff others miss. And for his Josiah, #GOJGO.

The first indie Mac app I ever purchased was Brent Simmons‘ NetNewsWire. He writes tech at inessential.com: weblog and I learn something new from each post.

In the same vein, Craig Hockenberry occasionally writes tech at furbo.org and when he drops a post, it hits with a resounding thud. Many a time his posts have saved me when I have gotten stuck on the same issues.

The mysterious Dr Drang writes at And now it’s all this. You never know what to expect next, a script, an analysis or an opinion piece. But they all make you think.

Ben Brooks hosts The Brooks Review and often has an alternative view of things. Which I like. Maybe too many of us follow the herd, Ben does not, and clearly presents his view.

News

The Beard, a.k.a. Jim Dalrymple presents The Loop with all the latest Apple news. It’s where I hear about most things first. And I just love it when Jim drops in with his own opinion pieces. I have a special supply of Heineken should he ever drop by.

Renee Richie heads the team at iMore and I seem to be reading them more and more. I suppose that’s because the are the anti-Verge.

Of all the big companies that write about tech, only one makes this list today, Ars Technica. Their in-depth articles are usually well balanced and they avoid the link-bait fluff pieces that seem to fill the tech press these days. Lets hope that does not change.

Maciej Ceglowski probably had no idea that the Pinboard: popular bookmarks page would be of any use, but it is. Whenever I am looking for something new that I know I will like, I check this page out. It’s like a good Hacker News because of the kind of people who use Pinboard.

Shopping

I hate shopping, but surprised myself this year by finding two recommendation sites that I actually like. The Sweet Setup is new, and recommends apps, sorting the wheat from the chaff; and The Wirecutter that goes through all the tech specs and mad model numbers to find the best devices so we do not have to.

Thank you all for inspiring, entertaining, informing and sharing. You may never hear from us, your audience, from me, your reader, so hear this now: thank you, we love and appreciate what you do.

Hiltmon Now

Note: I wanted to keep the list short, being part of a long list makes it less special. And I wanted to focus on individuals or small businesses (except for one), doing what they love, not what they are being paid to do.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

Cannot Find File

“Can you please send me that file again? I cannot find it”

“I’ll have to get back to you, I don’t know where I saved the file I need.”

I hear this every day. Even now, in 2013, more than 20 years after hierarchical file systems became the core and the norm of computing, users still struggle to understand something as simple as knowing the location of their own files in their own computer systems.

I blame the complexity of the past and I have only seen one promising idea that has a way to go but may solve this issue.

The Cause

The hierarchical file system is something that ordinary, non-geek users do actually understand, once they get to it. They do get the concept of folders and the concept of placing files in these folders. The issue is in the complexity of finding the starting folder to work with.

All current Operating Systems have an issue here.

Windows sucks because of drive letters. Instead of just giving users a clean file system like Unix, Microsoft followed the CP/M and MS-DOS model of assigning letters to disk drives, something that users struggle to comprehend. What the heck is A: vs C: vs D: vs J:? Before users can even think of which folder the file may be in, they need to think in terms of which drive letter may contain the folder. Yet some letters are CD Drives that cannot be written to, but which? Some are inscrutably empty, but which? Some only appear at work, but which? It’s too confusing, so users give up and just let the system save where it wills, thereby losing any chance of finding the file later.

Unix sucks because of the /mnt file system. Instead of having inscrutable drive letters, users are faced with inscrutable paths. Worse, these mounts are usually shared between users on the same system. Which means that other user’s mount points exist to confuse the current user. And even worse, ordinary users may have to use actual command-line commands to access files remotely, using tools like scp. Which means that before the user can get to the folder to start looking, they need to remember which computer, server or mount the folder is on. It does not end well.

And OS X sucks these days because of iCloud. Apple may have solved the /mnt mess with it’s GUI display of volumes mounted on the desktop, then taken two steps back by introducing iCloud. Now the user has to remember which application (even worse than drive letter or mount path) that they used to create the darn document before they can figure out the folder in order to find the file. Assuming they even created the file themselves.

In short, there is too much for the user to know before they can even get to the file system to find files.

The Workarounds

Search tools such as Spotlight on OS X or Windows Desktop search attempt to resolve this mess by indexing files on all available drives. But regular users do not name or label their files properly, or know how to phrase searches in a way that these engines can find what they need. So search engines may be good, but do not work as expected and are ignored.

Some users have worked around this using their email clients. If a file is in their recent emails, they can find it based on who sent it and when. Go beyond a few weeks, however, and it is lost there too.

Other users jam everything onto their desktops. Because that’s the only place they can find things again. And wonder why their computers get so slow.

Document management applications do exist, but your average user does not have them. Web served file stores, like SharePoint or Google Drive all provide search and folders too, but again, most users don’t even think to look there first.

Even the workarounds require more user effort.

The Promising Option

In reality, the majority of computer users hit save, name the file and hope for the best, that they can find the file again whenever it is needed.

The current model requires them to make a bunch of decisions when saving: which drive, which cloud, which mount point, which server and then which folder. It’s too hard and too confusing. So they don’t. And I believe that they shouldn’t have to make these decisions.

I believe the option that holds the most promise in my mind is the technology behind Dropbox. Imagine for a moment that the default save location, the documents folder, on every device on every Operating System was actually a shared Dropbox-like root folder. Then have the File Open and File Save dialogs all default to this starting point. All files, all user folders, all in one place. Which leaves only one concept for users to understand, folder trees, something they do get!

Add a new drive to your computer? It should invisibly enable more storage in this documents folder or show the folders it contains as a folder. Go to work? The shared folders you have access to should appear there, as folders. Heck, work from home and they should still be there (invisibly being routed over secure VPNs without user intervention).

From a user’s perspective, all their folders and all the folders they have access to appear all in one place. As folders, with their files in them. So much simpler.

How do we get there?

Most users do not have a Dropbox account, and will never get one. And old-school IT managers want people to use their server drives for saving, compliance, control and backups. And I know not of a Dropbox like sync engine that just works within companies and homes.

So, if Operating System vendors added a common sync framework, maybe even just buy Dropbox for its technology, then share it amongst themselves, it could work.

Imagine this.

A new user gets a company computer with the operating system of their choice. IT has already set up their corporate dropbox. All the user user needs to do is save and load files to their default documents folder and the network (Dropbox) takes care of sync, shared folders and backup. Corporate shared folders appear as user folders, with special icons.

At home, if the user gets a Time Capsule, server or networked drive, it can broadcast itself as a home documents/dropbox server. The first time the user’s computer sees the device on the network, it asks if the user wants to back up to it and it then takes care of syncing documents to itself. All done. If users want to share files at home, they can set up a shared folder in their Documents folder and the dropbox-like home or work server takes care of routing.

From this perspective, all users have to do is open and save files in the default location in the folders they know. No need to think about drive letters, applications or mount paths. Just folders and files. And their data.

I believe the technology exists to do this, but sadly, we cannot make it happen. The Operating System vendors need to bring it into existence, integrate it into their offerings and make it seamless and accessible to users and network administrators alike, wrapping it in necessary security using the reliability of the Dropbox model. For users, it should “just work”.

Here’s hoping it will happen for regular users in 2014. It cannot happen soon enough for those that work with and support them. Then maybe “I cannot find a file” can go the way of the floppy disk.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

Managing Multiple Project's Assembly Notes in One

The problem I have is that I regularly need to open a specific set of Markdown documents scattered all over my file system while I work and have them available at my fingertips. But I need a different set open when working at home and available the same way. The best solution I have found is to use BBEdit projects to manage these document sets. Here’s why and how to do it.

The Scenario

I work across several projects during the day at work and across a different set of projects at night. I would like to have all their Assembly Notes and TODO documents together in one application for reference and update.

But each project I work on follows my preferred Project Folder Layout in the file system which means these files are scattered all over the place. Each project has an Assembly Notes document and a TODO document stored in its documents folder but I want to work with them in one place.

The pain point comes in when I switch projects and need to update the assembly notes or TODO for that project, then get back to another project. Finding, opening and closing these scattered files all day is not optimal.

Also, storing these files in a central location does not work for me as these are all for different clients and are stored in different repositories. I want the documentation to remain part of each project so its easy to zip, back up, copy, sync, commit and share.

The Solution

Since all these files are plain text Markdown files and I usually have them open in BBEdit, I use BBEdit projects to manage each set that I need. During the day I open the work project in BBEdit that contains references to all my work project assembly notes and TODO documents no matter where they are. At night, I open the Noverse BBEdit project which contains references to all current Noverse project assembly notes and TODOS.

And the best part is that BBEdit has a separate section in File / Open Recent for just these project files, so its very quick to switch sets.

To set this up, create a new Project File in BBEdit from the File / New menu. Then drag and drop the assembly notes files and TODO files into the project area at the top left of BBEdit from all the various project folders. Then do the same for the next set.

At the start of the work day, open the work BBEdit project and all your project assembly notes will be available in BBEdit at your fingertips. At the end of the day, close that project window and open the evening Project file and all those files will be now be available in the same manner.

In my case, BBEdit is always running, with one window containing the current set of assembly notes and the other being used as my hammer tool for file and data processing.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

The Day He Walked to Freedom

I remember it as a clear day. Sunny, not hot, not windy, just crystal clear blue skies. The rumor mill had it that he would be released on that day. It had been building up for a while. But no-one knew for sure.

I was on campus at the University of Cape Town when it happened. Word spread quickly. He’s out. Really out. They actually let him go. We had no internet or video phones back then, just word. We did not see him walk along that dusty road to freedom until later. But we all just knew. And we knew that he would next be driven to Greenmarket Square, the central square in Cape Town, for all of us to welcome him.

We rushed to our cars and to trains and raced into town. Somehow we all managed to find parking above the central train station and trotted across the road to the square. We were early so we got a great spot, about 10 meters from the Town Hall balcony where speeches were given.

Over the next couple of hours or so the square filled up. People of all walks of life, of all races and ages were coming together. This was unlike any event in Apartheid South Africa, for once the people were coming together.

Up the side roads leading into the city were the riot police and the army. The big water cannon that fired purple dye were in front. Lined behind them like a bunch of deadly turtles were the big armored trucks that the police and army used to shoot people from. Big men with big guns filled the gaps. They were preparing for war.

As the crowd grew, the noise grew. Not with screaming or slogan shouting, not even song. The murmurs of voices in wonder that this day had come, and that all these people had come.

Finally, a few old men walked out on the stoop of the Cape Town Town Hall. They had been released earlier and we recognized them. The crowd quieted in anticipation.

It felt like there were a million people in that square on that day at that time. But it was silent. No coughs or fidgeting sounds, no wind, even the birds seemed silent. We waited patiently.

A smallish older man slowly walked out to join the group before us. None of us had ever seen him. We did not know his face. But we all knew him. We just knew. He stood there a moment, looking at all of us staring back at him. I think he was surprised to see the turn out, and by the silence.

He slowly reached into his jacket pocket and pulled out the papers of his speech. They looked like they had been torn out of a child’s school notebook. Then he patted his other pocket.

A flash of consternation touched his face. He patted his other pockets. We waited. Then he leaned over and whispered something in the ear of the man standing next to him. We waited. The other man took off his reading glasses and handed them over. The person we’d come to greet had left his reading glasses back in jail.

“Amandla” he croaked, a thin, weak, reedy voice that should not have carried. We all heard it.

“Awethu” replied a million people in one voice.

“Viva” he said, voice getting stronger.

“I Africa” a million voices replied in unison.

“Viva” his voice boomed, clearly heard across the square.

“I Africa” we yelled back.

“Viva” he said again. It was usually repeated three times.

“Mandela” a million voices broke the chain and said his name.

He started to read his speech to us. We all listened, absorbing each word. No one interrupted, no one shouted, I think we all stopped breathing. At some point he lost his nervousness and stopped reading from his papers. He just spoke to us. I could see his eyes from where I stood. It was as if he spoke to me and me alone.

And before we knew it, the speech was ending. And it ended in magic. I don’t remember the exact words. I remember one small man thanked a million people who had grown up hating each other and had come together to welcome him to freedom, and asked us to quietly turn around and go home.

The police had expected the crowd to riot and burn Cape Town to the ground after the speech. Maybe we did too.

Madiba had asked us in his soft voice to go home quietly.

A million people turned around and walked quietly back to the central train station. No riots, no violence, no anger, no hatred, no racism. All smiles, tears in our eyes. White people who hated black people were shaking hands with black people. Black people who had been down trodden all their lives were hugging their white oppressors and patting the hated cops on their backs. There was no Apartheid any more, it was over for the people there.

Nelson Mandela’s great reconciliation began there and then, on that day, in that place, for those people, only a few hours after his walk to freedom.

That day changed us all. Thank you, Madiba, for teaching us all to be better people. We knew you and will always remember you and what you believed in. Now its our turn to remember the lessons you taught us, to pass them on and to make this world a better place.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

TextMate 2 Basics

I have been using TextMate for years as my primary programmer’s editor, but it turns out that I only use a subset of its amazing features. I’ve been speaking with users of other awesome programmer’s editors and they wanted to know what it is in TextMate that I love so much and keeps me using it.

So here are the basic parts of TextMate 2 that I use regularly. Maybe you can divine some tips from it.

Note that it all starts with the fact that TextMate 2 is the most Mac-like of programmer’s editors. It uses all of the standard Apple keystrokes and text features so that I do not have to know another set. I’ll assume you know the platform keys and features and move on to the TextMate features straight away.

Bundles

The true power of TextMate can be found in its bundles. Whether you code in Ruby and Shell like I do, or in some esoteric language, chances are that there is a bundle that will not only provide syntax highlighting for that language but include a whole bunch of snippets to make programming easier. There are bundles for programming languages, build systems, source code control, testing languages and odd file formats. Even the themes are bundles.

I don’t install all the bundles, I only install them as needed (and remove the ones I do not need). In the good old days when TextMate 1 was still growing, this prevented keystroke and scope conflicts.

To install a language or feature bundle, open Preferences… or press ⌘, and click on Bundles. Check a bundle to install, uncheck to remove. Or just open an unrecognized file and TextMate 2 will recommend a bundle to install.

There are many ways to learn about what’s in a bundle. You could go to the Bundle / Edit Bundles… menu (or press ⌃⌥⌘B) and then click on a bundle to see its contents. Or you could click on the gear menu (see image right) at the bottom of an opened file to see what’s available.

Or best of all, use ⌘⌃T to bring up the bundle search dialog to find what’s there. Type in what you think could exist or what it may be called, and chances are, there it is. That is how I found the available snippets and keystrokes that I use every day.

Default file

⌘N is a programmer’s friend, it creates a new file in any editor. In TextMate 2, the shipped default is a plain text file which really makes no sense in a programmer’s editor. I set it to my most common format, Ruby, as most of the time, that’s what I need. To set the default file type, open Preferences… and on the Files tab, set your preferred new document type.

Favorites

I find myself opening the same projects over and over again in TextMate 2. In TextMate 1 we had project files, but I never really used them. So I used to open projects by opening a terminal, cd to the project folder and type the command mate . to open the project folder (and then created TextExpander scripts to do this, see TextExpander in Terminal).

In TextMate 2, the Favorites feature works better for me. To set up a favorite, use the file browser in TextMate 2 to navigate to the folder above your projects.

Then, to create a favorite, right click on the project folder and choose “Add <folder> to Favorites”.

Next time you want to open an project, just press ⇧⌘O (O not zero) to open the Open Favorite dialog, then type or click to open the project. I just love this. Note that in the current alphas of TextMate 2, you need to restart the application for the favorites to be updated.

mate and rmate

Since I spend most of my day in the shell, I use the mate command a lot to open and edit files in TextMate 2. To install the mate command, open Preferences…, click on the Terminal tab and click install. Then you can just type mate <filename> to open a file for editing from anywhere.

I also ssh to a lot of virtual servers to edit and maintain them. vi works great for simple edits, but I use rmate from these remote machines to launch TextMate 2 on my computer to perform more complex edits.

To set up rmate on the server, ssh into the server and use sudo if necessary. Type in the following commands to download the script and enable it’s execution:

curl -Lo /usr/local/bin/rmate https://raw.github.com/textmate/rmate/master/bin/rmate
chmod a+x /usr/local/bin/rmate

You also need to create a reverse tunnel when you ssh for rmate to work. In your ~/.ssh/config file, create an entry for each remote server as follows:

Host nimitz
    User hiltmon
    HostName 192.168.xx.xxx
    RemoteForward 52698 localhost:52698

You can now ssh to that server and use rmate to edit in TextMate 2. For example:

**local** $ ssh nimitz
**nimitz** $ rmate .bash_profile

will open the .bash_profile file from the remote server in your local TextMate 2.

Muscle memory keys

As a Mac user, I already have the Mac editing keys in my muscle memory. Here are some additional keys that I use all the time in TextMate 2 to maximize my productivity:

  • ⌘T is my favorite key, it brings up the Go to File dialog which allows you to type a few letters and TextMate 2 intelligently finds all matching files in the project for you. Common uses are for opening the routes.rb file (⌘T rou ↩) or the schema.rbfile (⌘T sch ↩) in Rails projects.

  • ⌘R runs the current file. I do not like the default where the results are displayed at the bottom of the editor window, so I change it to New window in the Preferences, Projects tab.

  • ⇧⌃D duplicates a line (or the current selection). I use this a lot to duplicate code patterns and then change the parameters. For example, when writing code from a CSV import to add to an ActiveRecord such as

1
2
3
Security.create!(
  cusip: row[0],
  isin: row[1]

I’ll often duplicate the last line to create

1
2
3
4
5
6
7
8
Security.create!(
  cusip: row[0],
  isin: row[1]
  isin: row[1]
  isin: row[1]
  isin: row[1]
  isin: row[1]
  isin: row[1]

And then use the below keys to navigate quickly around the duplicates to change the labels and indices.

  • ⌥→ moves to the next word (add to select) and ⌘→ moves to the end of a line which is the standard Mac way. So without learning anything new, I can move the caret (cursor to some) around the file a lot quicker. But TextMate 2 goes further, if you do not have it mapped in Mission Control, ⌃→ moves to the next part in a CamelCase word (I do not use this).
  • ⌥← moves to the previous word (add to select) and ⌘← moves to the beginning of a line.
  • ⌘↩ creates a new line from anywhere in the line which saves moving to the end to do so, especially since TextMate auto-inserts closing braces and quotes for you to the right of your caret.
  • ⌘] indents the current line. TextMate 2 usually does this for you when typing, but does not always clean up when deleting.
  • ⌘[ un-indents the current line.
  • ⌘/ comments out a line, or uncomments it if commented out. I use this a lot to activate debug print statements in code when testing and then deactivate them after the bug is found. Select multiple lines to comment out blocks with this keystroke.
  • ⇧⌘] navigates to the next open tab, useful when working between views, controllers and helpers. Interestingly, the Ruby bundle does have macros to switch between these, but I don’t use them either.
  • ⇧⌘[ navigates to the previous open tab

Column mode

I use column mode a lot in TextMate 2, especially when working with SQL fields. In TextMate 1 this was a real column mode where everything had to be squared up, but in TextMate 2, it really means having multiple carets.

Multiple carets are awesome, you can make the same changes all over the document with a few keystrokes. Once you get used to having and using multiple carets, you really feel the power of TextMate 2.

For example, lets say I have the following CREATE TABLE command from SQL Server:

CREATE TABLE Securities(
cusip varchar(9),
isin varchar(13),
name varchar(255),
...

And want to change it to an inline cusip, isin, name for use in an INSERT statement, I can do the following using multiple carets:

  1. Select to where I want the carets to be with the selection start being the first location and the selection end being the last.
  2. Tap the key to switch to multiple carets.
  3. ⌥→ to move to the end of the field name, then ⇧⌘→ to select to the end of the line.
  4. DELETE to remove the excess baggage.
  5. once to cross over the newline, DELETE to remove the new line.
  6. Type a comma to separate the elements, ⇧→ to highlight the excess spaces, SPACE to replace them and finally ⌘⇧L to select the line.

And now I have what I need to paste into the INSERT statement. Its a lot easier than typing out all the fields, especially for large tables.

Searching

As with all editors, TextMate has full search and replace functionality. For me, I use the in-file search and replace a lot, as well as the find in project features, so much so that I have memorized those keystrokes as well.

For searching and navigating within a file, I usually start with a ⌘E to use the current selection for a find. ⌘G then finds and moves the selection to the next occurrence of that text in the file. ⇧⌘G finds the previous occurrence. This is great for finding where a function is used in a file. Select the function name, ⌘Eto set it up for search, then ⌘G again and again to see where its used.

I also use this to make fast replacements. Use ⌘E to set the selection to search. Then replace the selection with the new text. Select that and press ⇧⌘E to tell TextMate that the new selection is the replacement text. If you are brave, press ⌃⌘G and all occurrences of the original text are replaced with the replacement text in the file. Very sweet.

If you do not have a selection to start with, ⌘F is your friend. Press ⌘F to open the Find dialog, type in the search phrase and press return to start searching. You can then use ⌘G and ⇧⌘G to find next and previous.

And to search across all files in a project, ⇧⌘F brings up the Find dialog scoped to all files in the project.

Cool tip: If you have already created a selection to find in a file using ⌘E or a replacement using ⇧⌘E, that will be used in ⌘F (Find in file) or ⇧⌘F (Find in project) as well.

HTML and ERB

The final set of keystrokes I’d like to share with you today are used to create HTML or ERB tags, and I use these a lot too.

⌃⇧, (⌃<) creates a new HTML tag and selects the tag name. Just type to change the start and end tag. So press ⌃⇧, to create a <p></p> tag and then type div to create a <div></div> tag. TextMate 2 also stops copying the tag name to the end tag on space.

For ERB tags, use ⌃⇧. (⌃>) to create a new <%= | %> tag with the cursor where the pipe is. Press it again to change the tag to <% | %>, again to get <%- -%> and again for <%# %> and again to cycle back. That is far faster than typing the whole thing.

Fin

So those the basic TextMate 2 features that I use all the time, its only just over 20 keystrokes to get used to using.

I am quite certain that you can do all of the above in your programmer’s editor of choice just as efficiently. But for me, it’s the Mac basics and TextMate features and keystroke muscle memory that makes me so productive in TextMate 2.

Update: I have written a follow-up post with even More TextMate 2 Basics. Check it out.

If you have any awesome TextMate 2 features or keys you cannot live without, please share them in the comments.

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

Merge Asana Projects Into OmniFocus

Note: Updated the script, see Merge Asana Projects Into OmniFocus Updated.

I use OmniFocus to manage my personal tasks and activities, but my company uses Asana to manage Project tasks and activities amongst the team. I am so used to using OmniFocus that I often forget to check or update Asana, and thus miss tasks assigned to me or progress by the team.

Wouldn’t it be nice if the projects and tasks in Asana also appeared in OmniFocus? And were updated when they changed in Asana?

So I wrote a script to import Asana Projects and Tasks, and then to update them as they change in Asana.

Download: Gist 9786291Gist “merge_asana_into_omnifocus.rb” Read below on how to set it up and run it. Warning: Alpha!

What the script does is:

  1. Scans Asana for all projects and creates an OmniFocus project in the folder Asana. It also ignores archived projects in Asana as you do not need to see those.
  2. Scans each project for tasks and creates an OmniFocus task for it. It also captures the notes and due date for the task. If the task is subsequently completed in Asana, it will complete the task in OmniFocus. If the task does not exist in OmniFocus and is already completed, it will not create it.
  3. If you are not the assignee (the person the task is associated with), it creates a context under People for the assignee (so you can see who the task is for - and leaves the context blank for yourself). This is necessary as the Asana API returns all tasks for a project no matter the assignee.
  4. If there are sub-tasks to the task, the script will create those as well (with notes, due dates, completion status and context). You can turn this off in the script if you do not use sub-tasks to speed up merges.
  5. You can run the script again and again. If a project or task exists, it will be updated and overwritten with the changes in Asana.

The script does not:

  1. The script does not sync data back to Asana. This is a one way merge-in. I have no way to track Asana id’s and keep things in sync. You still have to maintain and complete tasks in Asana
  2. The script screws up if the task or project name changes. It uses these names to match OmniFocus, not ID’s. Since most of the time we never change the project or task name, this works.
  3. The script does not track any tasks that are not part of a project. My needs were for the team tasks in projects to be visible to me in OmniFocus and I do not use the Asana Inbox for anything, its all in OmniFocus. If you have the skill, you may be able to use the Asana workspace call to get these too.
  4. The script also ignores all comments, history and attachments, that’s what Asana is good for.
  5. The script does not do your tasks for you.

Prerequisites

Make sure you have the following set up in OmniFocus:

  • A Project Folder called Asana in Library.
  • A context called People under all Contexts.

Setup

Log into Asana and go into your account (It’s at the bottom left just above the Help link).

Make a note of your Profile Name, you’ll need it in the script.

Click on Apps to get your API Key. Click on the API Key… link to see it. Note that down too.

Open the script in a text editor and insert your API key and Profile Name at the top.

...
class MergeAsanaIntoOmnifocus

  API_KEY = '<YOUR KEY HERE>'
  ASIGNEE_NAME = '<YOUR PROFILE NAME HERE>'
  SUBTASKS = true

  def get_json_data(url_string)
  ...

Download: Gist 9786291Gist “merge_asana_into_omnifocus.rb”

Running It

Make sure that the prerequisites are set up in OmniFocus and Asana.

Then just run it from the command line:

$ ruby merge_asana_into_omnifocus.rb

It will display each project as it finds it, and each task as it searches for subtasks so you can see progress.

The script is not very quick. It has to run a bunch of AppleScripts to talk to OmniFocus, so please be patient.

To update with the latest data from Asana, just run it again.

Tips

  • Make sure you run the script before you archive Asana projects so that their tasks are marked as completed in OmniFocus first.
  • I prefer to use short project names because they fit the sidebar in OmniFocus. You may want to clean up your Asana before running this the first time.
  • Since Asana and OmniFocus treat subtasks differently, I have chosen to use the assignee from the task on sub-tasks unless they are overridden. OmniFocus ignores context on parent tasks.

Warning: This is just my script, it may or may not work for you. Backup your OmniFocus before running it the first time. And use at your own peril. Inelegant, alpha-level, probably quite buggy, my first AppleScripts, you get the drift.

But if you do use it, let me know in the comments or via Twitter. And if you have any ideas for changes or enhancements, let me know.

Download: Gist 9786291Gist “merge_asana_into_omnifocus.rb”

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.

Apples and Rotten Oranges

So Frank Shaw, Corporate Vice President of Communications at Microsoft wrote a thing called Apples and Oranges in response to the Apple Laptop, iPad, iWork, iLife and Mac Pro event yesterday, and sadly, it got a lot of coverage.

So I put my snark hat on and decided to throw darts at his piece (almost completely reproduced here). The text in grey are his words, the rest is me attempting to be humorous, ridiculous or just plain snarky. Take it all with a grain or two of salt. This is the Internet after all

So let me try to clear some things up.

Translation: Testing, 1, 2, 3, Is this thing working. Turning on the FUD machine and cranking it up to level 9.

Surface and Surface 2 both include Office, the world’s most popular, most powerful productivity software…

It also contains Windows 8, nuff said.

…for free and are priced below both the iPad 2 and iPad Air respectively.

Um, dude, there is no price point below free. Unless you pay people to take the products off your hand.

Making Apple’s decision to build the price of their less popular and less powerful iWork into their tablets not a very big (or very good) deal.

Ain’t no better deal than free. So how is Apple’s free software a worse deal than Microsoft’s free software? And how is an easier to use touch optimized iWork not a big deal to users of a touch-based tablet?

Since we launched the Surface line of tablets last year,…

And what a launch it was. Then we sold so few of them that I won’t mention sales at all.

… one of the themes we’ve consistently used to talk about them is that they are a terrific blend of productivity and entertainment in one lightweight, affordable package. In fact, we’re confident that they offer the best combination of those capabilities available on the market today.

No-one was listening. But do go on.

That’s not an accident, it’s exactly what we set out to design.

And then compromised the gezundheidt out of it. (See Daring Fireball’s summary at The C Word).

We saw too many people carrying two devices around (one for work and one for play) and dealing with the excess cost, weight and complexity that “dual carrying” entails.

I live in New York and see lots and lots more people carrying one device for work and play, an iPad. They also carry phones, handbags, umbrellas, briefcases and pens. So it’s really “multi carrying” (and they have to pay for their pens here too).

We believed…

but no longer believe, note the past tense here.

… that there was another, better way: A tablet built to offer great touch-based entertainment activities combined with a productivity powerhouse that helps people crank through the stuff they have to get done before they watch zombies or flick birds.

I have been told that performing “touch-based entertainment activities” makes one blind and should not be performed in public while wearing nothing but a trench coat and fedora.

That’s what Surface is. A single, simple, affordable device that helps you both lean in and kick back.

Since it violates temporal theory to be in two places at once, leaning in and kicking back are separate activities. Or maybe he means leaning into the back of a chair and kicking back. Just try to use a Surface on your lap with the keyboardy covery thingie. Hilarity ensues.

Let’s be clear – helping folks kill time on a tablet is relatively easy.

And Microsoft does it so well, making them wait to boot, wait to log in, wait to launch apps, wait to scroll, wait to browse the web, and wait for apps that will never appear on the device to be not written. Yep, very good at killing time.

Give them books, music, videos and games, and they’ll figure out the rest. Pretty much all tablets do that.

You mean make them buy books, music, videos and games, that stuff ain’t free.

But helping people be productive on a tablet is a little trickier.

No, it is not tricky at all, heck even the Samsung Galaxy Tab and the Kindle Fire show how easy it is.

It takes an understanding of how people actually work, how they get things done, and how to best support the way they do things already. The good news is that Microsoft understands how people work better than anyone else on the planet.

If Microsoft understood how people worked, they would have created the iPad.

We created the personal computing revolution by giving people around the world a low-cost, powerful, easy-to-use device…

Er, IBM did that.

… that helped them accomplish an unbelievable array of tasks.

Er, Apple, Lotus, Adobe and lots and lots of others did that.

And together, Windows and Office ended up reaching every corner of the globe and powering every academic institution, industry and profession. Of course both Windows and Office are evolving all the time – to reflect the way people work today – more social, more mobile and connected through the cloud.

And so he downs a yard glass of the Kool Aid.

Did not Microsoft force PC makers to sell Windows, made the Office file formats proprietary and undercut all competition price wise to get all corporates to use it (thereby killing WordPerfect and Lotus), and made out that users chose to work this. You sure that’s just tobacco in your pipe?

We literally wrote the book on getting things done.

Er, no, that was literally David Allen.

And that’s how we knew that Surface needed to include three things to help people do their best work:

1. The gold standard in productivity software – Office.

Slow, bloated, non-touch, software that most people have moved past. Oh, I see, Outlook is not yellow, it’s gold.

2. Faster and more precise input methods like keyboard/trackpad.

Because fingers, pointing and touching, skills humans have had for thousands of years are so imprecise we cannot indicate direction or place food in our mouths.

3. The ability to use apps and documents side by side, allowing the comparisons, analysis and synthesis that happens frequently during content creation.

Walk around any office or coffee shop and see how many people use Microsoft Windows with only one app open in full screen mode. Yep, all of them.

That’s what we delivered. And it’s why the Surface is the most productive tablet you can buy today.

Unless you have an iPad, Samsung Galaxy, Kindle Fire or any other tablet that exists on the market.

We also knew that it would make our competitors take notice.

They did, ridiculed it (See Microflops: Microsoft Surface RT and 8 tablets) and got on with their lives.

That as consumers got a taste of devices that could really help them get things done, they would see alternatives as being more limited.

So they saw the limited alternative, being the Surface, and bought the alternatives, being iPads.

And so it’s not surprising that we see other folks now talking about how much “work” you can get done on their devices.

People are talking about getting work done on their devices for a simple reason: you can get real work (without scary quotes) done on tablets, and they do.

Adding watered down productivity apps.

That do the 15% of the product that everybody needs and works well on touch enabled devices without delays and bloat.

Bolting on aftermarket input devices.

Instead of clicking-on aftermarket input devices.

All in an effort to convince people that their entertainment devices are really work machines.

Which they are, since all of the Fortune 500 use iPads. But ah, the iPad is for consuming content only, that old hoary thing, sigh.

In that spirit, Apple announced yesterday that they were dropping their fees on their “iWork” suite of apps.

Ooh, scary quotes around iWork (but then not used before or again in the post, even in the next sentence).

Now, since iWork has never gotten much traction, and was already priced like an afterthought,…

And since Microsoft charges for Office like a wounded bull, and does not want to lose that revenue.

…it’s hardly that surprising or significant a move. And it doesn’t change the fact that it’s much harder to get work done on a device that lacks precision input and a desktop for true side-by-side multitasking.

You are right, touch enabled large buttons are so hard to prod with fingers designed to do aforementioned prodding. So Microsoft left them small in Office and needs a trackpad for a touch device.

But you wouldn’t know that from reading some of the coverage I’ve read today.

I’m guessing you were reading about the launch of the Nokia tablet, because the iPad event was all about productivity.

Perhaps attendees at Apple’s event were required to work on iOS devices that don’t allow them to have two windows open for side-by-side comparisons,…

Yep, the big bully Apple forced people to use iPads, whereas the nice Microsoft never “embraced and extended” a whole bunch of companies into dust.

… so let me help them out by highlighting the following facts:

Starting with the F in FUD ain’t for ‘Fact’.

• The Surface and Surface 2 are less expensive than the iPad 2 and iPad Air respectively, and yet offer more storage, both onboard and in the cloud.

They have to be cheaper, even people with IQ’s lower than the thickness of a Surface won’t pay more if they can get a real iPad for less.

• … come with full versions of Office 2013, including Outlook, not non-standard, non-cross-platform, imitation apps that can’t share docs with the rest of the world.

So, standards-based IMAP email clients on all other tablets that use open protocols, well known MIME formats and send emails using public and free services reliably is worse that a bloated, buggy, proprietary email product that sends emails via an even more expensive and complex server (that Microsoft just happens to sell and wants to make more money on).

• … offer additional native productivity enhancing capabilities like kickstands, USB ports, SD card slots and multiple keyboard options.

If your number one differentiation feature is a stand to make the tablet stand up (and which is totally useless when actually holding the device or resting it on your lap), you have a problem. Oh, and like there are no keyboards or keyboard support for iPads. Really?

• … include interfaces for opening multiple windows, either side by side or layered to fit the way most people actually work.

Back with the multiple windows thing that no-one ever uses. The donkey died, stop beating it. (and no, I am not referring to performing “touch-based entertainment activities”)

So, when I see Apple drop the price of their struggling, lightweight productivity apps, I don’t see a shot across our bow, I see an attempt to play catch up.

Yep, Apple would love to catch up, er down, to the sales of Surfaces. Looks like this is the part of the post where the FUD dredged up is from the bottom dregs of the FUD wastebasket.

I think they, like others, are waking up to the fact that we’ve built a better solution for people everywhere,…

No-one, ever, woke up and imagined the Surface to be a better solution (see the compromised link above and any, and I mean any review of it).

.. who are getting things done from anywhere, and who don’t have hard lines between their personal and professional lives. People who want a single, simple, affordable device with the power and flexibility to enhance and support their whole day. :)

Yep, like iPad users, Galaxy Tab and Kindle Fire users. Or iPhone and Android users. Heck, even Blackberry users have “a single, simple, affordable device with the power and flexibility to enhance and support their whole day.”

Ok, one final dart: The whole day? A day is 6 hours? Really? According to Zdnet

The Surface Pro 2 trails behind competition in both the web browsing and video playback battery life tests, achieving a runtime of 6.68 and 6.65 hours respectively. Compare this to the Nvidia Shield, which ran for 13.53 and 18.63 hours in the respective tests, or the iPad 4 which could manage 9.48 and 13.45 hours.

BIGGER SMILEY FACE

Follow the author as @hiltmon on Twitter and @hiltmon on App.Net. Mute #xpost on one.