September 2007 Archives

Since I have no formal degree in programming, I have been extremely fortunate to work on teams of very talented, smart and diligent coders. Wherever I have worked, the bar has been high, the code has been challenging, and the expectation is that everyone on the team will do their best work every time. The terminology used among these various teams has varied, but the general sentiment is always "Don't do something you're going to regret later." Don't tie the data to the code, don't make it hard to extend the program, don't tie code together unless you have to, don't make assumptions about what people will do with the application... in short, don't be dumb. When we were working on the REST interface for Socialtext it was one of the best projects, because decoupling the data from the functions was useful and fun and it just *felt right.* The words used to describe these concepts were general terms like abstraction, pluggability, configurability.

Coming to Applied Minds, I find myself awash in new vocabulary words and frequently find myself scurrying to look up a term that's been thrown my way. Fortunately, once I do that it generally turns out that yes, I do that, whatever it is. For instance, Data Driven Design falls under abstraction, configurability, and "Don't be dumb." But I didn't know that.

I have to learn a couple of new languages next week in order to create a demo application for us to play with, so I headed off to Borders to pick up a book or two. While I was there, I also picked up the "Pragmatic Programmer" book which I heard referred to so frequently by my cohorts at Socialtext. Yesterday and today I read through it and discovered that it is all full of "Don't be dumb." Things I have observed and internalized, but couldn't articulate quite yet.

The book is an excellent discussion of the way a good programmer behaves, the things they do and don't do, and how to incorporate them into your workflow. Many of the sections are likely to add more overhead to your project up front - test harnesses, contracts, decoupling, abstraction of metadata. But the return on investment is huge. *Not* doing these things, paying attention only to the here and now and neglecting the future, is where almost everybody gets caught.

I have a strong tendency to want to get to the finish line as quickly as possible, and am frequently stung by not doing these things. Here, I'm starting a new project. It's not a rush (well, the demo wants to be created in two weeks, but that will be a true "throwaway" prototype for the purposes of playing with the UI elements). So I'm pledging to myself that I'll keep the discipline necessary to do things right, take the extra time to think abstractly and control the interfaces between the moving parts.

It's a good book. If you are a programmer and you haven't read it, you probably should.

I'm a planner. I like to plan. It's very comforting to know how things will happen. When I started this job I planned to spend a couple of weeks brainstorming, I outlined what I'd brainstorm about and what I'd have when I was done. The first week I pretty much stuck to my plan and investigated things, and then this week I was going to spend writing proposals about applications.

This weekend I helped my dad move, which involved flying 500 miles away and then driving his car back up on Saturday (and then helping him unpack on Sunday). The Saturday drive gave me a lot of time for thinking, and I realized that I was much less interested in what the applications would do than I was in how they would help the user visualize the complex data space. Metaweb is a great backend for things, and MQL is a good language for querying graph data, but there's still not a great metaphor out there for exploring that kind of data, and that's the puzzle I find most intriguing.

As it turns out, I'm going to end up with something along the lines of what I said I'd end up with, but the focus will be very different, because the piece I'm critically interested in is the presentation methodology, making the content domain much less problematic.

This afternoon/evening I'm heading up to the Metaweb offices to attend their Freebase user group with a couple of my co-workers who are flying up for the occasion. Should be a lot of fun :-)

I installed Socialtext on a system at AMI (and hey, who knew, it's actually *less* work to install on debian etch than on ubuntu dapper) so that various groups could use it for project coordination and collaboration. So far my workspace is just me writing a boat-load of pages, but I have hope that it will turn out to be an excellent way to communicate progress and record our thoughts.

Once the system was set up, I decided to make an IRC server so that I can actually hang out with the people I work with, even though they're in a whole nother part of the state. Since I had a debian box, and since I am very lazy, I looked to see what I could get with apt and ended up with ircd-hybrid. Took some fussing and fighting but I got it all running and working, and then I decided to add a bot.

Socialtext uses PerpLog for our logging, but since I wasn't really planning to put in a purple wiki and I wanted something more actively loved (sorry, cdent) I checked the apt-cache for IRC bots. My first choice was "blootbot" because it's written in Perl and seems to have all the bells and whistles I might need - plus Perl means I can add functionality easily. Unfortunately, I didn't manage to find the documentation (cleverly hidden on the sourceforge site) until too late, and even then the docs are pretty darned skimpy.

I ended up using supybot, which is written in Python and has a phenomenal set of documentation on installing it, configuring it, writing plugins, playing with it, just generally anything. After installing it I decided to take a slight detour in my day to write a simple plugin for looking up directory information within our company:

kirsten: !cell Kirsten
soupy: kirsten: Kirsten Jones: Cell 831-123-4567

We have an LDAP server which doesn't have all of the info I want, and a directory page which does, so I entered the magical world of screen scraping. Python has a library called BeautifulSoup which parses an HTML document and builds a tree of objects. Once that's done, you just have to figure out how to navigate to the piece you want and spit it out. Took me a little while to get it working correctly, but it does, and now I have both some plugin experience (for making more) and a plugin to save me from the horrible fate of going to a web page to find someone's cell phone.

I certainly could have done the same thing with blootbot, but I didn't have all day to peer through the code to determine how it worked. The lesson here is that documentation is extremely important - all throughout the supybot docs the point is reiterated that the developers of the bot want input and help to make it the best bot ever...

Plus, I needed to brush up my Python skills.

Someone I know once referred to the state reached by shopping at Target as the "Target Fugue State." You go in there to buy a dishpan. While in the dishpan row, you realize you needed some measuring cups. The endcap reminds you that you needed some dish towels, which brings to mind the fact that you needed some bath towels for your guest bathroom. By the time you've left the store you've got 20 items, which you ended up grabbing because Target spends tons of time and money figuring out how to lay out their stores to create *exactly* this effect.

One of the things about browsing through information on the internet is that really, all you can do is search. There are hyperlinks here and there but they have no context. Truthfully, the internet is much closer to a dump truck than a series of tubes - a pile of URLs describing a massive number of information blobs. Companies like Yahoo! have tried to impose some structure, but it's clumsy and incomplete because it's created by humans who look for things and then categorize them appropriately. From any information bit you can't really poke up your head and look around and figure out what is "near" you in the information space.

As a result, when we move about on the internet mostly we use search to find reasonable possibilities, and filter with our brains to narrow the results and find the thing we were looking for. The tools available out there don't help much with true "browsing" in any meaningful way.

If we find ways to impose graph database philosophy on the internet, it would be possible to feel like there were "neighborhoods" in which sites or pages (or other data) lived, and then look around to see what other things were nearby. If a commercial site could actually create a Target Fugue state on the internet, it would make for an enormous improvement in the user experience.

Amazon is currently the site that does the best job trying to emulate this, with "people who bought this also bought" and "based on your purchases I think you want" but it still feels forced. When I go into a bookstore, I love to wander up and down the aisles and just look at what attracts my eye. Although Amazon has 'browse' as a choice I don't really feel like I'm browsing. I feel like I'm narrowing my search, and it doesn't at all have the same feel as wandering around in a bookstore feeling my way through the stock.

I don't have any answers to this, but I'm hoping that building applications on top of a graphical database such as Freebase will make it easier to create that state. It's a hard problem, but that's the kind I like best.

This week I started my new job at Applied Minds, and I've been down in Glendale this week meeting people, ramping up, and brainstorming what we can do. It's an amazing place, as you might imagine, filled with brilliant people with amazing ideas. But it's also a workplace with things that need improving, so I've managed, with some help, to scrounge up a server to install Socialtext on, so that I can collaborate with my management team and some other teams can use it for their projects.

It's funny, coming from a company which is all-collaboration-all-the-time to go to a company which is brilliant in so many other ways, but which hasn't spent much time thinking through ways to reduce redundancy and build shared understanding on things. Since I'm going to be a remote worker (basically the only one they have) I'm strongly motivated to create processes and systems to make it simple for the mother ship to know what I'm working on, provide feedback and participate in collaborative activities.

For the first few months, I plan to come down here for about a week each month, so that I can both establish myself as a useful AMI team member in general and take advantage of the mighty brainpower assembled here. The atmosphere here is really delightful, so that's a draw as well.

After Labor Day weekend, I was pretty darned tired, and I'm trying very hard to keep reminding myself that there's no need to rush here. I've put together a list of things I need to explore, and I've set aside two weeks to explore and experiment and write up ideas, and then we'll come back together at the end of the month to discuss the options, tune them further, and decide on an overall strategy.

It's really odd to come from a company where there was a true sense of urgency (the company struggled at various times) throughout my time there, compounded by a false sense of urgency contributed by some of the management team and by myself as well due to the nature of my position. Here, I'm supposed to take my time and be deliberate. Slowing down and taking my time isn't something I have a lot of experience with, so here's hoping I find my way there.

I was happy to see that my OpenID talk was accepted at the Pittsburgh Perl Workshop. Should be a lot of fun.