Notes From Thee Underground

Change Of Venue

2006-11-10T10:36:00.000-08:00

Needless to say, I've not been posting for quite sometime. Just a note, though -- The company I work(ed) for, Jotspot, was recently purchased by Google. We started at the massive campus this last week, and my intent is to post (to the extent that I can) a little bit about my experiences as a new sysadmin over here.

(And yes, I'm excited!)

Blast from the past.

2005-10-22T00:56:00.000-07:00

So way, way back in the early/mid-90's, I worked for one of the first really big commercial ISP's, Netcom. I started out when the company was small (~30ish people) and was with them through IPO and a hell of a lot of growth -- By the time I left, we'd gone from two small offices in a dental/medical office park to 3 floors of the Tisch building on Winchester Ave. in San Jose.

Quite a number of interesting things happened there, including the so-called "Green Card Spam" courtesy the cslaw folks and a power outage that's meat for another post. But probably the most notorious was the breakin of Kevin Mitnick into Netcom's network and theft of about 20k credit card numbers, and the subsequent 'takedown' by Mitnick through the efforts of the FBI and Tsutomu Shimomura.

Shimomura and his team worked with Netcom's senior admins (one of whom I was not, although my boss Robert Hood was) for a number of days in the office right next to my cube to actually catch Mitnick, although at the time I wasn't entirely sure what was going on -- Needless to say Netcom's management wasn't keen on the idea of letting a lot of people know they'd just had a bunch of credit card numbers stolen, and besides this was a criminal investigation with the FBI involved, so the full story was strictly need-to-know. I've always been vaguely annoyed, actually, that hoodr's involvement in the tracking down of Mitnick was never really advertised; he was and is still to this day one of the most brilliant people I've ever met, and from what I understand he was partly responsible for Shimomura et al's success, but that's neither here nor there.

The reason for all this lead-in is that I was digging through some old text files last night, and dug up this old email I'd sent the morning after the breakin. See, I had no idea at all what had just happened, all I knew was I came into work one morning, heard there was a breakin of some sort, and one of the systems I was responsible for was completely locked down. At the time I was hostmaster@netcom.com as one of my primary duries, this was one of our primary nameservers, and nobody was around to tell me what was happening, even a little. In my youthful indignant rage, I sent:

From nc0027@netcom.com Mon Dec 12 08:58:07 1994
Return-Path: 
Received: from office.netcom.com by mail3.netcom.com (8.6.9/Netcom)
id IAA08924; Mon, 12 Dec 1994 08:57:48 -0800
Received: from admin.netcom.com by office.netcom.com (8.6.9/Netcom)
id IAA16594; Mon, 12 Dec 1994 08:57:02 -0800
Received: from netcom20.netcom.com by admin.netcom.com (8.6.9/SMI-4.1)
id IAA28177; Mon, 12 Dec 1994 08:57:01 -0800
Received: by netcom20.netcom.com (8.6.9/Netcom)
id IAA04336; Mon, 12 Dec 1994 08:56:59 -0800
From: nc0027@netcom.com (Brian Thomas)
Message-Id: <199412121656.iaa04336@netcom20.netcom.com>
Subject: NNTP....
To: admins@admin.netcom.com
Date: Mon, 12 Dec 1994 08:56:59 -0800 (PST)
X-Mailer: ELM [version 2.4 PL23]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Content-Length: 1933
Status: RO


After conferring with John H. I have re-enabled telnet, rlogin, and ftp
to nntp. I have commented out the nologin lines from the rc file, and
the system was rebooted to insure it's cleanliness. Password for noc
was changed, and logins were disallowed for netnews, netmail, support, and
riz, and wolfie, and the appropriate people told to login as root and
set new passwords.

Although I understand the necessity of emergency action during a crisis,
it should be pointed out NNTP is one of our primary/authoritative name
servers, and our single NNTP server. Access being cutoff to that machine
in a crisis is fine, but it's not acceptable to walk in Monday morning
and find a crippled system with no sign of the administrators who crippled
it. I'm not blaming anyone in particular, as I know a lot of people worked
very hard because of this incident, but I'm saying we need to take steps
to make sure this doesn't happen again.

Y'know, I think I'm gona assign macros to my F keys and pass out the
assignation file. I mean, it's not like any of us say anything different
in these messages. :)

Also, in a seperate story, whoever was  moving around NAC servers in the
machine room managed to shove NAC1 up nice and hard against nac2, knocking
out NAC1's tranceiver. Apparently the moves were done last week; this would
lead me to guess the vibration from drills and stuff made the already
half-fallen-out tranceiver fall completely, or something like that. I'll
also point out those servers are getting VERY hot shoved together over
there, with fans from some servers blowing directly onto others.

In Netcom, no one can hear you scream...

Brian

What I didn't know is that Mitnick had used nntp.netcom.com as one of his intrusion points, and during the night when it was realized what was happening all access was shut off to machines that were thought to be compromised. It wasn't until I tracked down hoodr later in the day that I found out there'd been a breakin and the lockdown had been a reaction to it, and weeks later I learned the scope of what had happened when Mitnick was caught in North Carolina (I believe) and management came clean about what was happening.

One other interesting note: Those NAC servers I'm referring to are some of the very first NFS servers the company now known as Network Appliance ever sold. Netcom was an early adopter of their technology, and we had about 5 of those big black cube cases they sold way back when. I even remember you could buy those cases at Fry's, we even had one at home my roomate and I built a NeXTStep x86 box on. NetApp back then used commodity hardware to drive they're systems; I remember they were these monster EISA-bus based motherboards on the inside they probably assembled themselves.

So there you go. One of my dubious claims to fame is that Mitnick broke into one of my boxes to steal credit card numbers from Netcom. Not exactly the most illustrious event, but looking back on it now, it's kinda cool.

When I'm SUPPOSED to be working.

2005-04-28T13:51:00.000-07:00

IM gem of the day:

IT: So, it looks like Shannon and I will be living in the city (with the gay people!)

btWK: Wait... There are gay people in San Francisco?

btWK: Why am I just now being informed of this?

IT: Yep. You can tell by the rainbows -- it's their symbol.

IT: By the way, rainbows are gay. Gay people should have something cool for a symbol, like a tank or a fast car.

btWK: I'm not sure a 'Pride Tank' would really fit the bill.

btWK: As for a fast car, maybe a Miata.

btWK: That's pretty gay.

IT: It would say 'Don't Mess with Me. Really.'

IT: A tank would, I mean.

IT: I'd vote for the Miata, but strongly suggest the tank.

IT: Let me know if you make any progress on that.

btWK: It could say, "Don't Tread On Me". 'Cuz tanks have treads, making it a pun that would make people be both amused and scared at the same time.

btWK: I'll bring it up at the next gay board meeting.

btWK: Honestly it'll be a refreshing break from the constant ideas about how to indoctrinate youths into our deviant lifestyle.

IT: Heh!

btWK: You can only discuss infiltrating America's public school system so many times before it just gets boring.

IT: That made me laugh.

btWK: I'm going to put this conversation on my blog by the way. Fair warning. :)

IT: No problem.

IT: Talk to you later.

btWK: Later!

Does the Internet make you stupid?

2005-04-04T21:01:00.000-07:00

A lot of it probably does. Too much Fark will actually lower your IQ, given enough time.

But not all of it. One of the exceptions is The Becker-Posner Blog. Written in a very Economist voice by Gary S. Becker and Richard A. Posner, it covers current topics from Tort Reform reform to the War on Drugs in a refreshingly objective (or at least opposingly subjective) examination. They often disagree but bring up excellent points, and I find myself consistently impressed by the thoughtfulness they put into their entries.

It's definitely dry, but I promise well worth the work to read. The most recent entries on China's growth opportunities in the coming years is fascinating. Even the comments are worth reading, and they respond to them regularly. Great stuff.

...is the soul of wit.

2005-04-01T19:31:00.000-08:00

You may not find this amusing in the slightest. I found it blowing-milk-out-my-nose-even-though-I-wasn't-drinking-milk funny.

So those who know me (or read the post I did in January) know I read Metafilter on a way-too-regular basis, it's like crack for office workers. A few months ago at this point, someone posted a front page post that said just:

FREE BOXES!

Apparently there are people in the US who didn't know that the postal service gives away priority mail shipping boxes -- It's part of why they charge so frigging much for the service. Anyway, what followed was a rather silly thread about boxes, foxes, I wonder if they ship those boxes in another box, etc. Now, Metafilter is known for being somewhat liberal... I mean, not Daily KOS liberal, or Talking Points Memo liberal, but liberal enough for my tastes. A few posts in, a gentleman by the name of "horsewithnoname" wrote:

"Astounding. I can't take a moral argument seriously from any of you who jumped at the opportunity to take advantage of an honest mistake. I don't want to hear ever again about greedy corporations or crooked politicians. You people just showed that you're willing to turn a blind eye when it is to your benefit. How does that make you any different from those you rail against?"

Two posts down, Pretty_Generic wrote (and this is the punch line so pay attention):

How does that make you any different from those you rail against?

We have boxes.

Apparently no one told long-face that this isn't actually taking advantage of the USPS, they're offering them right there on the site or the post office for god's sake, and besides the boxes aren't exactly useful for much besides shipping things via USPS priority mail. But that's not really the point. The point is that, in three words that involve no double-entendres, puns, or twists of a phrase, Mr./Ms. Generic managed to make a joke that's side-splittingly funny.

That, right there, is gold. It's comedy at it's most pure, and I love it. I savor it. I still run this joke over through my head and it makes me grin. It's become a meme on Metafilter that will likely last for months to come. All because of timing, placement, and yes, brevity. How cool is that.

I live!

2005-03-31T22:06:00.000-08:00

I'm actually... Posting?

Bought a house. I'll use that as my excuse for being so quiet!

Watch this space.

Time Well Squandered

2005-01-01T10:45:00.000-08:00

I know I was yammmering on about various sites I read stuff on while I was up there for the holidays, I thought I'd send you all a list of stuff I find interesting to read when I have a few minutes of free time.

First, the two 'discussion' type sites I read are Metafilter and Plastic. The former is meant to be a 'best of the web' composium, a very large blog that virtually anyone can post to as long as you've been a 'member' of the site long enough to have made some comments here and there. Plastic on the other hand isn't a blog at all, but a closely-moderated discussion site with much more complex FPPs (Front Page Posts) and a moderation system for comments. Both are worth checking out, the former's I think more fun than the latter. Both are also more time-consuming to read than the stuff below, a single thread can take nearly an hour to read and process if you really want to see all the comments and reference material.

Metafilter also has the 'Ask Metafilter' sister site. I actually got the idea of sending this mail because of an Ask post I saw this morning, for Evelyn. It's a question about how to properly sear Creme Brulee.

For comment-less 'best of the web' sorts of sites, sort of check once-a-day updates, there's generic ones like BoingBoing, Cool Tools, and Cool Hunting, geek ones like Engadget and the venerable Slashdot, modern design like MocoLoco and FunFurde, and cooking information sites like Cooking For Engineers and the info-cum-recipe-site Epicurious.

There's what could be best referred to as Funny Shit. Comics for gamers like Player vs. Player, VG cats, and Penny Arcade are fetchable (as are many others) via a comics aggregator my friend manages -- Be aware that some of the comics are occassionally NSFW (Not Safe For Work) depending on the day.

There's of course The Onion which most people know about and check weekly, but did you know about The Onion A.V. Club? Not only are the reviews fantastic, but you can get your weekly Savage Love fix there and occasional funny stuff like DVD Commentary Tracks Of The Damned, Films That Time Forgot, etc.

Speaking of films, there's also the good ol' movie-snob Film Threat, the now-defunct paper magazine that's transformed itself into a pretty good web site. The reviews are often both insightful and conflicting, it's not uncommon to find reviews there with wildly diverging opinions on a particular film, something I like. There's always Rotten Tomatoes as well for movie review comparisons. Finally, if you're a film geek, you can't get by without reading at least SOME Roger Ebert commentary every once in awhile, even if he's kind of softened since Siskel's passed away.

Finally, in the 'Watching Government" category, there're a couple of sites the try to shed light on what's going on in US and world governments without getting all tinfoil-hatty. The first is Cryptome, which just lists documents fetched through FOIA and similar laws and posts them, verbatim and without comment. The second is The Memory Hole which gets a little more shrill, but has some fantastic pages of information you really won't find anywhere else.

Enjoy.

I don't want to be a professional...

2004-11-14T01:09:00.000-08:00

I just wanted to be as good as this guy. Maybe he IS a professional, I don't know. I do know I love his pictures. Check 'em out.

Been writing

2004-10-28T13:40:00.000-07:00

...But not posting.

This is perhaps one of the coolest things I've found recently, amazingly off of Slashdot, which I thought had actually become a black hole of coolness, sucking the cool out of everything around in. I was apparently mistaken.

Two men enter, one man leaves!

2004-10-20T11:03:00.000-07:00

Interview, or cage match? I'm guessing the interview ended abruptly because McNealy stood up, morphed into his 'Fighting Form', and him and Takahashi began an epic battle that destroyed half of Atherton.

Holy jumping christ on a pogo stick!

2004-10-18T09:09:00.000-07:00

The badassness to end all badassness in the digital camera realm. I give you the Hasselblad H1D. 22 Freakin' megapixels. Built-in 40GB HD (Image size? 132MB each.)

I think I need a towel.

Still alive.

2004-10-13T22:27:00.000-07:00

Well, what do you know, moving your blog to a new site doesn't make you magically start posting in it again! Whodathunkit?

I actually have a few proto-entries I've been wanting to write, but work has been kicking my ass up one side and down the other since I got back from NY. Which isn't to say that I spend every waking moment at work; it's just the hours I'm not spending there are spent trying to calm myself down.

I could get all whiny about how overwhelming it is, but the fact of the matter is I devote too much of my emotional energy to my work and always have. Some of it's passion, but some of it's also just my needing to get better at rolling with the punches. No matter how... Challenging the situations I run into, I need to somehow get my head wrapped around the fact that these problems are better solved through calm and ruthless manipulation* than through seething frustration.

On the upside, Amon Tobin has a live album out, and so far, this isn't pleasant and spacey Northstar Tobin. This is more like he's been spending evenings toking up with the likes of Photek and Bill Laswell, a new direction I'm happy as a clam with.

*For my coworkers reading that, when I say 'ruthless manipulation', I really mean 'being a team player'. Honest.

Graffiti #1

2004-09-28T18:58:00.000-07:00

Graffiti #1,
originally uploaded by Wolftrouble.

Where have I been? In New York, actually, for a week for work. Well, partially for work, and partially just to get my first look at the big apple. Take a gander at my Flickr page for more snapshots from around the city.

Major props to my buddy Doron for playing tour guide (and alcohol-guzzling enabler). Wouldn't have been the same trip without you, man.

Steve & Me

2004-09-09T21:47:00.000-07:00

I went looking for something yesterday.

I'm not sure, in retrospect, entirely what I was looking for. I mean I had an idea of what I wanted to find, just not entirely sure how I would go about it. I eventually found it, sort of, but it depressed me at the same time.

Set the wayback to machine to 1992. A friend of mine, Elf (don't ask), invited me to go with him to Sunnyvale, California. He was going out to visit his boyfriend at the time, who's roomate I desperately wanted to meet, an artist who's work I admired. I'd never been to Northern California before (Disneyland, only, from when I was a kid.), I was 18 with a weekend to kill, so I said sure. I knew, kind of, that Sunnyvale was in this place called silicon valley, and there were a lot of electronics companies down here, and frankly that was as appealing to me as meeting Elf's friends.

I'd been a computer geek since I was like 13 or 14, mostly PC's from back with PC's mostly meant IBM. In the way that I'n sure airplane geeks get all breathless when they visit Seattle (Oh my god, that's Boeing Field!!!) I remember, with crystal clarity, taking the 280 into Sunnyvale. The Toshiba building, the first high-tech company logo I saw, is still there. Infinite Loop, Apple's headquarters, had just been built right off the De Anza exit we took. Hewlitt-Packard was everywhere I looked, it seemed. Seagate in Milpitas. NetManage, across the street from Apple. Intel's gigantic buildings, churning out chips way before making them in foreign countries was the only way to make any money, in Santa Clara. Even McAfee Associates, where I'd end up working as my first job in the vallley and who's shareware I'd been using for years, was where two of the people I met worked.

And I was excited. Most of you reading this will have no idea what I'm talking about when I say that, why the fuck would anyone get excited seeing companies? But to me, this was Valhalla -- Not heaven so much as where the god's themselves walked. The programmers and engineers I held in awe, the software and hardware I tore apart to figure out how it worked over and over again, all of that either resided or came from *right here*, around me, and I was in the thick of it just being BEING there. I hadn't expected to be this thrilled with it all, but I walked around in a daze that entire weekend (although getting laid the first night I was there probably contributed a little to the mysticism),

It was only maybe 6 months before I moved down to the valley to work at the aforementioned McAfee Associates with my boyfriend and a couple friends. This experience at a tiny startup run by a maniacal founder (the first of many) further solidified my view of the valley, one that I find hard to articulate. It encompasses brilliance, excitement, camaraderie, promise, drive, and passion. Even back then before the tech boom (and some would say especially back then) there was this electricity (ahem) that just seemed to well up out of the ground in silicon valley. From Weird Stuff Warehouse, packed to the walls with used electronics, to Disk Drive Depot where you could get all they grey-market hard drives you needed, to Fry's, a place where you could get a new 16550 UART, a Playboy, a CD player and a copy of DrDos 5.0 all in one convenient shopping trip, to Golfland where the newest standup games from Atari and Capcom were demo'd, way before anyone else got them. (Interesting factoid: Golfland Sunnyvale was one of the first, if not the first place to get a DDR machine in the US. Apparently, for a time, the Asian kids who played regularly there were regarded as some of the best in the world.)

Since then, the valley's gone through quite a bit. The size and scope of the tech boom was staggering, as was the bubble's collapse, and during it's time spawned some of the most brilliant minds the industry had to offer. It provided a breeding ground for the kind of creativity and 'lateral thinking' (to use a hackneyed phrase) that allowed growth of people and ideas that never would have occurred otherwise, and I don't mean this in a happy, hippy sense -- This was a matter of companies having the resources to try untested ideas, solutions, and ways of doing business, and for every ten of them that failed, for every dozen engineers who were just in it for the money, you had one really overwhelmingly good idea, or one really brilliant engineer (or executive) that never would have had a chance otherwise to succeed. I bristle at the sentiment that the valley is or was a bunch of spoiled twenty-somethings who wanted money for free; I knew a lot more people who were addicted to their work to the exclusion of everything else than slackers looking for a quick buck. Most of us do this as a passion that isn't well understood by those who don't share it.

So what did I go looking for?

One of the things that's always struck me about the valley is it's juxtaposition of old and new -- Shiny glass and metal buildings housing flash-in-the-pan software companies, next to a ramshackle 60's era strip office, two or three offices of which are occupied by a company that just makes electronics components to order that have been doing it since the 70's. At Weird Stuff, surplus but nearly-new computers sit alongside some archaic massive chunk of metal and circuit board that probably came from some room-sized piece of computer equipment from 1972. You can drive along Fairchild Way (Named for Fairchild Semiconductor) in Mountain View, it'll take you past the AOL (formerly Netscape) building; look behind you and you see Moffett Field, (est. 1933) and next to Lockheed Martin's main campus. A building I worked in that housed Radius (remember the tilty monitors?) was originally built by Control Data Corporation in the 1960's, an old school supercomputing company.

It's that combination and endurance that gives this soulless place a... Center, for lack of a better term. A history and a heart that wouldn't exist otherwise, one that's easy to miss when driving past Generic Office Building #4356 along Arques Ave. Those bland walls hide a pulse that I feel like I can sense just walking around here, and still to this day imbue me with a sense of wonder, because I know what goes on inside of them (at least I think I do). Great ideas are born, cultivated, and come to life.

So really what I wanted to do, camera in hand, was to try and find a visual representation of... this. To find the hints, however sparse, of the energy that drives this place independent of the drab urban sprawl that most see when they travel into or through the valley. I know they're here, I've seen them every once in awhile, in a lighted window in an office in the middle of the night, with a group of people huddled around a table trying to solve a problem. At an all-night restaurant, where a surprisingly boisterous crowd of developers, working until dawn, break for a refuel of caffeine and sugar. At the ham swap meet at 5AM in the faces of a father and son digging through bins of electronics parts, looking for the exact stepper motor needed for a school electronics project. I wanted to capture the enduring timelessness of it; when most people look at Infinite Loop and see an embodiment of the valley, I see it as only one face of it -- And a Johnny-come-lately at that. It's the companies that have been around for decades, or were around until recently like Amdahl that round out what I was hunting for and really gives that substance I love and respect. Cisco, for example, could plop down a headquarters anywhere, but if they did, that wouldn't create the thing that is the valley.

The problem is, this lifeblood is hidden, or maybe I should say it's obscured very well. I've seen it enough times to know it's there, but like so many things, when you go looking for it it's virtually impossible to find. It's the kind of evidence you have to stumble upon, I suppose, which is what makes it all the more thrilling to me when I do see it. I can think of a thousand times I've felt a kind of surge and thought, "Yeah, this, THIS is the valley!" at a particular moment, but wandering around on Labor Day, in retrospect, was probably not the best way to go hunting for some. The moral of the story is to just make sure I have my camera with me more often.

The depressing part is the part that everyone talks about -- The emptiness of the valley, compared to how it was during the tech boom years. This hits home any number of ways, although not quite so hard nowadays; now that there's actually jobs, now that a company isn't closing it's doors every few days and now thats ads for auctions aren't outnumbering ads for technology companies. I haven't thought about it as much lately, since I now work in the limbo that is the peninsula -- not quite San Francisco, not quite silicon valley -- and I'm not prowling around the back streets of Sunnyvale and Mountain View, but I'm aware of it's continued blight upon the valley, if for no other reason than I can still think of more former companies than new ones.

When I went on this little expedition recently, I found myself driving on miles and miles of roads that housed nothing but empty buildings. Clean, well-kept (for now) empty buildings, with well-manicured landscaping and the windows kept freshly washed, but even more devoid of life than an occupied bland 1970's era office building is. And I wonder, is this what you see before the rotting really starts? One real estate sign after another, stretching all the way down the nice, clean street, kept in meticulous condition so as to appear ready for move-in at a moment's notice? Is this what they saw in Detroit, back when the auto companies started running for the hills? Does it always start out looking like better times are right around the corner, even when the scope of the rebound necessary to repopulate these buildings would have to eclipse even the last boom? When do the landlords stop bothering to keep up appearances, and just let the buildings start falling apart?

Then, towards the end of this holiday Monday, I drive past one of the few leased spaces, with half a dozen cars in the way-too-large parking lot, and see someone in the window huddled over their keyboard, typing as fast as they can, with only the monitor lighting the office. I'm still depressed, because the place I love is more ghosts than births. But that heartbeat's still there, I can feel it and I can see it, and I believe it's stronger than anything the valley's suffering from today. It predates many of us, and I think it's going to outlive us all, too.

Think you understand intellectual property?

2004-09-09T13:58:00.000-07:00

Think again.

Check out this excellent essay by Mark Lemley. It's got some great points, articulated well, that will likely sadly be ignored by those who most need to understand it.

Is this really interesting?

2004-09-06T13:30:00.000-07:00

The subject, no. The website, kind of, if only for the obsession it seems to evoke in this man. For fuck's sake, he takes pictures of induction loops. And he appears to be excited about them.

On a totally unrelated topic, I want one of these. I don't know what it is, but I know it would smite my enemies hardcore.

Get in, get out, a man alone.

2004-08-31T22:14:00.000-07:00

So my original idea for an entry this evening was along the lines of talking about my work, but not in the way that I think I'm going to now.

What changed my mind was a link posted off of Metafilter today, to someone's blog entry called "Tricks Of The Trade". It was snippets of interviews with various people about the kind of work they do with the subject being what you absolutely, positively should know if you're going to work in the field they do. Maybe that's not an entirely accurate description; but I can't think of anything better right now. RTFA.

There were some comments from a tech support guy, and it got me thinking -- How would I encapsulate a few of my nuggets of knowledge I've accumulated in my line of work? What do I consider the tricks of my trade?

For those who don't know me, I do unix systems administration work for internet Ops groups. That is to say, I work for companies who have (or who's clients have) an internet presence of some kind, and architect and manage the environment in which that system functions.

I always qualify my description of my job with the internet operations part, because what I do is, I believe, fundamentally different from most IT jobs. I don't work directly with 'users' in the traditional sense, most of my 'customers' are actually engineers, DBAs, and database developers who I interact with for the purposes of choreographing new or changes to existing applications that will eventually be customer-facing. I'm expected to write a decent amount of code, mostly perl and python although I've been known to get my feet wet in Java, or SQL, when the need arises.

Much of my involvement in an endeavor is at the beginning, planning, building a first incarnation, and trial running new processes we're evaulating, whether that means an updated version of the code the site runs that require a change in the way we do things, or being asked to come up with a solution to a high-level problem (The database is slow, we've optimized it all we can, we have no more money to spend, make it better.) If it sounds like I'm describing all this with a certain note of pride, I am -- I believe I'm good at what I do, and I enjoy it thoroughly. It's a frustrating and high-pressure job at times, but during the brief periods of my employment where things have been slow I've been reminded of just how much the pressure keeps me as keen and energized as I want. I love the technical parts, I love the project management parts, I even love the budgeting parts. For a mildly OCD-afflicted thirty-something, it's a great career.

So what I have I learned in the 12 or so years I've done this kind of work? Well, first and foremost, it's that I'm no expert, and I write this entry with the full knowledge that I may be putting my foot in my mouth and not even realizing it. My hope is that someday I will, maybe when I finally reach the point of knowing everything, but until then I'm acutely aware of my own shortcomings even in things I consider myself to excel in. Second, it's that my field is very immature, I'd say analogous to software engineering circa 1982. It's the source of much of my frustration, actually, that systems administration hasn't progressed as much as I'd hoped in terms of generally accepted practices; I feel like everywhere I go, and in everything I do, I'm re-inventing the wheel when I should be building on the experiences of others. But I also see this as an opportunity, one I hope to exploit (both for the good of my own development as well as for others) as my career moves forward.

That's enough about my naivete. What about what I do know? Originally this was going to be a long-winded essay on my thoughts on systems administration, but that's not really in the spirit of the original metafilter post. So instead, here's the bite-sized nuggets version that anyway is much more fun to write.

(By the way, as I type this I'm sitting in a coffee shop while this guy is wandering around the room singing in a falsetto voice for reasons unknown. I seriously hope it's not the coffee we're both drinking.)

On avoiding making mistakes:

Cut and paste is your friend... One of the first things I learned from watching a coworker in my early days of sysadminning is that when precision counts there's no substitute for c'n'p. You can think you're reading that 7 digit number properly, or that you can at a glance tell the difference between usernames 'abooker' and 'acooker', but when you're typing fast trying to fix something it's all to easy to mis-read or mis-type.

The corollary to this is: Command completion and substitution is NOT your friend. It can be a great shortcut, but when you're at the root prompt they each have their shortcomings. For command completion, it's all too easy to type the first couple characters of what you think you want, whack tab, and just give a cursory glance to what pops up. Do yourself a favor and either don't use it at all, or make sure you're on extra-high alert when you do, and read what the shell thinks you want very carefully.. For substitution/variable expansion (a la bash's !$), either don't use it at the root prompt, or use advanced shells' (like zsh) option of first showing you a fully-expanded commandline anytime you're using any substitution or variable expansion, forcing you to hit enter on the line that actually shows what's going to be done. The files you save could be your own.

One more thing: When you're doing dangerous commands, anything that could result in a large number things changing at once for example, do this: Type what you're going run, STOP, read it carefully, THEN hit enter. If you have to take your hands off the keyboard to make sure you actually pause, do so. I can't tell you how many times this has saved me.

On properly budgeting worktime:

Everything takes 3 times longer than you think it's going to. Everything.

On writing code, as a sysadmin:

Efficient and 'pretty' are almost never as important as readability and maintainability. This is true for career coders as well, but even more important for sysadmins. The chances are approaching 90% that someone is going to have to read and understand, and possibly modify, your code at some point in the future. Those of you who're fans of nested map() perl statements, I'm talking to you.

A neat trick in writing shell scripts that could possibly do dangerous things is NOT DOING THEM. Instead, just print out what they would do. The user can then use cut'n'paste to run the commands, all at once or one-by-one as they choose.

Comment your goddamn code. You're not so cool that you don't have to.

On hardware:

I have to give props to my boss, JW, for driving this point home: All hardware fails at some point. Period. There is no way around it at any price. There is no such thing as any one system that is insusceptible to dying, even incredibly expensive devices that have built into them full redundancy at every level. If something is critical, build or buy two of them, and keep their existences as disparate as possible. No shared storage, no cluster software, especially not on the same damn power strip. Build and maintain a system to failover between them as a matter of course, for example for maintenance. It's the only way I've ever seen five-9's uptime achieved.

On dealing with engineers who are also co-workers:

Almost every one thinks they're smarter than you. Pander to this and they'll do what you want them to, every time, as the more likely they are to believe in their natural superiority the more they want their ego stroked. I am allowed to say this, as I have dated engineers.

On visibility & job security:

It is the most laughable of fallacies: Keeping information to yourself will insure you both status and continued employment at your company or on an informal project. The biggest threat to a sysadmin's career, and frankly one of the most frustrating aspects of systems administration, is obscurity. If you seriously want people to recognize your greatness (perceived or real) and make sure you keep your job, document, document, document. Let the whole world know everything you're doing, how you're doing it, and why. Comment your goddamn code. Write HOWTOs on systems you've built. Put together little whitepapers about the approach you're taking to solving a problem, or building a system. You will be made fun of for your lengthy emails and apparent obsession with your own work, but you will be noticed and, as a side benefit, you'll be providing documentation to yourself for future reference.

On naming conventions:

Try just tacking on a '1' to the end of the name of any machine you set up. You never know when you're going to need to do another one, and rather than ending up with either multiple disparate names, or have to suffer through explaining 'nameserver' and 'nameserver2's origins.

Pick names that people can actually type and remember. Cool isn't as important as easily entered.

Remember, a host's purpose will likely change. Try to avoid names tied directly to the system's use, instead pick something generic for the canonical name, and use aliases for specific purposes.

On IP addressing:

For important systems such as databases, set up sub-interfaces to be used as the application's IP to disassociate them from the system's address. This makes the failover to backup systems possible, even if there aren't any backup systems yet, and migration in the future much, much easier.

On good things to learn, if you're junior:

Learn regular expressions. A (somewhat funny) comment on sysadminning is that it's all just editing textfiles and copying files. While a gross understatement, a large part of what you do can be done faster and better if you know how to cleverly and quickly manipulate textfiles.

Learn to read sniffer (tcpdump and snoop) output; yes, this means you have to learn the basics of IP. It's not that hard, and it will come up, often, in your career.

If you're working on Sparc systems, learn OpenBoot commands and syntax. The time to try and remember 'probe-scsi-all' is not at 3AM when the system is down.

If you're working on Linux systems, spend some time learning about and exploring the /proc directory. I'm amazed how many so-called Linux sysadmins I talk to who don't even know what the hell it's for, let alone have any experience in using it to their advantage.

Get comfortable enough with Perl syntax and basics to use it on the commandline. It's powerful and (surprisingly) concise to use for many things you might otherwise use awk, sed, cut, etc. for.

Finally:

Don't take yourself, or your job, too seriously. The urge to martyr yourself is strong in many sysadmins, to put in long hours plus a few extra for plenty of bemoaning their lack of a life. It's a tough job, and it makes a lot of demands on your personal time, but no one's going to be lying on their deathbed, staring up at the ceiling and thinking, "Wow, I really wish I'd spent more time at work."

I don't even know who you are anymore.

2004-08-23T22:38:00.000-07:00

I don't even know who you are anymore.,
originally uploaded by Wolftrouble.

I need a new Photoshop plugin -- "Wires-B-Gone".

Hot Dogging

2004-08-23T22:24:00.000-07:00

On a less politicized note:

I recently found out about Flickr. And boy, is it freaking cool.

First off, take the idea of a blog. Well, ok, don't take the whole idea of a blog, just take elements from it. Now, instead of text, think of things only in terms of photographs. Remix as appropriate to accomodate the new medium, add in a few amazing tools and clever concepts, and voila - You have Flickr.

Photographs, my own and others, have always been fascinating to me. They evoke memories of a time and place in a way that nothing else can; not writing, not movies. I love looking at pictures others have taken almost as much as my own, because so much is encapsulated in each one: The subject itself, the position from which the picture is taken, and maybe more important than the other two, why the person who took it wanted to share it. Photographs are intimate in ways that most people recognize, and they which ones they choose to let others witness tells as much about the person as the substance of the photograph itself does.

Flickr makes it easy (oh god, I sound like the dot commie I am) to indulge this bit of voyeurism while making it easy to engage in a little bit off exhibitionism myself. It doesn't lend itself to dumping every single picture you have (well, unless you don't have a lot, but that's not me) online due to what I think are very fair upload limits, but instead encourages sharing the slices of your life, realized through photos, that you want others to see.

They have the elements you'd expect from any modern blogging application; groups for example, with semi-forums although the text is definitely kept to a minimum (my favorite -- A recipe group, the recipes done in photo sequences with little notes for ingredients. Art meets arty food.) Easy ways to update and manage your pictures (Check out their Organizr, possibly the most useful flash application ever written). Comments, of course, with various notification methods (RSS feeds for comments on your photos -- Hell yeah.)

They even toss in a couple of other neat little things. Check out their Popuar Tags page, with text size acting as an indicator of popularity. (The tags are RSS feeds too.) Their Notes functionality is addicting, as you'll see if you play with it. Shared group let you add and remove pictures from a common pool of images. Automatic blogging of images you upload to a blog you select.

And all of this through an interface so intuitive it's almost scary. To say I'm addicted is an understatement. Which is why I have my camera with my virtually everywhere I go now; it's rekindled a somewhat flagging interest in just keeping my PowerShot handy and snapping things I see around me in everyday life. And to me, that alone makes it worth the exorbitant fee Flickr charges for their service -- Zero. Free, while in beta, and maybe afterwards too.

Now if you'll excuse me, I'm going to go take some photographs.

Roger Ebert's First Law Of Moviemaking

2004-08-23T20:19:00.000-07:00

Something to remember.

"A movie is not about what it is about, but about how it is about it."

Truer words have seldom been spoken.

Fear & Gay Marriage in DC

2004-08-22T22:26:00.000-07:00

This is belated. And it may be that it was said before, but if it was, I didn't read it. And many others may not have either, people who may need to, so with that caveat:

Does the whole flap over the gay marriage amendment thing just confuse the hell out of you? Gay or straight, are you one of those people who feel that the time spent debating (some would say posturing) the issue of gay marriage in the house and senate could have been spent better doing, well, just about anything?

First, read "Failure Is Not An Option, It's Mandatory" (I hope the article sticks around for awhile there, the only place it still shows up on the New York Times is in their archival section)This piece got me to thinking... Well, actually, it gave me hope.

The cliff notes version of the article, for those who can't RTFA, is that the gay marriage amendment issue is one that is not meant to achieve any of it's stated goals; namely, some consitutional amendment similar in spirit (if not in letter) to the Defense of Marriage Act (DOMA). It is the latest incarncation of a long-used political tactic, one most commonly used, because of it's efficacy amongst their constituents, by right-wing politicians.

It is a fight meant to be lost, for the purposes of demonstrating to a demographic (republicans, in this case) the scope of the challenge (the vast liberal conspiracy, in this case) at hand and motivating that demographic to act, by voting, or giving money, or whatever.

I'm trying to be as careful as I can here to not just peg this as a right-wing tactic, not because I have evidence of it being used by liberals and progressives, but because I claim no moral high ground for *any* politician or political group. I think it is used more often by conservatives not because I think it's reprehensible (I do), but because I think (and I'm phrasing this carefully again) that the convservative demograpic encompasses a number of groups who are under the impression, rightly or wrongly, that their ideals and values are under siege. If the 'targets' of this tactic don't already feel a certain sense of victimhood it's not going to work.

But my intent here isn't to dissect the tactic; it's to talk about what I think it represents. By any measure this was probably the most grandiose demonstration of this tactic, a hue and cry that echoed between the executive and legislative branches until for days it was deafening by media standards. And this was all before any formal debate was actually taking place! Bush demanded legislation, republican party leaders went on at length about why it was critical this fight was fought now, today, while we're supposedly in the middle of a war, and the media gobbled it up like it was that month's Lacy Peterson. All this press, all this effort, all this grandstanding, for something that was doomed from the start. On purpose.

But why?

Because the conservative leadership is, rightly or wrongly, scared shitless. As a body, they can see as much as anyone the tidal wave of loathing for Bush and his policies. The former US world record for sustained and absolute loathing of their leader, conservatives on Clinton, has been shattered faster than you can say election recount. The voices that have spoken out against the current leadership and the choruses supporting them has led the conservatives to believe that their worst fears might come true.

People, the wrong people, might actually fucking vote!.

Now I'm going to take a moment here and say that I'm not so liberal as to be breathless with excitement because I necessarily agree with this sentiment. Well, the hating Bush part I certainly do. And the belief that on some level there's likely to be a greater turnout for the elections this year due to the deepening of the split between right and left in American politics IS something I think is true. But whether or not this by itself will result in more votes for Kerry or Nader or whatever is what I'm refusing to cop to.

What I'm fascinated by is the depths to which conservatives must believe this to have undertaken a farce of this magnitude. Like any bold attack, it tells something about the attacker while it achieves it's goals; it sacrifices defense and obscurity for impact. The ones behind these machinations are aware of that before they even undertake it, and yet do it anyway.

Which is why I'm so convinced that they're really, really scared. They felt that they needed something really big, really huge, to mobilize their base; they found it in Massachusetts and in San Francisco, and they made their move.

So the question that remains, of course, is whether they'll be successful or not. That's something I refuse to try and call, especially since there's plenty of time between then and now, and plenty of time between now and the elections, for things to happen that will likely dilute the overall impact this issue has on the election.

But as a liberal, or progressive, or whatever you want to call it, I will say this: Please, people. Let's make them right to be afraid. :)

You keep eating your hand, you're not going to be hungry for lunch.

2004-08-22T17:05:00.000-07:00

So why am I doing this?

I have a LiveJournal account; I've had one for some years now. It works fine, I even had a pay one for awhile because I was happy enough with it. So what's the deal? Why here?

A couple of reasons. First, LJ has a definite vibe to it; one that's dominated heavily by teenaged-and-early-twenty-something goth girls who want you all to know just how uniquely shitting their lives are. This is not to say that *everyone* who posts to LJ is like this; I have a number of friends who do write insightful and articulate blogs, but they're still, almost to a one, just public diaries of their lives.

Commentary, voluminous and overly supportive, of course, are requisite additions to every post. As are replies to the posts, if you want anyone to continue to post, or they feel unloved. Oh yeah, and heaven forfend someone add you as a friend and you don't add them... That's just mean, man.

So this blog is my attempt to, well, escape from what I perceive as a limitation to the LJ approach to blogging. Well, ok, maybe every 'style' of blogging is limited in some manner, but for me the voice I want to have just isn't the kind that's commonly expressed there. So, I'm off to do my own thing. Here. And see how it works.

And so it begins...

2004-08-22T16:46:00.000-07:00

This should be a lot more fun than that farking Livejournal shit.