IT story? or HR story?

There’s been a lot of talk about the downfall of journalspace.com and their incredibly shortsited (sic) backup policy.

“It was the guy handling the IT (and, yes, the same guy who I caught stealing from the company, and who did a slash-and-burn on some servers on his way out) who made the choice to rely on RAID as the only backup mechanism for the SQL server. He had set up automated backups for the HTTP server which contains the PHP code, but, inscrutibly, had no backup system in place for the SQL data.”
Journal Space Blog

This isn’t a data backup strategy – it’s a strategy for failing over to another machine when things (temporarily) go awry.  The very nature of your setup is that anything (bad) that happens will immediately be applied to both sets of your data.  I would literally be afraid to work in an environment where an innocent accident (bug) could completely wipe out *all* of the data you have ever worked with!  Add on top of that that this data is not “theirs” – they are (were) in the business of hosting other people’s data… just completely unprofessional.

There are undeniably serious IT lessons to be learned her.  Not only do you need to have multiple copies of your data (which technically they did).  The copies need to be stored in disconnected systems/media and geographically disparate locations.

But you also need to protect yourself from sabotage – and your first line of defense there is clearly HR.

“A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can’t rule out the possibility of additional sabotage.”

journalspace.com

Sabotage is certainly preventable but can be much more difficult.  If you are going to have one “guy handling IT”, it’s quite conceivable that he’ll have the authority to destroy all the backups and clear data from your servers.  A person hired into that position must not only be up to the task technically but absolutely trustworthy.  You must be willing to say: I would bet my company that <<person we’re hiring>> would never do something like that.  If you’re not willing to say that, you need to pass on hiring him — or (more likely, since you may not know the person that well) provide a backup structure that you completely understand and cannot  be circumvented by one person.

Leave a Comment

Filed under HR

MacGreerver strikes again!

We’ve been having some issues lately with the heat (or lack there of) in our office.  With WI temps dipping down low it was time to get a space heater in there to supplement.  Unfortunately, the fan on the stupid thing didn’t work causing it not only to not spit out the heat like it should, but also shut itself down when it overheats itself.

<Cue the theme music>

It may not dazzle like the speaker phone, but using a couple garbage cans, a oscilating fan we were using this summer, and the cover of a (now defunct) Mac Mini, MacGreerver was able to provide fan power to the unit and restore circulation to the hands of grateful developers.

episode-ii

Leave a Comment

Filed under Uncategorized

We’re the greatest!

So, I’m using some holiday to catch up on my RSS feeds…

Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn’t seem to be the case in most types of work. When I worked in fast food, we didn’t prefer the busy times. And when I used to mow lawns, I definitely didn’t prefer it when the grass was long after a week of rain.

Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.

- Paul Graham, The Other Half of “Artists Ship”

I’ve heard exactly this type of sentiment many times but I’ve never really been able to buy in.

Don’t get me wrong: it’s 100% correct that the best programmers prefer to have an incredible product.  And since I don’t feel like presenting some corny argument about “working hard” vs. “working smart”, I’ll concede that the best programmers feel that if they’re not working hard… they’re not making the product as “incredible” as it could possibly be – so yeah, they prefer that too.

What I don’t agree with is this belief that programming is the place where (unlike any other profession) you’ll find that “preferring hard work” coincides with being “the best”.

When I worked in fast food, we didn’t prefer the busy times.

Coincidentally, Paul Graham is not running a successful fast food empire.  Why?  When presented with one of the “hard problems” in the fast food industry – How do you increase quality of service and customer throughput while maintaining the sanity of minimum wage workers during the time of day when the most people are looking to hand out their money?… he didn’t like it.

No shame in it. Instead, he looked for another industry.  A place where he could spend all night (long after his 8 hr day was long done) thinking about some problem he left unfinished – and come away happier for it.

If you won’t mow the lawn unless it’s 70F and sunny, your landscaping career will not end at the top.

Leave a Comment

Filed under Deep Thoughts

Speaker Phone

We recently went out and got a shiny new $160 speaker phone at work.  Long story short – it just didn’t work that well.  The sound quality wasn’t that great when we were varying distances from the thing and it just didn’t cut it… it got returned.

Today, I mentioned to Justin (one of the guys in the office) that I was about to start a conference call with one of our guys in a different office.  He strolled over to my desk with the new “speaker phone” that he had soldered together in his basement:

speaker phone

The binaural headphones worked AWESOME.  Great sound quality and the price is right too!

2 Comments

Filed under Uncategorized

And here I am *thinking* (like a sucker)

You may not know it by looking at me but I actually do a lot of thinking.  You know… about stuff.  Those days may be coming to a close though as Jim F (or “Thunderbird”, as the kids are calling him) has pointed me back to stackoverflow.com.

I had visited the site after the initial round of hype but never really spent any time there.  Until today when I posted a question and was rewarded with an answer <30 min later.  Very very nice.

1 Comment

Filed under Uncategorized

Session Expiration

Session expiration is simple, right?  If someone returns to their browser after a long idle period and clicks a link – you make em reauthenticate and then send them on their way.  In fact, it should be even easier than normal because the app I’m working on doesn’t involve users writing a books worth of data into a form that could be lost — so it’s acceptable to “lose” their progress in almost all situations.  So, why didn’t it take a half-day to implement?

To Expire or Not To Expire
Why do you really want to “expire sessions”?  Concerned that someone’s not “releasing” the tickets they were looking at back into the pool?  Worried about the size of your session data store (this uses db backed sessions, btw)?  Our only real concern is security, so we don’t really need to “expire” data in the session or any such fun – we just want them to reauthenticate to prove it’s not the janitor happening upon their (authenticated) browser, after they’ve left for the day.

The Simple Case
The core problem is pretty simple: When the app sees a request it needs to check if it’s from a session that’s been used recently.  Rails makes this pretty easy by allowing you to easily add a filter to check on each request received.  Something like this in application.rb:

before_filter :idle_check

And write that idle_check method to involve some sort of check along these lines:

session.model.updated_at < (Time.now.gmtime – MAX_IDLE_TIME)

If it’s been too long – record where they were headed – clear their credentials – and send em over to the login page with a nice message about being timed out.  Since we’re not actually going to mess with the session data on the db, we can even store that “attempted target” right in their session.

FYI: I’ve positioned that filter after all the authentication done on each request.  By the time this thing is checking your session status, I want to be sure you’re a valid user and worth actually checking for.

A Few Exceptions
There are some methods that we just don’t want to have this protection.  Like, if someone comes back to their browser after they’ve been away for an hour and clicks “logout”… are you really going to bother them with “please sign back in so we can log you out”?  Of course, there are more… login, forgot password requests, and others in that vein.  Most of these are pretty simply accomplished with :except on the before_filter.

before_filter :idle_check, :except => [ :login, :logout ]

We also have some exceptions to the exceptions (just to keep things interesting).  For example, I have an exception setup for :initial_confirmation.  When that method is called via get, it starts the initial confirmation process — so it’s in my :except list.  But when it’s called with a post, it’s really going to change data, so it “manually” (as opposed to using the filter) invokes the idle check.

Just a (tiny bit) off topic.  This talk of using a method as both a “before_filter” and “directly” calling it from another method reminds me of something I bumped into recently… filters no longer halt on returning false… just something that I didn’t realize but bumped into here.

Post Requests
It wouldn’t necessarily be a bad thing to go ahead with the post that they were attempting when they timed out.  In many systems it may even be vital,  but as I said earlier – here we have the luxury of not worrying about it.  So, after reauthenticating we simply do a get for the URI that they were looking for.  Here’s where you start to see some payoff for properly “protecting” your data-changing methods.  Protect your methods with something like this and you won’t end up redirecting to something that changes data.

verify :method => :post, :o nly => :update_profile, :redirect_to => {:action => :profile}

Ajaxy Requests
So, what about when an ajax request is the first thing you see after the timeout period?  If it’s a request that normally updates a portion of the page… you don’t want the login box appearing in some cludgy piece of the screen.  If it’s a request that normally doesn’t display any result to the user… they’re not going to see anything (and maybe assume it worked?).

To handle this, I’ve baked in a bit of smarts to determine if we’re dealing with an ajax request.  We look for the presence of “x-requested-with” in the request headers.  That header isn’t magically added with any use of XmlHttpRequest (so it may not work for you) but it’s sorta a defacto standard at this point, obeyed by several of the major js frameworks (including Prototype, which we’re relying on for most of our ajax work).  So, something like this to decide if it’s an ajax request:

request.headers["x-requested-with"] && request.headers["x-requested-with"] == “XMLHttpRequest”

Ok, so we know it’s an ajax request and we know they’ve been idle.  Now what?  Well, we send them a 403.  “403″, you say?  Yeah, I know, it’s not really perfect for what’s actually happening here but the fact of the matter here is that we don’t want to rely on the (varied) way that browsers may treat 401s.  So, 403 it is.  Now what?  A (prototype) handler for 403s, of course:

Ajax.Responders.register({
on403: function(t){
window.location = “/users/reauth?wanted=” + escape(window.location.href);
}
});

That guy will see 403s and redirect them to a reauthentication method that basically gets some nice messaging ready and gives them the login page (remembering where they were when it went down).  If that’s not working for you, make sure your version of Rails has a patch like this one.

It’s not 100% smooth, you might notice.  It doesn’t have the info to actually carry out their ajax request afterwards.  It doesn’t even have the smarts to know if the current window location is actually a good place to send them to, but (at least so far) I’m pretty happy with where people end up.  It may not be perfect, but you started it by ignoring me for half an hour!

Here’s another handy resource for this strategy.

iFrames
Right about now I’m starting to realize that there are a lot of “weird” situations that I didn’t think of before I dove into this little project…

Ok, so here we have a request that appears to the server to be a pretty normal get request (it could be something more complicated to but let’s ignore that for now).  In this app, it’s largely where we’re using Lightboxes.  The issue is that if I reply back with my normal reaction of “reauthenticate”, they’re going to render that in the iframe.  At first, I thought maybe that wasn’t so bad but after seeing a variety of shapes, sizes, and behaviors…. I wanted something a bit better.  So, instead I’m flagging my “iframe bound” actions (took a little refactoring to make sure they weren’t used in “normal” situations too).  Something like this:

skip_before_filter :idle_check, :o nly => :edit_multiple
before_filter :idle_check_for_iframe, :o nly => :edit_multiple

So, basically saying: that method shouldn’t be subjected to the normal idle check… but it should be subjected to the special one.  What does the special one do differently?  Instead of asking them to redirect to the login page, it sends them a little bit of script to have the “whole window” head over to the login page.

render :text => “<script type=’text/javascript’>self.top.location=’#{ uri_for_idle_redirect }’;</script>”

Our app has the luxury of requiring javascript, so we can get away with this.  Also, keep in mind that if they maliciously disabled js at just the right moment to avoid the redirection… they would have already had their credentials cleared, so it’s not like they’d be able to skip the reauth.  They’d just kinda have a dead app on their hands.

Uploading
There’s some hardcore uploading going on through this app.  Some of these things are expected to take longer than our allowable idle time (or at the very least someone may want to kick one off and come back later).  The messaging needs to be nice – don’t have them return to a “your session expired” message and not know how things really went.

This actually wasn’t too bad at all.  Our uploads (and the related status updates) are directed at a server outside of our Rails app, so they’re automatically not subject to this idle stuff.  That is… until the upload completes and the app gets pinged back with the success/failure message.  Easy enough though.  When those requests come in, they set an extra parm indicating the result of the action — we just pass that right along to the reauth page so that when the guy gets back, he’ll see: Your session timed out but your upload finished without a hitch.

Downloading
Downloads aren’t a problem, right?  As long as the session is alright at the beginning of the request (and we’re only checking on new requests), things should be fine.  Well, yeah sorta but like you just said – “as long as the session is alright at the beginning”.  What should the reaction be if you come back from lunch and immediately click a download link?  “You should reauthenticate and get the file” is a fairly reasonable answer, but I just don’t like it.  I think in that situation it’s more than reasonable to say: “whoa man.  First let’s reauthenticate and then let’s figure out if you’re serious about that file download”.  To accomplish this I’ve put in a couple “interceptors” in our app.

One deals with downloads from our external file servers and one to deal with downloads from the local machine.  Something like this:

def send_data_with_idle_check(data, options)
if session[:first_after_reauth]
forward_file_request
else
send_data_without_idle_check(data, options)
end

end
alias_method_chain :send_data, :idle_check

This is just saying: after they reauthenticate, if the first thing we’re dealing with is a send_data request – tell em to chill and they can re-request if they want.  Of course, if it’s not the first thing after reauthentication: go nuts.  What does forward_file_request do?  It sends them to a destination they specified (via request parm) would be good for such a situation — or, if they didn’t give us extra info, it takes it’s best guess at where would be a not-so-inconvenient spot.

So… that’s it?  Well, that’s at least the high notes.  Hope some of this helps you out.

Leave a Comment

Filed under Uncategorized

Ruby String Concatenation

Ran in to a bug tonight.  See it?

def display_value
display_value = notes.blank? ? "untitled" : notes
display_value << " (#{ shortcut })" unless shortcut.blank?
display_value
end

Need a hint?  The symptom was the +notes+ field being unexpectedly changed.

+display_value+ is pointed at +notes+. “<<” then goes ahead and changes that value (which both attributes are pointed at), thus both are effectively changed.

So, while it’s nice to avoid the extra String creation with “<<” where possible – sometimes, you’ve just gotta go with the “+=” to clone, then modify.


a and b ending up the same:
>> a = "a"
=> "a"
>> b = a
=> "a"
>> b << "asomething"
>> a.object_id == b.object_id
=> true


and different:
>> a = "a"
=> "a"
>> b = a
=> "a"
>> b += "something"
=> "asomething"
>> a.object_id == b.object_id
=> false

Leave a Comment

Filed under rails, Ruby