The Money Mustache Community

Learning, Sharing, and Teaching => Ask a Mustachian => Topic started by: trompowsky87 on March 28, 2019, 07:51:24 AM

Title: IT job with No On-call
Post by: trompowsky87 on March 28, 2019, 07:51:24 AM
Hey All,

I was curious if anyone had any suggestions on how to steer an IT career to a place where you are no longer on-call. Would switching to being a developer be the best bet? I look forward to your thoughts and suggestions.
Title: Re: IT job with No On-call
Post by: Tester on March 28, 2019, 08:04:09 AM
Depends on the place, you might ger oncall as a develper too.
Look into tester jobs - some get oncall but much less than the developers.
Title: Re: IT job with No On-call
Post by: Lady SA on March 28, 2019, 09:47:35 AM
at most companies I'm familiar with, software engineers/developers (especially backend services engineers) will rotate to be on call /the point of contact for any emergencies/problems that occur. My DH is "on call" a few days each month, and that was the same at the last 2 companies he worked at as well. However, as long as you have your laptop and an internet connection, you can just hop on, fix the problem, and then get back to what you were doing. You don't have to drive anywhere.

Title: Re: IT job with No On-call
Post by: NextTime on March 28, 2019, 09:57:01 AM
I'm a contractor on a federal government contract and the developers on my team are pretty much never on call.
The project lead, who also programs, is pretty much on call all the time.
As an analyst, I am never on call per say, however I do sometimes (quite often at the moment) have to validate production moves over the weekend when they occur. Though that is only usually a couple of hours of work.
But the developers are pretty solid 9to5ers with overtime available (straight pay) at times.

Of course this is just my specific contract, so YMMV.
Title: Re: IT job with No On-call
Post by: des999 on March 28, 2019, 11:10:16 AM
I moved to the architecture field a few years ago, more money and no oncall.  I recommend it, if that is something you feel like you could excel in and/or enjoy.  I happen to enjoy it more so than coding all day.

This is coming from a megacorp, not a silicon valley developer, as I think they are more highly compensated. 
Title: Re: IT job with No On-call
Post by: Louisville on March 28, 2019, 12:10:30 PM
Infrastructure jobs tend to have more on call situations than development jobs.
But it varies.
Once you get a resume built up, you could just not take any jobs that require you be on call. But you may have to suck it up initially.
Title: Re: IT job with No On-call
Post by: nessness on March 28, 2019, 02:27:52 PM
My husband works in IT security and is never on call. You could also look for government jobs - I'm a Fed and even our help desk people just work regular hours; if someone had a problem over the weekend, too bad (with rare exceptions).
Title: Re: IT job with No On-call
Post by: Dave1442397 on March 29, 2019, 09:00:18 AM
I'm a developer and haven't been on-call for the past 18 years.

My last job was at a small company where nothing much ran outside working hours.

My current job is at a large corporation where they have dedicated support teams who handle everything. If they find an issue that needs a code change, they open a change request and it goes from there.
Title: Re: IT job with No On-call
Post by: ChickenStash on March 30, 2019, 08:59:59 AM
The only advice I can give is to get as far away from operations roles as possible, particularly if you're working for a 24x7 company. BI/data science are less likely to be called. IT security (away from operations) is also unlikely to have regular on-call.

The worst place to be is working in infrastructure, particularly lower management since they wind up being called for anything significant. In my experience, server and network engineers get called for virtually every issue and wind up playing the role of incident manager for anything major that crosses teams. It's worse in organizations without strong application support.
Title: Re: IT job with No On-call
Post by: BicycleB on March 30, 2019, 09:19:26 AM
As a side note, a friend of mine working on a Computer Science degree was studying Linux the other day. He reported to me his newfound belief that SysOp people divide into two camps. The majority go through waves of actually responding on call to incidents because they don't automate their system maintenance, and automate responses that can be automated. He reported that automating the automatable aspects of the job would prevent the need for personal response to the vast majority of incidents, but that most SysOps don't develop that level of skill, much of which he claimed was simply learning a large-ish set of command line commands in Linux. Maybe in the short term you can learn skills that minimize your time spent?

Title: Re: IT job with No On-call
Post by: ChickenStash on March 30, 2019, 12:16:28 PM
As a side note, a friend of mine working on a Computer Science degree was studying Linux the other day. He reported to me his newfound belief that SysOp people divide into two camps. The majority go through waves of actually responding on call to incidents because they don't automate their system maintenance, and automate responses that can be automated. He reported that automating the automatable aspects of the job would prevent the need for personal response to the vast majority of incidents, but that most SysOps don't develop that level of skill, much of which he claimed was simply learning a large-ish set of command line commands in Linux. Maybe in the short term you can learn skills that minimize your time spent?

Your friend is right that there's time to be saved in automating alerting and making scripts to respond proactively to those situation. From my experience, though, calls of that nature in a well-established IT shop represent the minority of call-outs for the reason your friend gives - they are automated out of existence. That doesn't make the frustration of being on call stop, though. For example, I'm an AIX/linux admin in an almost-mega-corp and here's just two of the more time consuming things I've been called out to handle in the last month or so of on-call:

1) I was called at 11pm on a Saturday by the team that handles the automated file transfers saying that they were getting SMB (Windows file sharing/CIFS) permissions errors from one of my AIX (IBM's flavor of UNIX) servers. It took over two hours to explain to them that error was impossible since we don't install SMB on that platform and they need to look at the rest workflow for the real issue. Two hours... on a weekend... in the middle of the night. They finally scrolled down on the WebUI and saw that the other server in the workflow was a Windows file server and, after calling a Windows admin discovered someone changed the perms on a directory they used earlier that day. I lost a perfectly good night's sleep for absolutely no good reason.

2) I was called because someone rebooted a server and it wouldn't come back up. When I checked, I found a slew of odd, disk-related errors on the console. I did some fancy disk swapping and config changes to get the box to at least boot but without any of its data disks (just the OS). After poking around I found someone had deleted the software package that allows the OS to actually read/understand the data disks (LVM for those that are familiar). What in the actual...? Turns out a team of app support folks that we were forced to give root access to the servers decided to do some cleanup, removing software they didn't recognize, and that one was removed. "After the cleanup, the system acted very strange so I thought a reboot might fix it," the culprit explained after the fix. There went almost half a day because folks that shouldn't have admin perms on a serer decided to do things they didn't understand.

Sure, we still get calls for things like file systems unexpectedly running out of space or a machine not coming back up after routine patching but those are usually easy to fix and we add a permanent fix if we can but those are the easy ones that usually don't consume much time outside of work. The killers are the issues caused by things you either don't see coming or that you can't control, like other people.
 
Title: Re: IT job with No On-call
Post by: Tester on March 30, 2019, 02:00:51 PM
Even if the oncall is quiet the problem is that you have to be available.
Unplanned hike with no internet access? No.
Drink a lot? Or even more than a little? NO.
Home alone with significant other??? No, you don't want the pager then.
Did not sleep for 4 days because the kid is sick? Too bad, you need to solve this problem now, you are oncall.
Title: Re: IT job with No On-call
Post by: BicycleB on March 30, 2019, 08:03:14 PM
As a side note, a friend of mine working on a Computer Science degree was studying Linux the other day. He reported to me his newfound belief that SysOp people divide into two camps. The majority go through waves of actually responding on call to incidents because they don't automate their system maintenance, and automate responses that can be automated. He reported that automating the automatable aspects of the job would prevent the need for personal response to the vast majority of incidents, but that most SysOps don't develop that level of skill, much of which he claimed was simply learning a large-ish set of command line commands in Linux. Maybe in the short term you can learn skills that minimize your time spent?

Your friend is right that there's time to be saved in automating alerting and making scripts to respond proactively to those situation. From my experience, though, calls of that nature in a well-established IT shop represent the minority of call-outs for the reason your friend gives - they are automated out of existence. That doesn't make the frustration of being on call stop, though. For example, I'm an AIX/linux admin in an almost-mega-corp and here's just two of the more time consuming things I've been called out to handle in the last month or so of on-call:

1) I was called at 11pm on a Saturday by the team that handles the automated file transfers saying that they were getting SMB (Windows file sharing/CIFS) permissions errors from one of my AIX (IBM's flavor of UNIX) servers. It took over two hours to explain to them that error was impossible since we don't install SMB on that platform and they need to look at the rest workflow for the real issue. Two hours... on a weekend... in the middle of the night. They finally scrolled down on the WebUI and saw that the other server in the workflow was a Windows file server and, after calling a Windows admin discovered someone changed the perms on a directory they used earlier that day. I lost a perfectly good night's sleep for absolutely no good reason.

2) I was called because someone rebooted a server and it wouldn't come back up. When I checked, I found a slew of odd, disk-related errors on the console. I did some fancy disk swapping and config changes to get the box to at least boot but without any of its data disks (just the OS). After poking around I found someone had deleted the software package that allows the OS to actually read/understand the data disks (LVM for those that are familiar). What in the actual...? Turns out a team of app support folks that we were forced to give root access to the servers decided to do some cleanup, removing software they didn't recognize, and that one was removed. "After the cleanup, the system acted very strange so I thought a reboot might fix it," the culprit explained after the fix. There went almost half a day because folks that shouldn't have admin perms on a serer decided to do things they didn't understand.

Sure, we still get calls for things like file systems unexpectedly running out of space or a machine not coming back up after routine patching but those are usually easy to fix and we add a permanent fix if we can but those are the easy ones that usually don't consume much time outside of work. The killers are the issues caused by things you either don't see coming or that you can't control, like other people.

Great answer, @ChickenStash. I'll pass these examples to my friend for his understanding.
Title: Re: IT job with No On-call
Post by: Schaefer Light on April 01, 2019, 11:56:46 AM
Even if the oncall is quiet the problem is that you have to be available.
Unplanned hike with no internet access? No.
Drink a lot? Or even more than a little? NO.
Home alone with significant other??? No, you don't want the pager then.
Did not sleep for 4 days because the kid is sick? Too bad, you need to solve this problem now, you are oncall.
Yep.  That is the whole problem with on-call.  You'll inevitably get called at some of the worst possible times.  Christmas, Thanksgiving, while you're at a concert or ballgame, the list goes on.
Title: Re: IT job with No On-call
Post by: formerlydivorcedmom on April 02, 2019, 11:03:53 AM
A lot of this depends on your industry, your particular company, and your boss.

I worked at a megacorp as a developer, and I was on call for 4 months out of the year.  I supported the custom application the company used to do R&D budgeting, and I got called almost every night during that time period.  For years.  Then we got a new director.  He found out that 2 of us were doing that level of support and put his foot down with the business.  My on-call time went down drastically.  (*I will note that I was paid quite handsomely at this job, both in money and in delicious cake that showed up on my desk the day after a particularly heinous night.)

Then I worked at a hospital as a report writer.  My group was also responsible for supporting the application on the server, so I still had to be on-call for a week at a time.

I'm back in my original industry now, at a much smaller company, and I'm only sort of "on call".  If the reporting server goes down, they'll try to contact me, but if they can't reach me we do a collective shrug.  I'm working to convince them to let me train a guy in our overseas office to do basic support, so they don't even have to try to call me.  I have a work phone but I almost never remember to carry it after hours, and my boss is aware of that and okay with it.
Title: Re: IT job with No On-call
Post by: jeromedawg on April 02, 2019, 05:19:49 PM
Software Testing/QA is one area with low or no on-call duty. Only time I have to be on the phone past normal working hours is when we're doing a big release/deployment and they want us there to spot-check post-deployment. The way my group does things is a little out of the ordinary even regarding the scope of work too, since QA counterparts generally should never be involved in post-deployment testing. But we wear that hat anyway sort of as concession to the fact that every little bit helps, and it doesn't hurt our job security either. It's really very little to ask of us anyway.
The only time this was really a nightmare was when we had a lead developer with narcissistic personality disorder and who thought he was always right and everyone else had to be shown their place. He created and caused more problems for the group than not being there... the most recent release that he was involved in was a complete disaster that kept nearly all of us on multiple calls for hours and working days and nights beyond what it should have taken. He was making last minute changes and not telling anybody what he was doing -- just flying solo and trying to fix his mistake but was making it worse the entire time. He had operated this way leading up to this but it was the culmination of all his poor decision-making that really caused things to implode. My manager had enough of it and he was slowly pushed out of the group - somehow he ended up in someone else's organization at the same corporation... I have no clue how he managed to land another position but I guess he pulled some strings and had some last minute ditch options. I feel bad for that other team, but I'm so glad that he's out of our group now because even if we have another "long night" it'll be way better without that guy being on the call. LOL

I digress. IT Security *can* be one area with low on-call but when you are put on the spot it can be one of the more stressful calls because when (not if) there is an incident, you will likely have multiple higher-ups breathing down your neck for immediate RCA and resolution. It does depend on what IT Security position you're in too - if working in a SOC/NOC, etc that'll definitely be the case. But when (not if) they escalate it to you, you will feel that pressure. If you like that kind of pressure, by all means. But I doubt that's the case based on your original question.
Title: Re: IT job with No On-call
Post by: secondchance on April 05, 2019, 09:05:54 PM
Come to front end (what I apparently do now)!

It's boring as all get out, but no one ever woke me up from a dead sleep to move something two pixels to the left.