Author Topic: Job opportunity in IT Systems Operations - yay or nay?  (Read 2291 times)

Gone_Hiking

  • Stubble
  • **
  • Posts: 233
  • Location: Arizona
Job opportunity in IT Systems Operations - yay or nay?
« on: October 04, 2018, 10:34:32 PM »
Dear Mustachian Systems Administrators, managers and all who want to chime in,

I come to you for a sage advice about a career opportunity that has presented itself.

My last ten years were spent in IT, first as a software developer and more recently as a project manager.  I have an opportunity to be promoted to management from my current position.  The team I would manage would be Systems Operations: server administration and desktop support.   I am undecided whether to go for it or not.

I hesitate  for two reasons.   First one is the career track: from what I have observed in the past, it seems to me that managers in this area rise through the ranks.  That's not what I did.   I have been assisting with management of the team since the last manager was fired and, at the beginning, I knew very little about details of the group's everyday tasks.  The learning curve is steep, even for a former coder comfortable with sending jobs to server clusters and command line input.

The second reason for hesitation is risk.  The last manager was fired after about a year and half.  The next-to-last manager lasted three years before termination.  I could be setting up for failure and termination in the same fashion.   I've experienced a layoff less than two years ago and changed jobs twice since; I have no appetite for starting over anytime soon.  I'm also nearing 50, and ageism in tech is still a thing.

On the other hand, the team really seems to need a manager rather than a promoted tech lead.   Their level of expertise is not in question; ability to prioritize, however, is not their strong suit.  There is also an opportunity to learn more about system security, penetration testing, virtualization, hybrid cloud, firewalls and switches, and discovering new uses for shell scripting, which I had not done for the last several years.

With all this, here is my question to you:  is taking the job the logical next step or fool's errand?

Uturn

  • Pencil Stache
  • ****
  • Posts: 889
  • Age: 54
  • Location: Raleigh, NC
Re: Job opportunity in IT Systems Operations - yay or nay?
« Reply #1 on: October 05, 2018, 06:17:33 AM »
I've been in IT Ops for 25 years, mostly networking and the last 7 year in security.  Yeah, I know, but I did not embrace the MMM lifestyle very early.  Anyhow, the vast majority of the managers that I have had throughout my career really did not understand my job.  You are about to become one of those.  The good ones realized that they did not fully understand, the bad ones thought they did.  Find trusted advisors in each of your functional areas.  Make sure you get input from each team before making a decision on a specific technology.  Everything is intertwined, and you have to take into account how one system reacts to another.

Here is an example from just this week.  New storage.  At first thought, you need to talk to your systems (server/storage) folks.  But let's look at how each team views storage.  The systems folks are probably thinking about size, speed, and backups.  The network folks are thinking about jumbo frames, 10G or 40G interfaces, vlan isolation, WAN op to the DR site.  The security folks are thinking about controlling access, patch management, and data encryption, both in flight and at rest. 

tyler2016

  • Stubble
  • **
  • Posts: 125
    • Tyler's Guides
Re: Job opportunity in IT Systems Operations - yay or nay?
« Reply #2 on: October 05, 2018, 07:16:55 AM »
It will be a learning experience. Building software and maintaining the systems they interact with are completely different mindsets.  Here are some pointers from a seasoned sysad.

Here is an example I have seen first hand. A web developer using a python framework with root access (I inherited this host) to a dev machine pip installed a bunch of stuff that overwrote python libraries installed by the OS package manager. The ones from the OS were older, but tested and known to work with critical system management tools. I have seen package names changes between versions that cause things to break. Going outside of the OS packages also presents a patching issue, which is critical to security. I (the sysad) wasn't too thrilled when I discovered this. The developer just wanted the latest version of the libraries.

Users can overreact to down time.

You will have to work with other organizations that use horrid systems.

Paid support can be unhelpful. A fancy support with a good sounding response time can have things in their contract like "we don't guarantee we can fix your problem, but we will try"

Big well known companies can produce some really crappy products. I have used one that didn't work as advertised. Their support was pretty much trying to turn my org into QC and test engineers.

Software Licensing can be a pain.

Good security requires an effort from everyone.

Dogmatic views about commercial vs free software can bite you in the butt. Don't be afraid of free software, but there are cases where there is no viable free or open source tool.

Avoid vendor lock in.

Hire people that understand concepts rather than products. An admin that can get into the weeds with LDAP can easily learn AD, OpenLDAP, or DSEE. Someone who understands AD but doesn't know much about LDAP will have a harder time adjusting to a change.

Make sure you get the source code from any development that is outsourced.

Communication throughout your department is critical. Make sure everyone is aware of changes before implementing them. E.g. an ACL on a router could break something, everyone being aware will make troubleshooting easier and reduce your down time.

Understand the XY problem. Users are very prone to it.

Don't be user hostile and make good service and response time a priority. The last thing you want is shadow IT.

Good communication skills and a backbone will help tremendously. Telling users they can't  do something because it is best practice is good for security won't get as good of a response as explaining why. I had a PM (contractor) that wanted me to put a system into production on the internet with the default admin credentials. It handled PII. I told him not until it is changed. He tried to scare me by saying he would tell the CEO I am delaying the project. A C level walked by and over heard. I explained that it is common for hackers to run automated scanners looking for vulnerable software and default accounts. Needless to say, the CEO backed me on it when they realizes the inevitable disaster if I said yes.

Sometimes non technical solutions are the best. We have a product that requires an expensive plugin for single sign on. Users complained about multiple passwords. I suggested setting the requirements and change interval to be the same and then just setting them to the same password. Same end result, much cheaper, kept system complexity down.

Temporary solutions frequently become permanent.

I could keep going, but this should help if you venture into operations.


reeshau

  • Magnum Stache
  • ******
  • Posts: 2510
  • Location: Houston, TX
  • Former locations: Detroit, Indianapolis, Dublin
Re: Job opportunity in IT Systems Operations - yay or nay?
« Reply #3 on: October 05, 2018, 08:22:16 AM »
With all this, here is my question to you:  is taking the job the logical next step or fool's errand?

What I don't see from your description is where you want your career to go.  As you say, this opportunity fell into your lap.  If you want to get into IT management / leadership, I think a rotation into operations is a great way to round out your dev and project experience.  If you want to stay technical, or stay involved in projects / development, then you are moving away from that.  (not irrevocably, but it will be harder to turn back to that)  This is the first question, and must be answered thinking long term.  (You could answer that you don't know--in this case, trying it out may help to clarify it for you!)

In terms of filling a role with a poor promotion-to-firing ratio, I would ask two things.  First, do you trust that position's manager?  It could be a sign of a bad boss.  If they are trustworthy, I would sit down with them and ask them what led to the last two firings.  Then, don't do that.  Even better, find what is the *opposite* of that behavior, and try that.

More generally, @Uturn and @tyler2016 make good points:  you will be an outsider coming into the group, and they will know it.  (might you have some disappointed candidates in the population?)  Be humble, be supportive, and learn from them.  These are the ways to gain their trust.  And look to see if there *is* someone in the group who shows leadership skills that could eventually be your successor; if you can groom them early on, they may prove to be a valuable first mate to help your development, too.

Gone_Hiking

  • Stubble
  • **
  • Posts: 233
  • Location: Arizona
Re: Job opportunity in IT Systems Operations - yay or nay?
« Reply #4 on: October 19, 2018, 10:20:48 PM »
Thank you, @reeshau, @tyler2016, and @Uturn, for your thoughtful responses.  My observations mirror your feedback. 

The opportunity, indeed, fell into my lap; the previous manager was fired one day in July.  Before that happened, I was at first asked to assist the team in better organization as a project manager, then asked whether I knew possible replacement, then informed that the decision to fire the person was made, and then that I would be managing that group in the interim.  Two days after the manager was fired, my own boss went on vacation, leaving me to the sharks without protective lifeboat of his experience, and with the eyes of senior management firmly fixed on me and my new charges.

As predicted, resistance arose almost immediately from one member of the team; everyone else was too scared to question the decision.   Two months later, the dissatisfied person gave me administrative access to Lansweeper, while mumbling something about how he trusted that I knew when to stop.  Our internal customers have acquired a habit of coming first to me with any requests for assistance.  The number of Microsoft Office licenses is now known and the information about licenses and how to work with Microsoft Volume Licensing has been published internally; looks like we are lowering silo walls and making areas of expertise redundant.  I purposely seek knowledge and learning opportunities from the group.  I occasionally step in to work on support tickets, which gives me insight into the effort that is required in this area and possible headcount increase in that area.  I'm finding that backbone helps when dealing with team hermits who bark at users who happen to wander into their den.  I feel like I'm making progress; too slow for my satisfaction, but progress nevertheless.

I feel that getting experience in systems operations will give me a more rounded expertise that I will be able to use when an opportunity to become a head of  IT comes along.  I can also parlay the knowledge into speaking to local startup and small business groups about practical aspects of managing and securing business information.  I am planning to keep my PMP certification; a lot of things in IT seem to be projects and knowing how to organize complex effort will pay dividends.  Project management will remain a useful fallback should things not work out on the managerial path.

To address risks, I have held conversations with business leaders where reasons for dismissal of two previous managers were discussed.  These people indicated that they would support me stepping up to management; I fully trust one of them because the person is always blunt and tells it like it is.  My own boss, whom I judge to be trustworthy, believes that the only way I can be fired is if the top management changes and he loses his job, too.  Due to the way our budget works, the business leadership does not want to hire a manager for this group until early next year.

After considering risks and opportunities, and analyzing your insightful feedback, I am prepared to ask for promotion to the managerial position within two or three weeks.

 

Wow, a phone plan for fifteen bucks!