Welcome to the CozyNet Blog!
IT horror stories: The REAL print nightmare!
One day on the Krapban dashboard, I’m given a task to disable SNMP version 1 and version 2 on all printers across the network to meet the security compliance requirements of some security goon that ran a Nessus Tenable vulnerability scan. We have almost thirty printers because the office workflow is a dated model from the 90’s in present decade that’s paper dependent.
Since I’m totally a smart and handsome guy and know better than to just blindly follow orders without always questioning them first, I therefor submit a change request to one of our MSP’s responsible for managing some of our backend services. One of those services is a resource and usage statistics collector (FMAudit), to which I request that they enable SNMPv3 support along with the ID and password that I made up. The system is necessary for automating the orders of toner and paper with our print supply vendor. The only reason I know about this is because I use to manage it myself before it was one day pulled from out of my hands and outsourced, which was not my decision. I'll get into this topic in another post someday.
After completing my request, I move on to disabling SNMPv1 and v2 on the printers and enabling v3. A few days later I reach back out and verify that it’s all working and that the service is pulling logs. Task complete!
Or was it?
About one month later, “REEEEEEEEEE!!!!!1111 Why are we running out of toner!? WE NEED TONER! WE CAN’T PRINT! HELP HELP HELP!”
Apparently something changed, because it was working before and now it stopped. Since I disabled v1 and v2, FMAudit is no longer pulling stats and not able to relay usage reports to the print supply vendor. Usually when something goes wrong, the supply vendor sends an email. But they were sending their messages to the account of an employee that retired during the Coof. Nobody sees their emails and so it goes unnoticed. As a result, no toner and no paper was being delivered.
I scramble with snmpwalk to verify that I configured the printers correctly for SNMPv3, then I reached back out to the service provder to let them know that something isn’t right with FMAudit and to check it out pronto. They come back to me in the ticket comments stating that they no longer handle FMAudit anymore and to contact this other service provider that does, then canceled my incident!
I couldn’t believe it at first, but then queue the hazy flash backs of groggy early morning coffee and boring ass monotone meetings in Teams from a week or so ago. “Blah blah moving servers blah blah new MSP blah.” Right… I almost forgot. The company contracted and onboarded yet another service provider and is abandoning ship with the current one. Just great.
I find out that at the time of my old request from a month ago, they did handle FMAudit but then about a week later the virtual server it was hosted on was moved to the new service provider along with a whole slew of other things. After sleuthing thru a few hundred emails of auto generated Jira spam, I found their point of contact. I reach out to the new MSP and they have no idea what FMAudit was and thought I was looking for someone. They didn't even know their ass from the hole in the ground!
“This is Rafeem Ajhadeet. I do not understand you sir. Should I redeem?”
Aren’t these guys supposed to have a KB from our company or something? Also I’m not sure if these guys are literally in another country or just H-1B’s. If they really are in another country, I find it highly suspicious if not possibly down right illegal for an American bank and insurance company to be outsourcing its technical support in this manner. How is any of this secure? I wouldn't know because I'm not the one that onboards them, that's up to the wrecking crew. I must of called them at a bad time because now they suddenly know things.
Anyhow, after some finagling with GoToAssist and talking with one of their techs over the phone, I remote into their excruciatingly slow craptop. Some things are in a different language too… I instruct him over the phone to open the vSphere client to our VM running the FMAudit service which is back in the US and also excruciatingly slow. A few remote desktop layers latter, I open a browser to FMAudits web interface that’s running on localhost and find that the whole VM itself was obviously a gawd dang snapshot! Someone made a snapshot of the VM captured prior to the completion of my original SNMPv3 change request, and then restored it to that state after the migration! So I add the SNMPv3 credentials back in, verify that it works, and call it a day.
But then things took an even worse turn…
To preface this, it was someones big idea about a year ago to let the departments handle their own toner and paper supply. They would fetch what they needed and replace it themselves. The cool old boomer that retired recently and myself were against it, but we were overruled. Lemme tell you, it’s not a good idea bros; the users get pissy with the printers and break things in their tantrums with it all the time. However, this incident was a first for me.
So on the day we finally received a freight train load of toner and paper that was on backorder due to the delays caused by the FMAudit conundrum, desperate departments besieged shipping and receiving in a mad dash. They snatched up whatever they could get their grubby little hands on! I didn’t see it for myself, but I like to imagine a Black Friday stampede rolling thru and people having it out and brawling over toner.
Obviously they grabbed up the wrong toner, mixing the generic cartridges with the metered toner cartridges. While they may be the same exact shape and fit into the same printer models, there is a difference, and they won’t work depending on which payment model the printer is licensed for. They were trained on this detail, but didn’t bother to check during their frenzy; thus spawning numerous tickets of “toner not working” and users hoarding toner until they could find one that did work.
What an absolute nightmare.
What really grinds my gears in this situation was how the tosspot that oversaw the server migration was aware of the issues being reported and neither said nor did anything about it, save for the one global announcement that some servers were going down for maintenance on a weekend. We were wrestling with DNS and route issues for weeks, and he just let it go! Him and his all star group of halfwits are what I've come to refer to as the wrecking crew, because that's about all they're good for... That and yapping all day long in Teams meetings.
So this concludes my printer nightmare IT horror story. I hope you enjoyed it! I plan to write some more in the future, like the time when I worked IT for a hospital and watched a crazy homeless guy in the ER rip a computer wall mount from out of the wall and threw it across the room, lodging it into the other wall!
Thanks for reading my blog!
Date: 2023-02-01
Back to top!
Comments:
-
Feb 2, 2023
Permalink
Reply
-
Feb 2, 2023
Permalink
Reply