Tales from Tech Support

These stories are originally posted over the past decade on Reddits TalesfromTechSupport so I am copying over to Lemmy to help bring some life into this /c/

Some of you know I work for an ISP in a land down under. This incident took place a few ~~months~~ years ago when Apple ios 7.1 came out

Just got back from lunch one day and one of our layer 2 wholesalers call up to log a "fault"

Me: G'day slazer speaking
ResellerIT: Hi mate, I am wanting to log a speed fault with one of our private schools.
Me: no worries mate. What school?
ResellerIT: RegionalPrivateSchool. Your favourite one, they are only getting really high latency and between 5 to 10mb/s

damn it not those guys again

Back story. When this school went live their hardware firewall had a bug where after x amount of data was pushed, it could only do about 20mb/s in either direction.

Me: Considering previous problems with that school have they rebooted their firewall?
ResellerIT: Yes, odd thing happened though, when the firewall came up it ran at the 100mb/s for about 10 - 15 min before dropping back again.
Me: Odd, let me check it out.

I log onto the radio and see the school usage is bouncing between 80 to 100mb/s.

Me: Mate, have you looked at their current usage?
ResellerIT: No, why would I?
Me: Just look. You will work it out.
ResellerIT: Bugger me, that's quite a but of usage. I'll take it from here, sorry to call you mate.

/call

I kept the radio screen open in the background in case he called back and went back to my "active internet monitoring" AKA Reddit while listening to LRRLive on Twitch.

A few hours later I get an email from my boss asking what is happening at RegionalPrivateSchool, he got a call from the account manager. The only time the account manager gets involved is when he isn't getting in info out of his IT team (ResellerIT).

I flicked him an email back recapping my chat with ResellerIT and look at the radio it is still flatlining 80mb/s both ways.

I decided to take a look as to why a school with no students in it is still using 80% of their bandwidth in both directions. So I run the SuperSecretSexySpecial command on the radio that shows the top 20 source and destination IPs along with packets per second in real time.

When looking at the SuperSecretSexySpecial output I do some reverse look ups on the addresses. The school seemed to be pulling an arse tone of traffic from the local Akamai cache and pushing just as much up to addresses that map back to dsl services.

I start thinking, why is the school doing so much data? First thought, second Wednesday of the month Windows updates. But then I thought surely a school should run WSUS in case a bad patch comes out. As for the upload maybe some of the staff have discovered torrents aren't blocked on the firewall and let them run overnight.

I shoot my findings though to my boss, the account manager and ResellerIT. I include in the email that this is all speculation as well as some pointers for fixing it they can pass onto the schools IT guys. I get an email back from the account manager with some comments from the schools IT people saying they don't run windows, it is an Apple school and they are already running the apple version of WSUS. They also boasted that their school was one of the ipad trial schools. 1,300 students all with ipads, my second worse nightmare.

Then I remembered what my work iphone did this morning and an article I was reading at lunch, ios 7.1 for iphone, ipad and ipod came out a few days ago and we all know what happens next. The flood of app updates.

I decided to call the school and talk with their IT guys about running some tests for me. First step was to remove the apple update server network cable. When he did, the traffic dropped back from 80mb/s both ways to about 15mb/s. I asked them to plug the server back in and surely when it came back online the usage started again.

At that point I speculated that the student devices are calling back to the school to get the ios7.1 update and any apps that also require updates.

The following Friday I get an email from the account manager, thanking me for helping with the issue at the school. It turns out I was spot on with the student devices calling back to the school for app updates. After the schools IT guy reconfigured the apple server their speed tests were back up to 100mb/s both ways and sub 15ms response times.

The boss was so happy with my work he let me off early on Friday with a bottle of something special.