You fell victim to one of the classic blunders.
this post was submitted on 18 Nov 2024
165 points (97.7% liked)
Sysadmin
7682 readers
243 users here now
A community dedicated to the profession of IT Systems Administration
No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world
founded 1 year ago
MODERATORS
This is one reason I love test driven development. Once you have the solution, there's somehow never time to write tests. If you write the tests first, then find the solution, there is somehow time to write them.
The motto I got from working at an ISP was "There is nothing more permanent then a temporary fix."
Similarly, I’ve heard, “nothing more permanent than a 3/4 solution”, i.e. it works well enough that fixing other things always ends up taking priority.
You should read the Red hat article about decommissioning a data center. Over the period of 20 years there is a crazy amount of works well enough and no one documented it.
It will be in production for 10 years
In 25 years someone's going to discover a dust covered raspberry pi hosting the poc service in the back of a network cabinet and unplug it, bringing down the rest of production
Young people will be hired and spend months to learn "how it was done in the old days" (it was just old stuff piled upon other old stuff).
Don't let the perfect be the enemy of the good. That said, it's vanishingly rare that I've regretted designing something for long term use rather than just hacking it together.
Taking the extra time to document stuff, to add guard rails/error handling, to make a piece of script easier to re-use, to actually plan something rather than building it as you go... almost always a good use of time.
Amen, wizardbeard.
Always implement like you're going to turn it on tomorrow
Same with “lift-and-shift now and fix after migration” . Everyone knows nothing will be fixed and the same problems will exist, just in the new environment.
Several years ago I had a client who needed to be able to receive files from a client of theirs. This was before Dropbox and the like, so I just went in and set up a quick and dirty FTP server. Worked fine, client got their files, all was good with the world.
A couple weeks later: "hey, we have another client that needs to send us files, can you set them up on the FTP?" Sure, no problem.
Repeat for about the next six months. The thing turned into this huge glob of a mess that miraculously enough still worked.
Finally, I call up my contact there and tell him if he wants this to be a permanent solution we should go in and set it up right, and to do that I'm going to need a bunch of information from him, who needs access to what, who should not be able to access what, etc. He says fine, why don't you come on over and we'll hash it all out.
I go on-site, we end up having a 2-hour meeting (billed, of course) where we go over all this stuff, plenty of notes are taken, decisions are made. We wrap up, I tell him all I need now is the list of users and their access needs and I can clean the whole thing up.
I never got the list. The thing just continued to grow and grow into some kind of unearthly abomination. Fortunately, I left that job before the thing imploded completely. Someone else got to untangle that mess.
But somewhere inside of you, it hurts, doesn't it? One more pile of infrastructure and code that ain't right, and it could have been different.
And of course, the next guy that comes in thinks the mess is your fault as opposed to an expression of organisational dysfunction.
This is why I put an end date on anything temporary. If you want it longer than that, it'll have to be fully built out.
For me it is always the emergency script for fix a problem with production that ends up being forgotten about.
My boss convinced me to make a pricing calculator for our contracts using Excel with macros. I pushed back and said we need to use Power Apps, CRM, or anything other than Excel. He said it would just be a POC/wireframe we'd use to build a more permanent solution with later.
Well 5 years later, there are like 2 dozen versions floating around with different macros, pricing, etc. And of course I'm the one who gets messaged every time some makes a change and breaks the damn thing, or some sales person is convinced it's not calculating correctly. I fucking HATE that Excel sheet.
If it's on Excel365, you can edit permissions for specific cell ranges. That way your colleagues can only give input but they can't change formulas or fixed pricing.
This is why i always push back that if you want it, I'm going to implement it properly, or at least in a way I can change easily later. I'm not going to make my own life difficult on your whim.
Web developers are the worst. Especially the ones that call themselves full stack. </sysadmin rant>
What's poc?
"Proof of concept". Only meant to demonstrate something, not actually to be used.
You know things are going to get messy when the PoC are demonstrating. That's when the cops turn off their body cameras.
And this is why bug bounty hunters get paid.
Even as a PoC, you'll want a strong foundation to build on. Even as a temporary solution, you want to know the fix won't fail the next time its under pressure. Even under time constraints, you want a solution that you are confident works when it goes live.