Stop running in VM's and get a panel

I formally suggest that we reinstate some sort of panel such as Pterodactyl (https://pterodactyl.io)

Here's why:

• It is already integrated with TotalFreedoMod and the existing Discord bot
• The current "solution" we have is that the server is running in a VM under the Atlas infrastructure. The server is ran directly through a screen instance. This has multiple problems which I will now list:
• You can't scroll through the logs using screen. This makes looking through the logs, well, impossible. As a developer that's what I spend half of my time doing. The only solution is to manually look through the logs in the logs folder.
• No automatic restarts. Pterodactyl is able to detect when EITHER the server has stopped responding or when it gracefully stopped. The server is automatically able to restart whenever Paper hangs because that is controlled separately. The solution we now have in place will only restart the server if it is gracefully stopped (stopped using the /stop command). If it crashes, we have to resort to using some "kill" command which somehow seems like the hackiest solution ever and has weird instructions like only run it once or else it'll kill the entire VM. This isn't an issue with Pterodactyl which simply has a "Kill" button.
• A centralized place to manage backups, databases, and SFTP
• Port forwarding is done automatically. Just now, the beta server does not work. This was because of two reasons. One, the server.properties was still binding to 25565 and it still didn't work when changing it to what the port is supposed to be. I assume the port is not firewalled correctly. Pterodactyl automatically will update the server.properties file to what the correct server port should be. It will also automatically port forward the correct port through iptables, which as of now has to be done manually by Wild (and it didn't get done).

I've been told by Wild that the reason we won't use a panel is for three reasons:

1. We're running the server in a VM and Docker will not run on it
   I suggested separating TotalFreedom from Atlas' infrastructure. We've only had issues since moving to it with slow read/writes on CoreProtect and the forums being slow. Moving TF to a dedicated server has many benefits. Firstly, everything is centrally in one place, and therefore, more secure. Everything can be binded from localhost, so we don't have to go around exposing everything on the internet (regardless if there's a secure password or not). It also allows for less downtime. Everything is under a separate VM. The forums, server, and whatever else are all separate. This is, frankly, ridiculous as that means patching multiple OS's and keeping dependencies in sync. Running everything under one dedicated server allows for significantly less downtime as whenever an OS patch is needed, you only have to do it once. Finally, it gives us more flexibility on where we can get the server from. The server locations are mixed all over the place from Canada (The forums are hosted with OVH) and France (the server, which is hosted on Hetzner). We can locate a server that's demographically appropriate for TF so we don't have long ping times.
2. Docker is less efficient
   I don't think this is true at all. If there is any performance impact, I think it's very negligible. Running the servers in Docker is also much more secure as it is strictly off limits from all the other files on the system. Using Docker allows much more flexibility as well. Want to update Java? Just pull in the correct Docker image. You don't have to change the OS version of Java. The Docker images (at least mine, which are better than the default in my biased opinion) run Alpine Linux. The Docker container is extremely small at ~65MB.
3. We only use it for starting and stopping the server
   That's not true. Just look above. Pterodactyl does all of the technical stuff for us, offers much more security, and actually has a console that's scrollable. While it's true that most people used it to start / stop the server, it also allowed admins to have easy access to the console. It's possible to limit executing commands so that you can't (you can only view the console). This can be very useful for grabbing IPs for permban requests rather than downloading the logs or relying on telnet.

I understand that Wild has lots of sysadmin experience. What I've heard is that he does stuff because that's "the best way to do it". Well guess what, this is a Minecraft server, not an enterprise company. I think that ultimately it's more important to be flexible on what infrastructure you're willing to setup rather than what's technically the best on paper. It really isn't hard to secure a dedicated server. All you have to do is change the default SSH port, not use root for everything, install fail2ban, and optionally use a IP whitelist (you can only SSH in from my home IP or my VPN). Wild already has some of these securities in practice, and that's frankly all you need for a Minecraft server. (And common sense of course). This suggestion is to bring back the Pterodactyl panel, its integrations with TF, and (consider) moving all of TF to its own dedicated server and keeping it separate from Atlas. Doing this also has many benefits. That would actually allow there to be an official TotalFreedom VPN required to access the server if desired. This is a lot better than relying on the Atlas VPN where traffic can be monitored, and more importantly, you have to sign a NDA to access the VPN. I did not sign the NDA because I don't really want to be giving out my full name and home address.

TL;DR: Bring back Pterodactyl, move TF to its own server

Quote

@wild1145#1283 We've discussed this and I feel the way we have things setup now is better and more efficient.

I don't. I can't even scroll through the console using screen. This makes it impossible to get anything done efficiently because if there's a NPE it will be cut off. Not to mention that it doesn't have scroll stop so it'll jump forward on the next message.

Quote

@wild1145#1283 The temp server we ran performed well and didn't crash so there's clearly something broken when how the main server is setup and configured and if time was spent investigating that instead of arguing with me over to panel or not to panel we may have actually had a more stable serve by now already.

That's not true because I only got access to the server last night. If we had installed a panel that problem would have been sorted out and I could've gotten started on fixing things over the weekend. All you had to do was add a user account and I could've gotten started. I had to wait days for my VPN to get added to the IP whitelist, and even now that the beta server is up, it's still broken as I cannot connect to it.

Quote

@wild1145#1286 TF as a setup doesn't crash all the time, we know that and have proved that. The issue is somewhere in our configuration of it more recently.

That's a major contradiction. Above you said that it was a problem with how the server was configured and now it's an issue with the configuration of Atlas??

Quote

@wild1145#1286 but we get to expose another attack vector to the server while also adding additional overhead and taking a performance hit.

I don't consider it an attack vector at all. It's easy to keep the panel up to date as it doesn't require restarting the MC server or the VPS itself. Pterodactyl has an option to use 2FA, which I would recommend enabling. It's not insecure at all and out of its five years of existence there were, if I recall, only 3 security vulnerabilities. That's pretty damn good if you ask me.

Quote

@wild1145#1286 For the sake of admins wanting to start and stop the server, the discord bot is quicker and simpler, if admins aren't spotting the sever is down then that's not something the panel is going to fix.

But it's broken, so no, it's not. And if the bot were to ever fail like it consistently has been, it means we don't have a single point of failure. Right now we pretty much have to rely on an executive being online.

Quote

@wild1145#1286 I see no value in a panel right now and if and when that changes I would investigate what panel I feel is most suitable for what we want to do, the fact someone decided to deeply integrate tfm into pterodactyl in my view was a shirt sighted poor design decision and not something that should govern the future usage of the sever.

I don't. Pterodactyl actually worked 100% of the time. If it was a broken integration, sure, but it wasn't. Everything was actually working and now we're supposedly using an archived jAgent software you made 5 years ago which is finnicky.

Quote

@wild1145#1288 I will also post a more proper reply later. I need to go to work right now. But equally I've said what I will say to people more than once before so will just be a repeat of that.

Then you probably didn't read the thread in the first place, did you?

@SupItsDillon#1355 why not? competent people use the proper tools to get the job done

Quote

@wild1145#1367 That's great, Your opinion is noted but doesn't change my decision. It's not caused me or the hundreds of others of people issues, and ultimately if you have an NPE or similar, you should be pulling the whole log file down anyway to do a deeper dive. In addition that I don't want you or anyone else actively using the console, it's not necessary or appropriate in my view.

I think you misunderstand my word of "use" the console. I'm not talking about using it in terms of executing commands. I'm talking about reading it. I can't even do that because I can't scroll up and it will always keep going. You can't stop it like you can on Pterodactyl. There's usually no "deeper dive" for a NPE. Someone executes a command and there was a coding error and then there's a NPE.

Quote

@wild1145#1367 've actually been acting as the official owner, there have been more pressing priorities.

You spent a day made a fancy GUI for the punishments.yml file. Is that a high priority?? Because the password was changed not even an hour after it was setup and it's not even online anymore.

Quote

@wild1145#1367 also stand by my point that in theory you don't even need access to the server, having developers with access is a relatively new thing, especially with the level of access that we now grant. If you'd rather not use that access we can go back to you sending the owner plugins and config that needs changing.

Also not true. Developers have always primarily had access to the server. This is not a "new" thing at all. And by the way, you're granting this access to every executive now. So in reality MORE people have access to the server and its files now that anyone with the Atlas VPN can login to it.

Quote

@wild1145#1367 As for the beta server, the connectivity works fine, I've tested it and if you'd cared to read the message in both discord and teams it would explain how it works. The fact you've not bothered and are blaming that on me I find shows your attitude towards me pretty clearly...

I have read the messages. The port was still 25565 in the server.properties and when I changed it to what you posted on teams, it still timed out. Your assumption of me not reading those messages is incorrect.

Quote

@wild1145#1367 TF shouldn't be crashing all the time, and as I said the way it was setup on the temp server didn't. Which can only lead me to suspect it's something in how the live server Seth was running was setup. The fact it doesn't fully crash and hangs instead suggests to me something's properly fucked up, TF shouldn't do that, and we've proved already that it isn't a problem with the infrastructure provisioned for it. The fact you don't like the way I've set this up doesn't mean it's wrong.

Well also that's because everything was put on hold for a little bit while TF recovered. Now that the server is back in its "day to day" operation, people are coming back to crash the server and do their usual trolling.

Quote

@wild1145#1367 Then quite frankly that shows how little you understand cyber security and an approach to risk. It is an attack vector, same as SSH, same as SFTP, same as the bot's. They're all viable attack vectors, and it's about managing that risk in a sane and sensible way. The panel introduces a lot more risk than anything we've done before, with a net gain of zero, so it feels like a pretty poor setup...

Even if I have absolutely zero clue about anything whatsoever in cyber security, which isn't true, you haven't told me HOW it's an attack vector. How are we putting the server more at risk by using Pterodactyl? There were never any security issues within the software that anyone used to hack TF.

Quote

@wild1145#1367 Again, we've had this discussion. Patching is one part of a number of things, and doesn't take away from the fact it's a viable attack vector with no real gain over what we have.

It's not hard to do though. Patching is a pain in the ass for certain things like Windows. The panel has never broken because of an update.

Quote

@wild1145#1367 But it's not. The only reason it was broken this morning was because you fucked around on the VPS and ran things I had explicitly stated was not to be done, and in the process you not just fucked the server up but also the agent that talks to the bot's. Take some ownership for your fuckups as well if you're going to try to make me out to be the bad guy. The bot's needed some work, and that work has been done, we needed to do that work anyway so why not do it in a more rational way.

You never told me to explicitly not kill Java. I tried running the kill script and it didn't work. I had to kill Java to get it working again because it hung. This right here proves another reason why TF should be on its own server. If someone accidentally does something (that wasn't documented especially), it won't affect with your infrastructure. Also I don't really appreciate the way you handled that in the exec group chat. I would have 100% missed it if my window wasn't left open (and it's usually not). You could have simply DM'd me to not do that, but instead you dramatized it into something, but more importantly, you were indirect and rude about a genuine mistake.

Quote

@wild1145#1367 Yet the agent works perfectly until you go and fuck with the box... If I stopped the docker service on a box Ptero would fall over as well. I also didn't make the software for what it's worth, I just used it for previous projects.

I didn't "fuck with the box", I tried to get the server working after it crashed because of some hangup.

Quote

@wild1145#1367 So you're either calling our admin staff incompetent for not being able to type something in discord or suggesting Ptero's ability to start / stop a server is that much better than what I've done with Elmon to get the bot's working? Interesting, last I checked I had a solid few years of infrastructure engineering experience on you, and having talked the solution through with peers who while they acknowledge a panel is pretty, serves no additional functional purpose over what we have. What credentials exactly have you got that makes you an expert in this field?

I'm not calling anyone incompetent. Where did you pull that out from? I don't care about your experience. I frankly feel that you're being way to inflexible compared to how Seth actually listened to people. You know what's funny? How you accuse me of calling our staff members and yet you've called me incompetent twice in this post. Once with me not knowing anything about cybersecurity and now again right here because apparently you're a fucking god who's a infrastructure engineer and no one can be better than you. I don't need credentials to setup a working Minecraft server. When you step back for a second and look at it, that's ridiculous. You don't need credentials to run a Minecraft server. You don't need them to secure it either. I followed a few guides for securing my server. I bet you $100 you can't login. The IP is telesphoreo.me and the port is 2063. Username is root. There you go! You still can't login. I don't need credentials to operate a Minecraft server. What a fucking joke lmfao.

Edit:

Quote

@wild1145#1369 I've not however said that we will never have a panel, and if in the future we move in a direction where there would be a genuine and serious benefit to having one, I'm more than happy to install one, but it needs to be right for the server and it's future growth and needs to be something I've had proper time to research and test rather than throwing any old shit on the server.

Corporate lingo for "I really don't care"

Quote

@wild1145#1382 There are log files... Use them

I've already stated that they're inefficient as you have to unzip everyone and they're not easily distinguishable.

Quote

@wild1145#1382 For a personal project that took me a couple of hours to do as part of my day-job which pays for this server, quite frankly I'll do whatever the fuck I like in my own time. If you want to start paying me to own the server we can have a different conversation.

Right, then don't say that it was a waste of time having a genuine discussion about the future of the server.

Quote

@wild1145#1382 It's been something that has only existed since Seth took on ownership. Before then it was very rare for anyone other than the owner to have access. And yes, I've intentionally opened up access more within a trusted group of individuals because I feel that's a rational and sane thing to do.

Seth's been owner for almost two years. It's not a relatively new concept. It's something we've had for a long time and know works.

Quote

@wild1145#1382 Then you should have read what I said on teams more properly... Because you didn't need to make any changes.

Then why don't you explain to my why I couldn't connect?

Quote

@wild1145#1382 So you're telling me when we were averaging 30/40 players on none of them were out to crash it? I think you're talking out of your arse... The fact is something in the way the server is currently setup causes these hangs which didn't exist on the temp server. It doesn't take a genius to understand something is wrong there.

Well you're the one with the credentials and so much better than me.

Quote

@wild1145#1382 You're missing the point entirely and continue to drown on about nonsense that is utterly irrelevant. Anything installed and accessible to the public which exposes management access, is an attack vector. That's my point. That's what I've been saying since you started nagging me on this.

Not if we have 2FA. That makes it pretty much impossible to hijack a user account even in the event of a full on breach.

Quote

@wild1145#1382 Yet it's still something we would have to do that I don't now.

Well part of the job of being the owner is maintaining the server.

Quote

@wild1145#1382 Originally I was not sure who had done it, and I had no intention of naming names, but the fact you've come on here and accused me of not running things how you like it, despite it actually being you at fault changed that. I was trying to give a gentle nudge without having to take things more seriously, but if you want me to DM you individually when you fuck things up then I will, I was hoping you'd get the message without anyone getting into trouble, and to remind people not to do it.

And yet I would've missed it if I just hadn't happen to have left Discord open. At least I actually know now, but I would have not known at all unless you had pinged me. I don't tend to read group chat messages that I've missed while sleeping

Quote

@wild1145#1382 Then follow the instructions instead of thinking you know better.

I didn't know.

Quote

@wild1145#1382 Maybe you should re-read your own posts, the implications were pretty clear imo. And the fact you don't care about my experience of anyone else's is exactly the problem here, you aren't willing to listen to those who frankly know more than you do, and want to do things in a different way to what you're used to. You've tried to play yourself off here as an expert, the fact I've proved you wrong a number of times, I think you're the one putting the term incompetent in my mouth here, not me.

I'm not trying to sell myself as an expert. You ironically have the same problem of not being able to listen to others. I said in the first post right here:

Quote

@Telesphoreo#1282 I understand that Wild has lots of sysadmin experience. What I've heard is that he does stuff because that's "the best way to do it". Well guess what, this is a Minecraft server, not an enterprise company. I think that ultimately it's more important to be flexible on what infrastructure you're willing to setup rather than what's technically the best on paper.

@wild1145#1382 I never said you needed credentials, but if you know so much more than me, what's your basis and experience for that? So far it's a lot of "Because I said so" and not a lot of "And here's the proof and why". I also never said. I was a hacker and able to hack in, but I am trying to use industry best standards here and minimise risk and attack surface, which any sensible person should be doing if they want to run something which will naturally attract script kiddies and such.

I don't think you understand that it's not a pissing contest between who knows more about system administration. At the end of the day it boils down to what's best for the server. That's what my job is and why I'm making the suggestion.

Quote

@wild1145#1382 You're the one who's deciding to ignore my decision for your 5mins of fame, and feel free but it doesn't change anything, and it's just wasting both of our time and means instead of doing things to improve the server, we're having this conversation, I'll let the community judge if they think it's a valuable use of our time or not.

You're right, and I have every right to. I can make a legitimate suggestion. If I don't say anything, the answer is automatically no. If I make a deal about it, at least there's a chance.

Not to mention that I just tried to extract a log file. They all look the same and tab completion doesn't even work on it. It takes way to long to read through the logs to find an error when testing things. This is not efficient at all.