THPP Wiki Situation

[EXSariel]

So the Wiki has been getting a lot of attention, yes? Very good. It's really nice that it's starting to become a much more proper wiki and that it might soon start becoming a more comprehensive source of Touhou Puppet Play information. It's really nice for the metagame players as well, as it speeds up a lot of theorizing, crafting and other calculations that particularly helps in team building. It's getting big, really.

Too big.

I got a notification from my webhost that the SQL database it's using is becoming overloaded. This can't be helped, Mediawiki automatically trims and prunes it's own database usage and keeps it slim, but the wiki has gotten so big to the point to where that doesn't matter. The wiki hasn't gotten to the size limit yet, but at this pace, it'll hit it somewhat soon. At least soon enough to the point to where I need to consider alternative sources for an SQL database. I can't increase the size limit at this point, because it's already gotten really large, weighing in at 225 MB and climbing. Preferably, I would like an SQL server that is very quick to access and allows remote connections without slowing it down. However, I am also not made of money. All of this stuff is coming out of my pockets already as it is, and I'm not sure if I can sustain another pocket drainer, at least not right away.

I'd like suggestions on where I could host an SQL server though that is fast, reliable and will hold massive amounts of data first. That is my primary concern. I will do something about the price after that. If it's too expensive, I'll go onto the next one until I find some sort of good balance. I also am not going to tell anyone to stop updating or using the Wiki and I would like everyone to continue working on it as they have been, this is simply a warning of what's to come. I'll tell all of you when the size limit is getting dangerous close to being reached, at which point I will soon lock the wiki from editing in an attempt to keep it's size from ballooning too hard.

I'm sorry it's come to this, but this is what I have to tell you guys.

[Naï]

Wait... what. Seriously? I had honestly never imagined this scenario...



I have no idea about the SQL server and all (what does SQL even mean?), but would trying to trim the codes help?
What takes up more space? Is it text, is it images, or what?

Would taking down all of the current images, rehosting them on an external database (Photobucket) and then linking to them help, or would that be even heavier than the alternative? Also, is it viable coding-wise, or does the wiki code only recognize images hosted in its own database?

Would changing the Touhoudex from the a || b || c format to the old
| a
| b
| c
one help? After all, even if the new format is much easier to read and work with, I think changing it added about 9k characters.



EDIT: can we (well, mainly Rhet :<) at least continue the puppet page optimization project? Will the database hold? It was the most pressing issue we had, and it's what helps with navigation the most; that, and it's probably almost finished.

EDIT2: does this only concern the wiki, or is making forum posts trouble for the database, too?...

[shai_LP]

I have no idea about the SQL server and all (what does SQL even mean?), but would trying to trim the codes help?
What takes up more space? Is it text, is it images, or what?

Structured Query Language, the language of pulling data from Structured Databases with Queries

http://www.thesitewizard.com/faqs/what-is-mysql-database.shtml
http://simple.wikipedia.org/wiki/SQL
http://simple.wikipedia.org/wiki/Database

Inform yourself. _{might be overs-implication but...}

the tl;dr version of EX's announcement the database is running out of space. Or more accurately, we are running out of space for the Server that he's paid for out of his own pocket.

I doubt it's the images really, It just that you're adding so many STRING AND STRINGS AND STRINGS AND STRINGS AND STRINGS of data it just keeps filling tables and well...

The fact its already filled 255MB is a testament to how many lines of text you've guys put in the wiki

self-source: someone that works tables everyday, but knows little-to-nothing of DB structure


I've been meaning to say that I've been trying to get in contact with someone who has dealt with a similar  issue in the past [Webmaster of Soku Lobby], but he's giving me the cold-shoulder at the moment.

[Naï]

I understood his post for the most part. I don't know what's SQL, but I do have a general idea of how databases work, anyone would.

(meaning what you just told me was mostly repetition of what I had already understood, other than slapping me with wiki pages :< )

Hence,

I have no idea about the SQL server and all (what does SQL even mean?), but would trying to trim the codes help?
What takes up more space? Is it text, is it images, or what?

Would taking down all of the current images, rehosting them on an external database (Photobucket) and then linking to them help, or would that be even heavier than the alternative? Also, is it viable coding-wise, or does the wiki code only recognize images hosted in its own database?

Would changing the Touhoudex from the a || b || c format to the old
| a
| b
| c
one help? After all, even if the new format is much easier to read and work with, I think changing it added about 9k characters.

Which consists of me asking whether trimming down said so many STRING AND STRINGS AND STRINGS AND STRINGS AND STRINGS of data would help at all. That includes the images, and yet that doesn't mean it's limited to it.



Also,

I've been meaning to say that I've been trying to get in contact with someone who has dealt with a similar  issue in the past [Webmaster of Soku Lobby], but he's giving me the cold-shoulder at the moment.

I'm not Exs, but if you do manage to contact him, I think everyone will be thankful. He'd probably give helpful feedback if he's willing to.

[shai_LP]

trimming and re-hosting images elsewhere would be marginal at best

I doubt re-ordering the text would help much either.


[inspoiler]I'd like to get out in the open that there is bad blood between this webmaster and me but I'm hoping he'll be professional about this.[/inspoiler]

[Naï]

I understand that it wouldn't help much, but even then, if it means we can complete the puppet page project without Exs needing to find a new database, I'd be glad to do it.

The point isn't the reordering in itself; rather, it's just that, when I did the reorganization (not the reordering), that added exactly 9,592 characters to the script. The Touhoudex script is many pages long - there's a reason why the script reorganization took me more work than the switching itself.

[shai_LP]

I can't stop you from trying I guess

go for it?

[inspoiler]This came as a bit of a blow to me because I doubt we'll have space for ball sprites until we get a new server but oh well, still working on animating them on-and-off[/inspoiler]

[chaosakita]

This is such bad news. I would hate to see the wiki have to go away. I hope a problem can be found soon.

[Naï]

Actually, Rhet already completed the puppet page project while I wasn't looking, LOL.
(rhet y u so efficient, wat r u doin, stahp)

Either way, I'll just re-change the Touhoudex for now, that should at least secure some extra space in case of an emergency.

[EXSariel]

I've already mentioned this in the opening post, but I'll make it clear again; I want everyone who was working on the wiki to continue working on it as normal, and most importantly, not to freak out. The worst that will happen is that the wiki will have to locked for a while. Nothing will be lost, nothing will change otherwise, it'll be fine. I don't want anyone to go out of their way to keep the wiki from surpassing the size limit because that will inevitably have to be reversed when we make the transition to a bigger SQL database, something which will have to happen at some point.

All I really need is an SQL database server, everything on the HTTP side of things is handled by the webserver (which actually rests on a different space then the SQL database) so that doesn't really need to be fiddled with. The server I pay for now has amazing HTTP service but the SQL service is rather limited, which is why I need to look for an SQL server that won't limit me in such a restrictive way.

I'll continue looking into a better SQL server myself in the time being.

[shai_LP]

https://www.linode.com/

The soku lobby rests on a expensive as fuck dedicated server but I was recommended this.

Hope it helps!

[EXSariel]

Well, using some black magic voodoo, I managed to trim a good deal of fat from the wiki's SQL database and I brought it out of the danger zone... for now. I have no idea how long it will last, since the data I trimmed is meant to be regenerated by mediawiki when needed, and I don't know when exactly it will regenerate absolutely everything I deleted (and I'm pretty sure eventually it will) but for a while, hopefully, everything should be fine and you can use and edit the wiki at your leisure again. I'll unsticky this topic for now and bring it back up when the situation arises once more.