You are at netAirspace : Forum : NOTAMs - Site Related Forums : About netAirspace

Hi guys, since 07 o'clock (CET) is the site very slow sometimes not conactable.

Always the same problem. After 7 (CET) it´s impossible to open a photo for the next hours.

I'm investigating. Your patience is appreciated.

It seems that the server is choking on disk access, and that the nightly backup may be the trigger for that.

I'm not sure of the exact failure sequence, but the end result is a long queue of database queries, all waiting for disk access. Once one struggles, it's likely that the next will get backed up waiting, and the failure cascades from there.

A database restart "fixes" the problem by wiping out all those backed-up queries... until it happens again.

I have a temporary and hacky fix in place but am still digging.

Thank you so much to bring the site back on track. It´s very helpful.

Just for info, the same problem sice 07.00. Time now 13:23 CET.

Yep, the fix didn't take. Sorry about that.

It'll probably break again tomorrow morning because I'm out tonight and not working on a live server with beer in me.

A full optimisation of the backup itself is a scary job, looks like I need to eke out enough performance improvements from the rest of the site to prevent it all from piling up and falling over itself.

Still, for something that I said would probably be OK up to about 100k photos, it's not doing terribly

Just now I got following message, may be it could help you.

General error

SQL ERROR [mysqli]

Server shutdown in progress [1053]

Nevertheless thank you so much for trying to fix the problem.

I put a very hacky "fix" in place last night, and the site seems OK this morning. A proper solution will be needed.

Since noon the site is slow again, especially by opening of photos or "My Photos".

Same, its as though my browser is hanging when loading a page as there is no visible operation.

Database restart fixed it again. Don't know what happened at noon.

Survived today so far

If it survives the nightly jobs at 0700 tomorrow, but falls over later in the day, I think we'll have another piece of the puzzle. On Monday, Wednesday, and Friday, a script deletes all the copies of photos (thumbnails, full-sized, AvB-sized, etc., but obviously not the originals) in order to keep the disk space usage manageable. However, that obviously involves a lot of disk I/O. It's entirely possible that that's what killed the site on Monday, and might kill it tomorrow.

Quite why this is a problem now, I have no idea. It may simply be that there are so many photos that, even with lower traffic than we're used to (not least because the site keeps falling over), so many get viewed, appear in searches, etc., that deleting them takes us over some limit, especially when the backup is running. Paradoxically, the answer to this might be to run the job more often, giving it less to do each time.

Kind of feeling my way through this one, as you can tell. But we'll get there.

Just for info. Last days was everything fine, but today the same problem to open images.