tl;dr

Anything that depended on writing to the database between 2024/03/20 3:02AM (AWST) and 2024/03/20 8:06AM (AWST) was lost due to the server running out of storage.

Update

As of 7:47PM AWST on the 21st of March 2024, We're back!!

Currently being hosted by Contabo in Sydney, for a surprisingly cheap price :3

Just a reminder that we're running on donations now and I've paid 6mo out of pocket for Xenia (and I will continue to do so, unless I can't afford it anymore).

Explanation

The server that Xenia Bot and Dashboard run on, is also the same server that dariox.club (my Mastodon) is hosted on, and the database for that loves to chew up space.

Cause

I only found out about this issue from my DMs getting flodded with error messages saying that the database was timing out. I thought this was a connection issue, it was not. So I SSH'd into the server while on the train to work, which was when I noticed the server ran out of space and I gotta start deleting stuff so dariox.club doesn't go down.

I should've found out about the server running out of space like this. It only happened because I forgot to setup netdata when initially setting up the server.

Effect

Since the server ran out of space, MongoDB couldn't write anything to the disk, and thus connections were being dropped when writing to the database because there was no storage.

Solution

Tonight, around 10PM (AWST), Xenia will be temporarily moved to my home server until I can move it to a fresh VPS. During that peroid that Xenia is hosted on my home server, performance will be degraded, but Xenia will be functional. It is very important to note that all logging events (message/channel/user update/create/delete events) during the outage window have been lost permanently due.

Xenia will move to a VPS hosted in Sydney on 2024/03/22 at 8PM AWST. Xenia will start relying on donations for keeping the server up since I can't host it out-of-pocket for the rest of the project's life.

If you appreciate the work I've done with Xenia, then please consider donating to my ko-fi, contributing to Xenia, or start using it in your own discord servers.