Downtime Resolved

October 11th, 2024


Author: Kate Ward

Welp, I found the issue that was causing all the downtime and intermittent issues with Xenia.

Initially I thought that the issue was blocking the Discord.Net events, and not creating threads inside of an event handler, but that was only part of the issue. Xenia got hard rate limited, where it couldn't even sustain a WebSocket connection to discord.

To find the source of this issue, I added some better error handling (by implementing Sentry support), and some extra logging to figure out what the hell is going on. I added logging on every single event that is handled from discord in DiscordCacheService, and when I brought up the bot on my home network, I instantly found the issue.

So I checked the log files, and it seemed like someone thought it would be smart to have a fuck-ton of users in their server, and update their nicknames every ~15s to have the conversion rate of multiple crypto currencies (and real currencies).

Because of this, a new event saying that a Guild Member got updated would come in every ~300ms, and because of how Xenia is designed, it was hammering the Discord API quite a lot for information about that user, and hammering the Database a lot for previous history of that user (which at the moment is unavoidable).

screenshot of log files, showing the spam issues

What's stupid is that an easy fix for this is to ignore that guild for that specific event, but I'll implement a proper way to resolve this in the future (by blacklisting events/modules for specific guilds).

I've temporarily moved Xenia Bot to my home server, so there will be inactivity on the VPS's that I use for hosting Xenia so the rate limit can be lifted. Because of this, the Dashboard will be unavailable while Xenia Bot is hosted on my home server, which should be a few days (1 week maximum).

Thank you all for understanding. <3


<< go back