Engineering
Enhancing Countly with Dynamic Plugin Enabling and Disabling
Last updateD on
February 19, 2024
For almost a decade now, Countly has been extendable through the plugin system. But it also had its cost. In this article, we will explore how it worked before, what the limitations were, and what we did to improve the situation.
Unlike other languages (like PHP, which dynamically loads all files on each request), NodeJS loads the files once per process. And it imposes multiple restrictions when making it extendable by plugins that can be enabled or disabled at any time (not even including the development part of those plugins).
Since we wanted to get Countly off the ground with the plugin system as soon as possible, we opted for a more crude but easier approach. When you toggle any plugin, we would simply restart the NodeJS process and regenerate the whole frontend minified files again. The idea was that we would not do it that often anyway. And as usual, we were wrong.
However, multiple problems arose. Firstly, lots of people started using them as feature toggle, disabling if there were any performance or usage issues.
Then as the front end grew bigger and more complicated, regenerating became slower and slower. With the introduction of sass, it became unbearably slow.
There are also problems with browser-side caching, as it all needs to be updated on each regeneration.
And the straw that broke the camel's back was docker images, where the user could have any dynamic plugin set provided on docker image start, and as we did not know if any plugin was already installed, we had to reinstall all the plugins on every image start and regenerate all frontend minified files, which proved to be a very long booting time for docker images.
Eventually, we decided enough was enough, and something had to change. And that is what we did with our 23.03 release.
Since it is not possible to unrequire files in running the NodeJS process (well, it is possible, but it is not a clean thing to do), so with that out of the way, we knew we needed to disable the plugin while preserving everything as is (including minified frontend files). So we identified 5 main pain points in our architecture that prevent us from doing dynamic plugin toggling, and we tackled them one by one.
In the old approach, the plugin state was saved in the file. All servers/and docker images running on the same deployment would need to have the same configuration of the plugin file, which was manageable, since there are also other configurations, like connection to the database, etc.
But with a dynamic approach, we needed a way to share the plugin states across multiple servers in the same deployment because they could change run time. For that purpose, we added a plugin section in our shared config document in countly.plugins collection.
Now if the plugin is not installed, it would not appear in that document, and core would automatically run installation scripts for it. If the plugin is enabled, it would have a property with the plugin name as key and value true, indicating that. And, of course, if it is disabled, the value would be false. And all places checking for plugin states would reference that document.
Here is the code doing the initialization check:
https://github.com/Countly/countly-server/blob/23.03/plugins/pluginManager.js#L166-L180
On the Countly API backend, all plugins communicate through event-like systems. They register and dispatch events through the Countly plugin manager.
If the plugin is enabled, it is not a problem, and it will subscribe and receive events. But if the plugin is disabled, then it can subscribe to the events on process start but should not receive any event until it is enabled.
And as you can imagine, it was relatively easy to do so. We would know which plugin is listening and simply not propagate even to that plugin (the perks of going with our own event system, rather than using NodeJS built in one - more control).
Here is the code that does it in our plugin manager:
https://github.com/Countly/countly-server/blob/23.03/plugins/pluginManager.js#L649-L652
Another part that communicates with all the plugins is the backend for dashboards. Since it does not experience the heavy load that API is subjected to, we used express js there from the start, and adding plugins was just adding more middleware - which was easy. Changing this now to disabling middleware, unfortunately, was not easy.
The first thought was to add a check in each and every plugin if it should process the middleware or skip it. But it would mean modifying every single plugin, and we wanted to make it as backward compatible as possible, so this approach would not work for us.
The second way was to mangle with express js internals to skip middlewares automatically from the inside, but it would potentially break if express js changed something internally, so it was deemed to be too risky.
The approach we went with was actually to create a layer upon express js, which allowed us to wrap each added middleware in all our middleware, which would check if the plugin's middleware should be processed or not. It may not sound like an entirely clean approach, but it works and would not need any change on the plugin sides, and it would only break if Express JS changes its interface with middleware.
Here is the code where we wrap plugin middlewares in our own before passing them to express js:
https://github.com/Countly/countly-server/blob/23.03/plugins/pluginManager.js#L755-L826
Next Countly plugin interface is the background jobs that plugins can schedule and run code periodically. But since we already had a central place that managed all jobs, it was easy just to add checks before jobs were run.
Here is the code doing exactly that:
https://github.com/Countly/countly-server/blob/23.03/api/parts/jobs/manager.js#L265-L268
The last part, which is also the only one that is less backward compatible, is the browser part, which also has a state of minified files combined together. So for all plugins, there is only one single minified javascript file.
Since the minification process was very long and resource-consuming, we had to go at it from the perspective that minified files should not change. We just need to enable/disable plugin functionality there.
And there are three parts to that:
The only backward incompatible part is the browser side, and there are only two things you need to check.
The first one is adding a plugin menu/submenu/tab/etc. If your plugin permission matches your plugin name (plugin folder name), then all good, you don't have to do anything. But if they do not match, you will need to provide the pluginName property like this:
And the other part is to make sure you scope your plugin and perform operations that are outside of the plugin view (like adding links and parts for other plugins/sections) after checking if the plugin is enabled. Here is the code example of that:
Now, enabling and disabling plugins in Countly does not require any process restart or changes to the file structure.
This has been a big change with the best possible backward compatibility we could have. But the value of this is not obvious right away. But soon, more articles will come out that rely on this new feature.
Stay tuned to our blog and join our Discord community server to stay up-to-date with the latest technical product updates:
