Does This Look Like an AA Issue?

Sorry that I do not know what may be the source of the square grid of pixels, BUT I can help to try to find what may be wrong.

What is important is to tell what FF version you are using, if it is the same with different render resolutions.

There could be different reasons why this may be happening. What I do when something weird happens there are some things you can do

OPTION 1 - Go to the FF options and preferences and see if there is something that you have that may be the reason for this, maybe there is not antialiasing active, or is active with wrong setting, or something else

OPTION 2 - Another thing you can do is to render each one of the components of the filter and see and test if the problem appears,

1 - start testing fr om beginning of the filter components and stop when this effects you are telling may appear in the result
2 - when you find where is it, you must see what are the components involved in this result and what settings you have
3 - Try to use different settings to see if it continues or could be removed
4 - Try to use another way if possible
5 - If you can not find out yourself, Try to make it reproducible (how to get the wrong effect) and tell the steps to reach it, so we could help you here in the forum or you could upload the filter that shows the problem and see what it maybe

If the problem is inside the Math Script component, I do not have any experience in scripting, but you could try to see each of the lines of the script and see what each do, and if they could be the reason for the problem, and can test where in the script it happens. And also you can share the script here if other that know about scripting can help you

You can also put a thread in the FF scripting part of the forum for others to see with the script that gives the error

OPTION 3 - Maybe it could be a combination of things and not just only reason

OPTION 4 - If this may be a bug inside FF then you have to contact the FF team and be able that they can reproduce and see wh ere it fails.

Wish you best and that you can find what is wrong and fix it