droste effect spiral does not produce rounded spirals

do you have an example of what you get and you do not like?

And do you have an example of what you mean with rounded spirals?

If what you want is that the sides are not straight and linear, you need to load an image that has transparent background with the shape that you want

For example in the presets you find this



and if you want to remove the rectangular tiling you need to remove the gray background, so instead of loading the full lifesaver image, you need to load the same built in image WITHOUT the background

So you get this shown here below

Just an added note: For the built-in preview images you need to go under File/Built-In Preview Images/ and click on Use Transparency at the bottom. It is also possible to load a grayscale image as a Selection, also under File. You would do this if you're using Filter Forge in stand-alone mode and need to use a filter that requires a selection. Just remember that the image you are editing and the grayscale image you use as the Selection need to have the same resolution and aspect ratio.