It absolutely makes a difference.
A polarizer filter is used in many different ways. First, it is used to cut reflections from glass surfaces like windows and display counters. While looking through the lens (with the polarizer on obviously), rotate the polarizer and you will find a spot there the reflections mostly disappear and you can see right through the glass.
Another use is to affect skies and clouds. As you rotate the polarizer, you will see the sky darken and the clouds become more contrasty. Its an artistic thing. Too much of it and it looks wrong. You need to view the image through the lens (ie on your device) and then rotate the polarizer until you see the effect and amount you desire in your images. Then get airborn and snap a few to make sure you got what you want. The affect varies with your angle to the sun.
What a polarizer does is filter out (mostly) light that is not aligned to the filter's axis. Think of it this way, if you took a piece of paper and scribbled back and forth, up and down, diagnals all over it then you cut a slit in another piece of paper and placed it on top of the one you scribbled on, you would only see the scribbles that were aligned to the slit. As you rotate the slit, other scribbles become visible. Thats a basic description on what the polarizer does. Light is like the scribbles and the polarizer is the slit. As you rotate it, it excludes some like while letting others through.