You will have a hole, but it isn't that noticeable.
If you do a lot of driving on gravel roads, you will get more rock chips. I drive on gravel roads just about everyday, so from time to time I clean the rockers off real well and hit them with a couple coats of clear.
Even with more rock chips, you probably won't get any more rust than you would with them on. I noticed that when I took mine off, they had actually dug into the paint and there was a little bit of surface rust.