I don't like much smoke so I use very little wood and have tried no wood at all. I like it, but I can certainly tell the difference if I put a little wood in the fire. For me, a little hickory makes ribs taste really good and you don't get that from just charcoal. FWIW, I use lump. I would think you'd get even less flavor from a briquette, but I've never tried it so I'm not certain.
Like most things in BBQ, try it and see if you like it. Unlike using too much wood, you won't ruin anything using too little or none at all. In the end, it's personal preference.