Well that was only part of the problem, I just realized I did not have the bottom of the door seated correctly. While the top of the door was fine, all looked good, when I just went to add charcoal, I noticed a large gap, so it was getting air that I did not know about.
Lessons learned.