Lack of timely answer means that he could not think of graceful way to decline and was hoping you'd forget and move on to someone else. Or waiting to hear who else is attending. Just the three of you can be too much pressure for some people.
He might be thinking that Birthday Dinner is more intimate than he is interested.
Also, since it's your birthday, he might be thinking you expect him to pick up the tab.
Have a nice Birthday Dinner without him.
Maybe invite co-workers and/or neighbors out for a quick drink after your shift? 3 or 4 co-workers for 30 minutes is easy to 'endure' for those who are timid or uncertain.
Have a nice Birthday Dinner without him.
-Aude.