Sure, you can totally use infinitives after "давайте" (plural, formal) or "давай" (singular, informal). However, following all the grammar rules you should use infinitives only when talking about the actions in the present tense. Usually it's a specific offer/suggestion, which is not discussed, but rather reminded of.
Here are some examples:
давайте смотреть фильм - let's watch the movie (you already have a movie you are about to watch, so you just want to make everyone else watch, turn their attention to it, remind them of it)
давай пить чай - let's drink some tea (you already have your tea ready and just want to turn somebody's attention to it)
It is also very common to put verbs in the future tense with the prefix "по-" after "давайте". In this case you would be talking about unspecified future events, and your "давайте" would sound just like an offer, a suggestion, which can be accepted or denied:
Давайте пообедаем - let's have lunch (for example, you are in a park and got hungry). Compare to: "давайте обедать" - let's have lunch (in this case, you are already at the table, or have just finished cooking, and urging people to eat, give them a "start eating" command)