The Blues fell behind just before the whistle for the break after Bertrand Traore found the back of the net against his former club.
It was quickly 2-0 in the second-half when Anwar El Ghazi converted from the penalty spot and the game looked beyond Chelsea.
Chilwell did offer some sort of reply for the traveling Blues, who were playing in front of a reduced capacity at Villa Park, but it wasn’t enough on the day.
Chelsea did, however, still qualify for next season’s Champions League with Tottenham beating Leicester City at the King Power Stadium.
But Liverpool’s win over Crystal Palace means last season’s champions finish above Thomas Tuchel’s team this term.
Chilwell, who has been outstanding in his first season with Chelsea, has explained the high standards him and his colleagues set themselves and that was the reason they were left “angry” in the dressing room at half-time.
We were nowhere near out best but we have to take it,” the Blues full-back said.
“You’ve got to give Villa credit, first game back here with the fans they played really well – you can still hear the fans now, they were good for them today.
First-half, we weren’t good enough. We have created chances all game, not put the game to bed when we should have.
“I don’t know how many chances they’ve had but it seems like they’ve scored two of their only chances.
“They played well, we were nowhere near good enough but like I said, we’re in the Champions League so that’s the main thing.
“We were angry at ourselves [at half-time] because we should have been winning the game.
“It was all to play for today, literally, and we weren’t good enough.
“The main thing is we’re qualified, not down to ourselves, thankfully down to Spurs but we’re through and that’s the main thing.”
Chilwell also revealed that the Chelsea squad were not aware with what was going on elsewhere, in regards to the Liverpool and Leicester games.
“We didn’t know (about other results) until the last two minutes of the game, we didn’t have a clue,” the former Foxes left-back explained.
“As you can see in the second half we were fighting the whole half to try and get back into the game.
“We should have scored a number of goals in the second half, especially. We weren’t good enough overall, we should have won the game.
“It was a 38-game season, it’s been a difficult season for every team, a lot of games condensed.
“The main thing is we finished the season in the top four.”