Combine offline with online information to generate robot actions