I used to work at a vision store and helped fit people with their contact lenses. When I first started doing this, I was surprised at the number of people who didn't clean their contact lenses on a regular basis.
They would just throw them in the case with some saline solution and put them in their eyes the next day. The longer I worked there, I wasn't so surprised by it because so many people treated their lenses that way.
Your eyes will tell you when you are pushing the limits, but I like to take better care of my lenses and my eyes than that. Not only does regular cleaning of your contact lenses help your vision, but it is also so much better for the long term health of your eyes.