Every time I attempt a little gardening without wearing proper gardening clothes, I regret it. Gardening in casual shoes sounds easy enough, until you step in mud! The clean up is quite labour intensive. A pair of easy-on wellies that can be easily hosed down or cleaned in a utility sink would be a better choice.
Gardening without gloves is another time waster- thorns, dirt under nails, broken nails, skin injuries, etc. Obtain plenty of gardening gloves and a nice pair of rose gloves (gauntlet style) and you will be able to tend your garden in comfort and without dirtying your hands. I wash my garden gloves after each use and have a nice basket of them, as well as a pair of secateurs inside the front door. It encourages me to get out and trim a little each day.