XML Sitemap is helping you to provide information to Google! Your website is small and ranking well is so, that's really good.
First of all, XML sitemap will not create any negative effect to your existing ranking. So, you have to go for XML Sitemap.
XML Sitemap can help you to measure total number of pages and existence of indexed pages. If your indexing value will go down so, you can drill down more to improve it.
One another thing, you can provide image information in your XML sitemap and that will help you to improve impression and clicks from Google image search.
And bigger picture is that, you'll never be small ... Your website will be big in future so, why should not start to follow standard practice of SEO to get maximum in future.
I hope, it will work for you!