It isn't wrong to use question marks in URLs.
However - as Thomas says, short and descriptive URLs describe what is on the page, whereas the query parameters you use currently don't. "Keyword present in the page's URL" was also believed to be an important ranking factor by the participants in Moz's Ranking Factors survey earlier this year: https://moz.com/search-ranking-factors/survey
By mirroring URLs, will each product then exist at 2 URLs? E.g.:
example.com/product.php?id=3624
and
example.com/category/product
If so, you don't want to do that as it creates duplicate content. Using canonical tags to fix the duplication doesn't solve the problem completely.
If you change URLs, you should 301 redirect the old to the new, so
example.com/product.php?id=3624
would 301 redirect to
example.com/category/product
Just in case you aren't aware - a "301" is a permanent redirect, so when using a permanent redirect, even if other websites have linked to example.com/product.php?id=3624 - the browser will automatically redirect to the new URL, passing most of the authority built up from the links you mentioned.
Whether I'd advise you to change the URLs of existing products depends on whether you can implement the redirects. In the short-term you'll lose a slight bit of link authority through the redirect, but in the longer-term, the frendlier URLs should benefit your website.
Also, there's always a risk when making large scale changes to URL structure - you need to make sure customers can still navigate the site without issues, and search engine crawlers understand the changes you've made - 301 redirects are best-practice to ensure this.
Ideally, when URLs are changed, all internal links should be updated.