We present experimental results on the effects of using quantum or 'truly' random numbers, as opposed to pseudorandom numbers, in a system that exhibits computational creativity (given its ability to compose original chess problems). The results indicate that using quantum random numbers too often or too seldom in the composing process does not have any positive effect on the output generated. Interestingly, there is a 'sweet spot' of using quantum random numbers 15% of the time that results in fewer statistical outliers. Overall, it would appear that there may indeed be a slight advantage to using quantum random numbers in such a system and this may also be true in other systems that exhibit computational creativity. The benefits of doing so should, however, be weighed against the overhead of obtaining quantum random numbers in contrast to a pseudorandom number generator that is likely more convenient to incorporate.