A numerical wave tank (NWT) based on Computational Fluid Dynamics (CFD) provides a useful tool for the analysis of offshore renewable energy (ORE) systems, such as wave energy converters (WECs). NWT experiments, of WEC operation, rely on accurate wave generation and absorption at the NWT boundaries. To tackle this problem, different methodologies, termed as numerical wave makers (NWMs), are available. The performance of these NWMs are often sensitive to properties of the experiment being performed, such as the frequency spectrum of the input sea state, the CFD solver used and/or the internal settings of the NWM. This paper discusses the desired NWM capabilities, for effective analysis of ocean wave energy systems, and then proposes a set of test cases to assess these capabilities for a given NWM. Results are presented for a sample NWM, the OpenFOAM toolbox OLAFOAM, and demonstrate the sensitivity of the NWM to the desired wave conditions and the global solver settings. The assessment methodologies introduced in this paper, and demonstrated for a single type of NWM, lay the groundwork for future evaluation and comparison of different NWM types, enabling appropriate NWM selection for NWT analysis of WECs.