I'm currently running a test to compare the performance times of our new and old machines. Working with a substantial dataset—over 50,000 records—that leans toward the new machine being faster. Previously, I conducted a similar test but lacked clarity on certain technical aspects. Consequently, I opted for the Mann Whitney U test, yielding a binary outcome for the hypothesis.
Having dedicated time to study and better understand the fundamentals, I'm revisiting the test using the same dataset from a different period. My aim this time is to execute a more robust analysis by factoring in the percentage improvement of the new machine over the old one, considering the data's scale and characteristics. I'm considering employing the Two-Sample t-test from the Pingouin library, known for providing confidence intervals in its output and being suitable for large sample sizes under the Central Limit Theorem.
I've come across information suggesting that while t-tests are suitable for smaller sample sizes, like 30 or fewer, z-tests are preferred for larger samples, like 30 or more. However, I'm uncertain if this applies to the ttest function in Pingouin for such a sizable sample, because the library does not have a ztest function. How to know if I can confidently utilize the ttest from Pingouin for this substantial sample size? Any advice, suggestions, recommendations will be really helpful.
