Through Keypad Programming Menu 9-Installation Programming, 07-Globals System Definitions, G34 should be set to G34: 9 = 115200 Serial Port 0 Baud Rate. What version of ElkRP2 Programming Software are you running? If not running ElkRP2 version 2.0.38 I would suggest updating. Have you verified using the “Special M1XEP 2.0.34 Updater Utility” https://www.elkproducts.com/elk-m1xep-and-windows-compatibility/ the M1XEP is running Firmware 2.0.34 or higher?
For the Serial Connection under Windows, Device Manager, Ports (COM & LPT) what COM Port number has Windows assigned? In ElkRP2 under Setup, Communication, Direct Connections make sure the same Port Number is listed and you’ve selected the port to use.