Solution:
Read documentation about add_filter function.
The second parameter is:
$function_to_add (callable) (Required) The callback to be run when the filter is applied.
It should be string with function name. In your case following code should work:
function return_custom_error() {
$error = '<strong>ERROR:</strong> Code was invalid.';
return $error;
}
add_filter( 'login_errors', 'return_custom_error' );