Are you trying to do server authentication or client authentication?
If you're attempting server authentication then you don't need to call InitClientCertificates(), and you don't need to add the X-Roku-Reserved-Dev-Id header. They shouldn't do any harm; however they are only needed if you're doing client authentication (that is, if your server is trying to authenticate that the requests are coming from a legitimate Roku device). In this case, you'd also need to copy the Roku public key file (located in the SDK file: certificates\cacert.pem) to your server.
For server authentication, the only thing you should need on the Roku is the certificate chain for the signer of your server certificate. The file containing the certificate chain is the file specified in the call to SetCertificatesFile().
You don't need the server's public key file (I assume that's what you have in certificate.crt). When the Roku establishes an https connection, the firmware sends the Roku's public key (buried in the firmware) to the server. The server then responds with the server's public key. The Roku will use the server's public key that it got from the server for encrypting messages to the server that are decrypted by the server using the server's private key.
You absolutely should not need the server's private key for anything. (I assume that's your file, private_key.key.) The whole point of public key encryption is that everyone is allowed to know the server's public key, which they use to encrypt messages sent to the server, but only the server has access to the private key, which it uses to decrypt the messages.
The call to SetCertificatesFile() should specify a file containing the certificate chain used to verify the signature on the server's public key. For whichever Certificate Authority was used to sign the server's public key, there should be a certificate in this file. All the Certificate Authorities in turn used to sign that CA's certificate should also be in the certificate file, all the way up to the root Certificate Authority.
The certificate file should be in PEM format. The certificates corresponding to each Certificate Authority should be concatenated within this file, each one having this format:
-----BEGIN CERTIFICATE-----
MIICWjCCAcMCAgGlMA0GCSqGSIb3DQEBBAUAMHUxCzAJBgNVBAYTAlVTMRgwFgYDVQQKEw9HVEUg
Q29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNvbHV0aW9ucywgSW5jLjEjMCEG
A1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJvb3QwHhcNOTgwODEzMDAyOTAwWhcNMTgwODEz
MjM1OTAwWjB1MQswCQYDVQQGEwJVUzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQL
Ex5HVEUgQ3liZXJUcnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0
IEdsb2JhbCBSb290MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCVD6C28FCc6HrHiM3dFw4u
sJTQGz0O9pTAipTHBsiQl8i4ZBp6fmw8U+E3KHNgf7KXUwefU/ltWJTSr41tiGeA5u2ylc9yMcql
HHK6XALnZELn+aks1joNrI1CqiQBOeacPwGFVw1Yh0X404Wqk2kmhXBIgD8SFcd5tB8FLztimQID
AQABMA0GCSqGSIb3DQEBBAUAA4GBAG3rGwnpXtlR22ciYaQqPEh346B8pt5zohQDhT37qw4wxYMW
M4ETCJ57NE7fQMh017l93PR2VX2bY1QY6fDq81yx2YtCHrnAlU66+tXifPVoYb+O7AWXX1uw16OF
NMQkpw0PlZPvy5TYnh+dXIVtx6quTx8itc2VrbqnzPmrC3p/
-----END CERTIFICATE-----
This curl command should work if your certificates' file and your server are set up correctly:
curl https://service:port --cacert cafile.pem
If you're using an Apache server, for example, the location of the certificate chain file on the server is specified by the SSLCACertificateFile directive. A copy of this same file (if it's in PEM format, or converted to PEM format) should also work on your Roku as the file specified by SetCertificatesFile().
All of the above assumes that your server certificate was not signed by one of the commonly-known Certificate Authorities. If it was, then you should be able to just use the standard Roku CA bundle:
http.SetCertificatesFile ("common:/certs/ca-bundle.crt")
And if your server is using a self-signed certificate, use the server's self-signed certificate (converted to PEM format) as the certificates' file.