一旦工作,那就要努力的干,聪明的干,快速的干——用省下来的时间干自己喜欢干的事情。!

istio部署grpc服务时遇到的问题

linux lampnick 551℃ 0评论

问题一:mycaller服务请求myresponser服务时,http接口可以通,grpc服务不通

  • 使用sleep容器对mycaller服务进行请求
    # kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
    {"code":10000000,"error":"内部错误","detail":"grpc: error while marshaling: proto: Marshal called with nil"}

    返回错误,于是查看mycaller服务容器的日志

    [root@master /root]# kubectl logs -f mycaller-6659dc46b-h4jzp mycaller
    2020/02/28 22:07:49 [info] Start http server listen :59130
    mycaller请求==>1
    请求myresponser服务连接信息: {"ServerName":"myresponser","ServerPort":"53606","CertFile":"","CertServerName":"","IsSsl":false}
    mycaller get请求url==>http://myresponser:53605/testResponser/GetHello?type=1
    mycaller get请求fileBody==>{"res":"[2020-02-28 22:08:47] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:"}
    请求myresponser服务客户端-获取Hello失败 err: rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type "text/plain; charset=utf-8"
    ERROR: 2020/02/28 22:08:47 grpc: server failed to encode response:  rpc error: code = Internal desc = grpc: error while marshaling: proto: Marshal called with nil
    {"level":"error","caller":"/usr/local/go/src/git.myscrm.cn/trade/mycaller/vendor/git.myscrm.cn/golang/common/yklog/logger_context.go:202","time":"2020-02-28 22:08:47.084","msg":"grpc-gateway err: grpc: error while marshaling: proto: Marshal called with nil"}

    看日志通过http get请求返回了正常的数据,但是grpc调用Hello方法时报错:rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type "text/plain; charset=utf-8",于是再查看mycaller pod中的envoy日志

    2020-02-28T14:07:32.934346Z info    Envoy proxy is ready
    [2020-02-28T14:07:49.471Z] "- - -" 0 - "-" "-" 136 566 5 - "-" "-" "-" "-" "10.10.3.61:18011" PassthroughCluster 10.36.0.10:49416 10.10.3.61:18011 10.36.0.10:49414 - -
    [2020-02-28T14:07:49.483Z] "GET /v2/keys/?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 26283 2 1 "-" "Go-http-client/1.1" "45b99d06-d24a-4a9e-af17-430b582976c6" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
    [2020-02-28T14:07:49.486Z] "PUT /v2/keys/service.mycaller.default HTTP/1.1" 200 - "-" "-" 137 413 0 0 "-" "Go-http-client/1.1" "511e9a64-8294-4fa9-97db-2710474c6816" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
    [2020-02-28T14:08:47.063Z] "GET /v2/keys/service.myresponser.default?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 216 1 1 "-" "Go-http-client/1.1" "aeb99ac6-17bf-4aea-8f6b-b58b6f3d49c8" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
    [2020-02-28T14:08:47.066Z] "GET /testResponser/GetHello?type=1 HTTP/1.1" 200 - "-" "-" 0 113 9 9 "-" "Go-http-client/1.1" "82b0a59f-223c-4944-b630-59d2c728d059" "myresponser:53605" "10.36.0.11:53606" outbound|53605||myresponser.default.svc.cluster.local - 10.97.218.153:53605 10.36.0.10:35778 - default
    [2020-02-28T14:08:47.077Z] "POST /testResponser.TestService/GetHello HTTP/2" 404 - "-" "-" 8 10 6 5 "-" "grpc-go/1.26.0" "be930121-1b34-4697-a909-3ec80dccbf18" "myresponser:53606" "10.36.0.11:53606" outbound|53606||myresponser.default.svc.cluster.local - 10.97.218.153:53606 10.36.0.10:37922 - default
    [2020-02-28T14:08:47.060Z] "GET /testCaller/GetHello?type=1&orgcode=private&port=53605 HTTP/2" 500 - "-" "-" 0 112 24 23 "-" "curl/7.64.0" "744ba535-8d37-4fa4-8e87-2a7e755c432a" "mycaller.default:59130" "127.0.0.1:59130" inbound|59130|http2-59130|mycaller.default.svc.cluster.local - 10.36.0.10:59130 10.36.0.12:52096 outbound_.59130_.v1_.mycaller.default.svc.cluster.local default

    发现是HTTP/2协议报的404,于是查看svc,是配置的grpc协议,不是http2

    [root@master /root]# kubectl get svc myresponser -o yaml
    apiVersion: v1
    kind: Service
    metadata:
    ...
    labels:
    app: myresponser
    name: myresponser
    namespace: default
    resourceVersion: "1982215"
    selfLink: /api/v1/namespaces/default/services/myresponser
    uid: 6fbd7ac2-0895-4d5c-b99f-2cdb6fc07e0d
    spec:
    clusterIP: 10.97.218.153
    ports:
    - name: http-53605
    port: 53605
    protocol: TCP
    targetPort: 53606
    - name: grpc-53606
    port: 53606
    protocol: TCP
    targetPort: 53606
    selector:
    app: myresponser
    sessionAffinity: None
    type: ClusterIP
    status:
    loadBalancer: {}

    于是修改svc,将name从grpc-53606修改为http2-53606

    apiVersion: v1
    kind: Service
    metadata:
    name: myresponser
    namespace: default
    labels:
    app: myresponser
    spec:
    selector:
    app: myresponser
    ports:
    - name: http-53605
    port: 53605
    protocol: TCP
    targetPort: 53606
    - name: http2-53606
    port: 53606
    protocol: TCP
    targetPort: 53606

    再次请求mycaller服务,发现还是报一样的错误。搞了半天,索性把http部分去掉,只保留http2的port试试

    apiVersion: v1
    kind: Service
    metadata:
    name: myresponser
    namespace: default
    labels:
    app: myresponser
    spec:
    selector:
    app: myresponser
    ports:
    #  - name: http-53605
    #    port: 53605
    #    protocol: TCP
    #    targetPort: 53606
    - name: http2-53606
    port: 53606
    protocol: TCP
    targetPort: 53606

    再请求mycaller服务,发现grpc请求成功了

    # kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
    {"res":"[2020-02-28 22:30:31] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}

    mycaller pod日志, 由于取消了http的端口,所以http请求不通,grpc正常

mycaller请求==>1
请求myresponser服务连接信息: {"ServerName":"myresponser","ServerPort":"53606","CertFile":"","CertServerName":"","IsSsl":false}
mycaller get请求url==>http://myresponser:53605/testResponser/GetHello?type=1
mycaller get请求Do err:==> Get http://myresponser:53605/testResponser/GetHello?type=1: read tcp 10.36.0.10:46620->10.97.218.153:53605: read: connection reset by peer
[2020-02-28 22:30:29] mycaller请求==> hostname:mycaller-6659dc46b-h4jzp, req:1, myresponser响应==>[2020-02-28 22:30:31] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private

mycaller中envoy的日志

[2020-02-28T14:30:29.863Z] "GET /v2/keys/service.myresponser.default?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 216 1 1 "-" "Go-http-client/1.1" "0b71ef78-1cbc-4f40-a731-153d0833637d" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
[2020-02-28T14:30:29.866Z] "- - -" 0 UF,URX "-" "-" 0 0 1000 - "-" "-" "-" "-" "10.97.218.153:53605" PassthroughCluster - 10.97.218.153:53605 10.36.0.10:46620 - -
[2020-02-28T14:30:30.866Z] "POST /testResponser.TestService/GetHello HTTP/2" 200 - "-" "-" 8 117 301 300 "-" "grpc-go/1.26.0" "ea990043-c6f7-416b-8aeb-d1c7b59844ba" "myresponser:53606" "10.5.24.224:32483" outbound|53606||myresponser.default.svc.cluster.local - 10.97.218.153:53606 10.36.0.10:48764 - default
[2020-02-28T14:30:29.861Z] "GET /testCaller/GetHello?type=1&orgcode=private&port=53605 HTTP/2" 200 - "-" "-" 0 120 1307 1307 "-" "curl/7.64.0" "2ea39954-58cc-4a32-8db9-96a0ed1c160b" "mycaller.default:59130" "127.0.0.1:59130" inbound|59130|http2-59130|mycaller.default.svc.cluster.local - 10.36.0.10:59130 10.36.0.12:52096 outbound_.59130_.v1_.mycaller.default.svc.cluster.local default

问题二:当grpc通了之后,使用istio的VirtualService进行分流的时候出现流量没有按照规则进行

  • 当时的VirtualService规则如下
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: myresponser-virtualservice
    spec:
    hosts:
    - myresponser
    gateways:
    - myresponser-gateway
    http:
    - match:
    - headers:
        orgcode:
          exact: private 
    route:
    - destination:
        host: myresponser
        subset: v2
    - route:
    - destination:
        host: myresponser
        subset: v1

    请求及响应如下

    [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
    {"res":"[2020-03-02 10:57:55] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}[root@master /root/yunke-istio-manifests-problem]# 
    [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
    {"res":"[2020-03-02 10:58:02] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}
    [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kupe=1&orgcode=private&port=53605"sonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?typ
    {"res":"[2020-03-02 10:58:06] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}
    [root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubecpe=1&orgcode=private&port=53605"path='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?typ
    {"res":"[2020-03-02 10:58:10] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}[root@master /root/yunke-istio-manifests-problem]# 

    经过排查,发现上面的yaml文件多配置了网关,后来把网关去掉,问题解决

    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: myresponser-virtualservice
    spec:
    hosts:
    - myresponser
    #  gateways:
    #  - myresponser-gateway
    http:
    - match:
    - headers:
        orgcode:
          exact: private 
    route:
    - destination:
        host: myresponser
        subset: v2
    - route:
    - destination:
        host: myresponser
        subset: v1

    请求如下:

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-03 09:09:59] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-03 09:10:04] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-03 09:10:07] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=&port=53605"
{"res":"[2020-03-03 09:10:16] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:prod"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=ttt&port=53605"
{"res":"[2020-03-03 09:10:23] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:ttt"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=prod&port=53605"
{"res":"[2020-03-03 09:10:30] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:prod"}

转载请注明:lampNick » istio部署grpc服务时遇到的问题

喜欢 (1)or分享 (0)
头像
发表我的评论
取消评论
表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址