0

Ho guys, I am new to elasticsearch and I want to import a json file to elasticsearch/kibana.The json file is the output of a wireshark pcap data traffic capture.As I understand i should do a mapping for this file before being able to use _bulk(curl) to import it to elasticsearch. I read some sample for doing the mapping like the below document, but it is not still clear for me how to do this with my own file: https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana

My json file looks like:

[
  {
    "_index": "packets-2021-10-15",
    "_type": "doc",
    "_score": null,
    "_source": {
      "layers": {
        "frame": {
          "frame.interface_id": "0",
          "frame.interface_id_tree": {
            "frame.interface_name": "\\Device\\NPF_{9751C4A6-3584-467A-81DB-8E9E881967C3}",
            "frame.interface_description": "Ethernet"
          },
          "frame.encap_type": "1",
          "frame.time": "Oct 26, 2021 14:08:42.112764000 W. Europe Summer Time",
          "frame.offset_shift": "0.000000000",
          "frame.time_epoch": "1635250122.112764000",
          "frame.time_delta": "0.006664000",
          "frame.time_delta_displayed": "0.000000000",
          "frame.time_relative": "15.640148000",
          "frame.number": "18",
          "frame.len": "241",
          "frame.cap_len": "241",
          "frame.marked": "0",
          "frame.ignored": "0",
          "frame.protocols": "eth:ethertype:ip:tcp:tpkt:cotp:ses:pres:acse:mms",
          "frame.coloring_rule.name": "TCP",
          "frame.coloring_rule.string": "tcp"
        },
        "eth": {
          "eth.dst": "00:00:96:14:19:00",
          "eth.dst_tree": {
            "eth.dst_resolved": "MarconiE_14:19:00",
            "eth.dst.oui": "150",
            "eth.dst.oui_resolved": "Marconi Electronics Ltd.",
            "eth.addr": "00:00:96:14:19:00",
            "eth.addr_resolved": "MarconiE_14:19:00",
            "eth.addr.oui": "150",
            "eth.addr.oui_resolved": "Marconi Electronics Ltd.",
            "eth.dst.lg": "0",
            "eth.lg": "0",
            "eth.dst.ig": "0",
            "eth.ig": "0"
          },
          "eth.src": "20:47:47:b5:f2:62",
          "eth.src_tree": {
            "eth.src_resolved": "Dell_b5:f2:62",
            "eth.src.oui": "2115399",
            "eth.src.oui_resolved": "Dell Inc.",
            "eth.addr": "20:47:47:b5:f2:62",
            "eth.addr_resolved": "Dell_b5:f2:62",
            "eth.addr.oui": "2115399",
            "eth.addr.oui_resolved": "Dell Inc.",
            "eth.src.lg": "0",
            "eth.lg": "0",
            "eth.src.ig": "0",
            "eth.ig": "0"
          },
          "eth.type": "0x00000800"
        },
        "ip": {
          "ip.version": "4",
          "ip.hdr_len": "20",
          "ip.dsfield": "0x00000000",
          "ip.dsfield_tree": {
            "ip.dsfield.dscp": "0",
            "ip.dsfield.ecn": "0"
          },
          "ip.len": "227",
          "ip.id": "0x00001a1d",
          "ip.flags": "0x00000040",
          "ip.flags_tree": {
            "ip.flags.rb": "0",
            "ip.flags.df": "1",
            "ip.flags.mf": "0"
          },
          "ip.frag_offset": "0",
          "ip.ttl": "128",
          "ip.proto": "6",
          "ip.checksum": "0x00000000",
          "ip.checksum.status": "2",
          "ip.src": "192.168.1.92",
          "ip.addr": "192.168.1.92",
          "ip.src_host": "192.168.1.92",
          "ip.host": "192.168.1.92",
          "ip.dst": "192.168.1.93",
          "ip.addr": "192.168.1.93",
          "ip.dst_host": "192.168.1.93",
          "ip.host": "192.168.1.93"
        },
        "tcp": {
         ......

I would be thankful if someone can show me the best way to proceed.

TBA
  • 2,005
  • 3
  • 10
  • 23
Amir kamtarin
  • 1
  • 1

1 Answers1

0

One of the benefits of ES is that mapping in a great number of the cases is done automatically. So, you can try to do it and see if the types that it assigns to your keys are the same ones that you were expecting.

Basically, you need to build a request that meet the requirements presented in the ES documentation.

TL;DR

curl -X POST "localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
'

This answer might help too.

ixperdomo
  • 127
  • 2
  • 9
  • Thanks a lot for your answer. As you see there is no _id in the json file that I shared here, so ai do not understand what does id mean in this case Also I wonder if I can use "--data-binary @packets.json" instead of using " -d ..." to be able to import the whole file to elasticsearch? – Amir kamtarin Nov 16 '21 at 17:25
  • The `_id` is an internal ES id assigned to each document. You can give it whatever value you want. For example, you can build a script in which every bulk request uses the unix timestamp as document `_id`. Regarding the other question, that is more a curl usage question, but of course, you can use `--data-binary` instead of `-d`. – ixperdomo Nov 17 '21 at 08:08
  • Thank you sir. Whenever i tried to change my configuration to the form you advised,but when I run the curl command I gets an error curl (52) Empty reply from server. Do you know what can be the reason for this error? – Amir kamtarin Nov 17 '21 at 15:20
  • Update your question with what you have tried and I'll look into it. – ixperdomo Nov 17 '21 at 16:35