Skip to content
/ cbor Public

CBOR codec (RFC 8949) with CBOR tags, Go struct tags (toarray, keyasint, omitempty), float64/32/16, big.Int, and fuzz tested billions of execs.

License

Notifications You must be signed in to change notification settings

fxamacker/cbor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

author
Faye Amacker
Oct 3, 2019
79019c2 · Oct 3, 2019

History

47 Commits
May 15, 2019
Sep 1, 2019
May 15, 2019
Oct 1, 2019
May 16, 2019
Oct 2, 2019
Sep 15, 2019
Sep 15, 2019
Sep 15, 2019
Sep 3, 2019
Aug 28, 2019
Sep 15, 2019
Sep 15, 2019
Oct 3, 2019
Oct 2, 2019

Repository files navigation

Build Status codecov Go Report Card Release GoDoc License

cbor - CBOR encoding and decoding in Go

CBOR is a concise binary alternative to JSON. This library makes CBOR easy to use.

This library is designed to be:

  • Easy -- idiomatic API like encoding/json.
  • Safe and reliable -- no unsafe pkg, test coverage at 96%, and 10+ hrs of fuzzing with RFC 7049 test vectors.
  • Standards-compliant -- supports RFC 7049 and canonical CBOR encodings (both RFC 7049 and CTAP2).
  • Small and self-contained -- pkg compiles to under 0.5 MB with no external dependencies.

cbor balances speed, safety (no unsafe pkg) and compiled size. To keep size small, it doesn't use code generation. For speed, it caches struct field types, bypasses reflect when appropriate, and uses sync.Pool to reuse transient objects.

Current status

Sept 29, 2019: cbor v1.0 is released for Go 1.12+. It passed 10+ hours of fuzzing and is ready for production use on linux_amd64.

Size comparison

Program size comparison (linux_amd64, Go 1.12) doing the same CBOR encoding and decoding:

  • 2.6 MB program using fxamacker/cbor
  • 11.9 MB program using ugorji/go

Library size comparison (linux_amd64, Go 1.12):

  • 0.41 MB pkg -- fxamacker/cbor
  • 2.9 MB pkg -- ugorji/go without code generation (go install --tags "notfastpath")
  • 5.7 MB pkg -- ugorji/go with code generation (default build)

Features

  • Idiomatic API as in json package.
  • No external dependencies.
  • No use of unsafe package.
  • Tested with RFC 7049 test vectors.
  • Test coverage at 96%, and fuzzed 10+ hours using cbor-fuzz.
  • Decode slices, maps, and structs in-place.
  • Decode into struct with field name case-insensitive match.
  • Support canonical CBOR encoding for map/struct.
  • Support both "cbor" and "json" keys for struct field format tags.
  • Encode anonymous struct fields by json package struct fields visibility rules.
  • Encode and decode nil slice/map/pointer/interface values correctly.
  • Encode and decode indefinite length bytes/string/array/map ("streaming").
  • Encode and decode time.Time as RFC 3339 formatted text string or Unix time.

Standards

This library implements CBOR as specified in RFC 7049, with minor limitations.

It also supports canonical CBOR encodings (both RFC 7049 and CTAP2). CTAP2 canonical CBOR encoding is used by CTAP and WebAuthn in FIDO2 framework.

Limitations

  • CBOR tags (type 6) are ignored. Decoder simply decodes tagged data after ignoring the tags.
  • Signed integer values incompatible with Go's int64 are not supported. So RFC 7049 test vectors incompatible with Go's int64 are skipped. For example, test result -18446744073709551616 can't fit into int64.

Versions and API changes

This project uses Semantic Versioning, so the API is always backwards compatible unless the major version number changes.

API

See API docs for more details.

package cbor // import "github.com/fxamacker/cbor"

func Marshal(v interface{}, encOpts EncOptions) ([]byte, error)
func Unmarshal(data []byte, v interface{}) error
func Valid(data []byte) (rest []byte, err error)
type Decoder struct{ ... }
    func NewDecoder(r io.Reader) *Decoder
    func (dec *Decoder) Decode(v interface{}) (err error)
    func (dec *Decoder) NumBytesRead() int
type EncOptions struct{ ... }
type Encoder struct{ ... }
    func NewEncoder(w io.Writer, encOpts EncOptions) *Encoder
    func (enc *Encoder) Encode(v interface{}) error
    func (enc *Encoder) StartIndefiniteByteString() error
    func (enc *Encoder) StartIndefiniteTextString() error
    func (enc *Encoder) StartIndefiniteArray() error
    func (enc *Encoder) StartIndefiniteMap() error
    func (enc *Encoder) EndIndefinite() error
type InvalidUnmarshalError struct{ ... }
type SemanticError struct{ ... }
type SyntaxError struct{ ... }
type UnmarshalTypeError struct{ ... }
type UnsupportedTypeError struct{ ... }

Installation

go get github.com/fxamacker/cbor

Usage

See examples.

Using decoder:

// create a decoder
dec := cbor.NewDecoder(reader)

// decode into empty interface
var i interface{}
err = dec.Decode(&i)

// decode into struct 
var stru ExampleStruct
err = dec.Decode(&stru)

// decode into map
var m map[string]string
err = dec.Decode(&m)

// decode into primitives
var f float32
err = dec.Decode(&f)

Using encoder:

// create an encoder with canonical CBOR encoding enabled
enc := cbor.NewEncoder(writer, cbor.EncOptions{Canonical: true})

// encode struct
err = enc.Encode(stru)

// encode map
err = enc.Encode(m)

// encode primitives
err = enc.Encode(f)

Encoding indefinite length array:

enc := cbor.NewEncoder(writer, cbor.EncOptions{})

// start indefinite length array encoding
err = enc.StartIndefiniteArray()

// encode array element
err = enc.Encode(1)

// encode array element
err = enc.Encode([]int{2, 3})

// start nested indefinite length array as array element
err = enc.StartIndefiniteArray()

// encode nested array element
err = enc.Encode(4)

// encode nested array element
err = enc.Encode(5)

// close nested indefinite length array
err = enc.EndIndefinite()

// close outer indefinite length array
err = enc.EndIndefinite()

Benchmarks

See bench_test.go.

Unmarshal benchmarks are made on CBOR data representing the following values:

  • Boolean: true
  • Positive integer: 18446744073709551615
  • Negative integer: -1000
  • Float: -4.1
  • Byte string: h'0102030405060708090a0b0c0d0e0f101112131415161718191a'
  • Text string: "The quick brown fox jumps over the lazy dog"
  • Array: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]
  • Map: {"a": "A", "b": "B", "c": "C", "d": "D", "e": "E", "f": "F", "g": "G", "h": "H", "i": "I", "j": "J", "l": "L", "m": "M", "n": "N"}}

Marshal benchmarks are made on Go values representing the same values.

Benchmarks shows that decoding into struct is >50% faster than decoding into map, and encoding struct is >70% faster than encoding map.

BenchmarkUnmarshal/CBOR_boolean_to_Go_interface_{}-2         	                        10000000	       132 ns/op	      16 B/op	       1 allocs/op
BenchmarkUnmarshal/CBOR_boolean_to_Go_bool-2                 	                        20000000	      76.1 ns/op	       1 B/op	       1 allocs/op
BenchmarkUnmarshal/CBOR_positive_integer_to_Go_interface_{}-2         	                10000000	       159 ns/op	      24 B/op	       2 allocs/op
BenchmarkUnmarshal/CBOR_positive_integer_to_Go_uint64-2               	                20000000	      82.3 ns/op	       8 B/op	       1 allocs/op
BenchmarkUnmarshal/CBOR_negative_integer_to_Go_interface_{}-2         	                10000000	       160 ns/op	      24 B/op	       2 allocs/op
BenchmarkUnmarshal/CBOR_negative_integer_to_Go_int64-2                	                20000000	      83.8 ns/op	       8 B/op	       1 allocs/op
BenchmarkUnmarshal/CBOR_float_to_Go_interface_{}-2                    	                10000000	       159 ns/op	      24 B/op	       2 allocs/op
BenchmarkUnmarshal/CBOR_float_to_Go_float64-2                         	                20000000	      81.4 ns/op	       8 B/op	       1 allocs/op
BenchmarkUnmarshal/CBOR_byte_string_to_Go_interface_{}-2              	                10000000	       214 ns/op	      80 B/op	       3 allocs/op
BenchmarkUnmarshal/CBOR_byte_string_to_Go_[]uint8-2                   	                10000000	       159 ns/op	      64 B/op	       2 allocs/op
BenchmarkUnmarshal/CBOR_byte_string_indefinite_length_to_Go_interface_{}-2         	 2000000	       738 ns/op	     112 B/op	       3 allocs/op
BenchmarkUnmarshal/CBOR_byte_string_indefinite_length_to_Go_[]uint8-2              	 2000000	       687 ns/op	      96 B/op	       2 allocs/op
BenchmarkUnmarshal/CBOR_text_string_to_Go_interface_{}-2                           	 5000000	       248 ns/op	      80 B/op	       3 allocs/op
BenchmarkUnmarshal/CBOR_text_string_to_Go_string-2                                 	10000000	       176 ns/op	      64 B/op	       2 allocs/op
BenchmarkUnmarshal/CBOR_text_string_indefinite_length_to_Go_interface_{}-2         	 1000000	      1147 ns/op	     144 B/op	       4 allocs/op
BenchmarkUnmarshal/CBOR_text_string_indefinite_length_to_Go_string-2               	 1000000	      1075 ns/op	     128 B/op	       3 allocs/op
BenchmarkUnmarshal/CBOR_array_to_Go_interface_{}-2                                 	 1000000	      1156 ns/op	     672 B/op	      29 allocs/op
BenchmarkUnmarshal/CBOR_array_to_Go_[]int-2                                        	 1000000	      1075 ns/op	     272 B/op	       3 allocs/op
BenchmarkUnmarshal/CBOR_array_indefinite_length_to_Go_interface_{}-2               	 1000000	      1347 ns/op	     672 B/op	      29 allocs/op
BenchmarkUnmarshal/CBOR_array_indefinite_length_to_Go_[]int-2                      	 1000000	      1275 ns/op	     272 B/op	       3 allocs/op
BenchmarkUnmarshal/CBOR_map_to_Go_interface_{}-2                                   	  500000	      3094 ns/op	    1420 B/op	      30 allocs/op
BenchmarkUnmarshal/CBOR_map_to_Go_map[string]interface_{}-2                        	  300000	      4064 ns/op	     964 B/op	      19 allocs/op
BenchmarkUnmarshal/CBOR_map_to_Go_map[string]string-2                              	  500000	      2808 ns/op	     740 B/op	       5 allocs/op
BenchmarkUnmarshal/CBOR_map_to_Go_cbor_test.strc-2                                 	 1000000	      1624 ns/op	     208 B/op	       1 allocs/op
BenchmarkUnmarshal/CBOR_map_indefinite_length_to_Go_interface_{}-2                 	  300000	      4236 ns/op	    2607 B/op	      33 allocs/op
BenchmarkUnmarshal/CBOR_map_indefinite_length_to_Go_map[string]interface_{}-2      	  300000	      5819 ns/op	    2422 B/op	      22 allocs/op
BenchmarkUnmarshal/CBOR_map_indefinite_length_to_Go_map[string]string-2            	  300000	      4268 ns/op	    2183 B/op	       7 allocs/op
BenchmarkUnmarshal/CBOR_map_indefinite_length_to_Go_cbor_test.strc-2               	 1000000	      1860 ns/op	     208 B/op	       1 allocs/op
BenchmarkMarshal/Go_bool_to_CBOR_boolean-2                                         	20000000	      62.8 ns/op	       1 B/op	       1 allocs/op
BenchmarkMarshal/Go_uint64_to_CBOR_positive_integer-2                              	20000000	      73.4 ns/op	      16 B/op	       1 allocs/op
BenchmarkMarshal/Go_int64_to_CBOR_negative_integer-2                               	20000000	      64.7 ns/op	       3 B/op	       1 allocs/op
BenchmarkMarshal/Go_float64_to_CBOR_float-2                                        	20000000	      70.4 ns/op	      16 B/op	       1 allocs/op
BenchmarkMarshal/Go_[]uint8_to_CBOR_byte_string-2                                  	20000000	       105 ns/op	      32 B/op	       1 allocs/op
BenchmarkMarshal/Go_string_to_CBOR_text_string-2                                   	20000000	      95.0 ns/op	      48 B/op	       1 allocs/op
BenchmarkMarshal/Go_[]int_to_CBOR_array-2                                          	 3000000	       599 ns/op	      32 B/op	       1 allocs/op
BenchmarkMarshal/Go_map[string]string_to_CBOR_map-2                                	  500000	      3276 ns/op	     576 B/op	      28 allocs/op
BenchmarkMarshal/Go_cbor_test.strc_to_CBOR_map-2                                   	 2000000	       868 ns/op	      64 B/op	       1 allocs/op

License

Copyright (c) 2019 Faye Amacker

Licensed under MIT License