...
Run Format

Source file src/encoding/gob/doc.go

     1	// Copyright 2009 The Go Authors. All rights reserved.
     2	// Use of this source code is governed by a BSD-style
     3	// license that can be found in the LICENSE file.
     4	
     5	/*
     6	Package gob manages streams of gobs - binary values exchanged between an
     7	Encoder (transmitter) and a Decoder (receiver).  A typical use is transporting
     8	arguments and results of remote procedure calls (RPCs) such as those provided by
     9	package "net/rpc".
    10	
    11	The implementation compiles a custom codec for each data type in the stream and
    12	is most efficient when a single Encoder is used to transmit a stream of values,
    13	amortizing the cost of compilation.
    14	
    15	Basics
    16	
    17	A stream of gobs is self-describing.  Each data item in the stream is preceded by
    18	a specification of its type, expressed in terms of a small set of predefined
    19	types.  Pointers are not transmitted, but the things they point to are
    20	transmitted; that is, the values are flattened.  Recursive types work fine, but
    21	recursive values (data with cycles) are problematic.  This may change.
    22	
    23	To use gobs, create an Encoder and present it with a series of data items as
    24	values or addresses that can be dereferenced to values.  The Encoder makes sure
    25	all type information is sent before it is needed.  At the receive side, a
    26	Decoder retrieves values from the encoded stream and unpacks them into local
    27	variables.
    28	
    29	Types and Values
    30	
    31	The source and destination values/types need not correspond exactly.  For structs,
    32	fields (identified by name) that are in the source but absent from the receiving
    33	variable will be ignored.  Fields that are in the receiving variable but missing
    34	from the transmitted type or value will be ignored in the destination.  If a field
    35	with the same name is present in both, their types must be compatible. Both the
    36	receiver and transmitter will do all necessary indirection and dereferencing to
    37	convert between gobs and actual Go values.  For instance, a gob type that is
    38	schematically,
    39	
    40		struct { A, B int }
    41	
    42	can be sent from or received into any of these Go types:
    43	
    44		struct { A, B int }	// the same
    45		*struct { A, B int }	// extra indirection of the struct
    46		struct { *A, **B int }	// extra indirection of the fields
    47		struct { A, B int64 }	// different concrete value type; see below
    48	
    49	It may also be received into any of these:
    50	
    51		struct { A, B int }	// the same
    52		struct { B, A int }	// ordering doesn't matter; matching is by name
    53		struct { A, B, C int }	// extra field (C) ignored
    54		struct { B int }	// missing field (A) ignored; data will be dropped
    55		struct { B, C int }	// missing field (A) ignored; extra field (C) ignored.
    56	
    57	Attempting to receive into these types will draw a decode error:
    58	
    59		struct { A int; B uint }	// change of signedness for B
    60		struct { A int; B float }	// change of type for B
    61		struct { }			// no field names in common
    62		struct { C, D int }		// no field names in common
    63	
    64	Integers are transmitted two ways: arbitrary precision signed integers or
    65	arbitrary precision unsigned integers.  There is no int8, int16 etc.
    66	discrimination in the gob format; there are only signed and unsigned integers.  As
    67	described below, the transmitter sends the value in a variable-length encoding;
    68	the receiver accepts the value and stores it in the destination variable.
    69	Floating-point numbers are always sent using IEEE-754 64-bit precision (see
    70	below).
    71	
    72	Signed integers may be received into any signed integer variable: int, int16, etc.;
    73	unsigned integers may be received into any unsigned integer variable; and floating
    74	point values may be received into any floating point variable.  However,
    75	the destination variable must be able to represent the value or the decode
    76	operation will fail.
    77	
    78	Structs, arrays and slices are also supported. Structs encode and decode only
    79	exported fields. Strings and arrays of bytes are supported with a special,
    80	efficient representation (see below). When a slice is decoded, if the existing
    81	slice has capacity the slice will be extended in place; if not, a new array is
    82	allocated. Regardless, the length of the resulting slice reports the number of
    83	elements decoded.
    84	
    85	In general, if allocation is required, the decoder will allocate memory. If not,
    86	it will update the destination variables with values read from the stream. It does
    87	not initialize them first, so if the destination is a compound value such as a
    88	map, struct, or slice, the decoded values will be merged elementwise into the
    89	existing variables.
    90	
    91	Functions and channels will not be sent in a gob. Attempting to encode such a value
    92	at the top level will fail. A struct field of chan or func type is treated exactly
    93	like an unexported field and is ignored.
    94	
    95	Gob can encode a value of any type implementing the GobEncoder or
    96	encoding.BinaryMarshaler interfaces by calling the corresponding method,
    97	in that order of preference.
    98	
    99	Gob can decode a value of any type implementing the GobDecoder or
   100	encoding.BinaryUnmarshaler interfaces by calling the corresponding method,
   101	again in that order of preference.
   102	
   103	Encoding Details
   104	
   105	This section documents the encoding, details that are not important for most
   106	users. Details are presented bottom-up.
   107	
   108	An unsigned integer is sent one of two ways.  If it is less than 128, it is sent
   109	as a byte with that value.  Otherwise it is sent as a minimal-length big-endian
   110	(high byte first) byte stream holding the value, preceded by one byte holding the
   111	byte count, negated.  Thus 0 is transmitted as (00), 7 is transmitted as (07) and
   112	256 is transmitted as (FE 01 00).
   113	
   114	A boolean is encoded within an unsigned integer: 0 for false, 1 for true.
   115	
   116	A signed integer, i, is encoded within an unsigned integer, u.  Within u, bits 1
   117	upward contain the value; bit 0 says whether they should be complemented upon
   118	receipt.  The encode algorithm looks like this:
   119	
   120		var u uint
   121		if i < 0 {
   122			u = (^uint(i) << 1) | 1 // complement i, bit 0 is 1
   123		} else {
   124			u = (uint(i) << 1) // do not complement i, bit 0 is 0
   125		}
   126		encodeUnsigned(u)
   127	
   128	The low bit is therefore analogous to a sign bit, but making it the complement bit
   129	instead guarantees that the largest negative integer is not a special case.  For
   130	example, -129=^128=(^256>>1) encodes as (FE 01 01).
   131	
   132	Floating-point numbers are always sent as a representation of a float64 value.
   133	That value is converted to a uint64 using math.Float64bits.  The uint64 is then
   134	byte-reversed and sent as a regular unsigned integer.  The byte-reversal means the
   135	exponent and high-precision part of the mantissa go first.  Since the low bits are
   136	often zero, this can save encoding bytes.  For instance, 17.0 is encoded in only
   137	three bytes (FE 31 40).
   138	
   139	Strings and slices of bytes are sent as an unsigned count followed by that many
   140	uninterpreted bytes of the value.
   141	
   142	All other slices and arrays are sent as an unsigned count followed by that many
   143	elements using the standard gob encoding for their type, recursively.
   144	
   145	Maps are sent as an unsigned count followed by that many key, element
   146	pairs. Empty but non-nil maps are sent, so if the receiver has not allocated
   147	one already, one will always be allocated on receipt unless the transmitted map
   148	is nil and not at the top level.
   149	
   150	In slices and arrays, as well as maps, all elements, even zero-valued elements,
   151	are transmitted, even if all the elements are zero.
   152	
   153	Structs are sent as a sequence of (field number, field value) pairs.  The field
   154	value is sent using the standard gob encoding for its type, recursively.  If a
   155	field has the zero value for its type (except for arrays; see above), it is omitted
   156	from the transmission.  The field number is defined by the type of the encoded
   157	struct: the first field of the encoded type is field 0, the second is field 1,
   158	etc.  When encoding a value, the field numbers are delta encoded for efficiency
   159	and the fields are always sent in order of increasing field number; the deltas are
   160	therefore unsigned.  The initialization for the delta encoding sets the field
   161	number to -1, so an unsigned integer field 0 with value 7 is transmitted as unsigned
   162	delta = 1, unsigned value = 7 or (01 07).  Finally, after all the fields have been
   163	sent a terminating mark denotes the end of the struct.  That mark is a delta=0
   164	value, which has representation (00).
   165	
   166	Interface types are not checked for compatibility; all interface types are
   167	treated, for transmission, as members of a single "interface" type, analogous to
   168	int or []byte - in effect they're all treated as interface{}.  Interface values
   169	are transmitted as a string identifying the concrete type being sent (a name
   170	that must be pre-defined by calling Register), followed by a byte count of the
   171	length of the following data (so the value can be skipped if it cannot be
   172	stored), followed by the usual encoding of concrete (dynamic) value stored in
   173	the interface value.  (A nil interface value is identified by the empty string
   174	and transmits no value.) Upon receipt, the decoder verifies that the unpacked
   175	concrete item satisfies the interface of the receiving variable.
   176	
   177	The representation of types is described below.  When a type is defined on a given
   178	connection between an Encoder and Decoder, it is assigned a signed integer type
   179	id.  When Encoder.Encode(v) is called, it makes sure there is an id assigned for
   180	the type of v and all its elements and then it sends the pair (typeid, encoded-v)
   181	where typeid is the type id of the encoded type of v and encoded-v is the gob
   182	encoding of the value v.
   183	
   184	To define a type, the encoder chooses an unused, positive type id and sends the
   185	pair (-type id, encoded-type) where encoded-type is the gob encoding of a wireType
   186	description, constructed from these types:
   187	
   188		type wireType struct {
   189			ArrayT  *ArrayType
   190			SliceT  *SliceType
   191			StructT *StructType
   192			MapT    *MapType
   193		}
   194		type arrayType struct {
   195			CommonType
   196			Elem typeId
   197			Len  int
   198		}
   199		type CommonType struct {
   200			Name string // the name of the struct type
   201			Id  int    // the id of the type, repeated so it's inside the type
   202		}
   203		type sliceType struct {
   204			CommonType
   205			Elem typeId
   206		}
   207		type structType struct {
   208			CommonType
   209			Field []*fieldType // the fields of the struct.
   210		}
   211		type fieldType struct {
   212			Name string // the name of the field.
   213			Id   int    // the type id of the field, which must be already defined
   214		}
   215		type mapType struct {
   216			CommonType
   217			Key  typeId
   218			Elem typeId
   219		}
   220	
   221	If there are nested type ids, the types for all inner type ids must be defined
   222	before the top-level type id is used to describe an encoded-v.
   223	
   224	For simplicity in setup, the connection is defined to understand these types a
   225	priori, as well as the basic gob types int, uint, etc.  Their ids are:
   226	
   227		bool        1
   228		int         2
   229		uint        3
   230		float       4
   231		[]byte      5
   232		string      6
   233		complex     7
   234		interface   8
   235		// gap for reserved ids.
   236		WireType    16
   237		ArrayType   17
   238		CommonType  18
   239		SliceType   19
   240		StructType  20
   241		FieldType   21
   242		// 22 is slice of fieldType.
   243		MapType     23
   244	
   245	Finally, each message created by a call to Encode is preceded by an encoded
   246	unsigned integer count of the number of bytes remaining in the message.  After
   247	the initial type name, interface values are wrapped the same way; in effect, the
   248	interface value acts like a recursive invocation of Encode.
   249	
   250	In summary, a gob stream looks like
   251	
   252		(byteCount (-type id, encoding of a wireType)* (type id, encoding of a value))*
   253	
   254	where * signifies zero or more repetitions and the type id of a value must
   255	be predefined or be defined before the value in the stream.
   256	
   257	See "Gobs of data" for a design discussion of the gob wire format:
   258	https://blog.golang.org/gobs-of-data
   259	*/
   260	package gob
   261	
   262	/*
   263	Grammar:
   264	
   265	Tokens starting with a lower case letter are terminals; int(n)
   266	and uint(n) represent the signed/unsigned encodings of the value n.
   267	
   268	GobStream:
   269		DelimitedMessage*
   270	DelimitedMessage:
   271		uint(lengthOfMessage) Message
   272	Message:
   273		TypeSequence TypedValue
   274	TypeSequence
   275		(TypeDefinition DelimitedTypeDefinition*)?
   276	DelimitedTypeDefinition:
   277		uint(lengthOfTypeDefinition) TypeDefinition
   278	TypedValue:
   279		int(typeId) Value
   280	TypeDefinition:
   281		int(-typeId) encodingOfWireType
   282	Value:
   283		SingletonValue | StructValue
   284	SingletonValue:
   285		uint(0) FieldValue
   286	FieldValue:
   287		builtinValue | ArrayValue | MapValue | SliceValue | StructValue | InterfaceValue
   288	InterfaceValue:
   289		NilInterfaceValue | NonNilInterfaceValue
   290	NilInterfaceValue:
   291		uint(0)
   292	NonNilInterfaceValue:
   293		ConcreteTypeName TypeSequence InterfaceContents
   294	ConcreteTypeName:
   295		uint(lengthOfName) [already read=n] name
   296	InterfaceContents:
   297		int(concreteTypeId) DelimitedValue
   298	DelimitedValue:
   299		uint(length) Value
   300	ArrayValue:
   301		uint(n) FieldValue*n [n elements]
   302	MapValue:
   303		uint(n) (FieldValue FieldValue)*n  [n (key, value) pairs]
   304	SliceValue:
   305		uint(n) FieldValue*n [n elements]
   306	StructValue:
   307		(uint(fieldDelta) FieldValue)*
   308	*/
   309	
   310	/*
   311	For implementers and the curious, here is an encoded example.  Given
   312		type Point struct {X, Y int}
   313	and the value
   314		p := Point{22, 33}
   315	the bytes transmitted that encode p will be:
   316		1f ff 81 03 01 01 05 50 6f 69 6e 74 01 ff 82 00
   317		01 02 01 01 58 01 04 00 01 01 59 01 04 00 00 00
   318		07 ff 82 01 2c 01 42 00
   319	They are determined as follows.
   320	
   321	Since this is the first transmission of type Point, the type descriptor
   322	for Point itself must be sent before the value.  This is the first type
   323	we've sent on this Encoder, so it has type id 65 (0 through 64 are
   324	reserved).
   325	
   326		1f	// This item (a type descriptor) is 31 bytes long.
   327		ff 81	// The negative of the id for the type we're defining, -65.
   328			// This is one byte (indicated by FF = -1) followed by
   329			// ^-65<<1 | 1.  The low 1 bit signals to complement the
   330			// rest upon receipt.
   331	
   332		// Now we send a type descriptor, which is itself a struct (wireType).
   333		// The type of wireType itself is known (it's built in, as is the type of
   334		// all its components), so we just need to send a *value* of type wireType
   335		// that represents type "Point".
   336		// Here starts the encoding of that value.
   337		// Set the field number implicitly to -1; this is done at the beginning
   338		// of every struct, including nested structs.
   339		03	// Add 3 to field number; now 2 (wireType.structType; this is a struct).
   340			// structType starts with an embedded CommonType, which appears
   341			// as a regular structure here too.
   342		01	// add 1 to field number (now 0); start of embedded CommonType.
   343		01	// add 1 to field number (now 0, the name of the type)
   344		05	// string is (unsigned) 5 bytes long
   345		50 6f 69 6e 74	// wireType.structType.CommonType.name = "Point"
   346		01	// add 1 to field number (now 1, the id of the type)
   347		ff 82	// wireType.structType.CommonType._id = 65
   348		00	// end of embedded wiretype.structType.CommonType struct
   349		01	// add 1 to field number (now 1, the field array in wireType.structType)
   350		02	// There are two fields in the type (len(structType.field))
   351		01	// Start of first field structure; add 1 to get field number 0: field[0].name
   352		01	// 1 byte
   353		58	// structType.field[0].name = "X"
   354		01	// Add 1 to get field number 1: field[0].id
   355		04	// structType.field[0].typeId is 2 (signed int).
   356		00	// End of structType.field[0]; start structType.field[1]; set field number to -1.
   357		01	// Add 1 to get field number 0: field[1].name
   358		01	// 1 byte
   359		59	// structType.field[1].name = "Y"
   360		01	// Add 1 to get field number 1: field[1].id
   361		04	// struct.Type.field[1].typeId is 2 (signed int).
   362		00	// End of structType.field[1]; end of structType.field.
   363		00	// end of wireType.structType structure
   364		00	// end of wireType structure
   365	
   366	Now we can send the Point value.  Again the field number resets to -1:
   367	
   368		07	// this value is 7 bytes long
   369		ff 82	// the type number, 65 (1 byte (-FF) followed by 65<<1)
   370		01	// add one to field number, yielding field 0
   371		2c	// encoding of signed "22" (0x22 = 44 = 22<<1); Point.x = 22
   372		01	// add one to field number, yielding field 1
   373		42	// encoding of signed "33" (0x42 = 66 = 33<<1); Point.y = 33
   374		00	// end of structure
   375	
   376	The type encoding is long and fairly intricate but we send it only once.
   377	If p is transmitted a second time, the type is already known so the
   378	output will be just:
   379	
   380		07 ff 82 01 2c 01 42 00
   381	
   382	A single non-struct value at top level is transmitted like a field with
   383	delta tag 0.  For instance, a signed integer with value 3 presented as
   384	the argument to Encode will emit:
   385	
   386		03 04 00 06
   387	
   388	Which represents:
   389	
   390		03	// this value is 3 bytes long
   391		04	// the type number, 2, represents an integer
   392		00	// tag delta 0
   393		06	// value 3
   394	
   395	*/
   396	

View as plain text