JSON to Protobuf
Generate proto3 messages that protobuf's JSON mapping reads a JSON sample into and writes back. Every value is typed, json_name keeps each key, and a key that some objects lack is optional.
About JSON to Protobuf
The objects in the JSON become proto3 messages, which protobuf's JSON mapping, ProtoJSON, reads the JSON into and writes back. Fields are lower_snake_case and messages TitleCase, as the protobuf style guide asks. ProtoJSON names a field's key by the field's JSON name, its name in lowerCamelCase, so a field whose key is not that name gets the key as its json_name, as in string user_id = 1 [json_name = "user_id"];, and each key is read and written back as the JSON has it.
Every value at a place counts, not just the first. A whole number is an int32, or an int64 when one is beyond 32 bits, or a uint64 when one is beyond an int64 and none is below 0. ProtoJSON writes a 64-bit integer as a string, so 3000000000 in an int64 is written back as "3000000000", as a note says. A number written with a fraction or an exponent, as 10.0 and 1e3 are, makes a double: Python's json_format and Go's protojson read those into an integer field too, but a key written so can hold 10.5 in the next response, which they refuse for one.
The objects at one place, such as the items of a list, make one message, with the keys in the order first seen. ProtoJSON reads null, and a key the JSON lacks, as a field not set, which it leaves out when it writes the JSON. A scalar field cannot tell a field not set from its default value, 0, "" or false, so it leaves that out too, unless it is optional: a scalar field whose key some objects lack, or hold as null, is optional. A note names each null, default value and empty list the JSON has that is not written back.
A key that holds two kinds of value, such as a string and a number, or only null, is a google.protobuf.Value, which holds any value. A repeated field cannot hold null or a list, so items that are null in some lists, or of two kinds, are a repeated google.protobuf.Value, and lists of lists a repeated google.protobuf.ListValue; objects that are always empty are a google.protobuf.Struct. The file then imports google/protobuf/struct.proto. Those types hold every number as a double, and a note says why a field has each of them, and names each number a double changes.
A name already taken gets a number, as in user_id2, and the message of a list's items is singular, as Order is for orders. protoc 22 and later refuse two fields whose JSON names are one, and protoc 21 and before, which some systems still install, two whose names are one with the underscores dropped, as user_id and userid are; no two fields here are one with the underscores dropped, which rules out both. A key that holds half of a surrogate pair or the character U+0000, or that starts with [ and ends with ], as ProtoJSON writes the name of an extension, has no field, and a note says so.
When the JSON is an array of objects, the root message describes them, and a note says how to read the whole array. Choose Every field to make every scalar field optional, for an API that can leave any key out. The file has no package, which you add when it has its own place in a project, and it compiles with protoc 36.2 with no error or warning. Everything runs in your browser.
For Go, see JSON to Go Struct. For a schema of the JSON itself, try JSON Schema Generator, and for TypeScript, JSON to TypeScript.
Built and maintained by Meet Shah · Last updated
What this tool is used for
- Writing the proto3 messages for a gRPC or Connect service from a JSON response the API sends today.
- Moving an endpoint from JSON to protobuf, with ProtoJSON reading and writing each key as the JSON has it.
- Getting the field names and
json_nameoptions right for keys in camelCase, or with characters a field name cannot hold, such as spaces, dots and dashes. - Seeing which keys only some objects of a list carry, as the
optionalfields. - Checking, from the notes, what ProtoJSON writes back otherwise: 64-bit integers as strings, default values and empty lists left out, and numbers a double changes.
Frequently Asked Questions
- How do JSON types map to protobuf?
- Strings are
string, booleansbool, objects a message and arrays arepeatedfield. A whole number is anint32, or anint64or auint64when the integers need one, and a fraction or an exponent makes adouble, as in 10.0 and 1e3. A key with two kinds of value, or only null, is agoogle.protobuf.Value. - Why is an integer an int32 and not an int64?
- ProtoJSON writes an
int64or auint64as a string, so 1 in anint64is written back as "1", and code that expects a number gets a string. Anint32writes it back as a number, so a key is anint64only when one of its integers is beyond 32 bits, and a note says its integers are written back as strings. - How are keys named?
- Each field is its key in lower_snake_case, as the protobuf style guide asks, and each message is TitleCase. ProtoJSON reads and writes a field by its JSON name, its name in lowerCamelCase, so a field whose key is not that name gets the key as its
json_name, asuser_iddoes, and each key is written back unchanged. - How are null and missing keys handled?
- ProtoJSON reads null, and a key the JSON lacks, as a field not set, and leaves it out when it writes the JSON. A scalar field whose key some objects lack, or hold as null, is
optional, which tells a field not set from 0, "" or false; a scalar field withoutoptionalleaves those values out too, as a note says. - Which protoc reads optional in proto3?
- protoc 3.15 and later read it as it is. protoc 3.12 to 3.14 need the
--experimental_allow_proto3_optionalflag, and older ones refuse it. The file compiles with protoc 36.2 with no error or warning. - Should every field be optional?
- A sample shows only some of the keys an API can send as null or leave out: here a scalar field whose key is null in some objects, or that some objects lack, is
optional. The API's contract says which others are. Every field makes every scalar fieldoptional, so a 0, an empty string or false is written back too. - What do the field numbers mean?
- The wire format names a field by its number, and ProtoJSON by its JSON name. Fields are numbered from 1 in the order of their keys, skipping 19000 to 19999, which protobuf reserves. Once data is stored or sent, keep each field's number and name, and give a new key a new number.
Common errors and gotchas
- Generating from a sample where a decimal field happens to hold a whole number, such as a price of 12: the field is an
int32, and ProtoJSON then refuses 12.5 for it. Use a sample with a fraction, or write 12.0, which gives adouble. - Typing every whole number as an
int64: ProtoJSON writes it back as a string, so 1 becomes "1". Here a key is anint64only when one of its integers is beyond 32 bits. - Leaving out
json_namefor keys in snake_case: ProtoJSON writes the fielduser_idback as userId. Here such a field has[json_name = "user_id"]. - Typing a list that holds null as a
repeated string: ProtoJSON's readers refuse null in it. Here such a list is arepeated google.protobuf.Value, which reads null as a value, and a note says why. - Expecting 0, "" and false to be written back: a proto3 scalar field without
optionalleaves them out, as it leaves out a field not set. A note names each such field, and Every field makes every scalar fieldoptional. - Expecting a key that appears twice in an object to be read as JavaScript reads it: JSON.parse keeps the last value, but Python's json_format and Go's protojson refuse such JSON. A note names each place.